Re: [R-pkg-devel] socketConnection, delay when reading from

2021-11-27 Thread Jeff Newmiller
Google?

https://developer.r-project.org/Blog/public/2020/03/17/socket-connections-update/

https://www.digitalocean.com/community/tutorials/understanding-sockets

https://developer.ibm.com/tutorials/l-sockpit/

On November 27, 2021 2:36:48 PM PST, Ben Engbers  wrote:
>Hi,
>
>Looks promising! Where can I find more information or tutorials about 
>psock or parallel computing?
>
>Best,
>Ben
>
>Op 27-11-2021 om 20:19 schreef Tomas Kalibera:
>> On 11/27/21 8:05 PM, Tomas Kalibera wrote:
>> 
>> This is an extended demo with socketSelect() used to wait on the client 
>> for some data to be available, to avoid consuming too much CPU by 
>> polling. To be pasted into two R sessions running on the same computer.
>> You would have to replace the function done() with something figuring 
>> out from the data whether it is complete or not, based on the protocol.
>> 
>> Best
>> Tomas
>> 
>> 
>> # the client
>> 
>> con2 <- socketConnection("localhost", port = 6011, open = "rb")
>> cat("Connected...\n")
>> total <- 0
>> 
>> done <- function(n) {
>>    n >= 2e8
>> }
>> 
>> while(!done(total)) {
>>     cat("Waiting for data to become ready...\n")
>>     socketSelect(list(con2))
>>     cat("Reading data...\n")
>>     r <- readBin(con2, "raw", 1024)
>>     total <- total + length(r)
>>     cat("Read", length(r), "bytes (total ", total, ").\n")
>> }
>> close(con2)
>> 
>> # the server
>> 
>> n <- 1e8
>> w <- as.raw(runif(n, 0, 255))
>> con1 <- socketConnection(port = 6011, blocking = TRUE, server = TRUE, 
>> open="a+b")
>> cat("Connected...\n")
>> writeBin(w, con1)
>> cat("Sent data the first time, sleeping...\n")
>> Sys.sleep(10)
>> cat("Sending data the second time...\n")
>> writeBin(w, con1)
>> cat("Data sent to client...\n")
>> close(con1)
>> 
>>>
>>> Best
>>> Tomas
>>>
>>> # the client
>>>
>>> con2 <- socketConnection("localhost", port = 6011, open = "rb")
>>> cat("Connected...\n")
>>> total <- 0
>>>
>>> done <- function(n) {
>>>   n >= 1e8
>>> }
>>>
>>> while(!done(total)) {
>>>    r <- readBin(con2, "raw", 1024)
>>>    total <- total + length(r)
>>>    cat("Read", length(r), "bytes (total ", total, ").\n")
>>> }
>>> close(con2)
>>>
>>> # the server
>>>
>>> n <- 1e8
>>> w <- as.raw(runif(n, 0, 255))
>>> con1 <- socketConnection(port = 6011, blocking = TRUE, server = TRUE, 
>>> open="a+b")
>>> cat("Connected...\n")
>>> writeBin(w, con1)
>>> cat("Data sent to client...\n")
>>> close(con1)
>>>

 Ben

 __
 R-package-devel@r-project.org mailing list
 https://stat.ethz.ch/mailman/listinfo/r-package-devel

-- 
Sent from my phone. Please excuse my brevity.

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


Re: [R-pkg-devel] socketConnection, delay when reading from

2021-11-27 Thread Ben Engbers


I'll first finish the current development but then I will see if your 
suggestion is usefull.


Ben

Op 27-11-2021 om 20:19 schreef Tomas Kalibera:

On 11/27/21 8:05 PM, Tomas Kalibera wrote:


On 11/27/21 5:24 PM, Ben Engbers wrote:

Op 27-11-2021 om 17:03 schreef Jeff Newmiller:
This is a null-terminated message protocol [1]. It has to be 
processed one byte at a time.


[1] https://docs.basex.org/wiki/Server_Protocol

The message may contain embedded 0x00's. To distinguish these 
embedded 0x00's (and 0xFF's) from a terminating 0x00, embedded 0x00's 
and 0xFFare prefixed with a 0xFF byte. This means that when you 
process one byte at a time you have to perform a check on every byte. 
This results in totally unacceptable response times (My first version 
of this client was based on this approach)


The only alternative solution I can think off is to use C++ to create 
a socket and a function that reads from the socket. But since I have 
hardly any experience with C++ programming nor using the rcpp 
package



I think you could use non-blocking read. Read what is available in 
chunks (e.g. up to 1024 bytes long). And based on what is read 
already, figure out whether it is all data or not. Something along the 
lines as the demo below. The demo, though, is polling too much 
(re-running readBin to only find out no data is available yet). It 
should be possible to improve with socketSelect().


This is an extended demo with socketSelect() used to wait on the client 
for some data to be available, to avoid consuming too much CPU by 
polling. To be pasted into two R sessions running on the same computer.
You would have to replace the function done() with something figuring 
out from the data whether it is complete or not, based on the protocol.


Best
Tomas


# the client

con2 <- socketConnection("localhost", port = 6011, open = "rb")
cat("Connected...\n")
total <- 0

done <- function(n) {
   n >= 2e8
}

while(!done(total)) {
    cat("Waiting for data to become ready...\n")
    socketSelect(list(con2))
    cat("Reading data...\n")
    r <- readBin(con2, "raw", 1024)
    total <- total + length(r)
    cat("Read", length(r), "bytes (total ", total, ").\n")
}
close(con2)

# the server

n <- 1e8
w <- as.raw(runif(n, 0, 255))
con1 <- socketConnection(port = 6011, blocking = TRUE, server = TRUE, 
open="a+b")

cat("Connected...\n")
writeBin(w, con1)
cat("Sent data the first time, sleeping...\n")
Sys.sleep(10)
cat("Sending data the second time...\n")
writeBin(w, con1)
cat("Data sent to client...\n")
close(con1)



Best
Tomas

# the client

con2 <- socketConnection("localhost", port = 6011, open = "rb")
cat("Connected...\n")
total <- 0

done <- function(n) {
  n >= 1e8
}

while(!done(total)) {
   r <- readBin(con2, "raw", 1024)
   total <- total + length(r)
   cat("Read", length(r), "bytes (total ", total, ").\n")
}
close(con2)

# the server

n <- 1e8
w <- as.raw(runif(n, 0, 255))
con1 <- socketConnection(port = 6011, blocking = TRUE, server = TRUE, 
open="a+b")

cat("Connected...\n")
writeBin(w, con1)
cat("Data sent to client...\n")
close(con1)



Ben

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


Re: [R-pkg-devel] socketConnection, delay when reading from

2021-11-27 Thread Tomas Kalibera



On 11/27/21 8:05 PM, Tomas Kalibera wrote:


On 11/27/21 5:24 PM, Ben Engbers wrote:

Op 27-11-2021 om 17:03 schreef Jeff Newmiller:
This is a null-terminated message protocol [1]. It has to be 
processed one byte at a time.


[1] https://docs.basex.org/wiki/Server_Protocol

The message may contain embedded 0x00's. To distinguish these 
embedded 0x00's (and 0xFF's) from a terminating 0x00, embedded 0x00's 
and 0xFFare prefixed with a 0xFF byte. This means that when you 
process one byte at a time you have to perform a check on every byte. 
This results in totally unacceptable response times (My first version 
of this client was based on this approach)


The only alternative solution I can think off is to use C++ to create 
a socket and a function that reads from the socket. But since I have 
hardly any experience with C++ programming nor using the rcpp 
package



I think you could use non-blocking read. Read what is available in 
chunks (e.g. up to 1024 bytes long). And based on what is read 
already, figure out whether it is all data or not. Something along the 
lines as the demo below. The demo, though, is polling too much 
(re-running readBin to only find out no data is available yet). It 
should be possible to improve with socketSelect().


This is an extended demo with socketSelect() used to wait on the client 
for some data to be available, to avoid consuming too much CPU by 
polling. To be pasted into two R sessions running on the same computer.
You would have to replace the function done() with something figuring 
out from the data whether it is complete or not, based on the protocol.


Best
Tomas


# the client

con2 <- socketConnection("localhost", port = 6011, open = "rb")
cat("Connected...\n")
total <- 0

done <- function(n) {
  n >= 2e8
}

while(!done(total)) {
   cat("Waiting for data to become ready...\n")
   socketSelect(list(con2))
   cat("Reading data...\n")
   r <- readBin(con2, "raw", 1024)
   total <- total + length(r)
   cat("Read", length(r), "bytes (total ", total, ").\n")
}
close(con2)

# the server

n <- 1e8
w <- as.raw(runif(n, 0, 255))
con1 <- socketConnection(port = 6011, blocking = TRUE, server = TRUE, 
open="a+b")

cat("Connected...\n")
writeBin(w, con1)
cat("Sent data the first time, sleeping...\n")
Sys.sleep(10)
cat("Sending data the second time...\n")
writeBin(w, con1)
cat("Data sent to client...\n")
close(con1)



Best
Tomas

# the client

con2 <- socketConnection("localhost", port = 6011, open = "rb")
cat("Connected...\n")
total <- 0

done <- function(n) {
  n >= 1e8
}

while(!done(total)) {
   r <- readBin(con2, "raw", 1024)
   total <- total + length(r)
   cat("Read", length(r), "bytes (total ", total, ").\n")
}
close(con2)

# the server

n <- 1e8
w <- as.raw(runif(n, 0, 255))
con1 <- socketConnection(port = 6011, blocking = TRUE, server = TRUE, 
open="a+b")

cat("Connected...\n")
writeBin(w, con1)
cat("Data sent to client...\n")
close(con1)



Ben

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


Re: [R-pkg-devel] socketConnection, delay when reading from

2021-11-27 Thread Tomas Kalibera



On 11/27/21 5:24 PM, Ben Engbers wrote:

Op 27-11-2021 om 17:03 schreef Jeff Newmiller:
This is a null-terminated message protocol [1]. It has to be 
processed one byte at a time.


[1] https://docs.basex.org/wiki/Server_Protocol

The message may contain embedded 0x00's. To distinguish these embedded 
0x00's (and 0xFF's) from a terminating 0x00, embedded 0x00's and 
0xFFare prefixed with a 0xFF byte. This means that when you process 
one byte at a time you have to perform a check on every byte. This 
results in totally unacceptable response times (My first version of 
this client was based on this approach)


The only alternative solution I can think off is to use C++ to create 
a socket and a function that reads from the socket. But since I have 
hardly any experience with C++ programming nor using the rcpp package



I think you could use non-blocking read. Read what is available in 
chunks (e.g. up to 1024 bytes long). And based on what is read already, 
figure out whether it is all data or not. Something along the lines as 
the demo below. The demo, though, is polling too much (re-running 
readBin to only find out no data is available yet). It should be 
possible to improve with socketSelect().


Best
Tomas

# the client

con2 <- socketConnection("localhost", port = 6011, open = "rb")
cat("Connected...\n")
total <- 0

done <- function(n) {
  n >= 1e8
}

while(!done(total)) {
   r <- readBin(con2, "raw", 1024)
   total <- total + length(r)
   cat("Read", length(r), "bytes (total ", total, ").\n")
}
close(con2)

# the server

n <- 1e8
w <- as.raw(runif(n, 0, 255))
con1 <- socketConnection(port = 6011, blocking = TRUE, server = TRUE, 
open="a+b")

cat("Connected...\n")
writeBin(w, con1)
cat("Data sent to client...\n")
close(con1)



Ben

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


Re: [R-pkg-devel] socketConnection, delay when reading from

2021-11-27 Thread Jeff Newmiller
Maybe time to learn it. At least to assemble complete messages.

That said, the design of this protocol is intrinsically inefficient. Maybe they 
will upgrade if the software gets popular.

On November 27, 2021 8:24:36 AM PST, Ben Engbers  
wrote:
>Op 27-11-2021 om 17:03 schreef Jeff Newmiller:
>> This is a null-terminated message protocol [1]. It has to be processed one 
>> byte at a time.
>> 
>> [1] https://docs.basex.org/wiki/Server_Protocol
>> 
>The message may contain embedded 0x00's. To distinguish these embedded 
>0x00's (and 0xFF's) from a terminating 0x00, embedded 0x00's and 0xFFare 
>prefixed with a 0xFF byte. This means that when you process one byte at 
>a time you have to perform a check on every byte. This results in 
>totally unacceptable response times (My first version of this client was 
>based on this approach)
>
>The only alternative solution I can think off is to use C++ to create a 
>socket and a function that reads from the socket. But since I have 
>hardly any experience with C++ programming nor using the rcpp package
>
>Ben

-- 
Sent from my phone. Please excuse my brevity.

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


Re: [R-pkg-devel] socketConnection, delay when reading from

2021-11-27 Thread Ben Engbers

Op 27-11-2021 om 17:03 schreef Jeff Newmiller:

This is a null-terminated message protocol [1]. It has to be processed one byte 
at a time.

[1] https://docs.basex.org/wiki/Server_Protocol

The message may contain embedded 0x00's. To distinguish these embedded 
0x00's (and 0xFF's) from a terminating 0x00, embedded 0x00's and 0xFFare 
prefixed with a 0xFF byte. This means that when you process one byte at 
a time you have to perform a check on every byte. This results in 
totally unacceptable response times (My first version of this client was 
based on this approach)


The only alternative solution I can think off is to use C++ to create a 
socket and a function that reads from the socket. But since I have 
hardly any experience with C++ programming nor using the rcpp package


Ben

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


Re: [R-pkg-devel] socketConnection, delay when reading from

2021-11-27 Thread Jeff Newmiller
This is a null-terminated message protocol [1]. It has to be processed one byte 
at a time.

[1] https://docs.basex.org/wiki/Server_Protocol

On November 27, 2021 7:45:31 AM PST, Gabor Grothendieck 
 wrote:
>Whether the length is variable or not isn't relevant. The point is
>whether the message is prefaced by a length or command from which the
>length can be derived.  Maybe it is not and you will have to rely on
>inefficient methods but in many cases protocols are designed to avoid
>such problems.
>
>On Sat, Nov 27, 2021 at 9:40 AM Ben Engbers  wrote:
>>
>> No, according to the specification the minimal number of bytes that is
>> returned is 2. There is no maximum. (When querying a database you never
>> know on forehand how many records match the query so by definition you
>> can't calculate the size of the message).
>>
>> In some C, C++ or Java-code I found on internet, it was possible to
>> change the timeout settings so that there would be no delay. Of course
>> this would have as consequence that in your code you have to deal with
>> the possibility that the message has not been completely returned.
>>
>> In R you can set the timeout to 0 but that results in errors (at least
>> on Windows)
>>
>> Op 27-11-2021 om 14:57 schreef Gabor Grothendieck:
>> > Does the message start with a length or a command whose argument length is 
>> > known
>> > depending on the particular command?
>> > If so first read the length or command and from that the length of the
>> > remainder of
>> > the message can be determined.
>> >
>> > On Sat, Nov 27, 2021 at 4:09 AM Ben Engbers  
>> > wrote:
>> >>
>> >>
>> >> Hi,
>> >>
>> >> I have been working on a R-client for the BaseX XML-database and version
>> >> 0.9.2 is nearly finished (submitting version 0.9.0 was rejected by CRAN).
>> >> Version 0.3 of RBaseX can be found here
>> >> (https://cran.microsoft.com/web/packages/RBaseX/index.html).
>> >>
>> >> The client-server protocol specifies that the communication between the
>> >> client and the database is based on a socket. The code (below) shows how
>> >> I create that socket.
>> >>
>> >> Writing to the socket works perfect. Reading from the sockets (see
>> >> second codeblock) also produces correct results. The problem however is
>> >> that the timeout, as specified when initializing the socket, causes a 1
>> >> second delay for every read-operation.
>> >> I have experimented a lot with different settings and have been
>> >> searching a lot on internet, but I can't find any method to get rid of
>> >> that delay. (In C or C++ that should be easier but I have never before
>> >> had any need to use those languages).
>> >> The very first version of my client used a block-size of 1 when reading.
>> >> That gave acceptable response times for small query-results but reading
>> >> large responses from the database took very long time.
>> >>
>> >> Do you have any suggestions on how to cope with this problem?
>> >>
>> >> Ben Engbers
>> >>
>> >> -
>> >>   CreateSocket = function(host, port = 1984L, username, password) {
>> >> tryCatch(
>> >>   {conn <- private$conn <- socketConnection(
>> >> host = "localhost", port,
>> >> open = "w+b", server = FALSE, blocking = TRUE, encoding =
>> >> "UTF-8", timeout = 1)
>> >>   }, error = function(e) {
>> >> stop("Cannot open the connection")
>> >>   }
>> >> )
>> >>
>> >> -
>> >>
>> >> readBin_ <- function(conn) {
>> >> chars_read <- raw(0)
>> >> rd <- readBin(conn, what = "raw", 1024)
>> >> while(length(rd) == 1024) {
>> >>   chars_read <- c(chars_read, rd)
>> >>   rd <- readBin(conn, "raw", 1024)
>> >>   }
>> >> if (length(rd) > 0) chars_read <- c(chars_read, rd)
>> >> return(chars_read)
>> >> }
>> >>
>> >> __
>> >> R-package-devel@r-project.org mailing list
>> >> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>> >
>> >
>> >
>>
>
>

-- 
Sent from my phone. Please excuse my brevity.

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


Re: [R-pkg-devel] socketConnection, delay when reading from

2021-11-27 Thread Gabor Grothendieck
Whether the length is variable or not isn't relevant. The point is
whether the message is prefaced by a length or command from which the
length can be derived.  Maybe it is not and you will have to rely on
inefficient methods but in many cases protocols are designed to avoid
such problems.

On Sat, Nov 27, 2021 at 9:40 AM Ben Engbers  wrote:
>
> No, according to the specification the minimal number of bytes that is
> returned is 2. There is no maximum. (When querying a database you never
> know on forehand how many records match the query so by definition you
> can't calculate the size of the message).
>
> In some C, C++ or Java-code I found on internet, it was possible to
> change the timeout settings so that there would be no delay. Of course
> this would have as consequence that in your code you have to deal with
> the possibility that the message has not been completely returned.
>
> In R you can set the timeout to 0 but that results in errors (at least
> on Windows)
>
> Op 27-11-2021 om 14:57 schreef Gabor Grothendieck:
> > Does the message start with a length or a command whose argument length is 
> > known
> > depending on the particular command?
> > If so first read the length or command and from that the length of the
> > remainder of
> > the message can be determined.
> >
> > On Sat, Nov 27, 2021 at 4:09 AM Ben Engbers  
> > wrote:
> >>
> >>
> >> Hi,
> >>
> >> I have been working on a R-client for the BaseX XML-database and version
> >> 0.9.2 is nearly finished (submitting version 0.9.0 was rejected by CRAN).
> >> Version 0.3 of RBaseX can be found here
> >> (https://cran.microsoft.com/web/packages/RBaseX/index.html).
> >>
> >> The client-server protocol specifies that the communication between the
> >> client and the database is based on a socket. The code (below) shows how
> >> I create that socket.
> >>
> >> Writing to the socket works perfect. Reading from the sockets (see
> >> second codeblock) also produces correct results. The problem however is
> >> that the timeout, as specified when initializing the socket, causes a 1
> >> second delay for every read-operation.
> >> I have experimented a lot with different settings and have been
> >> searching a lot on internet, but I can't find any method to get rid of
> >> that delay. (In C or C++ that should be easier but I have never before
> >> had any need to use those languages).
> >> The very first version of my client used a block-size of 1 when reading.
> >> That gave acceptable response times for small query-results but reading
> >> large responses from the database took very long time.
> >>
> >> Do you have any suggestions on how to cope with this problem?
> >>
> >> Ben Engbers
> >>
> >> -
> >>   CreateSocket = function(host, port = 1984L, username, password) {
> >> tryCatch(
> >>   {conn <- private$conn <- socketConnection(
> >> host = "localhost", port,
> >> open = "w+b", server = FALSE, blocking = TRUE, encoding =
> >> "UTF-8", timeout = 1)
> >>   }, error = function(e) {
> >> stop("Cannot open the connection")
> >>   }
> >> )
> >>
> >> -
> >>
> >> readBin_ <- function(conn) {
> >> chars_read <- raw(0)
> >> rd <- readBin(conn, what = "raw", 1024)
> >> while(length(rd) == 1024) {
> >>   chars_read <- c(chars_read, rd)
> >>   rd <- readBin(conn, "raw", 1024)
> >>   }
> >> if (length(rd) > 0) chars_read <- c(chars_read, rd)
> >> return(chars_read)
> >> }
> >>
> >> __
> >> R-package-devel@r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-package-devel
> >
> >
> >
>


-- 
Statistics & Software Consulting
GKX Group, GKX Associates Inc.
tel: 1-877-GKX-GROUP
email: ggrothendieck at gmail.com

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


Re: [R-pkg-devel] socketConnection, delay when reading from

2021-11-27 Thread Ben Engbers
No, according to the specification the minimal number of bytes that is 
returned is 2. There is no maximum. (When querying a database you never 
know on forehand how many records match the query so by definition you 
can't calculate the size of the message).


In some C, C++ or Java-code I found on internet, it was possible to 
change the timeout settings so that there would be no delay. Of course 
this would have as consequence that in your code you have to deal with 
the possibility that the message has not been completely returned.


In R you can set the timeout to 0 but that results in errors (at least 
on Windows)


Op 27-11-2021 om 14:57 schreef Gabor Grothendieck:

Does the message start with a length or a command whose argument length is known
depending on the particular command?
If so first read the length or command and from that the length of the
remainder of
the message can be determined.

On Sat, Nov 27, 2021 at 4:09 AM Ben Engbers  wrote:



Hi,

I have been working on a R-client for the BaseX XML-database and version
0.9.2 is nearly finished (submitting version 0.9.0 was rejected by CRAN).
Version 0.3 of RBaseX can be found here
(https://cran.microsoft.com/web/packages/RBaseX/index.html).

The client-server protocol specifies that the communication between the
client and the database is based on a socket. The code (below) shows how
I create that socket.

Writing to the socket works perfect. Reading from the sockets (see
second codeblock) also produces correct results. The problem however is
that the timeout, as specified when initializing the socket, causes a 1
second delay for every read-operation.
I have experimented a lot with different settings and have been
searching a lot on internet, but I can't find any method to get rid of
that delay. (In C or C++ that should be easier but I have never before
had any need to use those languages).
The very first version of my client used a block-size of 1 when reading.
That gave acceptable response times for small query-results but reading
large responses from the database took very long time.

Do you have any suggestions on how to cope with this problem?

Ben Engbers

-
  CreateSocket = function(host, port = 1984L, username, password) {
tryCatch(
  {conn <- private$conn <- socketConnection(
host = "localhost", port,
open = "w+b", server = FALSE, blocking = TRUE, encoding =
"UTF-8", timeout = 1)
  }, error = function(e) {
stop("Cannot open the connection")
  }
)

-

readBin_ <- function(conn) {
chars_read <- raw(0)
rd <- readBin(conn, what = "raw", 1024)
while(length(rd) == 1024) {
  chars_read <- c(chars_read, rd)
  rd <- readBin(conn, "raw", 1024)
  }
if (length(rd) > 0) chars_read <- c(chars_read, rd)
return(chars_read)
}

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel






__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


Re: [R-pkg-devel] socketConnection, delay when reading from

2021-11-27 Thread Gabor Grothendieck
Does the message start with a length or a command whose argument length is known
depending on the particular command?
If so first read the length or command and from that the length of the
remainder of
the message can be determined.

On Sat, Nov 27, 2021 at 4:09 AM Ben Engbers  wrote:
>
>
> Hi,
>
> I have been working on a R-client for the BaseX XML-database and version
> 0.9.2 is nearly finished (submitting version 0.9.0 was rejected by CRAN).
> Version 0.3 of RBaseX can be found here
> (https://cran.microsoft.com/web/packages/RBaseX/index.html).
>
> The client-server protocol specifies that the communication between the
> client and the database is based on a socket. The code (below) shows how
> I create that socket.
>
> Writing to the socket works perfect. Reading from the sockets (see
> second codeblock) also produces correct results. The problem however is
> that the timeout, as specified when initializing the socket, causes a 1
> second delay for every read-operation.
> I have experimented a lot with different settings and have been
> searching a lot on internet, but I can't find any method to get rid of
> that delay. (In C or C++ that should be easier but I have never before
> had any need to use those languages).
> The very first version of my client used a block-size of 1 when reading.
> That gave acceptable response times for small query-results but reading
> large responses from the database took very long time.
>
> Do you have any suggestions on how to cope with this problem?
>
> Ben Engbers
>
> -
>  CreateSocket = function(host, port = 1984L, username, password) {
>tryCatch(
>  {conn <- private$conn <- socketConnection(
>host = "localhost", port,
>open = "w+b", server = FALSE, blocking = TRUE, encoding =
> "UTF-8", timeout = 1)
>  }, error = function(e) {
>stop("Cannot open the connection")
>  }
>)
>
> -
>
> readBin_ <- function(conn) {
>chars_read <- raw(0)
>rd <- readBin(conn, what = "raw", 1024)
>while(length(rd) == 1024) {
>  chars_read <- c(chars_read, rd)
>  rd <- readBin(conn, "raw", 1024)
>  }
>if (length(rd) > 0) chars_read <- c(chars_read, rd)
>return(chars_read)
> }
>
> __
> R-package-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel



-- 
Statistics & Software Consulting
GKX Group, GKX Associates Inc.
tel: 1-877-GKX-GROUP
email: ggrothendieck at gmail.com

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


Re: [R-pkg-devel] socketConnection, delay when reading from

2021-11-27 Thread Ben.Engbers
That's what I figured out also. The problem is that even when I reduce the 
size to 2 bytes and there is only 1byte available in the socket, the code will 
wait for 1 second which is the minimum time-out.BenVerzonden vanaf mijn Galaxy
 Oorspronkelijk bericht Van: Duncan Murdoch 
 Datum: 27-11-2021  12:02  (GMT+01:00) Aan: 
ben.engb...@be-logical.nl, r-package-devel@r-project.org Onderwerp: Re: 
[R-pkg-devel] socketConnection, delay when reading from On 27/11/2021 4:09 
a.m., Ben Engbers wrote:> > Hi,> > I have been working on a R-client for the 
BaseX XML-database and version> 0.9.2 is nearly finished (submitting version 
0.9.0 was rejected by CRAN).> Version 0.3 of RBaseX can be found here> 
(https://cran.microsoft.com/web/packages/RBaseX/index.html).> > The 
client-server protocol specifies that the communication between the> client and 
the database is based on a socket. The code (below) shows how> I create that 
socket.> > Writing to the socket works perfect. Reading from the sockets (see> 
second codeblock) also produces correct results. The problem however is> that 
the timeout, as specified when initializing the socket, causes a 1> second 
delay for every read-operation.> I have experimented a lot with different 
settings and have been> searching a lot on internet, but I can't find any 
method to get rid of> that delay. (In C or C++ that should be easier but I have 
never before> had any need to use those languages).> The very first version of 
my client used a block-size of 1 when reading.> That gave acceptable response 
times for small query-results but reading> large responses from the database 
took very long time.> > Do you have any suggestions on how to cope with this 
problem?You are attempting to read 1024 bytes.  If the socket returns fewer, 
the code presumably assumes it should wait for more.  You should specify the 
number to read to be no larger than you'll receive.  That's why reading a byte 
at a time works well for small reads.To handle larger ones, you need to know 
what size blocks you are going to receive with each read.  If that varies 
unpredictably, I doubt if there's much you can do other than reducing the 
timeout.Duncan Murdoch> > Ben Engbers> > ->   
CreateSocket = function(host, port = 1984L, username, password) {> 
tryCatch(>   {conn <- private$conn <- socketConnection(> 
host = "localhost", port,> open = "w+b", server = FALSE, blocking = 
TRUE, encoding => "UTF-8", timeout = 1)>   }, error = function(e) {>
 stop("Cannot open the connection")>   }> )> > 
-> > readBin_ <- function(conn) {> chars_read 
<- raw(0)> rd <- readBin(conn, what = "raw", 1024)> while(length(rd) == 
1024) {>   chars_read <- c(chars_read, rd)>   rd <- readBin(conn, 
"raw", 1024)>   }> if (length(rd) > 0) chars_read <- c(chars_read, rd)> 
    return(chars_read)> }> > __> 
R-package-devel@r-project.org mailing list> 
https://stat.ethz.ch/mailman/listinfo/r-package-devel> 
[[alternative HTML version deleted]]

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


Re: [R-pkg-devel] socketConnection, delay when reading from

2021-11-27 Thread Duncan Murdoch

On 27/11/2021 4:09 a.m., Ben Engbers wrote:


Hi,

I have been working on a R-client for the BaseX XML-database and version
0.9.2 is nearly finished (submitting version 0.9.0 was rejected by CRAN).
Version 0.3 of RBaseX can be found here
(https://cran.microsoft.com/web/packages/RBaseX/index.html).

The client-server protocol specifies that the communication between the
client and the database is based on a socket. The code (below) shows how
I create that socket.

Writing to the socket works perfect. Reading from the sockets (see
second codeblock) also produces correct results. The problem however is
that the timeout, as specified when initializing the socket, causes a 1
second delay for every read-operation.
I have experimented a lot with different settings and have been
searching a lot on internet, but I can't find any method to get rid of
that delay. (In C or C++ that should be easier but I have never before
had any need to use those languages).
The very first version of my client used a block-size of 1 when reading.
That gave acceptable response times for small query-results but reading
large responses from the database took very long time.

Do you have any suggestions on how to cope with this problem?


You are attempting to read 1024 bytes.  If the socket returns fewer, the 
code presumably assumes it should wait for more.  You should specify the 
number to read to be no larger than you'll receive.  That's why reading 
a byte at a time works well for small reads.


To handle larger ones, you need to know what size blocks you are going 
to receive with each read.  If that varies unpredictably, I doubt if 
there's much you can do other than reducing the timeout.


Duncan Murdoch



Ben Engbers

-
  CreateSocket = function(host, port = 1984L, username, password) {
tryCatch(
  {conn <- private$conn <- socketConnection(
host = "localhost", port,
open = "w+b", server = FALSE, blocking = TRUE, encoding =
"UTF-8", timeout = 1)
  }, error = function(e) {
stop("Cannot open the connection")
  }
)

-

readBin_ <- function(conn) {
chars_read <- raw(0)
rd <- readBin(conn, what = "raw", 1024)
while(length(rd) == 1024) {
  chars_read <- c(chars_read, rd)
  rd <- readBin(conn, "raw", 1024)
  }
if (length(rd) > 0) chars_read <- c(chars_read, rd)
return(chars_read)
}

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel



__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel


[R-pkg-devel] socketConnection, delay when reading from

2021-11-27 Thread Ben Engbers



Hi,

I have been working on a R-client for the BaseX XML-database and version 
0.9.2 is nearly finished (submitting version 0.9.0 was rejected by CRAN).
Version 0.3 of RBaseX can be found here 
(https://cran.microsoft.com/web/packages/RBaseX/index.html).


The client-server protocol specifies that the communication between the 
client and the database is based on a socket. The code (below) shows how 
I create that socket.


Writing to the socket works perfect. Reading from the sockets (see 
second codeblock) also produces correct results. The problem however is 
that the timeout, as specified when initializing the socket, causes a 1 
second delay for every read-operation.
I have experimented a lot with different settings and have been 
searching a lot on internet, but I can't find any method to get rid of 
that delay. (In C or C++ that should be easier but I have never before 
had any need to use those languages).
The very first version of my client used a block-size of 1 when reading. 
That gave acceptable response times for small query-results but reading 
large responses from the database took very long time.


Do you have any suggestions on how to cope with this problem?

Ben Engbers

-
CreateSocket = function(host, port = 1984L, username, password) {
  tryCatch(
{conn <- private$conn <- socketConnection(
  host = "localhost", port,
  open = "w+b", server = FALSE, blocking = TRUE, encoding = 
"UTF-8", timeout = 1)

}, error = function(e) {
  stop("Cannot open the connection")
}
  )

-

readBin_ <- function(conn) {
  chars_read <- raw(0)
  rd <- readBin(conn, what = "raw", 1024)
  while(length(rd) == 1024) {
chars_read <- c(chars_read, rd)
rd <- readBin(conn, "raw", 1024)
}
  if (length(rd) > 0) chars_read <- c(chars_read, rd)
  return(chars_read)
}

__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel