Re: Thin Client Protocol documentation
Hi Andrey, I’m afraid there are no plans for this features yet (at least I didn’t spot any relevant discussion on @dev). Feel free to start a separate discussion proposing the protocol expansion. It’s better not to intermix this with the documentation thread. — Denis > On Dec 15, 2017, at 11:12 AM, Andrey Kornev <andrewkor...@hotmail.com> wrote: > > Pavel, could you please at least share your time frame wrt adding compute, > clustering and services features to the spec? > > Thanks > Andrey > > From: Pavel Tupitsyn <ptupit...@gridgain.com> > Sent: Tuesday, December 5, 2017 11:15 PM > To: dev@ignite.apache.org > Subject: Re: Thin Client Protocol documentation > > Andrey, > > As I understand, you suggest to document every prospective feature right > now. > That would be (at least) compute, clustering, transactions, services, > messages, events, failover, data structures, metrics. > > I don't think it makes sense to do that. Another problem is that it will > take weeks :) > But, of course, you are welcome to take the initiative. > > Thanks, > Pavel > > On Tue, Dec 5, 2017 at 10:20 PM, Andrey Kornev <andrewkor...@hotmail.com> > wrote: > >> Pavel, >> >> I have absolutely no doubts that support for all those features will be >> added eventually. And if so, wouldn't it be the right thing to do to >> document the operations and their semantics right now without >> necessarily implementing them? It should really help to ensure that the >> protocol can accommodate all those use cases before it gets released to the >> public. >> >> Thanks >> Andrey >> ------ >> *From:* Pavel Tupitsyn <ptupit...@gridgain.com> >> *Sent:* Tuesday, December 5, 2017 12:07 AM >> *Cc:* dev@ignite.apache.org >> *Subject:* Re: Thin Client Protocol documentation >> >> Andrey, >> >> All of this is to come :) >> >> >> Prachi, >> >> 1) There are no flags, see the doc >> 2) String is simply 4 byte length (n) + n bytes >> 3) Op codes have changed >> >> writeByteLittleEndian(1051, out); // Type code >> writeIntLittleEndian(20, out); // String length >> out.writeUTF("myNewCache"); // Cache name >> >> On Tue, Dec 5, 2017 at 4:39 AM, Prachi Garg <pg...@gridgain.com> wrote: >> >>> Hi Pavel, >>> >>> I am trying to use the OP_CACHE_CREATE_WITH_NAME operation, but can't get >>> it to work. Digging deeper into the source code, it seems like I have to >>> provide a flag, string length, and position, in addition to the type code >>> and the actual string. Is that correct? >>> >>> Here is the request I am sending to the server- >>> >>> DataOutputStream out = new DataOutputStream(socket.getOutputStream()); >>> >>> // Message length >>> writeIntLittleEndian(24, out); >>> >>> // Op code = OP_CACHE_CREATE_WITH_NAME >>> writeShortLittleEndian(1051, out); >>> >>> // Request id >>> long reqId = 1; >>> writeLongLittleEndian(reqId, out); >>> >>> // String (cache name) >>> writeByteLittleEndian(9, out); // Type code >>> writeByteLittleEndian(0, out); // Flag >>> writeIntLittleEndian(20, out); // String length >>> writeIntLittleEndian(0, out); // Position >>> out.writeUTF("myNewCache"); // Cache name >>> >>> // Send request >>> out.flush(); >>> >>> But I get the following error on the server side. >>> >>> [2017-12-04 17:27:39,421][ERROR][client-connector-#53][ >> ClientListenerNioListener] >>> Failed to parse client request. >>> java.lang.StringIndexOutOfBoundsException: String index out of range: >> 2575 >>> at java.lang.String.checkBounds(String.java:385) >>> at java.lang.String.(String.java:462) >>> at org.apache.ignite.internal.binary.BinaryUtils. >>> doReadString(BinaryUtils.java:1314) >>> at org.apache.ignite.internal.binary.BinaryReaderExImpl. >>> readString(BinaryReaderExImpl.java:1055) >>> at org.apache.ignite.internal.processors.platform.client.cache. >>> ClientCacheCreateWithNameRequest.(ClientCacheCreateWithNameReque >>> st.java:43) >>> at org.apache.ignite.internal.processors.platform.client. >>> ClientMessageParser.decode(ClientMessageParser.java:318) >>> at org.apache.ignite.internal.processors.platform.client. >>> ClientMessageParser.decode(ClientMessageParser.java:220)
Re: Thin Client Protocol documentation
Pavel, could you please at least share your time frame wrt adding compute, clustering and services features to the spec? Thanks Andrey From: Pavel Tupitsyn <ptupit...@gridgain.com> Sent: Tuesday, December 5, 2017 11:15 PM To: dev@ignite.apache.org Subject: Re: Thin Client Protocol documentation Andrey, As I understand, you suggest to document every prospective feature right now. That would be (at least) compute, clustering, transactions, services, messages, events, failover, data structures, metrics. I don't think it makes sense to do that. Another problem is that it will take weeks :) But, of course, you are welcome to take the initiative. Thanks, Pavel On Tue, Dec 5, 2017 at 10:20 PM, Andrey Kornev <andrewkor...@hotmail.com> wrote: > Pavel, > > I have absolutely no doubts that support for all those features will be > added eventually. And if so, wouldn't it be the right thing to do to > document the operations and their semantics right now without > necessarily implementing them? It should really help to ensure that the > protocol can accommodate all those use cases before it gets released to the > public. > > Thanks > Andrey > -- > *From:* Pavel Tupitsyn <ptupit...@gridgain.com> > *Sent:* Tuesday, December 5, 2017 12:07 AM > *Cc:* dev@ignite.apache.org > *Subject:* Re: Thin Client Protocol documentation > > Andrey, > > All of this is to come :) > > > Prachi, > > 1) There are no flags, see the doc > 2) String is simply 4 byte length (n) + n bytes > 3) Op codes have changed > > writeByteLittleEndian(1051, out); // Type code > writeIntLittleEndian(20, out); // String length > out.writeUTF("myNewCache"); // Cache name > > On Tue, Dec 5, 2017 at 4:39 AM, Prachi Garg <pg...@gridgain.com> wrote: > > > Hi Pavel, > > > > I am trying to use the OP_CACHE_CREATE_WITH_NAME operation, but can't get > > it to work. Digging deeper into the source code, it seems like I have to > > provide a flag, string length, and position, in addition to the type code > > and the actual string. Is that correct? > > > > Here is the request I am sending to the server- > > > > DataOutputStream out = new DataOutputStream(socket.getOutputStream()); > > > > // Message length > > writeIntLittleEndian(24, out); > > > > // Op code = OP_CACHE_CREATE_WITH_NAME > > writeShortLittleEndian(1051, out); > > > > // Request id > > long reqId = 1; > > writeLongLittleEndian(reqId, out); > > > > // String (cache name) > > writeByteLittleEndian(9, out); // Type code > > writeByteLittleEndian(0, out); // Flag > > writeIntLittleEndian(20, out); // String length > > writeIntLittleEndian(0, out); // Position > > out.writeUTF("myNewCache"); // Cache name > > > > // Send request > > out.flush(); > > > > But I get the following error on the server side. > > > > [2017-12-04 17:27:39,421][ERROR][client-connector-#53][ > ClientListenerNioListener] > > Failed to parse client request. > > java.lang.StringIndexOutOfBoundsException: String index out of range: > 2575 > > at java.lang.String.checkBounds(String.java:385) > > at java.lang.String.(String.java:462) > > at org.apache.ignite.internal.binary.BinaryUtils. > > doReadString(BinaryUtils.java:1314) > > at org.apache.ignite.internal.binary.BinaryReaderExImpl. > > readString(BinaryReaderExImpl.java:1055) > > at org.apache.ignite.internal.processors.platform.client.cache. > > ClientCacheCreateWithNameRequest.(ClientCacheCreateWithNameReque > > st.java:43) > > at org.apache.ignite.internal.processors.platform.client. > > ClientMessageParser.decode(ClientMessageParser.java:318) > > at org.apache.ignite.internal.processors.platform.client. > > ClientMessageParser.decode(ClientMessageParser.java:220) > > at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener. > > onMessage(ClientListenerNioListener.java:119) > > at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener. > > onMessage(ClientListenerNioListener.java:40) > > at org.apache.ignite.internal.util.nio.GridNioFilterChain$ > > TailFilter.onMessageReceived(GridNioFilterChain.java:279) > > at org.apache.ignite.internal.util.nio.GridNioFilterAdapter. > > proceedMessageReceived(GridNioFilterAdapter.java:109) > > at org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3. > > body(GridNioAsyncNotifyFilter.java:97) > > at org.apache.ignite.internal.util.worker.GridWorker.run( > > GridWorker.java:110) > > at org.apache.ignite.internal.util.worker.GridW
Re: Thin Client Protocol documentation
>>> I don't think it makes sense to do that. Another problem is that it will >>> take weeks :) I'm sorry, Pavel. None of what you've said so far makes it obvious why it doesn't make sense to produce the complete spec. Would you care to elaborate, please? Also, what's the problem with the "another problem" of the effort taking weeks? If based on your best estimate the effort requires a few weeks of your time, why is this a problem then? >>> But, of course, you are welcome to take the initiative. Saying something like that is not very helpful, Pavel. But let me just say, I believe that as the current spec lead and an employee of the company behind Apache Ignite project you're in a much better position to continue to lead the effort (you've been doing great so far!). From my side, I'd be more than happy to review the spec and provide my feedback. Also, I'm very interested in becoming a contributor/early adopter of the Java-based implementation of the protocol. Regards Andrey From: Pavel Tupitsyn <ptupit...@gridgain.com> Sent: Tuesday, December 5, 2017 11:15 PM To: dev@ignite.apache.org Subject: Re: Thin Client Protocol documentation Andrey, As I understand, you suggest to document every prospective feature right now. That would be (at least) compute, clustering, transactions, services, messages, events, failover, data structures, metrics. I don't think it makes sense to do that. Another problem is that it will take weeks :) But, of course, you are welcome to take the initiative. Thanks, Pavel On Tue, Dec 5, 2017 at 10:20 PM, Andrey Kornev <andrewkor...@hotmail.com> wrote: > Pavel, > > I have absolutely no doubts that support for all those features will be > added eventually. And if so, wouldn't it be the right thing to do to > document the operations and their semantics right now without > necessarily implementing them? It should really help to ensure that the > protocol can accommodate all those use cases before it gets released to the > public. > > Thanks > Andrey > -- > *From:* Pavel Tupitsyn <ptupit...@gridgain.com> > *Sent:* Tuesday, December 5, 2017 12:07 AM > *Cc:* dev@ignite.apache.org > *Subject:* Re: Thin Client Protocol documentation > > Andrey, > > All of this is to come :) > > > Prachi, > > 1) There are no flags, see the doc > 2) String is simply 4 byte length (n) + n bytes > 3) Op codes have changed > > writeByteLittleEndian(1051, out); // Type code > writeIntLittleEndian(20, out); // String length > out.writeUTF("myNewCache"); // Cache name > > On Tue, Dec 5, 2017 at 4:39 AM, Prachi Garg <pg...@gridgain.com> wrote: > > > Hi Pavel, > > > > I am trying to use the OP_CACHE_CREATE_WITH_NAME operation, but can't get > > it to work. Digging deeper into the source code, it seems like I have to > > provide a flag, string length, and position, in addition to the type code > > and the actual string. Is that correct? > > > > Here is the request I am sending to the server- > > > > DataOutputStream out = new DataOutputStream(socket.getOutputStream()); > > > > // Message length > > writeIntLittleEndian(24, out); > > > > // Op code = OP_CACHE_CREATE_WITH_NAME > > writeShortLittleEndian(1051, out); > > > > // Request id > > long reqId = 1; > > writeLongLittleEndian(reqId, out); > > > > // String (cache name) > > writeByteLittleEndian(9, out); // Type code > > writeByteLittleEndian(0, out); // Flag > > writeIntLittleEndian(20, out); // String length > > writeIntLittleEndian(0, out); // Position > > out.writeUTF("myNewCache"); // Cache name > > > > // Send request > > out.flush(); > > > > But I get the following error on the server side. > > > > [2017-12-04 17:27:39,421][ERROR][client-connector-#53][ > ClientListenerNioListener] > > Failed to parse client request. > > java.lang.StringIndexOutOfBoundsException: String index out of range: > 2575 > > at java.lang.String.checkBounds(String.java:385) > > at java.lang.String.(String.java:462) > > at org.apache.ignite.internal.binary.BinaryUtils. > > doReadString(BinaryUtils.java:1314) > > at org.apache.ignite.internal.binary.BinaryReaderExImpl. > > readString(BinaryReaderExImpl.java:1055) > > at org.apache.ignite.internal.processors.platform.client.cache. > > ClientCacheCreateWithNameRequest.(ClientCacheCreateWithNameReque > > st.java:43) > > at org.apache.ignite.internal.processors.platform.client. > > ClientMessageParser.decode(ClientMessageParser.java:318) > > at org.apache.igni
Re: Thin Client Protocol documentation
Andrey, As I understand, you suggest to document every prospective feature right now. That would be (at least) compute, clustering, transactions, services, messages, events, failover, data structures, metrics. I don't think it makes sense to do that. Another problem is that it will take weeks :) But, of course, you are welcome to take the initiative. Thanks, Pavel On Tue, Dec 5, 2017 at 10:20 PM, Andrey Kornev <andrewkor...@hotmail.com> wrote: > Pavel, > > I have absolutely no doubts that support for all those features will be > added eventually. And if so, wouldn't it be the right thing to do to > document the operations and their semantics right now without > necessarily implementing them? It should really help to ensure that the > protocol can accommodate all those use cases before it gets released to the > public. > > Thanks > Andrey > -- > *From:* Pavel Tupitsyn <ptupit...@gridgain.com> > *Sent:* Tuesday, December 5, 2017 12:07 AM > *Cc:* dev@ignite.apache.org > *Subject:* Re: Thin Client Protocol documentation > > Andrey, > > All of this is to come :) > > > Prachi, > > 1) There are no flags, see the doc > 2) String is simply 4 byte length (n) + n bytes > 3) Op codes have changed > > writeByteLittleEndian(1051, out); // Type code > writeIntLittleEndian(20, out); // String length > out.writeUTF("myNewCache"); // Cache name > > On Tue, Dec 5, 2017 at 4:39 AM, Prachi Garg <pg...@gridgain.com> wrote: > > > Hi Pavel, > > > > I am trying to use the OP_CACHE_CREATE_WITH_NAME operation, but can't get > > it to work. Digging deeper into the source code, it seems like I have to > > provide a flag, string length, and position, in addition to the type code > > and the actual string. Is that correct? > > > > Here is the request I am sending to the server- > > > > DataOutputStream out = new DataOutputStream(socket.getOutputStream()); > > > > // Message length > > writeIntLittleEndian(24, out); > > > > // Op code = OP_CACHE_CREATE_WITH_NAME > > writeShortLittleEndian(1051, out); > > > > // Request id > > long reqId = 1; > > writeLongLittleEndian(reqId, out); > > > > // String (cache name) > > writeByteLittleEndian(9, out); // Type code > > writeByteLittleEndian(0, out); // Flag > > writeIntLittleEndian(20, out); // String length > > writeIntLittleEndian(0, out); // Position > > out.writeUTF("myNewCache"); // Cache name > > > > // Send request > > out.flush(); > > > > But I get the following error on the server side. > > > > [2017-12-04 17:27:39,421][ERROR][client-connector-#53][ > ClientListenerNioListener] > > Failed to parse client request. > > java.lang.StringIndexOutOfBoundsException: String index out of range: > 2575 > > at java.lang.String.checkBounds(String.java:385) > > at java.lang.String.(String.java:462) > > at org.apache.ignite.internal.binary.BinaryUtils. > > doReadString(BinaryUtils.java:1314) > > at org.apache.ignite.internal.binary.BinaryReaderExImpl. > > readString(BinaryReaderExImpl.java:1055) > > at org.apache.ignite.internal.processors.platform.client.cache. > > ClientCacheCreateWithNameRequest.(ClientCacheCreateWithNameReque > > st.java:43) > > at org.apache.ignite.internal.processors.platform.client. > > ClientMessageParser.decode(ClientMessageParser.java:318) > > at org.apache.ignite.internal.processors.platform.client. > > ClientMessageParser.decode(ClientMessageParser.java:220) > > at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener. > > onMessage(ClientListenerNioListener.java:119) > > at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener. > > onMessage(ClientListenerNioListener.java:40) > > at org.apache.ignite.internal.util.nio.GridNioFilterChain$ > > TailFilter.onMessageReceived(GridNioFilterChain.java:279) > > at org.apache.ignite.internal.util.nio.GridNioFilterAdapter. > > proceedMessageReceived(GridNioFilterAdapter.java:109) > > at org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3. > > body(GridNioAsyncNotifyFilter.java:97) > > at org.apache.ignite.internal.util.worker.GridWorker.run( > > GridWorker.java:110) > > at org.apache.ignite.internal.util.worker.GridWorkerPool$1. > > run(GridWorkerPool.java:70) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1142) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > > > What
Re: Thin Client Protocol documentation
Pavel, I have absolutely no doubts that support for all those features will be added eventually. And if so, wouldn't it be the right thing to do to document the operations and their semantics right now without necessarily implementing them? It should really help to ensure that the protocol can accommodate all those use cases before it gets released to the public. Thanks Andrey From: Pavel Tupitsyn <ptupit...@gridgain.com> Sent: Tuesday, December 5, 2017 12:07 AM Cc: dev@ignite.apache.org Subject: Re: Thin Client Protocol documentation Andrey, All of this is to come :) Prachi, 1) There are no flags, see the doc 2) String is simply 4 byte length (n) + n bytes 3) Op codes have changed writeByteLittleEndian(1051, out); // Type code writeIntLittleEndian(20, out); // String length out.writeUTF("myNewCache"); // Cache name On Tue, Dec 5, 2017 at 4:39 AM, Prachi Garg <pg...@gridgain.com> wrote: > Hi Pavel, > > I am trying to use the OP_CACHE_CREATE_WITH_NAME operation, but can't get > it to work. Digging deeper into the source code, it seems like I have to > provide a flag, string length, and position, in addition to the type code > and the actual string. Is that correct? > > Here is the request I am sending to the server- > > DataOutputStream out = new DataOutputStream(socket.getOutputStream()); > > // Message length > writeIntLittleEndian(24, out); > > // Op code = OP_CACHE_CREATE_WITH_NAME > writeShortLittleEndian(1051, out); > > // Request id > long reqId = 1; > writeLongLittleEndian(reqId, out); > > // String (cache name) > writeByteLittleEndian(9, out); // Type code > writeByteLittleEndian(0, out); // Flag > writeIntLittleEndian(20, out); // String length > writeIntLittleEndian(0, out); // Position > out.writeUTF("myNewCache"); // Cache name > > // Send request > out.flush(); > > But I get the following error on the server side. > > [2017-12-04 > 17:27:39,421][ERROR][client-connector-#53][ClientListenerNioListener] > Failed to parse client request. > java.lang.StringIndexOutOfBoundsException: String index out of range: 2575 > at java.lang.String.checkBounds(String.java:385) > at java.lang.String.(String.java:462) > at org.apache.ignite.internal.binary.BinaryUtils. > doReadString(BinaryUtils.java:1314) > at org.apache.ignite.internal.binary.BinaryReaderExImpl. > readString(BinaryReaderExImpl.java:1055) > at org.apache.ignite.internal.processors.platform.client.cache. > ClientCacheCreateWithNameRequest.(ClientCacheCreateWithNameReque > st.java:43) > at org.apache.ignite.internal.processors.platform.client. > ClientMessageParser.decode(ClientMessageParser.java:318) > at org.apache.ignite.internal.processors.platform.client. > ClientMessageParser.decode(ClientMessageParser.java:220) > at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener. > onMessage(ClientListenerNioListener.java:119) > at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener. > onMessage(ClientListenerNioListener.java:40) > at org.apache.ignite.internal.util.nio.GridNioFilterChain$ > TailFilter.onMessageReceived(GridNioFilterChain.java:279) > at org.apache.ignite.internal.util.nio.GridNioFilterAdapter. > proceedMessageReceived(GridNioFilterAdapter.java:109) > at org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3. > body(GridNioAsyncNotifyFilter.java:97) > at org.apache.ignite.internal.util.worker.GridWorker.run( > GridWorker.java:110) > at org.apache.ignite.internal.util.worker.GridWorkerPool$1. > run(GridWorkerPool.java:70) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > What am I missing here? Can you provide an example of how to send a > 'String' type request? > > -Prachi > > On Mon, Dec 4, 2017 at 1:06 PM, Andrey Kornev <andrewkor...@hotmail.com> > wrote: > >> Pavel, >> >> Thanks! While we're at it, are there any plans to add cluster-related >> operations? For example, I think it'd be nice to allow the thin clients to >> obtain a current topology snapshot. This would make it possible the clients >> to send requests directly to the affinity host for colocated computation. >> To make it even more useful, all server responses could optionally include >> the topology version the operation has been executed against. This would >> effectively give us a kind out-of-band topology change notification >> mechanism. This way the clients can detect a topology change and refresh >> the topology snapshot next time they need to compute affinity. >>
Re: Thin Client Protocol documentation
Andrey, All of this is to come :) Prachi, 1) There are no flags, see the doc 2) String is simply 4 byte length (n) + n bytes 3) Op codes have changed writeByteLittleEndian(1051, out); // Type code writeIntLittleEndian(20, out); // String length out.writeUTF("myNewCache"); // Cache name On Tue, Dec 5, 2017 at 4:39 AM, Prachi Garg <pg...@gridgain.com> wrote: > Hi Pavel, > > I am trying to use the OP_CACHE_CREATE_WITH_NAME operation, but can't get > it to work. Digging deeper into the source code, it seems like I have to > provide a flag, string length, and position, in addition to the type code > and the actual string. Is that correct? > > Here is the request I am sending to the server- > > DataOutputStream out = new DataOutputStream(socket.getOutputStream()); > > // Message length > writeIntLittleEndian(24, out); > > // Op code = OP_CACHE_CREATE_WITH_NAME > writeShortLittleEndian(1051, out); > > // Request id > long reqId = 1; > writeLongLittleEndian(reqId, out); > > // String (cache name) > writeByteLittleEndian(9, out); // Type code > writeByteLittleEndian(0, out); // Flag > writeIntLittleEndian(20, out); // String length > writeIntLittleEndian(0, out); // Position > out.writeUTF("myNewCache"); // Cache name > > // Send request > out.flush(); > > But I get the following error on the server side. > > [2017-12-04 > 17:27:39,421][ERROR][client-connector-#53][ClientListenerNioListener] > Failed to parse client request. > java.lang.StringIndexOutOfBoundsException: String index out of range: 2575 > at java.lang.String.checkBounds(String.java:385) > at java.lang.String.(String.java:462) > at org.apache.ignite.internal.binary.BinaryUtils. > doReadString(BinaryUtils.java:1314) > at org.apache.ignite.internal.binary.BinaryReaderExImpl. > readString(BinaryReaderExImpl.java:1055) > at org.apache.ignite.internal.processors.platform.client.cache. > ClientCacheCreateWithNameRequest.(ClientCacheCreateWithNameReque > st.java:43) > at org.apache.ignite.internal.processors.platform.client. > ClientMessageParser.decode(ClientMessageParser.java:318) > at org.apache.ignite.internal.processors.platform.client. > ClientMessageParser.decode(ClientMessageParser.java:220) > at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener. > onMessage(ClientListenerNioListener.java:119) > at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener. > onMessage(ClientListenerNioListener.java:40) > at org.apache.ignite.internal.util.nio.GridNioFilterChain$ > TailFilter.onMessageReceived(GridNioFilterChain.java:279) > at org.apache.ignite.internal.util.nio.GridNioFilterAdapter. > proceedMessageReceived(GridNioFilterAdapter.java:109) > at org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3. > body(GridNioAsyncNotifyFilter.java:97) > at org.apache.ignite.internal.util.worker.GridWorker.run( > GridWorker.java:110) > at org.apache.ignite.internal.util.worker.GridWorkerPool$1. > run(GridWorkerPool.java:70) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > What am I missing here? Can you provide an example of how to send a > 'String' type request? > > -Prachi > > On Mon, Dec 4, 2017 at 1:06 PM, Andrey Kornev <andrewkor...@hotmail.com> > wrote: > >> Pavel, >> >> Thanks! While we're at it, are there any plans to add cluster-related >> operations? For example, I think it'd be nice to allow the thin clients to >> obtain a current topology snapshot. This would make it possible the clients >> to send requests directly to the affinity host for colocated computation. >> To make it even more useful, all server responses could optionally include >> the topology version the operation has been executed against. This would >> effectively give us a kind out-of-band topology change notification >> mechanism. This way the clients can detect a topology change and refresh >> the topology snapshot next time they need to compute affinity. >> >> Regards >> Andrey >> >> From: Pavel Tupitsyn <ptupit...@apache.org> >> Sent: Sunday, December 3, 2017 9:23 AM >> To: dev@ignite.apache.org >> Subject: Re: Thin Client Protocol documentation >> >> Hi Andrey, >> >> Compute and other APIs are certainly planned, cache is just a start. >> We intentionally limit the scope to actually release something in 2.4 and >> not delay it further. >> >> Adding operations to existing protocol is relatively easy. >> Current focus is to make sure that the protocol itself is solid and >> future-proof. >> >> Thanks, >> Pavel >> > >
Re: Thin Client Protocol documentation
Hi Pavel, I am trying to use the OP_CACHE_CREATE_WITH_NAME operation, but can't get it to work. Digging deeper into the source code, it seems like I have to provide a flag, string length, and position, in addition to the type code and the actual string. Is that correct? Here is the request I am sending to the server- DataOutputStream out = new DataOutputStream(socket.getOutputStream()); // Message length writeIntLittleEndian(24, out); // Op code = OP_CACHE_CREATE_WITH_NAME writeShortLittleEndian(1051, out); // Request id long reqId = 1; writeLongLittleEndian(reqId, out); // String (cache name) writeByteLittleEndian(9, out); // Type code writeByteLittleEndian(0, out); // Flag writeIntLittleEndian(20, out); // String length writeIntLittleEndian(0, out); // Position out.writeUTF("myNewCache"); // Cache name // Send request out.flush(); But I get the following error on the server side. [2017-12-04 17:27:39,421][ERROR][client-connector-#53][ClientListenerNioListener] Failed to parse client request. java.lang.StringIndexOutOfBoundsException: String index out of range: 2575 at java.lang.String.checkBounds(String.java:385) at java.lang.String.(String.java:462) at org.apache.ignite.internal.binary.BinaryUtils.doReadString(BinaryUtils.java:1314) at org.apache.ignite.internal.binary.BinaryReaderExImpl.readString(BinaryReaderExImpl.java:1055) at org.apache.ignite.internal.processors.platform.client.cache.ClientCacheCreateWithNameRequest.(ClientCacheCreateWithNameRequest.java:43) at org.apache.ignite.internal.processors.platform.client.ClientMessageParser.decode(ClientMessageParser.java:318) at org.apache.ignite.internal.processors.platform.client.ClientMessageParser.decode(ClientMessageParser.java:220) at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:119) at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:40) at org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279) at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109) at org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.body(GridNioAsyncNotifyFilter.java:97) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at org.apache.ignite.internal.util.worker.GridWorkerPool$1.run(GridWorkerPool.java:70) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) What am I missing here? Can you provide an example of how to send a 'String' type request? -Prachi On Mon, Dec 4, 2017 at 1:06 PM, Andrey Kornev <andrewkor...@hotmail.com> wrote: > Pavel, > > Thanks! While we're at it, are there any plans to add cluster-related > operations? For example, I think it'd be nice to allow the thin clients to > obtain a current topology snapshot. This would make it possible the clients > to send requests directly to the affinity host for colocated computation. > To make it even more useful, all server responses could optionally include > the topology version the operation has been executed against. This would > effectively give us a kind out-of-band topology change notification > mechanism. This way the clients can detect a topology change and refresh > the topology snapshot next time they need to compute affinity. > > Regards > Andrey > > From: Pavel Tupitsyn <ptupit...@apache.org> > Sent: Sunday, December 3, 2017 9:23 AM > To: dev@ignite.apache.org > Subject: Re: Thin Client Protocol documentation > > Hi Andrey, > > Compute and other APIs are certainly planned, cache is just a start. > We intentionally limit the scope to actually release something in 2.4 and > not delay it further. > > Adding operations to existing protocol is relatively easy. > Current focus is to make sure that the protocol itself is solid and > future-proof. > > Thanks, > Pavel >
Re: Thin Client Protocol documentation
Pavel, Thanks! While we're at it, are there any plans to add cluster-related operations? For example, I think it'd be nice to allow the thin clients to obtain a current topology snapshot. This would make it possible the clients to send requests directly to the affinity host for colocated computation. To make it even more useful, all server responses could optionally include the topology version the operation has been executed against. This would effectively give us a kind out-of-band topology change notification mechanism. This way the clients can detect a topology change and refresh the topology snapshot next time they need to compute affinity. Regards Andrey From: Pavel Tupitsyn <ptupit...@apache.org> Sent: Sunday, December 3, 2017 9:23 AM To: dev@ignite.apache.org Subject: Re: Thin Client Protocol documentation Hi Andrey, Compute and other APIs are certainly planned, cache is just a start. We intentionally limit the scope to actually release something in 2.4 and not delay it further. Adding operations to existing protocol is relatively easy. Current focus is to make sure that the protocol itself is solid and future-proof. Thanks, Pavel
Re: Thin Client Protocol documentation
Prachi, Added "Cache flags" note to the doc. Thanks, Pavel On Mon, Dec 4, 2017 at 7:46 PM, Prachi Gargwrote: > Pavel, > > How do you specify the flags (withSkipStore, withExpiryPolicy, etc..) in > bytes? Do you have a code for each of them? > > On Sun, Dec 3, 2017 at 9:23 AM, Pavel Tupitsyn > wrote: > > > Hi Andrey, > > > > Compute and other APIs are certainly planned, cache is just a start. > > We intentionally limit the scope to actually release something in 2.4 and > > not delay it further. > > > > Adding operations to existing protocol is relatively easy. > > Current focus is to make sure that the protocol itself is solid and > > future-proof. > > > > Thanks, > > Pavel > > >
Re: Thin Client Protocol documentation
Pavel, How do you specify the flags (withSkipStore, withExpiryPolicy, etc..) in bytes? Do you have a code for each of them? On Sun, Dec 3, 2017 at 9:23 AM, Pavel Tupitsynwrote: > Hi Andrey, > > Compute and other APIs are certainly planned, cache is just a start. > We intentionally limit the scope to actually release something in 2.4 and > not delay it further. > > Adding operations to existing protocol is relatively easy. > Current focus is to make sure that the protocol itself is solid and > future-proof. > > Thanks, > Pavel >
Re: Thin Client Protocol documentation
Hi Andrey, Compute and other APIs are certainly planned, cache is just a start. We intentionally limit the scope to actually release something in 2.4 and not delay it further. Adding operations to existing protocol is relatively easy. Current focus is to make sure that the protocol itself is solid and future-proof. Thanks, Pavel
Re: Thin Client Protocol documentation
Pavel, Great job on the Thin Client protocol design! I'm a bit late to the party, but I'm wondering why none of the compute features are supported? Such omission is unfortunate. Is it intentional? If so, what are the reasons? I think it would be useful for the clients to be able to invoke already deployed tasks/services, or send a closure (if the client happens to be written in Java) for execution. Thanks Andrey From: Pavel Tupitsyn <ptupit...@apache.org> Sent: Friday, December 1, 2017 5:48 AM To: dev@ignite.apache.org Subject: Re: Thin Client Protocol documentation There are no legacy formats. JDBC and ODBC clients are not "legacy", quite the opposite. In future we may even want to combine JDBC Thin and general-purpose Thin clients since they have a lot in common. So let's keep the handshake format consistent across clients. > exceptions for message formats Handshake is an exception anyway, it does not have (or need) requestId, etc. On Fri, Dec 1, 2017 at 1:38 PM, Alexey Popov <tank2.a...@gmail.com> wrote: > Pavel, > > I believe ClientListenerNioListener.onHandshake() could support more than > one Handshake request format (be backward compatible), for instance, if we > will have a new handshake code = 0xABCD that differs from 0x01 byte. > > It is a design vs architecture question. > > I can’t understand why the legacy Handshake format should be used for a > new protocol. If this protocol is supposed to be public it should have no > exceptions for message formats. > > Thank you, > Alexey > > From: Pavel Tupitsyn > Sent: Thursday, November 30, 2017 12:11 PM > To: dev@ignite.apache.org > Subject: Re: Thin Client Protocol documentation > > Hi Alexey, > > 1,2,3 are related only to handshake. All other operations are consistent. > > Handshake request format is dictated by existing client connector that is > shared with ODBC and JDBC clients (see > ClientListenerNioListener.onHandshake). > so we can't add magic numbers or change operation code. > > But yes, we can add server version to the handshake response, and I think > this makes sense. > > > 4. The same comments for success flag (1 byte) and status code (4 bytes) > in responses. Let's leave only status code. > We don't have a success flag in responses, there is just a 4-byte status > code, 0 indicates success, everything else is an error. > > Thanks, > Pavel > > On Thu, Nov 30, 2017 at 12:01 PM, Alexey Popov <tank2.a...@gmail.com> > wrote: > > > Hi Pavel, > > > > Let me add my 5 cents. > > > > 1. It would be great if both Handshake request & response have some > > "magic" number (2 or 4 bytes) inside their msg body. That will simplify > > handling situations when non-Ignite client connects to Ignite server and > > vice versa. > > > > 2. It makes sense to add server version to successful Handshake response > > as well. It will help to understand & debug possible backward > compatibility > > issues in the field by *.pcap logs analysis & etc. > > > > 3. Can we have a more strict header for all message types? > > As far as I understand, > > Handshake request has: > > 1) length - 4 byte > > 2) Handshake code - 1 byte > > 3) body - (length - 1) bytes > > > > while OP_CACHE_GET request has: > > 1) length - 4 byte > > 2) OP_CACHE_GET code - 2 bytes > > 3) request id - 4 bytes > > 4) body - (length - 2 - 4) bytes > > > > Why some messages have Operation code with 1 byte while others - 2 bytes? > > Why some requests/responses have request-id while others don't? Let's > > simplify parser work ) > > > > 4. The same comments for success flag (1 byte) and status code (4 bytes) > > in responses. Let's leave only status code. > > > > Thank you, > > Alexey > > > > From: Pavel Tupitsyn > > Sent: Wednesday, November 22, 2017 4:04 PM > > To: dev@ignite.apache.org > > Subject: Thin Client Protocol documentation > > > > Igniters, > > > > I've put together a detailed description of our Thin Client protocol > > in form of IEP on wiki: > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > > 9+Thin+Client+Protocol > > > > > > To clarify: > > - Protocol implementation is in master (see ClientMessageParser class) > > - Protocol has not been released yet, so we are free to change anything > > - Protocol is only used by .NET Thin Client for now, but is supposed to > be > > used from other languages by third party contributors > > - More operations will be added in future, this is a first set of them, > > cache-related > > > > > > Please review the document and let me know your thoughts. > > Is there anything missing or wrong? > > > > We should make sure that the foundation is solid and extensible. > > > > > > Thanks, > > Pavel > > > > > >
Re: Thin Client Protocol documentation
There are no legacy formats. JDBC and ODBC clients are not "legacy", quite the opposite. In future we may even want to combine JDBC Thin and general-purpose Thin clients since they have a lot in common. So let's keep the handshake format consistent across clients. > exceptions for message formats Handshake is an exception anyway, it does not have (or need) requestId, etc. On Fri, Dec 1, 2017 at 1:38 PM, Alexey Popov <tank2.a...@gmail.com> wrote: > Pavel, > > I believe ClientListenerNioListener.onHandshake() could support more than > one Handshake request format (be backward compatible), for instance, if we > will have a new handshake code = 0xABCD that differs from 0x01 byte. > > It is a design vs architecture question. > > I can’t understand why the legacy Handshake format should be used for a > new protocol. If this protocol is supposed to be public it should have no > exceptions for message formats. > > Thank you, > Alexey > > From: Pavel Tupitsyn > Sent: Thursday, November 30, 2017 12:11 PM > To: dev@ignite.apache.org > Subject: Re: Thin Client Protocol documentation > > Hi Alexey, > > 1,2,3 are related only to handshake. All other operations are consistent. > > Handshake request format is dictated by existing client connector that is > shared with ODBC and JDBC clients (see > ClientListenerNioListener.onHandshake). > so we can't add magic numbers or change operation code. > > But yes, we can add server version to the handshake response, and I think > this makes sense. > > > 4. The same comments for success flag (1 byte) and status code (4 bytes) > in responses. Let's leave only status code. > We don't have a success flag in responses, there is just a 4-byte status > code, 0 indicates success, everything else is an error. > > Thanks, > Pavel > > On Thu, Nov 30, 2017 at 12:01 PM, Alexey Popov <tank2.a...@gmail.com> > wrote: > > > Hi Pavel, > > > > Let me add my 5 cents. > > > > 1. It would be great if both Handshake request & response have some > > "magic" number (2 or 4 bytes) inside their msg body. That will simplify > > handling situations when non-Ignite client connects to Ignite server and > > vice versa. > > > > 2. It makes sense to add server version to successful Handshake response > > as well. It will help to understand & debug possible backward > compatibility > > issues in the field by *.pcap logs analysis & etc. > > > > 3. Can we have a more strict header for all message types? > > As far as I understand, > > Handshake request has: > > 1) length - 4 byte > > 2) Handshake code - 1 byte > > 3) body - (length - 1) bytes > > > > while OP_CACHE_GET request has: > > 1) length - 4 byte > > 2) OP_CACHE_GET code - 2 bytes > > 3) request id - 4 bytes > > 4) body - (length - 2 - 4) bytes > > > > Why some messages have Operation code with 1 byte while others - 2 bytes? > > Why some requests/responses have request-id while others don't? Let's > > simplify parser work ) > > > > 4. The same comments for success flag (1 byte) and status code (4 bytes) > > in responses. Let's leave only status code. > > > > Thank you, > > Alexey > > > > From: Pavel Tupitsyn > > Sent: Wednesday, November 22, 2017 4:04 PM > > To: dev@ignite.apache.org > > Subject: Thin Client Protocol documentation > > > > Igniters, > > > > I've put together a detailed description of our Thin Client protocol > > in form of IEP on wiki: > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > > 9+Thin+Client+Protocol > > > > > > To clarify: > > - Protocol implementation is in master (see ClientMessageParser class) > > - Protocol has not been released yet, so we are free to change anything > > - Protocol is only used by .NET Thin Client for now, but is supposed to > be > > used from other languages by third party contributors > > - More operations will be added in future, this is a first set of them, > > cache-related > > > > > > Please review the document and let me know your thoughts. > > Is there anything missing or wrong? > > > > We should make sure that the foundation is solid and extensible. > > > > > > Thanks, > > Pavel > > > > > >
RE: Thin Client Protocol documentation
Pavel, I believe ClientListenerNioListener.onHandshake() could support more than one Handshake request format (be backward compatible), for instance, if we will have a new handshake code = 0xABCD that differs from 0x01 byte. It is a design vs architecture question. I can’t understand why the legacy Handshake format should be used for a new protocol. If this protocol is supposed to be public it should have no exceptions for message formats. Thank you, Alexey From: Pavel Tupitsyn Sent: Thursday, November 30, 2017 12:11 PM To: dev@ignite.apache.org Subject: Re: Thin Client Protocol documentation Hi Alexey, 1,2,3 are related only to handshake. All other operations are consistent. Handshake request format is dictated by existing client connector that is shared with ODBC and JDBC clients (see ClientListenerNioListener.onHandshake). so we can't add magic numbers or change operation code. But yes, we can add server version to the handshake response, and I think this makes sense. > 4. The same comments for success flag (1 byte) and status code (4 bytes) in responses. Let's leave only status code. We don't have a success flag in responses, there is just a 4-byte status code, 0 indicates success, everything else is an error. Thanks, Pavel On Thu, Nov 30, 2017 at 12:01 PM, Alexey Popov <tank2.a...@gmail.com> wrote: > Hi Pavel, > > Let me add my 5 cents. > > 1. It would be great if both Handshake request & response have some > "magic" number (2 or 4 bytes) inside their msg body. That will simplify > handling situations when non-Ignite client connects to Ignite server and > vice versa. > > 2. It makes sense to add server version to successful Handshake response > as well. It will help to understand & debug possible backward compatibility > issues in the field by *.pcap logs analysis & etc. > > 3. Can we have a more strict header for all message types? > As far as I understand, > Handshake request has: > 1) length - 4 byte > 2) Handshake code - 1 byte > 3) body - (length - 1) bytes > > while OP_CACHE_GET request has: > 1) length - 4 byte > 2) OP_CACHE_GET code - 2 bytes > 3) request id - 4 bytes > 4) body - (length - 2 - 4) bytes > > Why some messages have Operation code with 1 byte while others - 2 bytes? > Why some requests/responses have request-id while others don't? Let's > simplify parser work ) > > 4. The same comments for success flag (1 byte) and status code (4 bytes) > in responses. Let's leave only status code. > > Thank you, > Alexey > > From: Pavel Tupitsyn > Sent: Wednesday, November 22, 2017 4:04 PM > To: dev@ignite.apache.org > Subject: Thin Client Protocol documentation > > Igniters, > > I've put together a detailed description of our Thin Client protocol > in form of IEP on wiki: > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > 9+Thin+Client+Protocol > > > To clarify: > - Protocol implementation is in master (see ClientMessageParser class) > - Protocol has not been released yet, so we are free to change anything > - Protocol is only used by .NET Thin Client for now, but is supposed to be > used from other languages by third party contributors > - More operations will be added in future, this is a first set of them, > cache-related > > > Please review the document and let me know your thoughts. > Is there anything missing or wrong? > > We should make sure that the foundation is solid and extensible. > > > Thanks, > Pavel > >
Re: Thin Client Protocol documentation
Hi Alexey, 1,2,3 are related only to handshake. All other operations are consistent. Handshake request format is dictated by existing client connector that is shared with ODBC and JDBC clients (see ClientListenerNioListener.onHandshake). so we can't add magic numbers or change operation code. But yes, we can add server version to the handshake response, and I think this makes sense. > 4. The same comments for success flag (1 byte) and status code (4 bytes) in responses. Let's leave only status code. We don't have a success flag in responses, there is just a 4-byte status code, 0 indicates success, everything else is an error. Thanks, Pavel On Thu, Nov 30, 2017 at 12:01 PM, Alexey Popovwrote: > Hi Pavel, > > Let me add my 5 cents. > > 1. It would be great if both Handshake request & response have some > "magic" number (2 or 4 bytes) inside their msg body. That will simplify > handling situations when non-Ignite client connects to Ignite server and > vice versa. > > 2. It makes sense to add server version to successful Handshake response > as well. It will help to understand & debug possible backward compatibility > issues in the field by *.pcap logs analysis & etc. > > 3. Can we have a more strict header for all message types? > As far as I understand, > Handshake request has: > 1) length - 4 byte > 2) Handshake code - 1 byte > 3) body - (length - 1) bytes > > while OP_CACHE_GET request has: > 1) length - 4 byte > 2) OP_CACHE_GET code - 2 bytes > 3) request id - 4 bytes > 4) body - (length - 2 - 4) bytes > > Why some messages have Operation code with 1 byte while others - 2 bytes? > Why some requests/responses have request-id while others don't? Let's > simplify parser work ) > > 4. The same comments for success flag (1 byte) and status code (4 bytes) > in responses. Let's leave only status code. > > Thank you, > Alexey > > From: Pavel Tupitsyn > Sent: Wednesday, November 22, 2017 4:04 PM > To: dev@ignite.apache.org > Subject: Thin Client Protocol documentation > > Igniters, > > I've put together a detailed description of our Thin Client protocol > in form of IEP on wiki: > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > 9+Thin+Client+Protocol > > > To clarify: > - Protocol implementation is in master (see ClientMessageParser class) > - Protocol has not been released yet, so we are free to change anything > - Protocol is only used by .NET Thin Client for now, but is supposed to be > used from other languages by third party contributors > - More operations will be added in future, this is a first set of them, > cache-related > > > Please review the document and let me know your thoughts. > Is there anything missing or wrong? > > We should make sure that the foundation is solid and extensible. > > > Thanks, > Pavel > >
RE: Thin Client Protocol documentation
Hi Pavel, Let me add my 5 cents. 1. It would be great if both Handshake request & response have some "magic" number (2 or 4 bytes) inside their msg body. That will simplify handling situations when non-Ignite client connects to Ignite server and vice versa. 2. It makes sense to add server version to successful Handshake response as well. It will help to understand & debug possible backward compatibility issues in the field by *.pcap logs analysis & etc. 3. Can we have a more strict header for all message types? As far as I understand, Handshake request has: 1) length - 4 byte 2) Handshake code - 1 byte 3) body - (length - 1) bytes while OP_CACHE_GET request has: 1) length - 4 byte 2) OP_CACHE_GET code - 2 bytes 3) request id - 4 bytes 4) body - (length - 2 - 4) bytes Why some messages have Operation code with 1 byte while others - 2 bytes? Why some requests/responses have request-id while others don't? Let's simplify parser work ) 4. The same comments for success flag (1 byte) and status code (4 bytes) in responses. Let's leave only status code. Thank you, Alexey From: Pavel Tupitsyn Sent: Wednesday, November 22, 2017 4:04 PM To: dev@ignite.apache.org Subject: Thin Client Protocol documentation Igniters, I've put together a detailed description of our Thin Client protocol in form of IEP on wiki: https://cwiki.apache.org/confluence/display/IGNITE/IEP-9+Thin+Client+Protocol To clarify: - Protocol implementation is in master (see ClientMessageParser class) - Protocol has not been released yet, so we are free to change anything - Protocol is only used by .NET Thin Client for now, but is supposed to be used from other languages by third party contributors - More operations will be added in future, this is a first set of them, cache-related Please review the document and let me know your thoughts. Is there anything missing or wrong? We should make sure that the foundation is solid and extensible. Thanks, Pavel
Re: Thin Client Protocol documentation
Prachi, see "Response (failure)" table. Failed response includes server protocol version and an error message. On Thu, Nov 30, 2017 at 4:55 AM, Prachi Gargwrote: > Pavel, > > If the connection handshake fails, what should be the message length in > the response. When I try to fail the handshake, I get 32 as the message > length. Is this expected? > > -Prachi > > On Wed, Nov 29, 2017 at 8:20 AM, Pavel Tupitsyn > wrote: > >> Hi Prachi, >> >> Flags parameter in all cache operations is reserved for things like >> withSkipStore, withExpiryPolicy, withKeepBinary, withNoRetries, >> withPartitionRecover. >> See methods in IgniteCache interface. >> >> Thanks, >> Pavel >> >> On Wed, Nov 29, 2017 at 7:10 PM, Prachi Garg wrote: >> >>> Hi Pavel, >>> >>> In the operation request, what does the 'flags' parameter mean? >>> >>> >>> Thanks, >>> -Prachi >>> >>> On Wed, Nov 29, 2017 at 5:27 AM, Pavel Tupitsyn >>> wrote: >>> Sergey, good point, done. On Wed, Nov 29, 2017 at 2:30 PM, Sergey Kozlov wrote: > Pavel > > Could you update the page by following: > > - String, date, UUID arrays allow to put NULL. Due to that every item in > the array written as type code byte (default item type or null type code) + > type data. It should be detailed explained (looks like that the table > should have an addtional column called for instance "nullable") > > - UUID type takes 16 bytes length > > thanks > > > On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov > wrote: > > > Pavel > > > > Thanks for explanations! > > > > On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn < ptupit...@apache.org> > > wrote: > > > >> Sergey, > >> > >> 1. Code table size does not affect anything, as I understand, so there > is > >> no reason to introduce an extra byte. > >> 2. We have object arrays (code 23), I forgot to mention them, fixed. > >> 3. Also forgot, see code 25 in the updated document. > >> > >> Also note that operation codes have been updated (grouped by purpose) as > >> part of https://issues.apache.org/jira/browse/IGNITE-6989. > >> > >> Thanks, > >> Pavel > >> > >> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov < skoz...@gridgain.com> > >> wrote: > >> > >> > Pavel > >> > > >> > Thanks for the document and your efforts for new protocol. It was > really > >> > helpful for playing around the python thin client design. > >> > > >> > Could you explain some things that were still not clear for binary > >> object > >> > format: > >> > > >> > 1. What a reason to introduce separated type codes for arrays? Why > just > >> we > >> > can't use the following? > >> > *<1 byte universal array code>* > >> > *<1 byte primitive code>* > >> > *<4 bytes length>* > >> > ** > >> > > >> > We get 1 byte overhead but save 10 bytes in the code table. For arrays > >> the > >> > overhead is really insignificant:10 longs array takes now 1+4+4*10=45 > >> bytes > >> > vs 1+1+4+4*10=46 bytes for the approach > >> > Moreover for that appoach a new primitive code will be available for > >> using > >> > for array immediately. > >> > > >> > 2. Why the arrays force to use a selected type? For python there's no > >> > limitations to use different types across one array (list). Would be > >> good > >> > to introduce a new type that will allow that. It could be look like > >> > following > >> > *<1 byte universal array code>* > >> > *<1 byte no common type code*> <-- this says that every item must > >> provide > >> > its date type code like it does regular primitive data > >> > *<4 bytes length>* > >> > *<1 byte item 0 type code>* <-- item provides its code > >> > ** <-- item provides its data > >> > *<1 byte item 1 type code>* > >> > ** > >> > etc > >> > > >> > Also that allow to put nested arrays without changes in type code > table! > >> > For instance if we want to store 9 longs and 1 boolean it will take > >> > now 1+1+4+(1+9)*4+(1+1)=48 > >> > bytes (vs 45 bytes to store as 10 longs as usual). > >> > > >> > 3. Ther's only one way to store a dictionary (key-value) structure as > >> value > >> > in the cache via Complex Object. But it looks like overcomplicated. I > >> > suppose to introduce a code for that > >> > *<1 byte key-value dictionary code>* > >> > > >> > *<4 bytes length>* > >> > *<1 byte key 1 **name **type code>* > >> > **
Re: Thin Client Protocol documentation
Pavel, If the connection handshake fails, what should be the message length in the response. When I try to fail the handshake, I get 32 as the message length. Is this expected? -Prachi On Wed, Nov 29, 2017 at 8:20 AM, Pavel Tupitsynwrote: > Hi Prachi, > > Flags parameter in all cache operations is reserved for things like > withSkipStore, withExpiryPolicy, withKeepBinary, withNoRetries, > withPartitionRecover. > See methods in IgniteCache interface. > > Thanks, > Pavel > > On Wed, Nov 29, 2017 at 7:10 PM, Prachi Garg wrote: > >> Hi Pavel, >> >> In the operation request, what does the 'flags' parameter mean? >> >> >> Thanks, >> -Prachi >> >> On Wed, Nov 29, 2017 at 5:27 AM, Pavel Tupitsyn >> wrote: >> >>> Sergey, good point, done. >>> >>> On Wed, Nov 29, 2017 at 2:30 PM, Sergey Kozlov >>> wrote: >>> >>> > Pavel >>> > >>> > Could you update the page by following: >>> > >>> > - String, date, UUID arrays allow to put NULL. Due to that every item >>> in >>> > the array written as type code byte (default item type or null type >>> code) + >>> > type data. It should be detailed explained (looks like that the table >>> > should have an addtional column called for instance "nullable") >>> > >>> > - UUID type takes 16 bytes length >>> > >>> > thanks >>> > >>> > >>> > On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov >>> > wrote: >>> > >>> > > Pavel >>> > > >>> > > Thanks for explanations! >>> > > >>> > > On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn < >>> ptupit...@apache.org> >>> > > wrote: >>> > > >>> > >> Sergey, >>> > >> >>> > >> 1. Code table size does not affect anything, as I understand, so >>> there >>> > is >>> > >> no reason to introduce an extra byte. >>> > >> 2. We have object arrays (code 23), I forgot to mention them, fixed. >>> > >> 3. Also forgot, see code 25 in the updated document. >>> > >> >>> > >> Also note that operation codes have been updated (grouped by >>> purpose) as >>> > >> part of https://issues.apache.org/jira/browse/IGNITE-6989. >>> > >> >>> > >> Thanks, >>> > >> Pavel >>> > >> >>> > >> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov < >>> skoz...@gridgain.com> >>> > >> wrote: >>> > >> >>> > >> > Pavel >>> > >> > >>> > >> > Thanks for the document and your efforts for new protocol. It was >>> > really >>> > >> > helpful for playing around the python thin client design. >>> > >> > >>> > >> > Could you explain some things that were still not clear for binary >>> > >> object >>> > >> > format: >>> > >> > >>> > >> > 1. What a reason to introduce separated type codes for arrays? Why >>> > just >>> > >> we >>> > >> > can't use the following? >>> > >> > *<1 byte universal array code>* >>> > >> > *<1 byte primitive code>* >>> > >> > *<4 bytes length>* >>> > >> > ** >>> > >> > >>> > >> > We get 1 byte overhead but save 10 bytes in the code table. For >>> arrays >>> > >> the >>> > >> > overhead is really insignificant:10 longs array takes now >>> 1+4+4*10=45 >>> > >> bytes >>> > >> > vs 1+1+4+4*10=46 bytes for the approach >>> > >> > Moreover for that appoach a new primitive code will be available >>> for >>> > >> using >>> > >> > for array immediately. >>> > >> > >>> > >> > 2. Why the arrays force to use a selected type? For python >>> there's no >>> > >> > limitations to use different types across one array (list). Would >>> be >>> > >> good >>> > >> > to introduce a new type that will allow that. It could be look >>> like >>> > >> > following >>> > >> > *<1 byte universal array code>* >>> > >> > *<1 byte no common type code*> <-- this says that every item must >>> > >> provide >>> > >> > its date type code like it does regular primitive data >>> > >> > *<4 bytes length>* >>> > >> > *<1 byte item 0 type code>* <-- item provides its code >>> > >> > ** <-- item provides its data >>> > >> > *<1 byte item 1 type code>* >>> > >> > ** >>> > >> > etc >>> > >> > >>> > >> > Also that allow to put nested arrays without changes in type code >>> > table! >>> > >> > For instance if we want to store 9 longs and 1 boolean it will >>> take >>> > >> > now 1+1+4+(1+9)*4+(1+1)=48 >>> > >> > bytes (vs 45 bytes to store as 10 longs as usual). >>> > >> > >>> > >> > 3. Ther's only one way to store a dictionary (key-value) >>> structure as >>> > >> value >>> > >> > in the cache via Complex Object. But it looks like >>> overcomplicated. I >>> > >> > suppose to introduce a code for that >>> > >> > *<1 byte key-value dictionary code>* >>> > >> > >>> > >> > *<4 bytes length>* >>> > >> > *<1 byte key 1 **name **type code>* >>> > >> > ** >>> > >> > *<1 byte value 1 type code>* >>> > >> > ** >>> > >> > *<1 byte key 2 **name **type code>* >>> > >> > ** >>> > >> > *<1 byte value 2 type code>* >>> > >> > ** >>> > >> > etc >>> > >> > >>> > >> > Also that allow to put nested dictionaries without changes in type >>> > code >>> > >> > table! >>> > >> > Of course for the appoach above we get significat
Re: Thin Client Protocol documentation
Hi Prachi, Flags parameter in all cache operations is reserved for things like withSkipStore, withExpiryPolicy, withKeepBinary, withNoRetries, withPartitionRecover. See methods in IgniteCache interface. Thanks, Pavel On Wed, Nov 29, 2017 at 7:10 PM, Prachi Gargwrote: > Hi Pavel, > > In the operation request, what does the 'flags' parameter mean? > > > Thanks, > -Prachi > > On Wed, Nov 29, 2017 at 5:27 AM, Pavel Tupitsyn > wrote: > >> Sergey, good point, done. >> >> On Wed, Nov 29, 2017 at 2:30 PM, Sergey Kozlov >> wrote: >> >> > Pavel >> > >> > Could you update the page by following: >> > >> > - String, date, UUID arrays allow to put NULL. Due to that every item >> in >> > the array written as type code byte (default item type or null type >> code) + >> > type data. It should be detailed explained (looks like that the table >> > should have an addtional column called for instance "nullable") >> > >> > - UUID type takes 16 bytes length >> > >> > thanks >> > >> > >> > On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov >> > wrote: >> > >> > > Pavel >> > > >> > > Thanks for explanations! >> > > >> > > On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn > > >> > > wrote: >> > > >> > >> Sergey, >> > >> >> > >> 1. Code table size does not affect anything, as I understand, so >> there >> > is >> > >> no reason to introduce an extra byte. >> > >> 2. We have object arrays (code 23), I forgot to mention them, fixed. >> > >> 3. Also forgot, see code 25 in the updated document. >> > >> >> > >> Also note that operation codes have been updated (grouped by >> purpose) as >> > >> part of https://issues.apache.org/jira/browse/IGNITE-6989. >> > >> >> > >> Thanks, >> > >> Pavel >> > >> >> > >> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov > > >> > >> wrote: >> > >> >> > >> > Pavel >> > >> > >> > >> > Thanks for the document and your efforts for new protocol. It was >> > really >> > >> > helpful for playing around the python thin client design. >> > >> > >> > >> > Could you explain some things that were still not clear for binary >> > >> object >> > >> > format: >> > >> > >> > >> > 1. What a reason to introduce separated type codes for arrays? Why >> > just >> > >> we >> > >> > can't use the following? >> > >> > *<1 byte universal array code>* >> > >> > *<1 byte primitive code>* >> > >> > *<4 bytes length>* >> > >> > ** >> > >> > >> > >> > We get 1 byte overhead but save 10 bytes in the code table. For >> arrays >> > >> the >> > >> > overhead is really insignificant:10 longs array takes now >> 1+4+4*10=45 >> > >> bytes >> > >> > vs 1+1+4+4*10=46 bytes for the approach >> > >> > Moreover for that appoach a new primitive code will be available >> for >> > >> using >> > >> > for array immediately. >> > >> > >> > >> > 2. Why the arrays force to use a selected type? For python there's >> no >> > >> > limitations to use different types across one array (list). Would >> be >> > >> good >> > >> > to introduce a new type that will allow that. It could be look like >> > >> > following >> > >> > *<1 byte universal array code>* >> > >> > *<1 byte no common type code*> <-- this says that every item must >> > >> provide >> > >> > its date type code like it does regular primitive data >> > >> > *<4 bytes length>* >> > >> > *<1 byte item 0 type code>* <-- item provides its code >> > >> > ** <-- item provides its data >> > >> > *<1 byte item 1 type code>* >> > >> > ** >> > >> > etc >> > >> > >> > >> > Also that allow to put nested arrays without changes in type code >> > table! >> > >> > For instance if we want to store 9 longs and 1 boolean it will take >> > >> > now 1+1+4+(1+9)*4+(1+1)=48 >> > >> > bytes (vs 45 bytes to store as 10 longs as usual). >> > >> > >> > >> > 3. Ther's only one way to store a dictionary (key-value) structure >> as >> > >> value >> > >> > in the cache via Complex Object. But it looks like >> overcomplicated. I >> > >> > suppose to introduce a code for that >> > >> > *<1 byte key-value dictionary code>* >> > >> > >> > >> > *<4 bytes length>* >> > >> > *<1 byte key 1 **name **type code>* >> > >> > ** >> > >> > *<1 byte value 1 type code>* >> > >> > ** >> > >> > *<1 byte key 2 **name **type code>* >> > >> > ** >> > >> > *<1 byte value 2 type code>* >> > >> > ** >> > >> > etc >> > >> > >> > >> > Also that allow to put nested dictionaries without changes in type >> > code >> > >> > table! >> > >> > Of course for the appoach above we get significat overhead for key >> > >> > storing. But I think it is acceptable for some cases and >> definitely ok >> > >> for >> > >> > Python >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg >> > >> wrote: >> > >> > >> > >> > > Thanks Pavel! The document has good information. I'll create one >> on >> > >> > > readme.io; will also add some examples there. >> > >> > > >> > >> > > On
Re: Thin Client Protocol documentation
Hi Pavel, In the operation request, what does the 'flags' parameter mean? Thanks, -Prachi On Wed, Nov 29, 2017 at 5:27 AM, Pavel Tupitsynwrote: > Sergey, good point, done. > > On Wed, Nov 29, 2017 at 2:30 PM, Sergey Kozlov > wrote: > > > Pavel > > > > Could you update the page by following: > > > > - String, date, UUID arrays allow to put NULL. Due to that every item in > > the array written as type code byte (default item type or null type > code) + > > type data. It should be detailed explained (looks like that the table > > should have an addtional column called for instance "nullable") > > > > - UUID type takes 16 bytes length > > > > thanks > > > > > > On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov > > wrote: > > > > > Pavel > > > > > > Thanks for explanations! > > > > > > On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn > > > wrote: > > > > > >> Sergey, > > >> > > >> 1. Code table size does not affect anything, as I understand, so there > > is > > >> no reason to introduce an extra byte. > > >> 2. We have object arrays (code 23), I forgot to mention them, fixed. > > >> 3. Also forgot, see code 25 in the updated document. > > >> > > >> Also note that operation codes have been updated (grouped by purpose) > as > > >> part of https://issues.apache.org/jira/browse/IGNITE-6989. > > >> > > >> Thanks, > > >> Pavel > > >> > > >> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov > > >> wrote: > > >> > > >> > Pavel > > >> > > > >> > Thanks for the document and your efforts for new protocol. It was > > really > > >> > helpful for playing around the python thin client design. > > >> > > > >> > Could you explain some things that were still not clear for binary > > >> object > > >> > format: > > >> > > > >> > 1. What a reason to introduce separated type codes for arrays? Why > > just > > >> we > > >> > can't use the following? > > >> > *<1 byte universal array code>* > > >> > *<1 byte primitive code>* > > >> > *<4 bytes length>* > > >> > ** > > >> > > > >> > We get 1 byte overhead but save 10 bytes in the code table. For > arrays > > >> the > > >> > overhead is really insignificant:10 longs array takes now > 1+4+4*10=45 > > >> bytes > > >> > vs 1+1+4+4*10=46 bytes for the approach > > >> > Moreover for that appoach a new primitive code will be available for > > >> using > > >> > for array immediately. > > >> > > > >> > 2. Why the arrays force to use a selected type? For python there's > no > > >> > limitations to use different types across one array (list). Would be > > >> good > > >> > to introduce a new type that will allow that. It could be look like > > >> > following > > >> > *<1 byte universal array code>* > > >> > *<1 byte no common type code*> <-- this says that every item must > > >> provide > > >> > its date type code like it does regular primitive data > > >> > *<4 bytes length>* > > >> > *<1 byte item 0 type code>* <-- item provides its code > > >> > ** <-- item provides its data > > >> > *<1 byte item 1 type code>* > > >> > ** > > >> > etc > > >> > > > >> > Also that allow to put nested arrays without changes in type code > > table! > > >> > For instance if we want to store 9 longs and 1 boolean it will take > > >> > now 1+1+4+(1+9)*4+(1+1)=48 > > >> > bytes (vs 45 bytes to store as 10 longs as usual). > > >> > > > >> > 3. Ther's only one way to store a dictionary (key-value) structure > as > > >> value > > >> > in the cache via Complex Object. But it looks like overcomplicated. > I > > >> > suppose to introduce a code for that > > >> > *<1 byte key-value dictionary code>* > > >> > > > >> > *<4 bytes length>* > > >> > *<1 byte key 1 **name **type code>* > > >> > ** > > >> > *<1 byte value 1 type code>* > > >> > ** > > >> > *<1 byte key 2 **name **type code>* > > >> > ** > > >> > *<1 byte value 2 type code>* > > >> > ** > > >> > etc > > >> > > > >> > Also that allow to put nested dictionaries without changes in type > > code > > >> > table! > > >> > Of course for the appoach above we get significat overhead for key > > >> > storing. But I think it is acceptable for some cases and definitely > ok > > >> for > > >> > Python > > >> > > > >> > > > >> > > > >> > > > >> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg > > >> wrote: > > >> > > > >> > > Thanks Pavel! The document has good information. I'll create one > on > > >> > > readme.io; will also add some examples there. > > >> > > > > >> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn < > > ptupit...@apache.org > > >> > > > >> > > wrote: > > >> > > > > >> > > > Igniters, > > >> > > > > > >> > > > I've put together a detailed description of our Thin Client > > protocol > > >> > > > in form of IEP on wiki: > > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > > >> > > > 9+Thin+Client+Protocol > > >> > > > > > >> > > > > > >> > > > To clarify: > > >> > > > - Protocol implementation is in
Re: Thin Client Protocol documentation
Sergey, good point, done. On Wed, Nov 29, 2017 at 2:30 PM, Sergey Kozlovwrote: > Pavel > > Could you update the page by following: > > - String, date, UUID arrays allow to put NULL. Due to that every item in > the array written as type code byte (default item type or null type code) + > type data. It should be detailed explained (looks like that the table > should have an addtional column called for instance "nullable") > > - UUID type takes 16 bytes length > > thanks > > > On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlov > wrote: > > > Pavel > > > > Thanks for explanations! > > > > On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn > > wrote: > > > >> Sergey, > >> > >> 1. Code table size does not affect anything, as I understand, so there > is > >> no reason to introduce an extra byte. > >> 2. We have object arrays (code 23), I forgot to mention them, fixed. > >> 3. Also forgot, see code 25 in the updated document. > >> > >> Also note that operation codes have been updated (grouped by purpose) as > >> part of https://issues.apache.org/jira/browse/IGNITE-6989. > >> > >> Thanks, > >> Pavel > >> > >> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov > >> wrote: > >> > >> > Pavel > >> > > >> > Thanks for the document and your efforts for new protocol. It was > really > >> > helpful for playing around the python thin client design. > >> > > >> > Could you explain some things that were still not clear for binary > >> object > >> > format: > >> > > >> > 1. What a reason to introduce separated type codes for arrays? Why > just > >> we > >> > can't use the following? > >> > *<1 byte universal array code>* > >> > *<1 byte primitive code>* > >> > *<4 bytes length>* > >> > ** > >> > > >> > We get 1 byte overhead but save 10 bytes in the code table. For arrays > >> the > >> > overhead is really insignificant:10 longs array takes now 1+4+4*10=45 > >> bytes > >> > vs 1+1+4+4*10=46 bytes for the approach > >> > Moreover for that appoach a new primitive code will be available for > >> using > >> > for array immediately. > >> > > >> > 2. Why the arrays force to use a selected type? For python there's no > >> > limitations to use different types across one array (list). Would be > >> good > >> > to introduce a new type that will allow that. It could be look like > >> > following > >> > *<1 byte universal array code>* > >> > *<1 byte no common type code*> <-- this says that every item must > >> provide > >> > its date type code like it does regular primitive data > >> > *<4 bytes length>* > >> > *<1 byte item 0 type code>* <-- item provides its code > >> > ** <-- item provides its data > >> > *<1 byte item 1 type code>* > >> > ** > >> > etc > >> > > >> > Also that allow to put nested arrays without changes in type code > table! > >> > For instance if we want to store 9 longs and 1 boolean it will take > >> > now 1+1+4+(1+9)*4+(1+1)=48 > >> > bytes (vs 45 bytes to store as 10 longs as usual). > >> > > >> > 3. Ther's only one way to store a dictionary (key-value) structure as > >> value > >> > in the cache via Complex Object. But it looks like overcomplicated. I > >> > suppose to introduce a code for that > >> > *<1 byte key-value dictionary code>* > >> > > >> > *<4 bytes length>* > >> > *<1 byte key 1 **name **type code>* > >> > ** > >> > *<1 byte value 1 type code>* > >> > ** > >> > *<1 byte key 2 **name **type code>* > >> > ** > >> > *<1 byte value 2 type code>* > >> > ** > >> > etc > >> > > >> > Also that allow to put nested dictionaries without changes in type > code > >> > table! > >> > Of course for the appoach above we get significat overhead for key > >> > storing. But I think it is acceptable for some cases and definitely ok > >> for > >> > Python > >> > > >> > > >> > > >> > > >> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg > >> wrote: > >> > > >> > > Thanks Pavel! The document has good information. I'll create one on > >> > > readme.io; will also add some examples there. > >> > > > >> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn < > ptupit...@apache.org > >> > > >> > > wrote: > >> > > > >> > > > Igniters, > >> > > > > >> > > > I've put together a detailed description of our Thin Client > protocol > >> > > > in form of IEP on wiki: > >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > >> > > > 9+Thin+Client+Protocol > >> > > > > >> > > > > >> > > > To clarify: > >> > > > - Protocol implementation is in master (see ClientMessageParser > >> class) > >> > > > - Protocol has not been released yet, so we are free to change > >> anything > >> > > > - Protocol is only used by .NET Thin Client for now, but is > >> supposed to > >> > > be > >> > > > used from other languages by third party contributors > >> > > > - More operations will be added in future, this is a first set of > >> them, > >> > > > cache-related > >> > > > > >> > > > > >> > > > Please review the document and let me
Re: Thin Client Protocol documentation
Pavel Could you update the page by following: - String, date, UUID arrays allow to put NULL. Due to that every item in the array written as type code byte (default item type or null type code) + type data. It should be detailed explained (looks like that the table should have an addtional column called for instance "nullable") - UUID type takes 16 bytes length thanks On Mon, Nov 27, 2017 at 4:37 PM, Sergey Kozlovwrote: > Pavel > > Thanks for explanations! > > On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsyn > wrote: > >> Sergey, >> >> 1. Code table size does not affect anything, as I understand, so there is >> no reason to introduce an extra byte. >> 2. We have object arrays (code 23), I forgot to mention them, fixed. >> 3. Also forgot, see code 25 in the updated document. >> >> Also note that operation codes have been updated (grouped by purpose) as >> part of https://issues.apache.org/jira/browse/IGNITE-6989. >> >> Thanks, >> Pavel >> >> On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov >> wrote: >> >> > Pavel >> > >> > Thanks for the document and your efforts for new protocol. It was really >> > helpful for playing around the python thin client design. >> > >> > Could you explain some things that were still not clear for binary >> object >> > format: >> > >> > 1. What a reason to introduce separated type codes for arrays? Why just >> we >> > can't use the following? >> > *<1 byte universal array code>* >> > *<1 byte primitive code>* >> > *<4 bytes length>* >> > ** >> > >> > We get 1 byte overhead but save 10 bytes in the code table. For arrays >> the >> > overhead is really insignificant:10 longs array takes now 1+4+4*10=45 >> bytes >> > vs 1+1+4+4*10=46 bytes for the approach >> > Moreover for that appoach a new primitive code will be available for >> using >> > for array immediately. >> > >> > 2. Why the arrays force to use a selected type? For python there's no >> > limitations to use different types across one array (list). Would be >> good >> > to introduce a new type that will allow that. It could be look like >> > following >> > *<1 byte universal array code>* >> > *<1 byte no common type code*> <-- this says that every item must >> provide >> > its date type code like it does regular primitive data >> > *<4 bytes length>* >> > *<1 byte item 0 type code>* <-- item provides its code >> > ** <-- item provides its data >> > *<1 byte item 1 type code>* >> > ** >> > etc >> > >> > Also that allow to put nested arrays without changes in type code table! >> > For instance if we want to store 9 longs and 1 boolean it will take >> > now 1+1+4+(1+9)*4+(1+1)=48 >> > bytes (vs 45 bytes to store as 10 longs as usual). >> > >> > 3. Ther's only one way to store a dictionary (key-value) structure as >> value >> > in the cache via Complex Object. But it looks like overcomplicated. I >> > suppose to introduce a code for that >> > *<1 byte key-value dictionary code>* >> > >> > *<4 bytes length>* >> > *<1 byte key 1 **name **type code>* >> > ** >> > *<1 byte value 1 type code>* >> > ** >> > *<1 byte key 2 **name **type code>* >> > ** >> > *<1 byte value 2 type code>* >> > ** >> > etc >> > >> > Also that allow to put nested dictionaries without changes in type code >> > table! >> > Of course for the appoach above we get significat overhead for key >> > storing. But I think it is acceptable for some cases and definitely ok >> for >> > Python >> > >> > >> > >> > >> > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg >> wrote: >> > >> > > Thanks Pavel! The document has good information. I'll create one on >> > > readme.io; will also add some examples there. >> > > >> > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn > > >> > > wrote: >> > > >> > > > Igniters, >> > > > >> > > > I've put together a detailed description of our Thin Client protocol >> > > > in form of IEP on wiki: >> > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- >> > > > 9+Thin+Client+Protocol >> > > > >> > > > >> > > > To clarify: >> > > > - Protocol implementation is in master (see ClientMessageParser >> class) >> > > > - Protocol has not been released yet, so we are free to change >> anything >> > > > - Protocol is only used by .NET Thin Client for now, but is >> supposed to >> > > be >> > > > used from other languages by third party contributors >> > > > - More operations will be added in future, this is a first set of >> them, >> > > > cache-related >> > > > >> > > > >> > > > Please review the document and let me know your thoughts. >> > > > Is there anything missing or wrong? >> > > > >> > > > We should make sure that the foundation is solid and extensible. >> > > > >> > > > >> > > > Thanks, >> > > > Pavel >> > > > >> > > >> > >> > >> > >> > -- >> > Sergey Kozlov >> > GridGain Systems >> > www.gridgain.com >> > >> > > > > -- > Sergey Kozlov > GridGain Systems > www.gridgain.com > -- Sergey Kozlov GridGain
Re: Thin Client Protocol documentation
Pavel Thanks for explanations! On Mon, Nov 27, 2017 at 2:46 PM, Pavel Tupitsynwrote: > Sergey, > > 1. Code table size does not affect anything, as I understand, so there is > no reason to introduce an extra byte. > 2. We have object arrays (code 23), I forgot to mention them, fixed. > 3. Also forgot, see code 25 in the updated document. > > Also note that operation codes have been updated (grouped by purpose) as > part of https://issues.apache.org/jira/browse/IGNITE-6989. > > Thanks, > Pavel > > On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlov > wrote: > > > Pavel > > > > Thanks for the document and your efforts for new protocol. It was really > > helpful for playing around the python thin client design. > > > > Could you explain some things that were still not clear for binary object > > format: > > > > 1. What a reason to introduce separated type codes for arrays? Why just > we > > can't use the following? > > *<1 byte universal array code>* > > *<1 byte primitive code>* > > *<4 bytes length>* > > ** > > > > We get 1 byte overhead but save 10 bytes in the code table. For arrays > the > > overhead is really insignificant:10 longs array takes now 1+4+4*10=45 > bytes > > vs 1+1+4+4*10=46 bytes for the approach > > Moreover for that appoach a new primitive code will be available for > using > > for array immediately. > > > > 2. Why the arrays force to use a selected type? For python there's no > > limitations to use different types across one array (list). Would be good > > to introduce a new type that will allow that. It could be look like > > following > > *<1 byte universal array code>* > > *<1 byte no common type code*> <-- this says that every item must provide > > its date type code like it does regular primitive data > > *<4 bytes length>* > > *<1 byte item 0 type code>* <-- item provides its code > > ** <-- item provides its data > > *<1 byte item 1 type code>* > > ** > > etc > > > > Also that allow to put nested arrays without changes in type code table! > > For instance if we want to store 9 longs and 1 boolean it will take > > now 1+1+4+(1+9)*4+(1+1)=48 > > bytes (vs 45 bytes to store as 10 longs as usual). > > > > 3. Ther's only one way to store a dictionary (key-value) structure as > value > > in the cache via Complex Object. But it looks like overcomplicated. I > > suppose to introduce a code for that > > *<1 byte key-value dictionary code>* > > > > *<4 bytes length>* > > *<1 byte key 1 **name **type code>* > > ** > > *<1 byte value 1 type code>* > > ** > > *<1 byte key 2 **name **type code>* > > ** > > *<1 byte value 2 type code>* > > ** > > etc > > > > Also that allow to put nested dictionaries without changes in type code > > table! > > Of course for the appoach above we get significat overhead for key > > storing. But I think it is acceptable for some cases and definitely ok > for > > Python > > > > > > > > > > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg wrote: > > > > > Thanks Pavel! The document has good information. I'll create one on > > > readme.io; will also add some examples there. > > > > > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn > > > wrote: > > > > > > > Igniters, > > > > > > > > I've put together a detailed description of our Thin Client protocol > > > > in form of IEP on wiki: > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > > > > 9+Thin+Client+Protocol > > > > > > > > > > > > To clarify: > > > > - Protocol implementation is in master (see ClientMessageParser > class) > > > > - Protocol has not been released yet, so we are free to change > anything > > > > - Protocol is only used by .NET Thin Client for now, but is supposed > to > > > be > > > > used from other languages by third party contributors > > > > - More operations will be added in future, this is a first set of > them, > > > > cache-related > > > > > > > > > > > > Please review the document and let me know your thoughts. > > > > Is there anything missing or wrong? > > > > > > > > We should make sure that the foundation is solid and extensible. > > > > > > > > > > > > Thanks, > > > > Pavel > > > > > > > > > > > > > > > -- > > Sergey Kozlov > > GridGain Systems > > www.gridgain.com > > > -- Sergey Kozlov GridGain Systems www.gridgain.com
Re: Thin Client Protocol documentation
Sergey, 1. Code table size does not affect anything, as I understand, so there is no reason to introduce an extra byte. 2. We have object arrays (code 23), I forgot to mention them, fixed. 3. Also forgot, see code 25 in the updated document. Also note that operation codes have been updated (grouped by purpose) as part of https://issues.apache.org/jira/browse/IGNITE-6989. Thanks, Pavel On Sun, Nov 26, 2017 at 9:54 PM, Sergey Kozlovwrote: > Pavel > > Thanks for the document and your efforts for new protocol. It was really > helpful for playing around the python thin client design. > > Could you explain some things that were still not clear for binary object > format: > > 1. What a reason to introduce separated type codes for arrays? Why just we > can't use the following? > *<1 byte universal array code>* > *<1 byte primitive code>* > *<4 bytes length>* > ** > > We get 1 byte overhead but save 10 bytes in the code table. For arrays the > overhead is really insignificant:10 longs array takes now 1+4+4*10=45 bytes > vs 1+1+4+4*10=46 bytes for the approach > Moreover for that appoach a new primitive code will be available for using > for array immediately. > > 2. Why the arrays force to use a selected type? For python there's no > limitations to use different types across one array (list). Would be good > to introduce a new type that will allow that. It could be look like > following > *<1 byte universal array code>* > *<1 byte no common type code*> <-- this says that every item must provide > its date type code like it does regular primitive data > *<4 bytes length>* > *<1 byte item 0 type code>* <-- item provides its code > ** <-- item provides its data > *<1 byte item 1 type code>* > ** > etc > > Also that allow to put nested arrays without changes in type code table! > For instance if we want to store 9 longs and 1 boolean it will take > now 1+1+4+(1+9)*4+(1+1)=48 > bytes (vs 45 bytes to store as 10 longs as usual). > > 3. Ther's only one way to store a dictionary (key-value) structure as value > in the cache via Complex Object. But it looks like overcomplicated. I > suppose to introduce a code for that > *<1 byte key-value dictionary code>* > > *<4 bytes length>* > *<1 byte key 1 **name **type code>* > ** > *<1 byte value 1 type code>* > ** > *<1 byte key 2 **name **type code>* > ** > *<1 byte value 2 type code>* > ** > etc > > Also that allow to put nested dictionaries without changes in type code > table! > Of course for the appoach above we get significat overhead for key > storing. But I think it is acceptable for some cases and definitely ok for > Python > > > > > On Wed, Nov 22, 2017 at 9:14 PM, Prachi Garg wrote: > > > Thanks Pavel! The document has good information. I'll create one on > > readme.io; will also add some examples there. > > > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn > > wrote: > > > > > Igniters, > > > > > > I've put together a detailed description of our Thin Client protocol > > > in form of IEP on wiki: > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > > > 9+Thin+Client+Protocol > > > > > > > > > To clarify: > > > - Protocol implementation is in master (see ClientMessageParser class) > > > - Protocol has not been released yet, so we are free to change anything > > > - Protocol is only used by .NET Thin Client for now, but is supposed to > > be > > > used from other languages by third party contributors > > > - More operations will be added in future, this is a first set of them, > > > cache-related > > > > > > > > > Please review the document and let me know your thoughts. > > > Is there anything missing or wrong? > > > > > > We should make sure that the foundation is solid and extensible. > > > > > > > > > Thanks, > > > Pavel > > > > > > > > > -- > Sergey Kozlov > GridGain Systems > www.gridgain.com >
Re: Thin Client Protocol documentation
Pavel Thanks for the document and your efforts for new protocol. It was really helpful for playing around the python thin client design. Could you explain some things that were still not clear for binary object format: 1. What a reason to introduce separated type codes for arrays? Why just we can't use the following? *<1 byte universal array code>* *<1 byte primitive code>* *<4 bytes length>* ** We get 1 byte overhead but save 10 bytes in the code table. For arrays the overhead is really insignificant:10 longs array takes now 1+4+4*10=45 bytes vs 1+1+4+4*10=46 bytes for the approach Moreover for that appoach a new primitive code will be available for using for array immediately. 2. Why the arrays force to use a selected type? For python there's no limitations to use different types across one array (list). Would be good to introduce a new type that will allow that. It could be look like following *<1 byte universal array code>* *<1 byte no common type code*> <-- this says that every item must provide its date type code like it does regular primitive data *<4 bytes length>* *<1 byte item 0 type code>* <-- item provides its code ** <-- item provides its data *<1 byte item 1 type code>* ** etc Also that allow to put nested arrays without changes in type code table! For instance if we want to store 9 longs and 1 boolean it will take now 1+1+4+(1+9)*4+(1+1)=48 bytes (vs 45 bytes to store as 10 longs as usual). 3. Ther's only one way to store a dictionary (key-value) structure as value in the cache via Complex Object. But it looks like overcomplicated. I suppose to introduce a code for that *<1 byte key-value dictionary code>* *<4 bytes length>* *<1 byte key 1 **name **type code>* ** *<1 byte value 1 type code>* ** *<1 byte key 2 **name **type code>* ** *<1 byte value 2 type code>* ** etc Also that allow to put nested dictionaries without changes in type code table! Of course for the appoach above we get significat overhead for key storing. But I think it is acceptable for some cases and definitely ok for Python On Wed, Nov 22, 2017 at 9:14 PM, Prachi Gargwrote: > Thanks Pavel! The document has good information. I'll create one on > readme.io; will also add some examples there. > > On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsyn > wrote: > > > Igniters, > > > > I've put together a detailed description of our Thin Client protocol > > in form of IEP on wiki: > > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > > 9+Thin+Client+Protocol > > > > > > To clarify: > > - Protocol implementation is in master (see ClientMessageParser class) > > - Protocol has not been released yet, so we are free to change anything > > - Protocol is only used by .NET Thin Client for now, but is supposed to > be > > used from other languages by third party contributors > > - More operations will be added in future, this is a first set of them, > > cache-related > > > > > > Please review the document and let me know your thoughts. > > Is there anything missing or wrong? > > > > We should make sure that the foundation is solid and extensible. > > > > > > Thanks, > > Pavel > > > -- Sergey Kozlov GridGain Systems www.gridgain.com
Re: Thin Client Protocol documentation
Thanks Pavel! The document has good information. I'll create one on readme.io; will also add some examples there. On Wed, Nov 22, 2017 at 5:03 AM, Pavel Tupitsynwrote: > Igniters, > > I've put together a detailed description of our Thin Client protocol > in form of IEP on wiki: > https://cwiki.apache.org/confluence/display/IGNITE/IEP- > 9+Thin+Client+Protocol > > > To clarify: > - Protocol implementation is in master (see ClientMessageParser class) > - Protocol has not been released yet, so we are free to change anything > - Protocol is only used by .NET Thin Client for now, but is supposed to be > used from other languages by third party contributors > - More operations will be added in future, this is a first set of them, > cache-related > > > Please review the document and let me know your thoughts. > Is there anything missing or wrong? > > We should make sure that the foundation is solid and extensible. > > > Thanks, > Pavel >