Re: [gemfire-dev] New Client-Server Protocol Proposal

On Thu, May 4, 2017 at 2:14 PM, Jacob Barrett wrote: > > > One benefit of messageHeader with chunk is that, it gives us ability to > > > write different messages(multiplexing) on same socket. And if thread is > > > ready it can write its message. Though we are not there yet,

Re: [gemfire-dev] New Client-Server Protocol Proposal

I think we should be presenting the current proposal as the API and message structure, as would be laid out in an IDL. This way, we can experiment with Protobuf, Thrift serialization, for message structure without having to exactly specify the message structure. This will make designing a

Re: [gemfire-dev] New Client-Server Protocol Proposal

+1 William, an even better example - that kind of representation will make it so much better/easier for geode users to implement against, regardless of language. On Mon, May 8, 2017 at 2:48 PM, William Markito Oliveira < william.mark...@gmail.com> wrote: > +1 > > I think I've shared this before,

Re: [gemfire-dev] New Client-Server Protocol Proposal

+1 I think I've shared this before, but Kafka also has good (tabular) representation for messages on their protocol. - https://kafka.apache.org/protocol#protocol_messages - https://kafka.apache.org/protocol#protocol_message_sets On Mon, May 8, 2017 at 4:44 PM, Ernest Burghardt

Re: [gemfire-dev] New Client-Server Protocol Proposal

Hello Geodes! Good discussion on what/how the messages will be/handled once a connection is established. +1 to a simple initial handshake to establish version/supported features that client/server will be communicating. >From what I've seen so far in the proposal it is missing a definition for

Re: [gemfire-dev] New Client-Server Protocol Proposal

On Fri, May 5, 2017 at 2:09 PM Hitesh Khamesra wrote: > > 0. In first phase we are not doing chunking/fragmentation. And even this > will be option for client.( >

Re: [gemfire-dev] New Client-Server Protocol Proposal

5, 2017 7:29 AM Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal On Thu, May 4, 2017 at 2:52 PM Hitesh Khamesra <hitesh...@yahoo.com.invalid> wrote: Basically, thread/layer should not hold any resources while serializing the object or chunk.  We should be able to see this f

Re: [gemfire-dev] New Client-Server Protocol Proposal

In either case you packetize (serialize the message protocol) to buffers (fixed sizes and pooled) and flush buffers to the socket. Preferably using a async socket framework to do all the heavy lifting for you. Sent from my iPhone > On May 5, 2017, at 11:07 AM, Bruce Schuchardt

Re: [gemfire-dev] New Client-Server Protocol Proposal

Yes, of course it does but we don't serialize directly to a socket output stream because it's slow. I agree that this could be left out and added later as an optimization. Le 5/5/2017 à 10:33 AM, Galen M O'Sullivan a écrit : I think TCP does exactly this for us. On Fri, May 5, 2017 at 9:08

Re: [gemfire-dev] New Client-Server Protocol Proposal

It does! Both fragmenting and multiple channels as multiple sockets. Sent from my iPhone > On May 5, 2017, at 10:33 AM, Galen M O'Sullivan wrote: > > I think TCP does exactly this for us. > > On Fri, May 5, 2017 at 9:08 AM, Bruce Schuchardt >

Re: [gemfire-dev] New Client-Server Protocol Proposal

I think TCP does exactly this for us. On Fri, May 5, 2017 at 9:08 AM, Bruce Schuchardt wrote: > This is very similar to how peer-to-peer messaging is performed in Geode. > Messages are serialized to a stream that knows how to optimally "chunk" the > bytes into fixed-size

Re: [gemfire-dev] New Client-Server Protocol Proposal

I would leave it for a later optimization. Sent from my iPhone > On May 5, 2017, at 9:08 AM, Bruce Schuchardt wrote: > > This is very similar to how peer-to-peer messaging is performed in Geode. > Messages are serialized to a stream that knows how to optimally "chunk"

Re: [gemfire-dev] New Client-Server Protocol Proposal

This is very similar to how peer-to-peer messaging is performed in Geode. Messages are serialized to a stream that knows how to optimally "chunk" the bytes into fixed-size packets. On the receiving side these are fed into a similar input stream for deserialization. The message only contains

Re: [gemfire-dev] New Client-Server Protocol Proposal

On Thu, May 4, 2017 at 2:52 PM Hitesh Khamesra wrote: > Basically, thread/layer should not hold any resources while serializing > the object or chunk. We should be able to see this flow (ms1-chunk1, > msg2-chunk1, ms1-chunk2, msg3-chunk, msg2-chunk2, so on ...) >

Re: [gemfire-dev] New Client-Server Protocol Proposal

+1 for that as well On Thu, May 4, 2017 at 5:21 PM Dan Smith wrote: > > > > I wouldn't tackle that at this layer. I would suggest adding a layer > > between the message and TCP that creates channels that are opaque to the > > message layer above. The message layer wouldn't

Re: [gemfire-dev] New Client-Server Protocol Proposal

> > I wouldn't tackle that at this layer. I would suggest adding a layer > between the message and TCP that creates channels that are opaque to the > message layer above. The message layer wouldn't know if it was talking to > multiple sockets to the client or single socket with multiple channels.

Re: [gemfire-dev] New Client-Server Protocol Proposal

ject: Re: [gemfire-dev] New Client-Server Protocol Proposal Sent from my iPhone > On May 4, 2017, at 12:03 PM, Hitesh Khamesra <hitesh...@yahoo.com.INVALID> > wrote: > > And len 0 would indicate end of the message? > > > a. Now these two chunks will go co

Re: [gemfire-dev] New Client-Server Protocol Proposal

this better. Thanks. Hitesh From: Anthony Baker <aba...@pivotal.io> To: Hitesh Khamesra <hitesh...@yahoo.com> Cc: "dev@geode.apache.org" <dev@geode.apache.org> Sent: Thursday, May 4, 2017 11:20 AM Subject: Re: [gemfire-dev] New Cli

Re: [gemfire-dev] New Client-Server Protocol Proposal

t; two messages? And other side will combine those two messages using > "correlationId" ? > > Thanks. > HItesh > > > > > > From: Anthony Baker <aba...@pivotal.io> > To: dev@geode.apache.org > Cc: Hitesh Kha

Re: [gemfire-dev] New Client-Server Protocol Proposal

ssages using "correlationId" ? Thanks. HItesh From: Anthony Baker <aba...@pivotal.io> To: dev@geode.apache.org Cc: Hitesh Khamesra <hitesh...@yahoo.com> Sent: Wednesday, May 3, 2017 5:42 PM Subject: Re: [gemfire-dev] New Client-Server Protocol Proposa

Re: [gemfire-dev] New Client-Server Protocol Proposal

It's becoming clear that the document needs a section on EventIds. EventIds aren't opaque to the server. They are comparable objects and are used by the cache to prevent replay of older (operation eventId < recorded eventId) operations on the cache and on subscription queues. They are also

Re: [gemfire-dev] New Client-Server Protocol Proposal

+1 On Wed, May 3, 2017 at 5:43 PM Anthony Baker wrote: > > 2) I think we should pull out the message fragmentation support to avoid > some significant complexity. We can later add a fragmentation / envelope > layer on top without disrupting the current proposal. I do think

Re: [gemfire-dev] New Client-Server Protocol Proposal

+1 to what Anthony has laid out! I think this is a better way to handle value encodings, and it's also better to be putting message specific details like event id with those messages. I do wonder whether this proposal actually needs metadata headers at all? What will eventually go in there? -Dan

Re: [gemfire-dev] New Client-Server Protocol Proposal

Good point Dan !! that needs to document. From: Dan Smith <dsm...@pivotal.io> To: dev@geode.apache.org Sent: Wednesday, May 3, 2017 5:31 PM Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal Okay but how do I has an implementer of a driver know what message

Re: [gemfire-dev] New Client-Server Protocol Proposal

> On May 3, 2017, at 1:33 PM, Galen M O'Sullivan wrote: > > On Tue, May 2, 2017 at 11:52 AM, Hitesh Khamesra < > hitesh...@yahoo.com.invalid> wrote: > >> Absolutely its a implementation detail. >> > This doesn't answer Dan's comment. Do you think fragmentation should be

Re: [gemfire-dev] New Client-Server Protocol Proposal

Okay but how do I has an implementer of a driver know what messages need an event id and which don't? It seems like maybe this belongs with those message types, rather than in a generic header. Or maybe you need to start organizing messages into classes - eg messages that change state and

Re: [gemfire-dev] New Client-Server Protocol Proposal

Correct, I did miss that. @Dan, if you look at https://cwiki.apache.org/confluence/display/GEODE/Message+Structure+and+Definition#MessageStructureandDefinition-MetaDataforRequests specifies how we provide EventId information. On 5/3/17 09:53, Bruce Schuchardt wrote: I believe Hitesh put

Re: [gemfire-dev] New Client-Server Protocol Proposal

quot;dev@geode.apache.org" <dev@geode.apache.org> Sent: Wednesday, May 3, 2017 10:01 AM Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal Here are the few things we need to consider.. 1. key, value, callbackarg can be required to interpret as JSON-to-pdx2. client calls &quo

Re: [gemfire-dev] New Client-Server Protocol Proposal

The TCP fragmentation is fine for what it is, but it is *not* paging, and paging has long been something that we have wanted to get around to. -- Mike Stolz Principal Engineer, GemFire Product Manager Mobile: +1-631-835-4771 On Wed, May 3, 2017 at 1:33 PM, Galen M O'Sullivan

Re: [gemfire-dev] New Client-Server Protocol Proposal

On Tue, May 2, 2017 at 11:52 AM, Hitesh Khamesra < hitesh...@yahoo.com.invalid> wrote: > Absolutely its a implementation detail. > This doesn't answer Dan's comment. Do you think fragmentation should be taken care of by the TCP layer or the protocol should deal with it specifically?

Re: [gemfire-dev] New Client-Server Protocol Proposal

o json or vice-versa. Any thoughts? Thanks. HItesh From: Hitesh Khamesra <hitesh...@yahoo.com.INVALID> To: "dev@geode.apache.org" <dev@geode.apache.org> Sent: Wednesday, May 3, 2017 10:01 AM Subject: Re: [gemfire-dev] New Client-Server Pr

Re: [gemfire-dev] New Client-Server Protocol Proposal

From: Jacob Barrett <jbarr...@pivotal.io> To: dev@geode.apache.org Cc: Udo Kohlmeyer <ukohlme...@pivotal.io> Sent: Tuesday, May 2, 2017 8:11 PM Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal I agree completely with Dan. There is no reason to have flags for val

Re: [gemfire-dev] New Client-Server Protocol Proposal

I believe Hitesh put EventId in the metadata section. Le 5/2/2017 à 2:22 PM, Udo Kohlmeyer a écrit : We are considering the function service, but again, this should not detract from the proposed message specification proposal. You are also correct in your observation of list of error codes

Re: [gemfire-dev] New Client-Server Protocol Proposal

From: Michael Stolz <mst...@pivotal.io> To: dev@geode.apache.org Cc: Udo Kohlmeyer <ukohlme...@pivotal.io> Sent: Wednesday, May 3, 2017 8:55 AM Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal I'm not seeing any mention of versioning of the serializatio

Re: [gemfire-dev] New Client-Server Protocol Proposal

I agree completely with Dan. There is no reason to have flags for value encoding type in the message. I would argue that should be part of the value serialization layer. If something was placed in the message layer it should be more generic and allow for an unrestricted set of encodings by some

Re: [gemfire-dev] New Client-Server Protocol Proposal

We are considering the function service, but again, this should not detract from the proposed message specification proposal. You are also correct in your observation of list of error codes not being complete nor exhaustive. Maybe the first page needs to highlight that this is a proposal and

Re: [gemfire-dev] New Client-Server Protocol Proposal

I guess the value of building other messages on top of the function service mostly comes into play when we start talking about smarter clients that can do single hop. At that point it's really nice to have have a layer that lets us send a message to a single primary, or all of the members that

Re: [gemfire-dev] New Client-Server Protocol Proposal

ent: Tuesday, May 2, 2017 11:39 AM Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal > IsPartialMessage: This flag gives us ability to send partial message > without serializing the whole key-value(request). lets say I execute > function at server and function just returns

Re: [gemfire-dev] New Client-Server Protocol Proposal

> IsPartialMessage: This flag gives us ability to send partial message > without serializing the whole key-value(request). lets say I execute > function at server and function just returns "arraylist of object". And the > serialized size of ""arraylist of object"" is quite big( > 2gb). > My point

Re: [gemfire-dev] New Client-Server Protocol Proposal

Hey Dan, Imo, having a standardized, versioned definition for GET, PUT, PUTALL, etc. message, that is encoded/decoded in a manner that multiple clients (written in many other languages) can encode/decode these messages, is paramount. Having the standardized operational

Re: [gemfire-dev] New Client-Server Protocol Proposal

I think the downside of having a single generic message type is that we lose “type safety” and some efficiency. The message definition would essentially become: String functionName; byte[][] args; It’s a little more challenging for an author of a Geode Driver to fill in the args correctly

Re: [gemfire-dev] New Client-Server Protocol Proposal

t can send JSON string and we want to save that JSON string into pdx. From: Dan Smith <dsm...@pivotal.io> To: Udo Kohlmeyer <ukohlme...@pivotal.io> Cc: dev@geode.apache.org Sent: Monday, May 1, 2017 5:53 PM Subject: Re: [gemfire-dev] New Client-Server Protocol Proposal

Re: [gemfire-dev] New Client-Server Protocol Proposal

I think the current proposal seems to be gloming some things together that probably belong in different layers. The message header currently is specified to have things like correlation id, isPartial message, and also metdatadata about whether the key or the value is JSON. Fragmenting messages

Re: [gemfire-dev] New Client-Server Protocol Proposal

I think any new client driver or server we develop might want to incorporate function execution at lower level than region operations like get and put, etc. We could then easily build operations like GET, PUT, PUTALL, etc. on top of that by making them functions. The original client protocol isn't