Re: Cayenne ROP Improvements

[Aristedes Maniatis]

Sure, but let's not lose sight of what makes Cayenne ROP interesting and 
different. It exposes the whole stack of Cayenne contexts, nested contexts, 
rollback, prefetch, pagination, etc and not just the data objects themselves.

Otherwise it is just another restful, json, routing stack of which there are 
quite a few. Not that those simple rest stacks aren't useful, but what we have 
in Cayenne ROP is really quite different and interesting.


By coincidence I'm just today thinking about the lifecycle of data in the 
backbonejs library. That a javascript library for creating a data model and 
dealing with routing. It already has connectors for LocalStorage (storing data 
in the browser db). And of course for json/ajax. The question is, should we 
overlay a concept of Context to group the data in the model, track changes and 
move it from LocalStorage to ajax onto the server?


Ari


On 8/07/2016 10:16pm, Andrus Adamchik wrote:
> I suspect we'll need to step away from "blind" serialization of our 
> server-side data structures and devise a JSON object protocol for any JS 
> integrations to become practical. This may actually be simpler then it sounds 
> (LinkRest does a good deal of it already: 
> http://linkrest.io/docs/protocol.html ) and if there's a right client-side 
> framework to work with such protocol, it would certainly be a big motivation.
> 
> Andrus
> 
>> On Jul 8, 2016, at 3:11 PM, Michael Gentry  wrote:
>>
>> Another item on my ever-growing to-do list is to explore if BreezeJS could
>> be used nicely with Cayenne.  From my initial glance at the project, it
>> seems to have the spirit of Cayenne in a JS stack.
>>
>>
>> http://breeze.github.io/doc-js/
>>
>> http://www.getbreezenow.com/
>>
>> http://learn.breezejs.com/
>>
>>
>> mrg
>>
>>
>>
>> On Thu, Jul 7, 2016 at 9:01 PM, Aristedes Maniatis  wrote:
>>
>>> On 30/06/2016 8:36pm, Aristedes Maniatis wrote:
 I'll report more after a few weeks once we have real customers using
>>> this in production.
>>>
>>> We now have this version of Cayenne (pretty close to what is now trunk) in
>>> production at around 20 sites with more to come over the next weeks.
>>>
>>> Not a single reported problem and significant performance improvements
>>> across the board. Any site with more than 40ms latency to the data centre
>>> (that is, between the Cayenne ROP client and server) is seeing noticeable
>>> improvements.
>>>
>>> Savva, very nice work indeed.
>>>
>>>
>>>
>>> Andrus, if you have json workflows protostuff also supports json encoding.
>>> [1]  Of course that means a whole bunch of json encoded Cayenne objects
>>> like ObjectContext and I have no idea what you'd do with them in
>>> javascript, but if someone wanted to code a js version of Cayenne client, I
>>> guess they could.
>>>
>>> That would be an interesting idea to layer on top of backbonejs or similar.
>>>
>>>
>>> Ari
>>>
>>>
>>>
>>>
>>> [1]
>>> https://github.com/protostuff/protostuff/tree/master/protostuff-json/src/main/java/io/protostuff
>>>
>>> --
>>> -->
>>> Aristedes Maniatis
>>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>>>
> 

-- 
-->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A

Re: Cayenne ROP Improvements

[Andrus Adamchik]

I suspect we'll need to step away from "blind" serialization of our server-side 
data structures and devise a JSON object protocol for any JS integrations to 
become practical. This may actually be simpler then it sounds (LinkRest does a 
good deal of it already: http://linkrest.io/docs/protocol.html ) and if there's 
a right client-side framework to work with such protocol, it would certainly be 
a big motivation.

Andrus

> On Jul 8, 2016, at 3:11 PM, Michael Gentry  wrote:
> 
> Another item on my ever-growing to-do list is to explore if BreezeJS could
> be used nicely with Cayenne.  From my initial glance at the project, it
> seems to have the spirit of Cayenne in a JS stack.
> 
> 
> http://breeze.github.io/doc-js/
> 
> http://www.getbreezenow.com/
> 
> http://learn.breezejs.com/
> 
> 
> mrg
> 
> 
> 
> On Thu, Jul 7, 2016 at 9:01 PM, Aristedes Maniatis  wrote:
> 
>> On 30/06/2016 8:36pm, Aristedes Maniatis wrote:
>>> I'll report more after a few weeks once we have real customers using
>> this in production.
>> 
>> We now have this version of Cayenne (pretty close to what is now trunk) in
>> production at around 20 sites with more to come over the next weeks.
>> 
>> Not a single reported problem and significant performance improvements
>> across the board. Any site with more than 40ms latency to the data centre
>> (that is, between the Cayenne ROP client and server) is seeing noticeable
>> improvements.
>> 
>> Savva, very nice work indeed.
>> 
>> 
>> 
>> Andrus, if you have json workflows protostuff also supports json encoding.
>> [1]  Of course that means a whole bunch of json encoded Cayenne objects
>> like ObjectContext and I have no idea what you'd do with them in
>> javascript, but if someone wanted to code a js version of Cayenne client, I
>> guess they could.
>> 
>> That would be an interesting idea to layer on top of backbonejs or similar.
>> 
>> 
>> Ari
>> 
>> 
>> 
>> 
>> [1]
>> https://github.com/protostuff/protostuff/tree/master/protostuff-json/src/main/java/io/protostuff
>> 
>> --
>> -->
>> Aristedes Maniatis
>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>> 

Re: Cayenne ROP Improvements

[Michael Gentry]

Another item on my ever-growing to-do list is to explore if BreezeJS could
be used nicely with Cayenne.  From my initial glance at the project, it
seems to have the spirit of Cayenne in a JS stack.


http://breeze.github.io/doc-js/

http://www.getbreezenow.com/

http://learn.breezejs.com/


mrg



On Thu, Jul 7, 2016 at 9:01 PM, Aristedes Maniatis  wrote:

> On 30/06/2016 8:36pm, Aristedes Maniatis wrote:
> > I'll report more after a few weeks once we have real customers using
> this in production.
>
> We now have this version of Cayenne (pretty close to what is now trunk) in
> production at around 20 sites with more to come over the next weeks.
>
> Not a single reported problem and significant performance improvements
> across the board. Any site with more than 40ms latency to the data centre
> (that is, between the Cayenne ROP client and server) is seeing noticeable
> improvements.
>
> Savva, very nice work indeed.
>
>
>
> Andrus, if you have json workflows protostuff also supports json encoding.
> [1]  Of course that means a whole bunch of json encoded Cayenne objects
> like ObjectContext and I have no idea what you'd do with them in
> javascript, but if someone wanted to code a js version of Cayenne client, I
> guess they could.
>
> That would be an interesting idea to layer on top of backbonejs or similar.
>
>
> Ari
>
>
>
>
> [1]
> https://github.com/protostuff/protostuff/tree/master/protostuff-json/src/main/java/io/protostuff
>
> --
> -->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>

Re: Cayenne ROP Improvements

[Savva Kolbachev]

> Not a single reported problem and significant performance improvements
across the board. Any site with more than 40ms latency to the data > centre
(that is, between the Cayenne ROP client and server) is seeing noticeable
improvements.
>
> Savva, very nice work indeed.

Thanks! I'm very happy that it works well for you. I hope it will be
helpful for other Cayenne users as well.

2016-07-08 10:55 GMT+03:00 Aristedes Maniatis :

> On 8/07/2016 4:24pm, Andrus Adamchik wrote:
> > One possible usage is ROP debugging. It was noticed before that due to
> the blind serialization nature of ROP, occasionally unintended server-side
> objects are sent to the client and vice versa (like someone patches a query
> to "precompile" its state, adding a reference to EntityResolver or
> something). Wonder if switching things to JSON in development/testing would
> help in visual tracking of those extraneous serialized objects?
>
>
> That's a nice idea, especially if it is possible to dump the json stream
> to disk while still using the protostuff binary stream to the client.
>
> Ari
>
>
> --
> -->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>



-- 
Best Regards,
Savva Kolbachev

Re: Cayenne ROP Improvements

[Aristedes Maniatis]

On 8/07/2016 4:24pm, Andrus Adamchik wrote:
> One possible usage is ROP debugging. It was noticed before that due to the 
> blind serialization nature of ROP, occasionally unintended server-side 
> objects are sent to the client and vice versa (like someone patches a query 
> to "precompile" its state, adding a reference to EntityResolver or 
> something). Wonder if switching things to JSON in development/testing would 
> help in visual tracking of those extraneous serialized objects?


That's a nice idea, especially if it is possible to dump the json stream to 
disk while still using the protostuff binary stream to the client.

Ari


-- 
-->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A

Re: Cayenne ROP Improvements

[Andrus Adamchik]

> Not a single reported problem and significant performance improvements across 
> the board. Any site with more than 40ms latency to the data centre (that is, 
> between the Cayenne ROP client and server) is seeing noticeable improvements.
> 
> Savva, very nice work indeed.

Nice!

> Andrus, if you have json workflows protostuff also supports json encoding. 
> [1]  Of course that means a whole bunch of json encoded Cayenne objects like 
> ObjectContext

One possible usage is ROP debugging. It was noticed before that due to the 
blind serialization nature of ROP, occasionally unintended server-side objects 
are sent to the client and vice versa (like someone patches a query to 
"precompile" its state, adding a reference to EntityResolver or something). 
Wonder if switching things to JSON in development/testing would help in visual 
tracking of those extraneous serialized objects?

Andrus



> On Jul 8, 2016, at 4:01 AM, Aristedes Maniatis  wrote:
> 
> On 30/06/2016 8:36pm, Aristedes Maniatis wrote:
>> I'll report more after a few weeks once we have real customers using this in 
>> production.
> 
> We now have this version of Cayenne (pretty close to what is now trunk) in 
> production at around 20 sites with more to come over the next weeks.
> 
> Not a single reported problem and significant performance improvements across 
> the board. Any site with more than 40ms latency to the data centre (that is, 
> between the Cayenne ROP client and server) is seeing noticeable improvements.
> 
> Savva, very nice work indeed.
> 
> 
> 
> Andrus, if you have json workflows protostuff also supports json encoding. 
> [1]  Of course that means a whole bunch of json encoded Cayenne objects like 
> ObjectContext and I have no idea what you'd do with them in javascript, but 
> if someone wanted to code a js version of Cayenne client, I guess they could.
> 
> That would be an interesting idea to layer on top of backbonejs or similar.
> 
> 
> Ari
> 
> 
> 
> 
> [1] 
> https://github.com/protostuff/protostuff/tree/master/protostuff-json/src/main/java/io/protostuff
> 
> -- 
> -->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A

Re: Cayenne ROP Improvements

[Aristedes Maniatis]

On 30/06/2016 8:36pm, Aristedes Maniatis wrote:
> I'll report more after a few weeks once we have real customers using this in 
> production.

We now have this version of Cayenne (pretty close to what is now trunk) in 
production at around 20 sites with more to come over the next weeks.

Not a single reported problem and significant performance improvements across 
the board. Any site with more than 40ms latency to the data centre (that is, 
between the Cayenne ROP client and server) is seeing noticeable improvements.

Savva, very nice work indeed.



Andrus, if you have json workflows protostuff also supports json encoding. [1]  
Of course that means a whole bunch of json encoded Cayenne objects like 
ObjectContext and I have no idea what you'd do with them in javascript, but if 
someone wanted to code a js version of Cayenne client, I guess they could.

That would be an interesting idea to layer on top of backbonejs or similar.


Ari




[1] 
https://github.com/protostuff/protostuff/tree/master/protostuff-json/src/main/java/io/protostuff

-- 
-->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A

Re: Cayenne ROP Improvements

[Andrus Adamchik]

Nice! 

With Thrift and Protobuf gathering more acceptance as JSON alternatives for web 
services in general, perhaps ROP has a brighter future then we thought :)

Andrus

> On Jun 30, 2016, at 6:36 AM, Aristedes Maniatis  wrote:
> 
> On 30/06/2016 7:21pm, Savva Kolbachev wrote:
>> BTW, as I know Ari already uses http2 and Protostuff. Please, let us know
>> how it's going. If it's OK, I'll notify user-list about this features.
> 
> We've actually just deployed the upgrade into our staging environment are 
> will likely roll out the first customers early next week. If our preliminary 
> (very non-scientific) results are anything to go by, almost everything is 
> faster across a slow link.
> 
> Some parts of the UI (opening a certain window) went from 14 seconds to 5 
> seconds. Another from 87 seconds to 10 seconds. And one from 8 to 5. The 
> difference is likely how effectively we used prefetches already. In some 
> places where we follow a lot of relation joins, we could easily generate 50 
> separate fetches across the network. And on a slow link (say over 40ms 
> latency and bandwidth under 2Mb/s) with SSL handshaking that adds up very 
> quickly.
> 
> I'll report more after a few weeks once we have real customers using this in 
> production.
> 
> Because we deployed both protocol-buffers and HTTP/2 at the same time, I 
> can't really tell which one had more effect, but the combination looks great.
> 
> Ari
> 
> -- 
> -->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A

Re: Cayenne ROP Improvements

[Aristedes Maniatis]

On 30/06/2016 7:21pm, Savva Kolbachev wrote:
> BTW, as I know Ari already uses http2 and Protostuff. Please, let us know
> how it's going. If it's OK, I'll notify user-list about this features.

We've actually just deployed the upgrade into our staging environment are will 
likely roll out the first customers early next week. If our preliminary (very 
non-scientific) results are anything to go by, almost everything is faster 
across a slow link.

Some parts of the UI (opening a certain window) went from 14 seconds to 5 
seconds. Another from 87 seconds to 10 seconds. And one from 8 to 5. The 
difference is likely how effectively we used prefetches already. In some places 
where we follow a lot of relation joins, we could easily generate 50 separate 
fetches across the network. And on a slow link (say over 40ms latency and 
bandwidth under 2Mb/s) with SSL handshaking that adds up very quickly.

I'll report more after a few weeks once we have real customers using this in 
production.

Because we deployed both protocol-buffers and HTTP/2 at the same time, I can't 
really tell which one had more effect, but the combination looks great.

Ari

-- 
-->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A

Re: Cayenne ROP Improvements

[Savva Kolbachev]

I've finished work on Jetty Client implementation for Cayenne ROP by
https://github.com/apache/cayenne/commit/34646fd5baf3640208399b2f21afc7afff6def51


So now we have cayenne-client-jetty module which works as follows:
There is a one JettyHttpROPConnector

which
takes and uses Jetty HttpClient (high-level API). It could be configured by
JettyHttpClientConnectionProvider

for
HTTP/1.1 and by JettyHttp2ClientConnectionProvider

for
HTTP/2. The last one initializes Jetty HttpClient over HTTP2Client
through HttpClientTransportOverHTTP2 and works without ALPN by default.
ALPN could be configured
via Constants.ROP_SERVICE_USE_ALPN_PROPERTY. However, I'm not sure it's the
best way to do it.

Those ConnectionProviders are bound by ClientJettyHttpModule

 and ClientJettyHttp2Module

accordingly.
So, users just need to add the appropriate Module to their ClientRuntime.

I think all those things could be used by themselves or as examples to
providing custom implementations. Also, I've added the simple ROP tutorial
with both ClientJettyHttp2Module and ProtostuffModule:
Http2Client

Http2Server


BTW, as I know Ari already uses http2 and Protostuff. Please, let us know
how it's going. If it's OK, I'll notify user-list about this features.


2016-06-21 3:54 GMT+03:00 Aristedes Maniatis :

> On 13/06/2016 6:40pm, Savva Kolbachev wrote:
> > After some time I think that my current HTTP/2 ROP implementation for
> > Cayenne [1] is cumbersome and complicated. I want to refactor it and make
> > it as clear as possible, so it will be simple to use out of box.
> > So, I suggest to rename module to something like "cayenne-jetty-client"
> and
> > implement only two ROPConnectors: JettyHttpConnector and
> > JettyHttp2Connector for HTTP/1.1 and HTTP/2 accordingly. Both will use
> > high-level Jetty Client API.
>
> When I look at https://github.com/apache/cayenne/pull/96/files I think
> I'm seeing two separate things.
>
> 1. Adjustments and improvements to the Cayenne ROP implementation. For
> example changes to the exception handling in cayenne/rop/ROPConnector.java
>
> 2. An implementation of HTTP2/jetty.
>
>
> It would be interesting to split the two. First to commit the ROP
> improvements, but then to see what is left over. If it is only a couple of
> classes HighHttp2... LowHttp2... etc then another option is not to
> implement this at all as a compiled jar which users  add to their own
> project. Rather, they could be some code examples that users copy and paste.
>
>
> I'm not convinced this is the best approach, but I wanted to raise it as
> an option. I guess the question is whether users would find the
> org.apache.cayenne.configuration.Constants approach to configuration
> easier, or would prefer just editing the relevant files to suit their own
> config system.
>
>
>
> At any rate, now that the Jetty project have fixed the bug you found
> there, it looks like HTTP2-Cayenne-ROP is ready for production! Nice work.
>
>
> Ari
>
>
>
>
> --
> -->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>



-- 
Best Regards,
Savva Kolbachev

Re: Cayenne ROP Improvements

[Aristedes Maniatis]

On 13/06/2016 6:40pm, Savva Kolbachev wrote:
> After some time I think that my current HTTP/2 ROP implementation for
> Cayenne [1] is cumbersome and complicated. I want to refactor it and make
> it as clear as possible, so it will be simple to use out of box.
> So, I suggest to rename module to something like "cayenne-jetty-client" and
> implement only two ROPConnectors: JettyHttpConnector and
> JettyHttp2Connector for HTTP/1.1 and HTTP/2 accordingly. Both will use
> high-level Jetty Client API.

When I look at https://github.com/apache/cayenne/pull/96/files I think I'm 
seeing two separate things.

1. Adjustments and improvements to the Cayenne ROP implementation. For example 
changes to the exception handling in cayenne/rop/ROPConnector.java

2. An implementation of HTTP2/jetty.


It would be interesting to split the two. First to commit the ROP improvements, 
but then to see what is left over. If it is only a couple of classes 
HighHttp2... LowHttp2... etc then another option is not to implement this at 
all as a compiled jar which users  add to their own project. Rather, they could 
be some code examples that users copy and paste.


I'm not convinced this is the best approach, but I wanted to raise it as an 
option. I guess the question is whether users would find the 
org.apache.cayenne.configuration.Constants approach to configuration easier, or 
would prefer just editing the relevant files to suit their own config system.



At any rate, now that the Jetty project have fixed the bug you found there, it 
looks like HTTP2-Cayenne-ROP is ready for production! Nice work.


Ari




-- 
-->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A

Re: Cayenne ROP Improvements

[Andrus Adamchik]

> Please, let me know what do you think about all this things?

Sounds reasonable to me. 

> Finally, as Cayenne is becoming more modular, it will be helpful to have
> server and client test jars. It is sometimes frustrating to write
> integration tests for new modules.

No objections here. Let's make sure we set deploy plugin "skip" property to 
"true". We don't want them to be part of releases.

Andrus



> On Jun 13, 2016, at 11:40 AM, Savva Kolbachev  wrote:
> 
> After some time I think that my current HTTP/2 ROP implementation for
> Cayenne [1] is cumbersome and complicated. I want to refactor it and make
> it as clear as possible, so it will be simple to use out of box.
> So, I suggest to rename module to something like "cayenne-jetty-client" and
> implement only two ROPConnectors: JettyHttpConnector and
> JettyHttp2Connector for HTTP/1.1 and HTTP/2 accordingly. Both will use
> high-level Jetty Client API.
> If someone wants something specific, they could provide their own similar
> implementation. I think the most common case for this is to share one
> HttpClient between ROP and other parts of a project.
> In addition, we will be open to make more modules for other clients, like
> Apache HTTP Client, Netty, OkHttp and etc.
> 
> BTW, I got rid of Hessian ServiceContext in ROPServlet [2] by implementing
> something very similar in Cayenne
> https://github.com/apache/cayenne/commit/85852f92ee05f5e1ae70dae7315beda32f2b4077
> So now Hessian is used only for serialization purposes. As we want to move
> all ROP functionality to the separate module and Hessian serialization has
> problems with Java8, we should decide which serialization service should be
> used by default. For example Protostuff or Java serialization. Hessian one
> we could remain as pluggable.
> 
> Finally, as Cayenne is becoming more modular, it will be helpful to have
> server and client test jars. It is sometimes frustrating to write
> integration tests for new modules.
> 
> Please, let me know what do you think about all this things?
> 
> [1] https://github.com/apache/cayenne/pull/96
> [2] https://issues.apache.org/jira/browse/CAY-2090
> 
> 
> 2016-05-06 3:29 GMT+03:00 Andrus Adamchik :
> 
>> Good idea. The server part should probably be split from cayenne-server.
>> 
>> Andrus
>> 
>>> On May 5, 2016, at 8:26 PM, Aristedes Maniatis  wrote:
>>> 
>>> Perhaps the whole of ROP becomes an optional module?
>>> 
>>> Ari
>>> 
>>> 
>>> On 6/05/2016 10:18am, Andrus Adamchik wrote:
 It's been a while since I touched the ROP code. Back in the day Java
>> serialization "kind of worked", but not completely. So you are probably
>> right that it is not a real option. I am just trying to avoid new
>> dependencies (even optional) on third-party libs in the Cayenne core. So
>> perhaps we can simply leave out any "default" serialization and always
>> require an explicit serialization provider.
 
 Andrus
 
> On May 5, 2016, at 8:12 PM, Aristedes Maniatis 
>> wrote:
> 
> Maybe I'm not understanding correctly, but I don't think Java
>> serialisation has been implemented in ROP. The work Dima did was to move
>> away from the Hessian servlet stuff for making the HTTP connection, to
>> plain Java with the option for plugging in Jetty libraries for HTTP/2.
> 
> The work Savva did just now was to use protostuff for serialisation,
>> but I'm not sure what's now needed if we wanted plain Java serialisation or
>> whether that's even possible without some sort of library to handle an
>> object graph with cycles.
> 
> Or at least that's my understanding.
> 
> Ari
> 
> 
> On 6/05/2016 9:55am, Andrus Adamchik wrote:
>> Thanks for clarification. I would say use Java serialization as a
>> default, and make it easy to plugin Hessian and Protostuff as separate
>> modules.
>> 
>> A.
>> 
>>> On May 5, 2016, at 5:39 PM, Savva Kolbachev 
>> wrote:
>>> 
>>> Hi Andrus,
>>> 
 So which one is the default, Hessian or Java?
>>> We still use Hessian for serialization by default
>>> 
>> https://github.com/apache/cayenne/blob/master/cayenne-server/src/main/java/org/apache/cayenne/rop/HessianROPSerializationService.java
>>> But we use java.net.URLConnection for establish connection and
>> sending
>>> messages from client to server
>>> 
>> https://github.com/apache/cayenne/blob/master/cayenne-client/src/main/java/org/apache/cayenne/rop/http/HttpROPConnector.java
>>> So we have escaped from Hessian only in connectivity layer.
>>> 
 I don't have a problem with Protostuff being a recommended default,
>> but
>>> for dependency management purposes I'd rather we split all
>> third-party
>>> integrations in separate modules, and use whatever provider is
>> hooked up in
>>> runtime. Kind of what we do with Joda/Java8 extensions.
>>> I already did it in this way. I created separate module for
>> Protostuff
>>> serialization.
>>> 
>>

Re: Cayenne ROP Improvements

[Savva Kolbachev]

After some time I think that my current HTTP/2 ROP implementation for
Cayenne [1] is cumbersome and complicated. I want to refactor it and make
it as clear as possible, so it will be simple to use out of box.
So, I suggest to rename module to something like "cayenne-jetty-client" and
implement only two ROPConnectors: JettyHttpConnector and
JettyHttp2Connector for HTTP/1.1 and HTTP/2 accordingly. Both will use
high-level Jetty Client API.
If someone wants something specific, they could provide their own similar
implementation. I think the most common case for this is to share one
HttpClient between ROP and other parts of a project.
In addition, we will be open to make more modules for other clients, like
Apache HTTP Client, Netty, OkHttp and etc.

BTW, I got rid of Hessian ServiceContext in ROPServlet [2] by implementing
something very similar in Cayenne
https://github.com/apache/cayenne/commit/85852f92ee05f5e1ae70dae7315beda32f2b4077
So now Hessian is used only for serialization purposes. As we want to move
all ROP functionality to the separate module and Hessian serialization has
problems with Java8, we should decide which serialization service should be
used by default. For example Protostuff or Java serialization. Hessian one
we could remain as pluggable.

Finally, as Cayenne is becoming more modular, it will be helpful to have
server and client test jars. It is sometimes frustrating to write
integration tests for new modules.

Please, let me know what do you think about all this things?

[1] https://github.com/apache/cayenne/pull/96
[2] https://issues.apache.org/jira/browse/CAY-2090


2016-05-06 3:29 GMT+03:00 Andrus Adamchik :

> Good idea. The server part should probably be split from cayenne-server.
>
> Andrus
>
> > On May 5, 2016, at 8:26 PM, Aristedes Maniatis  wrote:
> >
> > Perhaps the whole of ROP becomes an optional module?
> >
> > Ari
> >
> >
> > On 6/05/2016 10:18am, Andrus Adamchik wrote:
> >> It's been a while since I touched the ROP code. Back in the day Java
> serialization "kind of worked", but not completely. So you are probably
> right that it is not a real option. I am just trying to avoid new
> dependencies (even optional) on third-party libs in the Cayenne core. So
> perhaps we can simply leave out any "default" serialization and always
> require an explicit serialization provider.
> >>
> >> Andrus
> >>
> >>> On May 5, 2016, at 8:12 PM, Aristedes Maniatis 
> wrote:
> >>>
> >>> Maybe I'm not understanding correctly, but I don't think Java
> serialisation has been implemented in ROP. The work Dima did was to move
> away from the Hessian servlet stuff for making the HTTP connection, to
> plain Java with the option for plugging in Jetty libraries for HTTP/2.
> >>>
> >>> The work Savva did just now was to use protostuff for serialisation,
> but I'm not sure what's now needed if we wanted plain Java serialisation or
> whether that's even possible without some sort of library to handle an
> object graph with cycles.
> >>>
> >>> Or at least that's my understanding.
> >>>
> >>> Ari
> >>>
> >>>
> >>> On 6/05/2016 9:55am, Andrus Adamchik wrote:
>  Thanks for clarification. I would say use Java serialization as a
> default, and make it easy to plugin Hessian and Protostuff as separate
> modules.
> 
>  A.
> 
> > On May 5, 2016, at 5:39 PM, Savva Kolbachev 
> wrote:
> >
> > Hi Andrus,
> >
> >> So which one is the default, Hessian or Java?
> > We still use Hessian for serialization by default
> >
> https://github.com/apache/cayenne/blob/master/cayenne-server/src/main/java/org/apache/cayenne/rop/HessianROPSerializationService.java
> > But we use java.net.URLConnection for establish connection and
> sending
> > messages from client to server
> >
> https://github.com/apache/cayenne/blob/master/cayenne-client/src/main/java/org/apache/cayenne/rop/http/HttpROPConnector.java
> > So we have escaped from Hessian only in connectivity layer.
> >
> >> I don't have a problem with Protostuff being a recommended default,
> but
> > for dependency management purposes I'd rather we split all
> third-party
> > integrations in separate modules, and use whatever provider is
> hooked up in
> > runtime. Kind of what we do with Joda/Java8 extensions.
> > I already did it in this way. I created separate module for
> Protostuff
> > serialization.
> >
> > As Hessian serialization has some troubles with Java8 types and
> provide
> > less efficient serialization than Protostuff, I suggest to use
> Protostuff
> > as default serialization service or to use Java serialization. So I
> just
> > suggest to escape from Hessian :)
> >
> > 2016-05-05 19:41 GMT+03:00 Savva Kolbachev :
> >
> >> Hi Ari,
> >>
> >> Looks like Protostuff works faster than Protobuf in some cases. For
> >> example Serializers (no shared refs) and Cross Lang Binary
> Serializers
> >> sections here
> http://hperadin.github

Re: Cayenne ROP Improvements

[Andrus Adamchik]

Good idea. The server part should probably be split from cayenne-server.

Andrus

> On May 5, 2016, at 8:26 PM, Aristedes Maniatis  wrote:
> 
> Perhaps the whole of ROP becomes an optional module?
> 
> Ari
> 
> 
> On 6/05/2016 10:18am, Andrus Adamchik wrote:
>> It's been a while since I touched the ROP code. Back in the day Java 
>> serialization "kind of worked", but not completely. So you are probably 
>> right that it is not a real option. I am just trying to avoid new 
>> dependencies (even optional) on third-party libs in the Cayenne core. So 
>> perhaps we can simply leave out any "default" serialization and always 
>> require an explicit serialization provider.
>> 
>> Andrus
>> 
>>> On May 5, 2016, at 8:12 PM, Aristedes Maniatis  wrote:
>>> 
>>> Maybe I'm not understanding correctly, but I don't think Java serialisation 
>>> has been implemented in ROP. The work Dima did was to move away from the 
>>> Hessian servlet stuff for making the HTTP connection, to plain Java with 
>>> the option for plugging in Jetty libraries for HTTP/2.
>>> 
>>> The work Savva did just now was to use protostuff for serialisation, but 
>>> I'm not sure what's now needed if we wanted plain Java serialisation or 
>>> whether that's even possible without some sort of library to handle an 
>>> object graph with cycles.
>>> 
>>> Or at least that's my understanding.
>>> 
>>> Ari
>>> 
>>> 
>>> On 6/05/2016 9:55am, Andrus Adamchik wrote:
 Thanks for clarification. I would say use Java serialization as a default, 
 and make it easy to plugin Hessian and Protostuff as separate modules.
 
 A.
 
> On May 5, 2016, at 5:39 PM, Savva Kolbachev  wrote:
> 
> Hi Andrus,
> 
>> So which one is the default, Hessian or Java?
> We still use Hessian for serialization by default
> https://github.com/apache/cayenne/blob/master/cayenne-server/src/main/java/org/apache/cayenne/rop/HessianROPSerializationService.java
> But we use java.net.URLConnection for establish connection and sending
> messages from client to server
> https://github.com/apache/cayenne/blob/master/cayenne-client/src/main/java/org/apache/cayenne/rop/http/HttpROPConnector.java
> So we have escaped from Hessian only in connectivity layer.
> 
>> I don't have a problem with Protostuff being a recommended default, but
> for dependency management purposes I'd rather we split all third-party
> integrations in separate modules, and use whatever provider is hooked up 
> in
> runtime. Kind of what we do with Joda/Java8 extensions.
> I already did it in this way. I created separate module for Protostuff
> serialization.
> 
> As Hessian serialization has some troubles with Java8 types and provide
> less efficient serialization than Protostuff, I suggest to use Protostuff
> as default serialization service or to use Java serialization. So I just
> suggest to escape from Hessian :)
> 
> 2016-05-05 19:41 GMT+03:00 Savva Kolbachev :
> 
>> Hi Ari,
>> 
>> Looks like Protostuff works faster than Protobuf in some cases. For
>> example Serializers (no shared refs) and Cross Lang Binary Serializers
>> sections here 
>> http://hperadin.github.io/jvm-serializers-report/report.html
>> 
>> In our case we need to serialize graph of objects (Full Object Graph
>> Serializers section in link above). Protobuf can't do it out of the box
>> but Protostuff can. In my implementation I use protostuff-graph-runtime
>> which generates a schema from objects at runtime and caches it.
>> 
>> Protostuff schema is something like .proto files but in Java:
>> http://www.protostuff.io/documentation/schema/
>> Runtime schema: http://www.protostuff.io/documentation/runtime-schema/
>> 
>> As you could see in benchmarks there is a small difference in efficiency
>> between protostuff-graph and protostuff-graph-runtime. The ser/deser
>> overhead is related to runtime schema generation. The size penalty is 
>> that
>> Protostuff adds class name for objects and than uses those for find
>> appropriate classes via reflection.
>> Hessian also adds fields names so the size of Hessian serialization is
>> much bigger. In my small example with selection of 6 objects Hessian
>> serialization size is more than 2400 bytes while Protostuff runtime is
>> about 800 bytes.
>> 
>> If we don't want to have ser/deser and size overhead we could find a way
>> to generate schemas via Velocity. And we should provide schemas for some
>> Cayenne classes. But it will require a lot of efforts.
>> 
>> 
>> 2016-05-05 13:44 GMT+03:00 Aristedes Maniatis :
>> 
>>> On 5/05/2016 7:35pm, Savva Kolbachev wrote:
 Protostuff (licensed under Apache 2.0 licence) is based on Google's
 Protocol-Buffers (Protobuf) but has some optimizations and some cool
>>> things
 like runtime seria

Re: Cayenne ROP Improvements

[Aristedes Maniatis]

Perhaps the whole of ROP becomes an optional module?

Ari


On 6/05/2016 10:18am, Andrus Adamchik wrote:
> It's been a while since I touched the ROP code. Back in the day Java 
> serialization "kind of worked", but not completely. So you are probably right 
> that it is not a real option. I am just trying to avoid new dependencies 
> (even optional) on third-party libs in the Cayenne core. So perhaps we can 
> simply leave out any "default" serialization and always require an explicit 
> serialization provider.
> 
> Andrus
> 
>> On May 5, 2016, at 8:12 PM, Aristedes Maniatis  wrote:
>>
>> Maybe I'm not understanding correctly, but I don't think Java serialisation 
>> has been implemented in ROP. The work Dima did was to move away from the 
>> Hessian servlet stuff for making the HTTP connection, to plain Java with the 
>> option for plugging in Jetty libraries for HTTP/2.
>>
>> The work Savva did just now was to use protostuff for serialisation, but I'm 
>> not sure what's now needed if we wanted plain Java serialisation or whether 
>> that's even possible without some sort of library to handle an object graph 
>> with cycles.
>>
>> Or at least that's my understanding.
>>
>> Ari
>>
>>
>> On 6/05/2016 9:55am, Andrus Adamchik wrote:
>>> Thanks for clarification. I would say use Java serialization as a default, 
>>> and make it easy to plugin Hessian and Protostuff as separate modules.
>>>
>>> A.
>>>
 On May 5, 2016, at 5:39 PM, Savva Kolbachev  wrote:

 Hi Andrus,

> So which one is the default, Hessian or Java?
 We still use Hessian for serialization by default
 https://github.com/apache/cayenne/blob/master/cayenne-server/src/main/java/org/apache/cayenne/rop/HessianROPSerializationService.java
 But we use java.net.URLConnection for establish connection and sending
 messages from client to server
 https://github.com/apache/cayenne/blob/master/cayenne-client/src/main/java/org/apache/cayenne/rop/http/HttpROPConnector.java
 So we have escaped from Hessian only in connectivity layer.

> I don't have a problem with Protostuff being a recommended default, but
 for dependency management purposes I'd rather we split all third-party
 integrations in separate modules, and use whatever provider is hooked up in
 runtime. Kind of what we do with Joda/Java8 extensions.
 I already did it in this way. I created separate module for Protostuff
 serialization.

 As Hessian serialization has some troubles with Java8 types and provide
 less efficient serialization than Protostuff, I suggest to use Protostuff
 as default serialization service or to use Java serialization. So I just
 suggest to escape from Hessian :)

 2016-05-05 19:41 GMT+03:00 Savva Kolbachev :

> Hi Ari,
>
> Looks like Protostuff works faster than Protobuf in some cases. For
> example Serializers (no shared refs) and Cross Lang Binary Serializers
> sections here http://hperadin.github.io/jvm-serializers-report/report.html
>
> In our case we need to serialize graph of objects (Full Object Graph
> Serializers section in link above). Protobuf can't do it out of the box
> but Protostuff can. In my implementation I use protostuff-graph-runtime
> which generates a schema from objects at runtime and caches it.
>
> Protostuff schema is something like .proto files but in Java:
> http://www.protostuff.io/documentation/schema/
> Runtime schema: http://www.protostuff.io/documentation/runtime-schema/
>
> As you could see in benchmarks there is a small difference in efficiency
> between protostuff-graph and protostuff-graph-runtime. The ser/deser
> overhead is related to runtime schema generation. The size penalty is that
> Protostuff adds class name for objects and than uses those for find
> appropriate classes via reflection.
> Hessian also adds fields names so the size of Hessian serialization is
> much bigger. In my small example with selection of 6 objects Hessian
> serialization size is more than 2400 bytes while Protostuff runtime is
> about 800 bytes.
>
> If we don't want to have ser/deser and size overhead we could find a way
> to generate schemas via Velocity. And we should provide schemas for some
> Cayenne classes. But it will require a lot of efforts.
>
>
> 2016-05-05 13:44 GMT+03:00 Aristedes Maniatis :
>
>> On 5/05/2016 7:35pm, Savva Kolbachev wrote:
>>> Protostuff (licensed under Apache 2.0 licence) is based on Google's
>>> Protocol-Buffers (Protobuf) but has some optimizations and some cool
>> things
>>> like runtime serialization graph of objects (like Hessian). It also
>> could
>>> generate schema on runtime so we shouldn't define .proto files although
>> it
>>> might increase efficiency. It works faster than Hessian and could handle
>>> Java8 Date and Time types. Here is some benchmarks. 

Re: Cayenne ROP Improvements

[Andrus Adamchik]

It's been a while since I touched the ROP code. Back in the day Java 
serialization "kind of worked", but not completely. So you are probably right 
that it is not a real option. I am just trying to avoid new dependencies (even 
optional) on third-party libs in the Cayenne core. So perhaps we can simply 
leave out any "default" serialization and always require an explicit 
serialization provider.

Andrus

> On May 5, 2016, at 8:12 PM, Aristedes Maniatis  wrote:
> 
> Maybe I'm not understanding correctly, but I don't think Java serialisation 
> has been implemented in ROP. The work Dima did was to move away from the 
> Hessian servlet stuff for making the HTTP connection, to plain Java with the 
> option for plugging in Jetty libraries for HTTP/2.
> 
> The work Savva did just now was to use protostuff for serialisation, but I'm 
> not sure what's now needed if we wanted plain Java serialisation or whether 
> that's even possible without some sort of library to handle an object graph 
> with cycles.
> 
> Or at least that's my understanding.
> 
> Ari
> 
> 
> On 6/05/2016 9:55am, Andrus Adamchik wrote:
>> Thanks for clarification. I would say use Java serialization as a default, 
>> and make it easy to plugin Hessian and Protostuff as separate modules.
>> 
>> A.
>> 
>>> On May 5, 2016, at 5:39 PM, Savva Kolbachev  wrote:
>>> 
>>> Hi Andrus,
>>> 
 So which one is the default, Hessian or Java?
>>> We still use Hessian for serialization by default
>>> https://github.com/apache/cayenne/blob/master/cayenne-server/src/main/java/org/apache/cayenne/rop/HessianROPSerializationService.java
>>> But we use java.net.URLConnection for establish connection and sending
>>> messages from client to server
>>> https://github.com/apache/cayenne/blob/master/cayenne-client/src/main/java/org/apache/cayenne/rop/http/HttpROPConnector.java
>>> So we have escaped from Hessian only in connectivity layer.
>>> 
 I don't have a problem with Protostuff being a recommended default, but
>>> for dependency management purposes I'd rather we split all third-party
>>> integrations in separate modules, and use whatever provider is hooked up in
>>> runtime. Kind of what we do with Joda/Java8 extensions.
>>> I already did it in this way. I created separate module for Protostuff
>>> serialization.
>>> 
>>> As Hessian serialization has some troubles with Java8 types and provide
>>> less efficient serialization than Protostuff, I suggest to use Protostuff
>>> as default serialization service or to use Java serialization. So I just
>>> suggest to escape from Hessian :)
>>> 
>>> 2016-05-05 19:41 GMT+03:00 Savva Kolbachev :
>>> 
 Hi Ari,
 
 Looks like Protostuff works faster than Protobuf in some cases. For
 example Serializers (no shared refs) and Cross Lang Binary Serializers
 sections here http://hperadin.github.io/jvm-serializers-report/report.html
 
 In our case we need to serialize graph of objects (Full Object Graph
 Serializers section in link above). Protobuf can't do it out of the box
 but Protostuff can. In my implementation I use protostuff-graph-runtime
 which generates a schema from objects at runtime and caches it.
 
 Protostuff schema is something like .proto files but in Java:
 http://www.protostuff.io/documentation/schema/
 Runtime schema: http://www.protostuff.io/documentation/runtime-schema/
 
 As you could see in benchmarks there is a small difference in efficiency
 between protostuff-graph and protostuff-graph-runtime. The ser/deser
 overhead is related to runtime schema generation. The size penalty is that
 Protostuff adds class name for objects and than uses those for find
 appropriate classes via reflection.
 Hessian also adds fields names so the size of Hessian serialization is
 much bigger. In my small example with selection of 6 objects Hessian
 serialization size is more than 2400 bytes while Protostuff runtime is
 about 800 bytes.
 
 If we don't want to have ser/deser and size overhead we could find a way
 to generate schemas via Velocity. And we should provide schemas for some
 Cayenne classes. But it will require a lot of efforts.
 
 
 2016-05-05 13:44 GMT+03:00 Aristedes Maniatis :
 
> On 5/05/2016 7:35pm, Savva Kolbachev wrote:
>> Protostuff (licensed under Apache 2.0 licence) is based on Google's
>> Protocol-Buffers (Protobuf) but has some optimizations and some cool
> things
>> like runtime serialization graph of objects (like Hessian). It also
> could
>> generate schema on runtime so we shouldn't define .proto files although
> it
>> might increase efficiency. It works faster than Hessian and could handle
>> Java8 Date and Time types. Here is some benchmarks. Take a look at Full
>> Object Graph Serializers section.
>> http://hperadin.github.io/jvm-serializers-report/report.html
>> https://github.com/eishay/jvm-serializers/wiki
> 
>>

Re: Cayenne ROP Improvements

[Aristedes Maniatis]

Maybe I'm not understanding correctly, but I don't think Java serialisation has 
been implemented in ROP. The work Dima did was to move away from the Hessian 
servlet stuff for making the HTTP connection, to plain Java with the option for 
plugging in Jetty libraries for HTTP/2.

The work Savva did just now was to use protostuff for serialisation, but I'm 
not sure what's now needed if we wanted plain Java serialisation or whether 
that's even possible without some sort of library to handle an object graph 
with cycles.

Or at least that's my understanding.

Ari


On 6/05/2016 9:55am, Andrus Adamchik wrote:
> Thanks for clarification. I would say use Java serialization as a default, 
> and make it easy to plugin Hessian and Protostuff as separate modules.
> 
> A.
> 
>> On May 5, 2016, at 5:39 PM, Savva Kolbachev  wrote:
>>
>> Hi Andrus,
>>
>>> So which one is the default, Hessian or Java?
>> We still use Hessian for serialization by default
>> https://github.com/apache/cayenne/blob/master/cayenne-server/src/main/java/org/apache/cayenne/rop/HessianROPSerializationService.java
>> But we use java.net.URLConnection for establish connection and sending
>> messages from client to server
>> https://github.com/apache/cayenne/blob/master/cayenne-client/src/main/java/org/apache/cayenne/rop/http/HttpROPConnector.java
>> So we have escaped from Hessian only in connectivity layer.
>>
>>> I don't have a problem with Protostuff being a recommended default, but
>> for dependency management purposes I'd rather we split all third-party
>> integrations in separate modules, and use whatever provider is hooked up in
>> runtime. Kind of what we do with Joda/Java8 extensions.
>> I already did it in this way. I created separate module for Protostuff
>> serialization.
>>
>> As Hessian serialization has some troubles with Java8 types and provide
>> less efficient serialization than Protostuff, I suggest to use Protostuff
>> as default serialization service or to use Java serialization. So I just
>> suggest to escape from Hessian :)
>>
>> 2016-05-05 19:41 GMT+03:00 Savva Kolbachev :
>>
>>> Hi Ari,
>>>
>>> Looks like Protostuff works faster than Protobuf in some cases. For
>>> example Serializers (no shared refs) and Cross Lang Binary Serializers
>>> sections here http://hperadin.github.io/jvm-serializers-report/report.html
>>>
>>> In our case we need to serialize graph of objects (Full Object Graph
>>> Serializers section in link above). Protobuf can't do it out of the box
>>> but Protostuff can. In my implementation I use protostuff-graph-runtime
>>> which generates a schema from objects at runtime and caches it.
>>>
>>> Protostuff schema is something like .proto files but in Java:
>>> http://www.protostuff.io/documentation/schema/
>>> Runtime schema: http://www.protostuff.io/documentation/runtime-schema/
>>>
>>> As you could see in benchmarks there is a small difference in efficiency
>>> between protostuff-graph and protostuff-graph-runtime. The ser/deser
>>> overhead is related to runtime schema generation. The size penalty is that
>>> Protostuff adds class name for objects and than uses those for find
>>> appropriate classes via reflection.
>>> Hessian also adds fields names so the size of Hessian serialization is
>>> much bigger. In my small example with selection of 6 objects Hessian
>>> serialization size is more than 2400 bytes while Protostuff runtime is
>>> about 800 bytes.
>>>
>>> If we don't want to have ser/deser and size overhead we could find a way
>>> to generate schemas via Velocity. And we should provide schemas for some
>>> Cayenne classes. But it will require a lot of efforts.
>>>
>>>
>>> 2016-05-05 13:44 GMT+03:00 Aristedes Maniatis :
>>>
 On 5/05/2016 7:35pm, Savva Kolbachev wrote:
> Protostuff (licensed under Apache 2.0 licence) is based on Google's
> Protocol-Buffers (Protobuf) but has some optimizations and some cool
 things
> like runtime serialization graph of objects (like Hessian). It also
 could
> generate schema on runtime so we shouldn't define .proto files although
 it
> might increase efficiency. It works faster than Hessian and could handle
> Java8 Date and Time types. Here is some benchmarks. Take a look at Full
> Object Graph Serializers section.
> http://hperadin.github.io/jvm-serializers-report/report.html
> https://github.com/eishay/jvm-serializers/wiki

 According to those benchmarks there appears to be no performance or size
 penalty to using protostuff over protobuffers. Am I reading that right?

 I don't really understand... doesn't the serialiser have to construct a
 .proto definition and then include it in the message? So shouldn't it be
 faster/smaller to predefine these?

 If we did, we could create them with velocity in the same way we create
 Java _superclasses today. Fairly trivial I'm guessing.

 Ari


 --
 -->
 Aristedes Maniatis
 GPG 

Re: Cayenne ROP Improvements

[Andrus Adamchik]

Thanks for clarification. I would say use Java serialization as a default, and 
make it easy to plugin Hessian and Protostuff as separate modules.

A.

> On May 5, 2016, at 5:39 PM, Savva Kolbachev  wrote:
> 
> Hi Andrus,
> 
>> So which one is the default, Hessian or Java?
> We still use Hessian for serialization by default
> https://github.com/apache/cayenne/blob/master/cayenne-server/src/main/java/org/apache/cayenne/rop/HessianROPSerializationService.java
> But we use java.net.URLConnection for establish connection and sending
> messages from client to server
> https://github.com/apache/cayenne/blob/master/cayenne-client/src/main/java/org/apache/cayenne/rop/http/HttpROPConnector.java
> So we have escaped from Hessian only in connectivity layer.
> 
>> I don't have a problem with Protostuff being a recommended default, but
> for dependency management purposes I'd rather we split all third-party
> integrations in separate modules, and use whatever provider is hooked up in
> runtime. Kind of what we do with Joda/Java8 extensions.
> I already did it in this way. I created separate module for Protostuff
> serialization.
> 
> As Hessian serialization has some troubles with Java8 types and provide
> less efficient serialization than Protostuff, I suggest to use Protostuff
> as default serialization service or to use Java serialization. So I just
> suggest to escape from Hessian :)
> 
> 2016-05-05 19:41 GMT+03:00 Savva Kolbachev :
> 
>> Hi Ari,
>> 
>> Looks like Protostuff works faster than Protobuf in some cases. For
>> example Serializers (no shared refs) and Cross Lang Binary Serializers
>> sections here http://hperadin.github.io/jvm-serializers-report/report.html
>> 
>> In our case we need to serialize graph of objects (Full Object Graph
>> Serializers section in link above). Protobuf can't do it out of the box
>> but Protostuff can. In my implementation I use protostuff-graph-runtime
>> which generates a schema from objects at runtime and caches it.
>> 
>> Protostuff schema is something like .proto files but in Java:
>> http://www.protostuff.io/documentation/schema/
>> Runtime schema: http://www.protostuff.io/documentation/runtime-schema/
>> 
>> As you could see in benchmarks there is a small difference in efficiency
>> between protostuff-graph and protostuff-graph-runtime. The ser/deser
>> overhead is related to runtime schema generation. The size penalty is that
>> Protostuff adds class name for objects and than uses those for find
>> appropriate classes via reflection.
>> Hessian also adds fields names so the size of Hessian serialization is
>> much bigger. In my small example with selection of 6 objects Hessian
>> serialization size is more than 2400 bytes while Protostuff runtime is
>> about 800 bytes.
>> 
>> If we don't want to have ser/deser and size overhead we could find a way
>> to generate schemas via Velocity. And we should provide schemas for some
>> Cayenne classes. But it will require a lot of efforts.
>> 
>> 
>> 2016-05-05 13:44 GMT+03:00 Aristedes Maniatis :
>> 
>>> On 5/05/2016 7:35pm, Savva Kolbachev wrote:
 Protostuff (licensed under Apache 2.0 licence) is based on Google's
 Protocol-Buffers (Protobuf) but has some optimizations and some cool
>>> things
 like runtime serialization graph of objects (like Hessian). It also
>>> could
 generate schema on runtime so we shouldn't define .proto files although
>>> it
 might increase efficiency. It works faster than Hessian and could handle
 Java8 Date and Time types. Here is some benchmarks. Take a look at Full
 Object Graph Serializers section.
 http://hperadin.github.io/jvm-serializers-report/report.html
 https://github.com/eishay/jvm-serializers/wiki
>>> 
>>> According to those benchmarks there appears to be no performance or size
>>> penalty to using protostuff over protobuffers. Am I reading that right?
>>> 
>>> I don't really understand... doesn't the serialiser have to construct a
>>> .proto definition and then include it in the message? So shouldn't it be
>>> faster/smaller to predefine these?
>>> 
>>> If we did, we could create them with velocity in the same way we create
>>> Java _superclasses today. Fairly trivial I'm guessing.
>>> 
>>> Ari
>>> 
>>> 
>>> --
>>> -->
>>> Aristedes Maniatis
>>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>>> 
>> 
>> 
>> 
>> --
>> Thanks and Regards
>> Savva Kolbachev
>> 
> 
> 
> 
> -- 
> Thanks and Regards
> Savva Kolbachev

Re: Cayenne ROP Improvements

[Savva Kolbachev]

Hi Andrus,

> So which one is the default, Hessian or Java?
We still use Hessian for serialization by default
https://github.com/apache/cayenne/blob/master/cayenne-server/src/main/java/org/apache/cayenne/rop/HessianROPSerializationService.java
But we use java.net.URLConnection for establish connection and sending
messages from client to server
https://github.com/apache/cayenne/blob/master/cayenne-client/src/main/java/org/apache/cayenne/rop/http/HttpROPConnector.java
So we have escaped from Hessian only in connectivity layer.

> I don't have a problem with Protostuff being a recommended default, but
for dependency management purposes I'd rather we split all third-party
integrations in separate modules, and use whatever provider is hooked up in
runtime. Kind of what we do with Joda/Java8 extensions.
I already did it in this way. I created separate module for Protostuff
serialization.

As Hessian serialization has some troubles with Java8 types and provide
less efficient serialization than Protostuff, I suggest to use Protostuff
as default serialization service or to use Java serialization. So I just
suggest to escape from Hessian :)

2016-05-05 19:41 GMT+03:00 Savva Kolbachev :

> Hi Ari,
>
> Looks like Protostuff works faster than Protobuf in some cases. For
> example Serializers (no shared refs) and Cross Lang Binary Serializers
> sections here http://hperadin.github.io/jvm-serializers-report/report.html
>
> In our case we need to serialize graph of objects (Full Object Graph
> Serializers section in link above). Protobuf can't do it out of the box
> but Protostuff can. In my implementation I use protostuff-graph-runtime
> which generates a schema from objects at runtime and caches it.
>
> Protostuff schema is something like .proto files but in Java:
> http://www.protostuff.io/documentation/schema/
> Runtime schema: http://www.protostuff.io/documentation/runtime-schema/
>
> As you could see in benchmarks there is a small difference in efficiency
> between protostuff-graph and protostuff-graph-runtime. The ser/deser
> overhead is related to runtime schema generation. The size penalty is that
> Protostuff adds class name for objects and than uses those for find
> appropriate classes via reflection.
> Hessian also adds fields names so the size of Hessian serialization is
> much bigger. In my small example with selection of 6 objects Hessian
> serialization size is more than 2400 bytes while Protostuff runtime is
> about 800 bytes.
>
> If we don't want to have ser/deser and size overhead we could find a way
> to generate schemas via Velocity. And we should provide schemas for some
> Cayenne classes. But it will require a lot of efforts.
>
>
> 2016-05-05 13:44 GMT+03:00 Aristedes Maniatis :
>
>> On 5/05/2016 7:35pm, Savva Kolbachev wrote:
>> > Protostuff (licensed under Apache 2.0 licence) is based on Google's
>> > Protocol-Buffers (Protobuf) but has some optimizations and some cool
>> things
>> > like runtime serialization graph of objects (like Hessian). It also
>> could
>> > generate schema on runtime so we shouldn't define .proto files although
>> it
>> > might increase efficiency. It works faster than Hessian and could handle
>> > Java8 Date and Time types. Here is some benchmarks. Take a look at Full
>> > Object Graph Serializers section.
>> > http://hperadin.github.io/jvm-serializers-report/report.html
>> > https://github.com/eishay/jvm-serializers/wiki
>>
>> According to those benchmarks there appears to be no performance or size
>> penalty to using protostuff over protobuffers. Am I reading that right?
>>
>> I don't really understand... doesn't the serialiser have to construct a
>> .proto definition and then include it in the message? So shouldn't it be
>> faster/smaller to predefine these?
>>
>> If we did, we could create them with velocity in the same way we create
>> Java _superclasses today. Fairly trivial I'm guessing.
>>
>> Ari
>>
>>
>> --
>> -->
>> Aristedes Maniatis
>> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>>
>
>
>
> --
> Thanks and Regards
> Savva Kolbachev
>



-- 
Thanks and Regards
Savva Kolbachev

Re: Cayenne ROP Improvements

[Savva Kolbachev]

Hi Ari,

Looks like Protostuff works faster than Protobuf in some cases. For
example Serializers
(no shared refs) and Cross Lang Binary Serializers sections here
http://hperadin.github.io/jvm-serializers-report/report.html

In our case we need to serialize graph of objects (Full Object Graph
Serializers section in link above). Protobuf can't do it out of the box but
Protostuff can. In my implementation I use protostuff-graph-runtime which
generates a schema from objects at runtime and caches it.

Protostuff schema is something like .proto files but in Java:
http://www.protostuff.io/documentation/schema/
Runtime schema: http://www.protostuff.io/documentation/runtime-schema/

As you could see in benchmarks there is a small difference in efficiency
between protostuff-graph and protostuff-graph-runtime. The ser/deser
overhead is related to runtime schema generation. The size penalty is that
Protostuff adds class name for objects and than uses those for find
appropriate classes via reflection.
Hessian also adds fields names so the size of Hessian serialization is much
bigger. In my small example with selection of 6 objects Hessian
serialization size is more than 2400 bytes while Protostuff runtime is
about 800 bytes.

If we don't want to have ser/deser and size overhead we could find a way to
generate schemas via Velocity. And we should provide schemas for some
Cayenne classes. But it will require a lot of efforts.


2016-05-05 13:44 GMT+03:00 Aristedes Maniatis :

> On 5/05/2016 7:35pm, Savva Kolbachev wrote:
> > Protostuff (licensed under Apache 2.0 licence) is based on Google's
> > Protocol-Buffers (Protobuf) but has some optimizations and some cool
> things
> > like runtime serialization graph of objects (like Hessian). It also could
> > generate schema on runtime so we shouldn't define .proto files although
> it
> > might increase efficiency. It works faster than Hessian and could handle
> > Java8 Date and Time types. Here is some benchmarks. Take a look at Full
> > Object Graph Serializers section.
> > http://hperadin.github.io/jvm-serializers-report/report.html
> > https://github.com/eishay/jvm-serializers/wiki
>
> According to those benchmarks there appears to be no performance or size
> penalty to using protostuff over protobuffers. Am I reading that right?
>
> I don't really understand... doesn't the serialiser have to construct a
> .proto definition and then include it in the message? So shouldn't it be
> faster/smaller to predefine these?
>
> If we did, we could create them with velocity in the same way we create
> Java _superclasses today. Fairly trivial I'm guessing.
>
> Ari
>
>
> --
> -->
> Aristedes Maniatis
> GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A
>



-- 
Thanks and Regards
Savva Kolbachev

Re: Cayenne ROP Improvements

[Andrus Adamchik]

> On May 5, 2016, at 5:35 AM, Savva Kolbachev  wrote:
> Also we moved from Hessian as a default implementation for connectivity
> layer to the standard Java implementation. Although, Hessian is still used
> for serialization by default.

So which one is the default, Hessian or Java?

> Also I want to suggest to use Protostuff serialization instead of Hessian
> by default. What do you think about it?

I don't have a problem with Protostuff being a recommended default, but for 
dependency management purposes I'd rather we split all third-party integrations 
in separate modules, and use whatever provider is hooked up in runtime. Kind of 
what we do with Joda/Java8 extensions. 

Going forward we may simplify dependency integration to be "drag and drop" by 
using Java ServiceLoader API. But even without it, it is very simple to switch 
between "providers" of various things.

Andrus

Re: Cayenne ROP Improvements

[Aristedes Maniatis]

On 5/05/2016 7:35pm, Savva Kolbachev wrote:
> Protostuff (licensed under Apache 2.0 licence) is based on Google's
> Protocol-Buffers (Protobuf) but has some optimizations and some cool things
> like runtime serialization graph of objects (like Hessian). It also could
> generate schema on runtime so we shouldn't define .proto files although it
> might increase efficiency. It works faster than Hessian and could handle
> Java8 Date and Time types. Here is some benchmarks. Take a look at Full
> Object Graph Serializers section.
> http://hperadin.github.io/jvm-serializers-report/report.html
> https://github.com/eishay/jvm-serializers/wiki

According to those benchmarks there appears to be no performance or size 
penalty to using protostuff over protobuffers. Am I reading that right?

I don't really understand... doesn't the serialiser have to construct a .proto 
definition and then include it in the message? So shouldn't it be 
faster/smaller to predefine these?

If we did, we could create them with velocity in the same way we create Java 
_superclasses today. Fairly trivial I'm guessing.

Ari


-- 
-->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A

Re: Cayenne ROP Improvements

[Aristedes Maniatis]

On 5/05/2016 7:35pm, Savva Kolbachev wrote:
> Also I want to suggest to use Protostuff serialization instead of Hessian
> by default. What do you think about it?

Nicely done Savva. I know that our application will be the first off the rank 
to adopt this code, so once we run it through our testing and QA, I think that 
Cayenne should be switched to protostuff by default.

There are no downsides and plenty of upsides: better performance and fewer 
known bugs.

We'll know once we get the entire stack together inside a real application, but 
I've got high hopes that the combination of the new serialiser and HTTP/2 will 
have great benefits for applications running over high latency links ( > 50ms).

For others thinking about ROP, we've got it working really well once you keep 
in mind a combination of Cayenne pagination (which allows you to efficiently 
send a 100,000 hollow object list over the internet in a second or two) and 
prefetching (which dramatically improves performance by combining many high 
latency 'chatty' connections into a single request-response).

With HTTP/2 we can reduce the latency of establishing SSL connections (and add 
compression to the headers). And with protocol-buffers we'll be able to send 
LocalDate and other JDK8 objects without problem.


Ari


-- 
-->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A