1 last thing... If you run the PeerCacheFunctionExecutionResultStreamingIntegrationTests test class, you will see the effects of Function Stream Result processing in you IDE (or on the command-line).
Enjoy! -j On Thu, Nov 30, 2017 at 12:31 PM, John Blum <[email protected]> wrote: > Hi Amit, all- > > 2 things... > > > 1. Have a look at... DATAGEODE-36 - "*Add Async, Reactive and Streaming > capabilities to the Function Annotation support.*" [1] > > I already started experimenting with Streaming Function results here [2]. > Essentially the only thing preventing Apache Geode from Streaming results > now is the default o.a.g.cache.execute.ResultCollector implementation > (i.e. o.a.g.internal.cache.execute.DefaultResultCollector [3]) provided > by Apache Geode OOTB. > > > 2. *Dan is correct* about the batching functionality provided in SDG! > > You can see how it work by looking at the o.s.d.g.function. > BatchingResultSender class [4], which is used by the o.s.d.g.function. > PojoFunctionWrapper class [5] (which is how the entire Function > "implementation" annotated POJO methods work, BTW! Well, that class and > some *Spring (Data Geode)* infrastructure to enable it, ;-). > > > Hope this helps! > > -John > > [1] https://jira.spring.io/browse/DATAGEODE-36 > [2] https://github.com/jxblum/spring-gemfire-tests/blob/ > master/src/test/java/org/spring/data/gemfire/cache/ > PeerCacheFunctionExecutionResultStreamingIntegrationTests.java > [3] https://github.com/apache/geode/blob/rel/v1.3.0/geode- > core/src/main/java/org/apache/geode/internal/cache/execute/ > DefaultResultCollector.java > [4] https://github.com/spring-projects/spring-data-gemfire/ > blob/2.0.2.RELEASE/src/main/java/org/springframework/data/ > gemfire/function/BatchingResultSender.java > [5] https://github.com/spring-projects/spring-data-gemfire/ > blob/2.0.2.RELEASE/src/main/java/org/springframework/data/ > gemfire/function/PojoFunctionWrapper.java#L143-L148 > > > On Thu, Nov 30, 2017 at 7:59 AM, Anthony Baker <[email protected]> wrote: > >> FWIW, I really like this suggestion. There is some interaction with the >> `isHA()` behavior but I would love to have a reactive Function that returns >> a mono / flux [1]. >> >> Anthony >> >> [1] https://spring.io/blog/2016/04/19/understanding-reactive-types >> >> >> On Nov 28, 2017, at 10:23 PM, Amit Pandey <[email protected]> >> wrote: >> >> Hey Guys, >> >> I filed this JIRA ticket :- https://issues.apache.org/j >> ira/browse/GEODE-4027 >> >> Regards >> >> On Wed, Nov 29, 2017 at 2:01 AM, Amit Pandey <[email protected]> >> wrote: >> >>> Sure. >>> >>> On Wed, Nov 29, 2017 at 1:43 AM, Swapnil Bawaskar <[email protected]> >>> wrote: >>> >>>> Amit, >>>> Can you please file a JIRA? >>>> >>>> Thanks! >>>> >>>> On Tue, Nov 28, 2017 at 12:02 PM Amit Pandey <[email protected]> >>>> wrote: >>>> >>>>> Hey Udo, >>>>> >>>>> if I was confident I wouldn't have griped about the non availability. >>>>> I think though that it will be a good addition. Because for large data >>>>> collections it's not enough to batch it from the server side. >>>>> >>>>> Without a streaming result collector its futile >>>>> >>>>> On Wed, Nov 29, 2017 at 1:05 AM, Udo Kohlmeyer <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi there Amit. >>>>>> >>>>>> Geode does also work for PRs.... We actually look forward to the >>>>>> community contributing to the product... :) >>>>>> >>>>>> But on a more serious note... a streaming results collector does >>>>>> require more thought and potentially significant (positive) changes to >>>>>> the >>>>>> system behavior. >>>>>> >>>>>> --Udo >>>>>> >>>>>> On Tue, Nov 28, 2017 at 11:21 AM, Amit Pandey < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Thanks Dan. >>>>>>> >>>>>>> A Streaming Result Collector in Geode is needed now . This will be >>>>>>> great addition. I know we can make it, but its pretty error prone to do >>>>>>> it >>>>>>> >>>>>>> On Wed, Nov 29, 2017 at 12:05 AM, Dan Smith <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Amit, >>>>>>>> >>>>>>>> Looking at the code, it looks like if your annotated method returns >>>>>>>> an array, or an Iteratable (like a List), and you have batchSize set, >>>>>>>> then >>>>>>>> spring will send your data in batches back to the caller of the >>>>>>>> function. >>>>>>>> Underneath the covers is it using geode's ResultSender and passing the >>>>>>>> batches to the ResultSender sendResult method. >>>>>>>> >>>>>>>> So it looks like this would reduce the amount of data that gets >>>>>>>> sent to the client in a single message. >>>>>>>> >>>>>>>> -Dan >>>>>>>> >>>>>>>> On Sun, Nov 26, 2017 at 9:11 PM, Amit Pandey < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi John and everyone, >>>>>>>>> >>>>>>>>> How does Spring Data Geode Function annotation batch size work? >>>>>>>>> >>>>>>>>> So if we set it up like @GemfireFunction(id="lotMapService",HA=true, >>>>>>>>> batchSize =1000) >>>>>>>>> >>>>>>>>> >>>>>>>>> what are the implications ? What happens if there is a million >>>>>>>>> data to be streamed on the server as well as on the client side. >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Kindest Regards >>>>>> ----------------------------- >>>>>> *Udo Kohlmeyer* | *Pivotal* >>>>>> [email protected] >>>>>> <http://www.gopivotal.com/> >>>>>> www.pivotal.io >>>>>> >>>>> >>>>> >>> >> >> > > > -- > -John > john.blum10101 (skype) > -- -John john.blum10101 (skype)
