Amit, Attached is a function and result collector that sends batches of values from each server to the client using sendResults / endResults.
The GetValueBatchesFunction iterates the local primary entries, creates a ValueBatch for each 100 values and sends them to the caller. The internal API used in this function prevents deserialization of the values on the server. The client ValueBatchResultCollector just logs the ValueBatches, but it could be changed to do anything with each batch. It doesn't save the batches; it just processes them. The ValueBatch is just a helper to contain the values. It dumps output like: Function Execution Thread-3: Processing batch 1 member boglesbymac-2(4131)<v37>:46709 containing 100 values (total=100) Function Execution Thread-1: Processing batch 2 member boglesbymac-2(4138)<v38>:63789 containing 100 values (total=200) Function Execution Thread-2: Processing batch 3 member boglesbymac-2(4145)<v39>:3184 containing 100 values (total=300) Function Execution Thread-1: Processing batch 4 member boglesbymac-2(4138)<v38>:63789 containing 100 values (total=400) Function Execution Thread-2: Processing batch 5 member boglesbymac-2(4145)<v39>:3184 containing 100 values (total=500) Function Execution Thread-3: Processing batch 6 member boglesbymac-2(4131)<v37>:46709 containing 100 values (total=600) Function Execution Thread-1: Processing batch 7 member boglesbymac-2(4138)<v38>:63789 containing 100 values (total=700) Function Execution Thread-2: Processing batch 8 member boglesbymac-2(4145)<v39>:3184 containing 100 values (total=800) Function Execution Thread-3: Processing batch 9 member boglesbymac-2(4131)<v37>:46709 containing 100 values (total=900) Function Execution Thread-1: Processing batch 10 member boglesbymac-2(4138)<v38>:63789 containing 40 values (total=940) Function Execution Thread-2: Processing batch 11 member boglesbymac-2(4145)<v39>:3184 containing 35 values (total=975) Function Execution Thread-3: Processing batch 12 member boglesbymac-2(4131)<v37>:46709 containing 25 values (total=1000) Thanks, Barry Oglesby On Mon, Aug 14, 2017 at 9:14 AM, Amit Pandey <[email protected]> wrote: > Hey Udo, > > I can do that. However I was using @GemfireFunction and didn't understand > how I can do it. Anyways I understand how to do it from the server. > Is there any example code for the Streaming result collector? > > John, > > Does Spring Data Geode have any helpers for this ? > > Regards > > On Mon, Aug 14, 2017 at 9:40 PM, Udo Kohlmeyer <[email protected]> wrote: > >> Hi there Amit. >> >> Have you looked at the ResultSender.sendResult() method on the function? >> You can use sendResult() as often as you like to send chunks of 1000 >> results. You just have to ensure that you "close" the resultSender by >> calling sendLast(). >> >> As for the streaming result collector... Geode does not have a streaming >> interface, but you can implement a custom result collector. In this custom >> result collector you can embed your processing of chunks in the >> "addResult". This way you can process data as soon as the collector >> receives it. >> >> The one caveat here is that you have to deal with failure and possible >> duplicates when the function is marked as HA and it might retry/restart >> upon detection of failure. >> >> --Udo >> >> On 8/14/17 00:14, Amit Pandey wrote: >> >> Also in Spring Data Geode is it possible to send data as soon as I have a >> chunk of say 1000/ I know I can specify batch size but I don't see how I >> can do it like streaming >> >> On Sun, Aug 13, 2017 at 3:08 PM, Amit Pandey <[email protected]> >> wrote: >> >>> Hi All, >>> >>> I have a function which can potentially return a very large data sets. >>> >>> I want to stream data via the functions. Now the default result >>> collector of Geode collects all the data in one large chunk, This might >>> result in very slow operation times. How can I use a streaming result >>> collector? Is there any example of it given? >>> >>> I am using spring-data-geode so if there is something available there >>> that will be great too. >>> >>> Regards >>> >> >> >> >
GetValueBatchesFunction.java
Description: Binary data
ValueBatch.java
Description: Binary data
ValueBatchResultCollector.java
Description: Binary data
