I actually ran into the same issue, and would love some guidance! I had a list of avro files within folders in GCS, each folder representing a single day, and I needed to de-dupe events per day (by a key). I didn't want a GroupByKey to hold billions of events when it didn't matter, so I added a timestamp to each folder, then tried windowing. I thought perhaps the windowed event passed into the ReadAll would mean windows could proceed down the pipeline without having to load the entire batch of files into memory.
I was wrong, and saw the same behavior of Neha. Is there a better way of doing this? I don't see a *technical* reason why the dataflow runner couldn't be implemented to have this behavior, but I could be mistaken. *~Vincent* On Thu, Apr 16, 2020 at 10:13 AM Neha Sharma <[email protected]> wrote: > Hi Luke, > > It is the order the record appears in the source file. > > Basically each record corresponding to a key depends on the previous > occurrence of the same key and hence parallel processing does not seem to > be a good idea. > > Is there a possibility where bounded source + fixed window based on the > timestamp in the record can be used to somehow batch the whole data into > smaller chunks for processing and at the same time can maintain the > ordering provided a sorting based on the timestamp? > > Something like this: > > Read from Bounded Source -> > Fixed window to make smaller batches -> > Sorting based on timestamp -> > Processing > > > Regards, > Neha > > > On Thu, Apr 16, 2020, 6:57 PM Luke Cwik <[email protected]> wrote: > >> What do you mean by in sequential order, order across files, keys, ...? >> Is this an ordering that is based on data such as a timestamp of the >> record or the order in which the records appear in the source files? >> Do you have a lot of keys or very few? >> >> If you want to process all the data across all the files in sequential >> order with no parallelism then Apache Beam may not provide much value since >> its basis is all about parallel data processing. >> >> On Wed, Apr 15, 2020 at 10:30 PM Neha Sharma <[email protected]> >> wrote: >> >>> Hello, >>> >>> I have a use case where I have a bounded source and I am reading Avro >>> files from Google Cloud Storage. I am also using group by transform.The >>> amount of data is huge and I need to process the data in sequential order. >>> >>> But as Bounded source reads everything it seemed to be a good idea fixed >>> window on top of the global window. But it does not seem to be working as >>> expected. >>> >>> Can you please tell me how to handle such scenarios where a bounded >>> source with large dataset can be broken down into smaller chunks for >>> processing, using windows such that the window for a key should always be >>> processed in order. >>> >>> >>> Regards, >>> Neha >>> >>
