IIRC the Initial Restrictions method gives you an element and you return the restrictions relative to that element.
It's entirely appropriate to stat files or query databases in order to determine the initial restrictions and partitions of the data. On Tue, May 18, 2021, 3:21 PM Miguel Anzo Palomo <miguel.a...@wizeline.com> wrote: > Hi, I’m looking at how to implement a reader as a SplittableDoFn and I'm > having some problems with the initial restriction, specifically, how do you > set the initial restriction if you don’t know the size of the data? > The DoFn that I'm working on takes a PCollection of Spanner *ReadOperations > *and splits the read operation query into a list of *Partitions* to query > against the database. > I’m currently setting the *InitialRestriction* to an OffsetRange(0L, > Long.MAX_VALUE); which is currently giving me this error in unit tests Last > attempted offset was 0 in range [0, 9223372036854775807), claiming work in > [1, 9223372036854775807) was not attempted. and it makes sense I think, > because I am setting up the range to max long value. > So, if I don't know how many partitions are going to be created until it's > being processed, how can I set the initial restriction or what initial > restriction do I need to set? > > -- > > Miguel Angel Anzo Palomo | WIZELINE > > Software Engineer > > miguel.a...@wizeline.com > > Remote Office > > > > > > > > > *This email and its contents (including any attachments) are being sent > toyou on the condition of confidentiality and may be protected by > legalprivilege. Access to this email by anyone other than the intended > recipientis unauthorized. If you are not the intended recipient, please > immediatelynotify the sender by replying to this message and delete the > materialimmediately from your system. Any further use, dissemination, > distributionor reproduction of this email is strictly prohibited. Further, > norepresentation is made with respect to any content contained in this > email.*
