Hi Ismaël, What I’ve currently been working on locally is almost 100% based on that Java implementation. I suppose I need to create Jira issue and make the contribution.
On 2022/02/15 09:19:33 Ismaël Mejía wrote: > Oh, forgot to add also the link to the tests that cover most of those > unexpected cases: > [2] > https://github.com/apache/beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTrackerTest.java > > > On Tue, Feb 15, 2022 at 10:17 AM Ismaël Mejía > <[email protected]<mailto:[email protected]>> wrote: > > > Great idea, please take a look at the Java ByteKeyRestrictionTracker > > implementation for consistency [1] > > I remember we had to deal with lots of corner cases so probably worth a > > look. > > > > [1] > > https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/ByteKeyRangeTracker.java > > > > > > On Mon, Feb 14, 2022 at 6:39 PM Robert Bradshaw > > <[email protected]<mailto:[email protected]>> > > wrote: > > > >> +1 to being forward looking and making restriction trackers. > >> Hopefully the restriction tracker and existing range tracker could share > >> 90% of their code. > >> > >> On Mon, Feb 14, 2022 at 9:36 AM Sami Niemi > >> <[email protected]<mailto:[email protected]>> wrote: > >> > >>> Hello Robert, > >>> > >>> > >>> > >>> Beam has documented only OffsetRangeTracker [1] for new SDF API. Since > >>> Beam is moving away from Source API, I thought it would be nice to develop > >>> IO connectors by using new SDFs. For this I need to create restriction > >>> tracker that follows new SDF API. > >>> > >>> > >>> > >>> So I propose adding ByteKeyRange as new restriction class and > >>> ByteKeyRestrictionTracker as new restriction tracker class. In my > >>> implementation I’ve also used ByteKey class which are given to > >>> restriction. > >>> > >>> > >>> > >>> 1. > >>> > >>> https://github.com/apache/beam/blob/7eb7fd017a43353204eb8037603409dda7e0414a/sdks/python/apache_beam/io/restriction_trackers.py#L76 > >>> > >>> > >>> > >>> On 2022/02/11 18:27:23 Robert Bradshaw wrote: > >>> > >>> > Hi Sam! Glad to hear you're willing to contribute. > >>> > >>> > > >>> > >>> > Though the name is a bit different, I'm wondering if this is already > >>> > >>> > present as LexicographicKeyRangeTracker. > >>> > >>> > > >>> https://github.com/apache/beam/blob/release-2.35.0/sdks/python/apache_beam/io/range_trackers.py#L349 > >>> > >>> > > >>> > >>> > On Fri, Feb 11, 2022 at 9:54 AM Ahmet Altay > >>> > <[email protected]<mailto:[email protected]>> wrote: > >>> > >>> > > > >>> > >>> > > Hi Sami. Thank you for your interest. > >>> > >>> > > > >>> > >>> > > Adding people who might be able to comment: @Chamikara Jayalath > >>> @Lukasz Cwik > >>> > >>> > > > >>> > >>> > > On Thu, Feb 10, 2022 at 8:38 AM Sami Niemi > >>> > > <[email protected]<mailto:[email protected]>> wrote: > >>> > >>> > >> > >>> > >>> > >> Hello, > >>> > >>> > >> > >>> > >>> > >> > >>> > >>> > >> > >>> > >>> > >> I noticed that Python SDK only has implementation for > >>> OffsetRangeTracker and OffsetRange while Java also has ByteKeyRange and > >>> -Tracker. > >>> > >>> > >> > >>> > >>> > >> > >>> > >>> > >> > >>> > >>> > >> I have currently created simple implementations of following Python > >>> classes: > >>> > >>> > >> > >>> > >>> > >> ByteKey > >>> > >>> > >> ByteKeyRange > >>> > >>> > >> ByteKeyRestrictionTracker > >>> > >>> > >> > >>> > >>> > >> > >>> > >>> > >> > >>> > >>> > >> I would like to make contribution and make these available in > >>> Python SDK in addition to OffsetRange and -Tracker. I would like to hear > >>> any thoughts about this and should I make a contribution. > >>> > >>> > >> > >>> > >>> > >> > >>> > >>> > >> > >>> > >>> > >> Thank you, > >>> > >>> > >> > >>> > >>> > >> Sami Niemi > >>> > >>> > > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> *SAMI NIEMI* > >>> Data Engineer > >>> +358 50 412 2115 <+358%2050%204122115> > >>> [email protected]<mailto:[email protected]> > >>> > >>> > >>> > >>> *SOLITA* > >>> Eteläesplanadi 8 > >>> 00130 Helsinki > >>> solita.fi <https://www.solita.fi><https://www.solita.fi%3e> > >>> > >>> > >>> > >> >
