Currently, Pasan is working on a design for adding a couple implementations to the Filesystem interface in Python, and it's not necessary to consider SDF here. IMHO.
On the other hand, Python's fileio[1] could probably use SDF-based improvements to split when many files are being matched. Best -P. On Mon, Apr 8, 2019 at 10:00 AM Alex Amato <[email protected]> wrote: > +Lukasz Cwik <[email protected]>, +Boyuan Zhang <[email protected]>, +Lara > Schmidt <[email protected]> > > Should splittable DoFn be considered in this design? In order to split and > scale the source step properly? > > On Mon, Apr 8, 2019 at 9:11 AM Ahmet Altay <[email protected]> wrote: > >> +dev <[email protected]> +Pablo Estrada <[email protected]> +Chamikara >> Jayalath <[email protected]> +Udi Meiri <[email protected]> >> >> Thank you Pasan. I quickly looked at the proposal and it looks good. >> Added a few folks who could offer additional feedback. >> >> On Mon, Apr 8, 2019 at 12:13 AM Pasan Kamburugamuwa < >> [email protected]> wrote: >> >>> Hi, >>> >>> I have updated the project proposal according to the given feedback. So >>> can you guys check my proposal again and give me your feedback about >>> corrections I have done. >>> >>> Here is the link to the updated project proposal >>> >>> https://docs.google.com/document/d/1i_PoIrbmhNgwKCS1TYWC28A9RsyZQFsQCJic3aCXO-8/edit?usp=sharing >>> >>> Thank you >>> Pasan Kamburugamuwa >>> >>
