I'd push towards waiting until SDF is working end to end to begin converting things. Unless it's something like Text.ReadAll batch API that gets benefits without a SDF implementation. I don't have a lot of context on what file APIs python already supports.
On Mon, Apr 8, 2019 at 10:06 AM Pablo Estrada <[email protected]> wrote: > Currently, Pasan is working on a design for adding a couple > implementations to the Filesystem interface in Python, and it's not > necessary to consider SDF here. IMHO. > > On the other hand, Python's fileio[1] could probably use SDF-based > improvements to split when many files are being matched. > Best > -P. > > On Mon, Apr 8, 2019 at 10:00 AM Alex Amato <[email protected]> wrote: > >> +Lukasz Cwik <[email protected]>, +Boyuan Zhang <[email protected]>, +Lara >> Schmidt <[email protected]> >> >> Should splittable DoFn be considered in this design? In order to split >> and scale the source step properly? >> >> On Mon, Apr 8, 2019 at 9:11 AM Ahmet Altay <[email protected]> wrote: >> >>> +dev <[email protected]> +Pablo Estrada <[email protected]> +Chamikara >>> Jayalath <[email protected]> +Udi Meiri <[email protected]> >>> >>> Thank you Pasan. I quickly looked at the proposal and it looks good. >>> Added a few folks who could offer additional feedback. >>> >>> On Mon, Apr 8, 2019 at 12:13 AM Pasan Kamburugamuwa < >>> [email protected]> wrote: >>> >>>> Hi, >>>> >>>> I have updated the project proposal according to the given feedback. So >>>> can you guys check my proposal again and give me your feedback about >>>> corrections I have done. >>>> >>>> Here is the link to the updated project proposal >>>> >>>> https://docs.google.com/document/d/1i_PoIrbmhNgwKCS1TYWC28A9RsyZQFsQCJic3aCXO-8/edit?usp=sharing >>>> >>>> Thank you >>>> Pasan Kamburugamuwa >>>> >>>
