Hi Pablo, Thanks for that example, it would be great to be able to use fileio.WriteToFiles transform to write files with filenames that are based on their PaneInfo.
Thanks @Charles Chen, for adding the remaining work on the issue - the emission of PaneInfo in the Python implementation of GBK (either in the Python FnAPIRunner or the old Python DirectRunner / triggers.py). I will certainly make sure that [BEAM-3759] is completed after my GSoC project is implemented. It's a good opportunity to get into the runner code. Regards - TT On Fri, May 31, 2019 at 2:35 AM Pablo Estrada <[email protected]> wrote: > Hi Tanay, > thanks for bringing this to the mailing list. I believe this is certainly > useful, and necessary. As an example, the fileio.WriteToFiles transform > does not work well without PaneInfo data (since we can't know how many > firings there are for each window, and we can't give names to files based > on this). > > Best > -P. > > On Thu, May 30, 2019 at 1:00 PM Tanay Tummalapalli <[email protected]> > wrote: > >> Hi everyone, >> >> The PR linked in [BEAM-3759] - "Add support for PaneInfo descriptor in >> Python SDK"[1] was merged, but, the issue is still open. >> There might be some work left on this for full support for PaneInfo. Eg: >> Although the PaneInfo class exists, it is not accessible in a DoFn via a >> kwarg(PaneInfoParam) like TimestampParam or WindowParam. >> >> Please let me know the remaining work to be done on this issue as this >> may be needed in the near future. >> >> Regards >> Tanay Tummalapalli >> >> [1] https://issues.apache.org/jira/browse/BEAM-3759 >> >
