Yep! I'm working on tools to generate the API docs here [1]. For the most part we'll just populate these with the pydoc from pandas, but the tooling will give us the ability to include a note about anything that isn't implemented in our API yet, for example operations that are sensitive to order.
[1] https://github.com/apache/beam/pull/14382 On Thu, Apr 8, 2021 at 2:30 PM Pablo Estrada <[email protected]> wrote: > Looking at [1] I see all the methods on the DF but not the pydoc. Will the > Pydoc be included later somehow? > Best > -P. > > [1] > https://theneuralbit.github.io/beam-site/pydoc/inherited-members/apache_beam.dataframe.frames.html > > On Thu, Apr 8, 2021 at 1:05 PM Brian Hulette <[email protected]> wrote: > >> I found a slightly hacky way to enable :inherited-members: just for the >> DataFrame API. I can add the option to the .rst output generated by >> sphinx-apidoc, before we run sphinx-build [1]. >> >> I'm fine just doing that instead of turning it on globally. >> >> [1] >> https://github.com/TheNeuralBit/beam/blob/e26760937f7a34fd72578b65f716098c74e4380b/sdks/python/scripts/generate_pydoc.sh#L86 >> >> On Tue, Apr 6, 2021 at 1:50 PM Brian Hulette <[email protected]> wrote: >> >>> Sure, I can try cutting out PTransform. >>> >>> We could also look into reducing noise by: >>> - removing undoc-members from the config [1] (this would make it so only >>> objects with a docstring are added to the generated docs) >>> - adding :meta private:` to docstrings for objects we don't want >>> publicly visible >>> >>> [1] >>> https://github.com/apache/beam/blob/243128a8fc52798e1b58b0cf1a271d95ee7aa241/sdks/python/scripts/generate_pydoc.sh#L48 >>> >>> On Tue, Apr 6, 2021 at 1:17 PM Robert Bradshaw <[email protected]> >>> wrote: >>> >>>> Way too many things are inherited from PTransform, can we at least cut >>>> that out? >>>> >>>> On Tue, Apr 6, 2021 at 1:09 PM Brian Hulette <[email protected]> >>>> wrote: >>>> >>>>> Just wanted to bump this - does anyone have concerns with the way the >>>>> API docs look when inherited members are included? >>>>> >>>>> On Wed, Mar 31, 2021 at 5:23 PM Brian Hulette <[email protected]> >>>>> wrote: >>>>> >>>>>> I staged my current working copy built from head here [1], see >>>>>> CombinePerKey here [2]. Note it also has a few other changes, most >>>>>> notably >>>>>> I excluded several internal-only modules that are currently in our API >>>>>> docs >>>>>> (I will PR this soon regardless). >>>>>> >>>>>> > are these inherited members grouped in such a way that it makes it >>>>>> easy to ignore them once they get to "low" in the stack? >>>>>> There doesn't seem to be any grouping, but it does look like >>>>>> inherited members are added at the end. >>>>>> >>>>>> > If it can't be per-module, is there a "nice" set of ancestors to >>>>>> avoid (as it seems this option takes such an argument). >>>>>> Ah good point, I missed this. I suppose we could avoid basic >>>>>> constructs like PTransform, DoFn, etc. I'm not sure how realistic that is >>>>>> though. It would be nice if this argument worked the other way >>>>>> >>>>>> [1] https://theneuralbit.github.io/beam-site/pydoc/inherited-members >>>>>> [2] >>>>>> https://theneuralbit.github.io/beam-site/pydoc/inherited-members/apache_beam.transforms.core.html#apache_beam.transforms.core.CombinePerKey >>>>>> >>>>>> On Wed, Mar 31, 2021 at 4:45 PM Robert Bradshaw <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> +1 to an example. In particular, are these inherited members grouped >>>>>>> in such a way that it makes it easy to ignore them once they get to >>>>>>> "low" >>>>>>> in the stack? If it can't be per-module, is there a "nice" set of >>>>>>> ancestors >>>>>>> to avoid (as it seems this option takes such an argument). >>>>>>> >>>>>>> On Wed, Mar 31, 2021 at 4:23 PM Pablo Estrada <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Do you have an example of what it would look like when released? >>>>>>>> >>>>>>>> On Wed, Mar 31, 2021 at 4:16 PM Brian Hulette <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I'm working on generating useful API docs for the DataFrame API >>>>>>>>> (BEAM-12074). In doing so, one thing I've found would be very helpful >>>>>>>>> is if >>>>>>>>> we could include docstrings for inherited members in the API docs. >>>>>>>>> That way >>>>>>>>> docstrings for operations defined in DeferredDataFrameOrSeries [1], >>>>>>>>> will be >>>>>>>>> propagated to DeferredDataFrame [2] and DeferredSeries, and the >>>>>>>>> former can >>>>>>>>> be hidden entirely. This would be more consistent with the pandas >>>>>>>>> documentation [3]. >>>>>>>>> >>>>>>>>> It looks like we can do this by specifying :inherited-members: >>>>>>>>> [4], but this will apply to _all_ of our API docs, there doesn't seem >>>>>>>>> to be >>>>>>>>> a way to restrict it to a particular module. This seems generally >>>>>>>>> useful to >>>>>>>>> me, but it would be a significant change, so I wanted to see if there >>>>>>>>> are >>>>>>>>> any objections from dev@ before doing this. >>>>>>>>> >>>>>>>>> An example of the kind of change this would produce: any >>>>>>>>> PTransform sub-classes, e.g. CombinePerKey [5], would now include >>>>>>>>> docstrings for every PTransform member, e.g. with_input_types [6], and >>>>>>>>> display_data [7]. >>>>>>>>> >>>>>>>>> Would there be any objections to that? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Brian >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> https://beam.apache.org/releases/pydoc/2.27.0/apache_beam.dataframe.frames.html#apache_beam.dataframe.frames.DeferredDataFrameOrSeries >>>>>>>>> [2] >>>>>>>>> https://beam.apache.org/releases/pydoc/2.27.0/apache_beam.dataframe.frames.html#apache_beam.dataframe.frames.DeferredDataFrame >>>>>>>>> [3] https://pandas.pydata.org/docs/reference/frame.html >>>>>>>>> [4] >>>>>>>>> https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html >>>>>>>>> [5] >>>>>>>>> https://beam.apache.org/releases/pydoc/2.27.0/apache_beam.transforms.core.html?highlight=combineperkey#apache_beam.transforms.core.CombinePerKey >>>>>>>>> [6] >>>>>>>>> https://beam.apache.org/releases/pydoc/2.27.0/apache_beam.transforms.ptransform.html#apache_beam.transforms.ptransform.PTransform.with_input_types >>>>>>>>> [7] >>>>>>>>> https://beam.apache.org/releases/pydoc/2.27.0/apache_beam.transforms.display.html#apache_beam.transforms.display.HasDisplayData.display_data >>>>>>>>> >>>>>>>>
