Hey Benjamin, That sounds really awesome. Thank you.
Sorry if this was already a well known thing as I am fairly new to the Arrow ecosystem. Is there a way to track a roadmap for Arrow 4 and be involved in that? Is there anywhere I can read more just general information on that? -Michael From: Benjamin Kietzman <bengil...@gmail.com> Date: Friday, March 19, 2021 at 9:14 AM To: dev <dev@arrow.apache.org> Subject: Re: [DISCUSS] How to encode table_pivot information state in Arrow Hi Michael, We are targeting grouped aggregation for 4.0 as part of a general query engine buildout. We also intend to bring DataFrame functionality into core Arrow (which would probably include an analog of pandas' pivot_table), but the query engine work is a prerequisite. Ben Kietzman On Fri, Mar 19, 2021, 08:19 Michael Lavina <michael.lav...@factset.com> wrote: > Hey Team, > > Sorry if this is answered already somewhere I tried searching emails and > issues but couldn’t find anything. I am wondering if there is a standard > way to encode row or column pivots in Arrow? > > I know Pandas does it already some way > https://urldefense.com/v3/__https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.pivot_table.html__;!!PBKjc0U4!aRYySdE5nJFh6JBpP7YXqwFlAXpHj81USUsUKdIyHn_ryLYJxyKobsgdrfhI8e_Ejvqp$<https://urldefense.com/v3/__https:/pandas.pydata.org/pandas-docs/stable/reference/api/pandas.pivot_table.html__;!!PBKjc0U4!aRYySdE5nJFh6JBpP7YXqwFlAXpHj81USUsUKdIyHn_ryLYJxyKobsgdrfhI8e_Ejvqp$> > and there are libraries using Arrow like Perspective that may have their > internal solution for representation of pivots > https://urldefense.com/v3/__https://perspective.finos.org/docs/md/view.html*row-pivots__;Iw!!PBKjc0U4!aRYySdE5nJFh6JBpP7YXqwFlAXpHj81USUsUKdIyHn_ryLYJxyKobsgdrfhI8cXbLZNA$<https://urldefense.com/v3/__https:/perspective.finos.org/docs/md/view.html*row-pivots__;Iw!!PBKjc0U4!aRYySdE5nJFh6JBpP7YXqwFlAXpHj81USUsUKdIyHn_ryLYJxyKobsgdrfhI8cXbLZNA$> > > I am wondering if there is already a discussion or already a best practice > or standard for encoding this information. Or alternatively is this not > something that should be at all associated with Arrow. > > -Michael > > P.S. If anyone on the Perspective team or anyone who might know is on this > thread I would be interested in understanding more how Perspective, > specifically, encodes pivot information in Arrow. > >
