In this case it's fine to discuss whether this would fit in Spark DataFrames' high level direction before putting it in JIRA. Otherwise we might end up creating a lot of tickets just for querying whether something might be a good idea.
About this specific feature -- I'm not sure what it means in general given we don't have axis in Spark DataFrames. But I think it'd probably be good to be able to shift a column by one so we can support the end time / begin time case, although it'd require two passes over the data. On Wed, Apr 29, 2015 at 1:08 PM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > I can't comment on the direction of the DataFrame API (that's more for > Reynold or Michael I guess), but I just wanted to point out that the JIRA > would be the recommended way to create a central place for discussing a > feature add like that. > > Nick > > On Wed, Apr 29, 2015 at 3:43 PM Olivier Girardot < > o.girar...@lateral-thoughts.com> wrote: > > > Hi Nicholas, > > yes I've already checked, and I've just created the > > https://issues.apache.org/jira/browse/SPARK-7247 > > I'm not even sure why this would be a good feature to add except the fact > > that some of the data scientists I'm working with are using it, and it > > would be therefore useful for me to translate Pandas code to Spark... > > > > Isn't the goal of Spark Dataframe to allow all the features of Pandas/R > > Dataframe using Spark ? > > > > Regards, > > > > Olivier. > > > > Le mer. 29 avr. 2015 à 21:09, Nicholas Chammas < > nicholas.cham...@gmail.com> > > a écrit : > > > >> You can check JIRA for any existing plans. If there isn't any, then feel > >> free to create a JIRA and make the case there for why this would be a > good > >> feature to add. > >> > >> Nick > >> > >> On Wed, Apr 29, 2015 at 7:30 AM Olivier Girardot < > >> o.girar...@lateral-thoughts.com> wrote: > >> > >>> Hi, > >>> Is there any plan to add the "shift" method from Pandas to Spark > >>> Dataframe, > >>> not that I think it's an easy task... > >>> > >>> c.f. > >>> > >>> > http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.shift.html > >>> > >>> Regards, > >>> > >>> Olivier. > >>> > >> >
