Pandas UDF cogroup.applyInPandas with multiple dataframes

2023-01-24 Thread Santosh Pingale
Hey all I have an interesting problem in hand. We have cases where we want to pass multiple(20 to 30) data frames to cogroup.applyInPandas function. RDD currently supports cogroup with upto 4 dataframes (ZippedPartitionsRDD4) where as cogroup with pandas can handle only 2 dataframes (with

Re: Time for Spark 3.4.0 release?

2023-01-24 Thread Xinrong Meng
Hi All, Apache Spark 3.4 is cut as https://github.com/apache/spark/tree/branch-3.4. Thanks, Xinrong Meng On Wed, Jan 18, 2023 at 3:45 PM Hyukjin Kwon wrote: > Yeah, these more look like something we should discuss around RC timing. > See "Spark 3.4 release window" in >

Re: Time for Spark 3.4.0 release?

2023-01-24 Thread Hyukjin Kwon
Thanks Xinrong. On Wed, 25 Jan 2023 at 12:01, Xinrong Meng wrote: > Hi All, > > Apache Spark 3.4 is cut as https://github.com/apache/spark/tree/branch-3.4 > . > > Thanks, > > Xinrong Meng > > On Wed, Jan 18, 2023 at 3:45 PM Hyukjin Kwon wrote: > >> Yeah, these more look like something we

Re: Time for Spark 3.4.0 release?

2023-01-24 Thread Dongjoon Hyun
Great! Thank you so much, Xinrong! Dongjoon On Tue, Jan 24, 2023 at 7:17 PM Hyukjin Kwon wrote: > Thanks Xinrong. > > On Wed, 25 Jan 2023 at 12:01, Xinrong Meng > wrote: > >> Hi All, >> >> Apache Spark 3.4 is cut as >> https://github.com/apache/spark/tree/branch-3.4. >> >> Thanks, >> >>

Re: Time for Spark 3.4.0 release?

2023-01-24 Thread yangjie01
Thanks Xinrong, 发件人: Dongjoon Hyun 日期: 2023年1月25日 星期三 15:49 收件人: Hyukjin Kwon 抄送: Xinrong Meng , "dev@spark.apache.org" 主题: Re: Time for Spark 3.4.0 release? Great! Thank you so much, Xinrong! Dongjoon On Tue, Jan 24, 2023 at 7:17 PM Hyukjin Kwon mailto:gurwls...@gmail.com>> wrote: Thanks