Created  a PR: https://github.com/apache/spark/pull/39902 
<https://github.com/apache/spark/pull/39902>


> On 24 Jan 2023, at 15:04, Santosh Pingale <santosh.ping...@adyen.com> wrote:
> 
> Hey all
> 
> I have an interesting problem in hand. We have cases where we want to pass 
> multiple(20 to 30) data frames to cogroup.applyInPandas function.
> 
> RDD currently supports cogroup with upto 4 dataframes (ZippedPartitionsRDD4)  
> where as cogroup with pandas can handle only 2 dataframes (with 
> ZippedPartitionsRDD2). In our use case, we do not have much control over how 
> many data frames we may need in the cogroup.applyInPandas function.
> 
> To achieve this, we can:
> (a) Implement ZippedPartitionsRDD5, 
> ZippedPartitionsRDD..ZippedPartitionsRDD30..ZippedPartitionsRDD50 with 
> respective iterators, serializers and so on. This ensures we keep type safety 
> intact but a lot more boilerplate code has to be written to achieve this.
> (b) Do not use cogroup.applyInPandas, rather use RDD.keyBy.cogroup and then 
> getItem in a nested fashion. Then convert data to pandas df in the python 
> function. This looks like a good workaround but mistakes are very easy to 
> happen. We also don't look at typesafety here from user's point of view.
> (c) Implement ZippedPartitionsRDDN and NaryLike with childrenNodes type set 
> to Seq[T] which allows for arbitrary number of children to be set. Here we 
> have very little boilerplate but we sacrifice type safety.
> (d) ... some new suggestions... ?
> 
> I have done preliminary work on option (c). It works like a charm but before 
> I proceed, is my concern about sacrificed type safety overblown, and do we 
> have an approach (d)?
> (a) is something that is too much of an investment for it to be useful. (b) 
> is okay enough workaround, but it is not very efficient.
> 

Attachment: signature.asc
Description: Message signed with OpenPGP

Reply via email to