[GitHub] [spark] EnricoMi commented on pull request #38624: [SPARK-40559][PYTHON] Add applyInArrow to groupBy and cogroup
EnricoMi commented on PR #38624: URL: https://github.com/apache/spark/pull/38624#issuecomment-1685936615 > qq, can't we workaround by `df.repartitionByExpression().mapInArrow()` for `groupby` case? That would require the user to implement what `GroupedIterator` in Spark does. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #38624: [SPARK-40559][PYTHON] Add applyInArrow to groupBy and cogroup
EnricoMi commented on PR #38624: URL: https://github.com/apache/spark/pull/38624#issuecomment-1685933606 > I get that `cogroup` might not be possible tho. But we can just convert pandas back to arrow batches easily. Is this really required for some scenario? IIRC this is only useful for addressing nested types. Going through Pandas requires users to install Pandas though they are using a different Arrow-based dataset API. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #38624: [SPARK-40559][PYTHON] Add applyInArrow to groupBy and cogroup
EnricoMi commented on PR #38624: URL: https://github.com/apache/spark/pull/38624#issuecomment-1640086388 @Kimahriman you mentioned this would fix a 2GB memory limit? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #38624: [SPARK-40559][PYTHON] Add applyInArrow to groupBy and cogroup
EnricoMi commented on PR #38624: URL: https://github.com/apache/spark/pull/38624#issuecomment-1640085866 @xinrong-meng @HyukjinKwon rebased with master and conflicts resolved -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #38624: [SPARK-40559][PYTHON] Add applyInArrow to groupBy and cogroup
EnricoMi commented on PR #38624: URL: https://github.com/apache/spark/pull/38624#issuecomment-1477426095 CC @xinrong-meng -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] EnricoMi commented on pull request #38624: [SPARK-40559][PYTHON] Add applyInArrow to groupBy and cogroup
EnricoMi commented on PR #38624: URL: https://github.com/apache/spark/pull/38624#issuecomment-1335369584 @HyukjinKwon what do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
