[GitHub] [spark] BryanCutler commented on issue #24926: [SPARK-28128][PYTHON][SQL] Pandas Grouped UDFs skip empty partitions

BryanCutler commented on issue #24926: [SPARK-28128][PYTHON][SQL] Pandas Grouped UDFs skip empty partitions URL: https://github.com/apache/spark/pull/24926#issuecomment-505099961 Thanks @dongjoon-hyun and @HyukjinKwon ! This

[GitHub] [spark] BryanCutler commented on issue #24926: [SPARK-28128][PYTHON][SQL] Pandas Grouped UDFs skip empty partitions

BryanCutler commented on issue #24926: [SPARK-28128][PYTHON][SQL] Pandas Grouped UDFs skip empty partitions URL: https://github.com/apache/spark/pull/24926#issuecomment-504546730 Looks like this cut almost 30s off of test time lol, happy friday @shaneknapp !

[GitHub] [spark] BryanCutler commented on issue #24926: [SPARK-28128][PYTHON][SQL] Pandas Grouped UDFs skip empty partitions

BryanCutler commented on issue #24926: [SPARK-28128][PYTHON][SQL] Pandas Grouped UDFs skip empty partitions URL: https://github.com/apache/spark/pull/24926#issuecomment-504490665 >do we have a test that process empty partitions? We have one for pandas scalar udfs, but not

[GitHub] [spark] BryanCutler commented on issue #24926: [SPARK-28128][PYTHON][SQL] Pandas Grouped UDFs skip empty partitions

BryanCutler commented on issue #24926: [SPARK-28128][PYTHON][SQL] Pandas Grouped UDFs skip empty partitions URL: https://github.com/apache/spark/pull/24926#issuecomment-504256446 cc @HyukjinKwon @icexelloss This is an

[GitHub] [spark] BryanCutler commented on issue #24926: [SPARK-28128][PYTHON][SQL] Pandas Grouped UDFs skip empty partitions

BryanCutler commented on issue #24926: [SPARK-28128][PYTHON][SQL] Pandas Grouped UDFs skip empty partitions URL: https://github.com/apache/spark/pull/24926#issuecomment-504256239 This probably won't have an impact if data is not small, but given how many tests there are for these it