Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19634
Fewer combined splits might not matter in this case.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user vgankidi commented on the issue:
https://github.com/apache/spark/pull/19634
We will end up having fewer combined splits. That reduces the number of
files that the job produces and also reduces the number of tasks in the
downstream jobs. In some tests I have noticed about
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19634
@vgankidi Does it help the performance of our file reading?
---
-
To unsubscribe, e-mail:
Github user vgankidi commented on the issue:
https://github.com/apache/spark/pull/19634
@gatorsmile I also wanted to discuss if we should consider other bin
packing algorithms. According to this
http://www.math.unl.edu/~s-sjessie1/203Handouts/Bin%20Packing.pdf, next fit
decreasing
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19634
Thanks! Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19634
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19634
**[Test build #3973 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3973/testReport)**
for PR 19634 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19634
**[Test build #3973 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3973/testReport)**
for PR 19634 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19634
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
