Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
```
This approach only works if the first level glob pattern matches a lot of
directories.
```
Yep, actually in our internal usage, we leave the problem to user and they
should use
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17702
This approach only works if the first level glob pattern matches a lot of
directories, e.g. `/my_path/*/*`. Otherwise, we can't apply it, e.g.
`/my_path/{ab, cd}/*`.
My proposal: think
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
ping @vanzin
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86583/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #86583 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86583/testReport)**
for PR 17702 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #86583 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86583/testReport)**
for PR 17702 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/188/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86573/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #86573 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86573/testReport)**
for PR 17702 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/178/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #86573 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86573/testReport)**
for PR 17702 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86558/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #86558 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86558/testReport)**
for PR 17702 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/165/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #86558 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86558/testReport)**
for PR 17702 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86517/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #86517 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86517/testReport)**
for PR 17702 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #86517 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86517/testReport)**
for PR 17702 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/129/
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
gental ping @zsxwing
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
@zsxwing Thanks for your comments, ready for review.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83837/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #83837 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83837/testReport)**
for PR 17702 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #83837 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83837/testReport)**
for PR 17702 at commit
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83822/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #83822 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83822/testReport)**
for PR 17702 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #83822 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83822/testReport)**
for PR 17702 at commit
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83819/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #83819 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83819/testReport)**
for PR 17702 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #83819 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83819/testReport)**
for PR 17702 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83783/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #83783 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83783/testReport)**
for PR 17702 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #83783 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83783/testReport)**
for PR 17702 at commit
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/17702
cc @zsxwing
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82378/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #82378 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82378/testReport)**
for PR 17702 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #82378 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82378/testReport)**
for PR 17702 at commit
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82360/
Test FAILed.
---
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82351/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #82351 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82351/testReport)**
for PR 17702 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #82351 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82351/testReport)**
for PR 17702 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78171/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #78171 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78171/testReport)**
for PR 17702 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #78171 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78171/testReport)**
for PR 17702 at commit
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
Test failed may because of the env? `process was terminated by signal 9` in
jenkins log.
retest it please
---
If your project is set up for it, you can reply to this email and have your
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78156/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #78156 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78156/testReport)**
for PR 17702 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78137/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #78137 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78137/testReport)**
for PR 17702 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #78137 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78137/testReport)**
for PR 17702 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17702
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
@cloud-fan Thanks for your reply!
It's possible to consolidate them but may be not so necessary? I can
consolidate them by replace the logic in `getGlobbedPaths` list below to
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17702
The logic looks very similar to `InMemoryFileIndex.bulkListLeafFiles`, is
it possible to consolidate them?
---
If your project is set up for it, you can reply to this email and have your
reply
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77634/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #77634 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77634/testReport)**
for PR 17702 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #77634 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77634/testReport)**
for PR 17702 at commit
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
ping @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
@gatorsmile @cloud-fan, do we need other performance test?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76378/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #76378 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76378/testReport)**
for PR 17702 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #76378 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76378/testReport)**
for PR 17702 at commit
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
Thanks for your review. @gatorsmile @cloud-fan
`Can you show us the performance difference?`
No problem, I reproduce our online case offline like below
## Without
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17702
yea I'm also wondering how useful it is
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17702
Can you show us the performance difference?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
ping @cloud-fan and @gatorsmile , could you have a look about this ?
Thanks :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17702
Not sure. Probably, @cloud-fan or @gatorsmile ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
@HyukjinKwon Can you help me to find a appropriate reviewer about this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
cc @zsxwing @tdas, can you review this? Founded the relative code of yours
before. Thanks :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user xuanyuanking commented on the issue:
https://github.com/apache/spark/pull/17702
@marmbrus Can you take a look of this? Thanks :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75984/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17702
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17702
**[Test build #75984 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75984/testReport)**
for PR 17702 at commit
1 - 100 of 101 matches
Mail list logo
