Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/19294
Thanks for the fix @szhem, great work !
Merged to master and 2.2.1
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82529/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82529 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82529/testReport)**
for PR 19294 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82529 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82529/testReport)**
for PR 19294 at commit
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19294
@mridulm sql-related tests were removed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82504/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82504 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82504/testReport)**
for PR 19294 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82504 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82504/testReport)**
for PR 19294 at commit
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/19294
@gatorsmile Sounds good, @szhem can we remove the spark sql tests you added
(due to my request).
Once build passes, I will commit this - it will definitely help spark core
users.
---
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19294
Since this is not related to Spark SQL, please do not add the test cases to
the Spark SQL side.
---
-
To unsubscribe,
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/19294
@gatorsmile have your concerns been addressed ? If yes, I will merge this
into master and 2.2.1
This patch is clearly better than existing state for 2.2 and master - for
spark core and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82412/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82412 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82412/testReport)**
for PR 19294 at commit
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/19294
@szhem that null path support in `FileOutputCommitter` came with the App
Master recovery work of
[MAPREDUCE-3711](https://issues.apache.org/jira/browse/MAPREDUCE-3711); its,
trying to
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82412 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82412/testReport)**
for PR 19294 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19294
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82411 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82411/testReport)**
for PR 19294 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82411/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82411 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82411/testReport)**
for PR 19294 at commit
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19294
@gatorsmile I believe that in Spark SQL code path `path` cannot be null,
because in that case `FileFormatWriter` [fails even
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19294
If `path` could be `null`, [this
line](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SQLHadoopMapReduceCommitProtocol.scala#L54)
Github user weiqingy commented on the issue:
https://github.com/apache/spark/pull/19294
@szhem @mridulm Sorry for the late reply. Just come back from vacation.
Sure. I'll try this PR with SHC.
---
-
To unsubscribe,
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/19294
Adding folks who have context about spark sql's `FileFormatWriter`
@ericchiang @rxin @cloud-fan @steveloughran @jiangxb1987
---
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/19294
@szhem Unfortunately, I do not have enough context about how sql writes
paths to comment on when which branch will be taken.
I was going purely from code in `HadoopMapReduceCommitProtocol`, if
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19294
@mridulm Regarding FileFormatWriter I've implemented some basic tests which
show that
1. [FileFormatWriter
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/19294
@szhem Did you resolve issues with newTaskTempFile, newTaskTempFileAbsPath,
etc potentially still throwing NPE due to path being null ?
I saw multiple invocations in spark sql which are calling
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19294
Hello guys, are there a change for this patch to be merged to master?
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82137/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82137 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82137/testReport)**
for PR 19294 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82136/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82136 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82136/testReport)**
for PR 19294 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82135/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82135 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82135/testReport)**
for PR 19294 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82134/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82134 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82134/testReport)**
for PR 19294 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82137 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82137/testReport)**
for PR 19294 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82133/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82133 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82133/testReport)**
for PR 19294 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82136 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82136/testReport)**
for PR 19294 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82130/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82130 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82130/testReport)**
for PR 19294 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82135 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82135/testReport)**
for PR 19294 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82134 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82134/testReport)**
for PR 19294 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82133 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82133/testReport)**
for PR 19294 at commit
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19294
@mridulm Updated `FileFormatWriterSuite` [to
cover](https://github.com/apache/spark/pull/19294/files#diff-bc98a3d91cf4f95f4f473146400044aa)
both branches of the [committer
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82131/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82131 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82131/testReport)**
for PR 19294 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82131 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82131/testReport)**
for PR 19294 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82130 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82130/testReport)**
for PR 19294 at commit
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/19294
@szhem You are correct, currently it fails in the driver itself.
So failures in executor are not seen - since job submission fails.
With this pr, the job submission should succeed - but
Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19294
@mridulm
> incorporating a test for the sql part will also help in this matter.
What should be the expected behaviour in case of sql?
I'm asking because [the sql part seems to fail
Github user steveloughran commented on the issue:
https://github.com/apache/spark/pull/19294
As I play with commit logic all the way through the stack, I can' t help
thinking everyone's lives would be better if we tagged the MRv1 commit APIs as
deprecated in Hadoop 3. and uses of the
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/19294
+CC @weiqingy
Can you try this PR with SHC and see if it works ?
That is, remove your current workaround for SPARK-21549 from SHC and try
writing to hbase with a spark version patched with
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/19294
IMO it should be fine to not provide output directory if you are not using
absolute output paths, I also don't think we should always create absolute
output paths in
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/19294
@szhem Did you try this patch with sql ?
A cursory look at
`org.apache.spark.sql.execution.datasources.FileFormatWriter` and
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82020/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82020 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82020/testReport)**
for PR 19294 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82020 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82020/testReport)**
for PR 19294 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82015/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82015 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82015/testReport)**
for PR 19294 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19294
**[Test build #82015 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82015/testReport)**
for PR 19294 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19294
cc @jiangxb1987 who I believe is interested in this. Without a super close
look, it looks making sense.
---
-
To
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19294
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19294
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
77 matches
Mail list logo
