Github user yhuai commented on the issue:
https://github.com/apache/spark/pull/14634
LGTM. Merging to master and branch 2.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14634
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65629/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14634
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14634
**[Test build #65629 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65629/consoleFull)**
for PR 14634 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14634
**[Test build #65629 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65629/consoleFull)**
for PR 14634 at commit
Github user yhuai commented on the issue:
https://github.com/apache/spark/pull/14634
This change looks good. Let's add a regression test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14634
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65620/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14634
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14634
**[Test build #65620 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65620/consoleFull)**
for PR 14634 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14634
**[Test build #65620 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65620/consoleFull)**
for PR 14634 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14634
@JoshRosen this is a regression in 2.0(it works in 1.6), so I think we
should target it to 2.0
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14634
cc @yhuai , In `InsertIntoHiveTable` we already called `newHadoopConf`, I
think it's safer to get these hive confs from `hadoopConf` instead of
`sqlConf`, to respect hive confs in
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14634
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user JoshRosen commented on the issue:
https://github.com/apache/spark/pull/14634
@yhuai, @cloud-fan What's the status of this issue? Should this still be
targeted for 2.0.1 (which it is currently in JIRA)?
---
If your project is set up for it, you can reply to this email and
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14634
Based on my understanding, after this PR, we will respect the conf values
of `hive.exec.dynamic.partition`, `hive.exec.dynamic.partition.mode` and
`hive.exec.compress.output` that are specified
Github user yhuai commented on the issue:
https://github.com/apache/spark/pull/14634
Sorry. What's the necessity to make this change?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14634
uh, I see. Thank you! No more question. : )
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14634
1. `hadoopConf` contains all the confs from SQL conf, see
`SessionState.newHadoopConf`
2. users can change `hadoopConf` at runtime.
3. the default value is from hive, I'm ok to not follow
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14634
Sorry, let me rephrase the potential issue.
- `insertInto` API forces users to set `hive.exec.dynamic.partition` to
`true` and `hive.exec.dynamic.partition.mode` to `nonstrict`. This
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14634
If users want to use the `DataFrameWriter`'s `insertInto` API for
partitioned Hive table, they have to set `hive.exec.dynamic.partition` to
`true` and `hive.exec.dynamic.partition.mode` to
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14634
> hive.exec.dynamic.partition also impacts our regular writing paths
I think it's hive only conf? Normal data source relation should not read
this conf.
---
If your project is set up
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14634
`hive.exec.dynamic.partition` also impacts our regular writing paths
(DataFrameWriter APIs). I remember DataFrameWriter APIs always assume this conf
is true, right?
If it is controlled
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14634
Great! It resolves my original concern. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14634
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63746/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14634
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14634
**[Test build #63746 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63746/consoleFull)**
for PR 14634 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14634
**[Test build #63746 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63746/consoleFull)**
for PR 14634 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14634
cc @yhuai
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
28 matches
Mail list logo
