[jira] [Commented] (HIVE-11427) Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
[
https://issues.apache.org/jira/browse/HIVE-11427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15244534#comment-15244534
]
Yongzhi Chen commented on HIVE-11427:
-
The failures are not related. Thanks [~spena] for reviewing the code.
> Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
>
>
> Key: HIVE-11427
> URL: https://issues.apache.org/jira/browse/HIVE-11427
> Project: Hive
> Issue Type: Bug
>Reporter: Grisha Trubetskoy
>Assignee: Yongzhi Chen
> Attachments: HIVE-11427.1.patch, HIVE-11427.2.patch
>
>
> If a user _does not_ have HDFS write permissions to the _default_ database,
> and attempts to create a table in a _private_ database to which the user
> _does_ have permissions using CREATE TABLE AS SELECT from a table in the
> default database, the following happens:
> {code}
> use default;
> create table grisha.blahblah as select * from some_table;
> FAILED: SemanticException 0:0 Error creating temporary folder on:
> hdfs://nn.example.com/user/hive/warehouse. Error encountered near token
> 'TOK_TMP_FILE’
> {code}
> I've edited this issue because my initial explanation was completely bogus. A
> more likely explanation is in
> https://github.com/apache/hive/commit/1614314ef7bd0c3b8527ee32a434ababf7711278
> {code}
> -fname = ctx.getExternalTmpPath(
> +fname = ctx.getExtTmpPathRelTo(
> // and then something incorrect happens in getExtTmpPathRelTo()
> {code}
> In any event - the bug is that the location chosen for the temporary storage
> is not in the same place as the target table. It should be same as the target
> table (/user/hive/warehouse/grisha.db in the above example) because this is
> where presumably the user running the query would have write permissions to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11427) Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
[
https://issues.apache.org/jira/browse/HIVE-11427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15244103#comment-15244103
]
Hive QA commented on HIVE-11427:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798617/HIVE-11427.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9968 tests executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-vector_distinct_2.q-load_dyn_part2.q-join1.q-and-12-more -
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_coltype_literals
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7614/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7614/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7614/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12798617 - PreCommit-HIVE-TRUNK-Build
> Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
>
>
> Key: HIVE-11427
> URL: https://issues.apache.org/jira/browse/HIVE-11427
> Project: Hive
> Issue Type: Bug
>Reporter: Grisha Trubetskoy
>Assignee: Yongzhi Chen
> Attachments: HIVE-11427.1.patch, HIVE-11427.2.patch
>
>
> If a user _does not_ have HDFS write permissions to the _default_ database,
> and attempts to create a table in a _private_ database to which the user
> _does_ have permissions using CREATE TABLE AS SELECT from a table in the
> default database, the following happens:
> {code}
> use default;
> create table grisha.blahblah as select * from some_table;
> FAILED: SemanticException 0:0 Error creating temporary folder on:
> hdfs://nn.example.com/user/hive/warehouse. Error encountered near token
> 'TOK_TMP_FILE’
> {code}
> I've edited this issue because my initial explanation was completely bogus. A
> more likely explanation is in
> https://github.com/apache/hive/commit/1614314ef7bd0c3b8527ee32a434ababf7711278
> {code}
> -fname = ctx.getExternalTmpPath(
> +fname = ctx.getExtTmpPathRelTo(
> // and then something incorrect happens in getExtTmpPathRelTo()
> {code}
> In any event - the bug is that the location chosen for the temporary storage
> is not in the same place as the target table. It should be same as the target
> table (/user/hive/warehouse/grisha.db in the above example) because this is
> where presumably the user running the query would have write permissions to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11427) Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
[
https://issues.apache.org/jira/browse/HIVE-11427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241787#comment-15241787
]
Sergio Peña commented on HIVE-11427:
You're right. At the end, the security issue was still there before.
So the code looks right.
+1
Let's wait for tests before doing the commit.
> Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
>
>
> Key: HIVE-11427
> URL: https://issues.apache.org/jira/browse/HIVE-11427
> Project: Hive
> Issue Type: Bug
>Reporter: Grisha Trubetskoy
>Assignee: Yongzhi Chen
> Attachments: HIVE-11427.1.patch, HIVE-11427.2.patch
>
>
> If a user _does not_ have HDFS write permissions to the _default_ database,
> and attempts to create a table in a _private_ database to which the user
> _does_ have permissions using CREATE TABLE AS SELECT from a table in the
> default database, the following happens:
> {code}
> use default;
> create table grisha.blahblah as select * from some_table;
> FAILED: SemanticException 0:0 Error creating temporary folder on:
> hdfs://nn.example.com/user/hive/warehouse. Error encountered near token
> 'TOK_TMP_FILE’
> {code}
> I've edited this issue because my initial explanation was completely bogus. A
> more likely explanation is in
> https://github.com/apache/hive/commit/1614314ef7bd0c3b8527ee32a434ababf7711278
> {code}
> -fname = ctx.getExternalTmpPath(
> +fname = ctx.getExtTmpPathRelTo(
> // and then something incorrect happens in getExtTmpPathRelTo()
> {code}
> In any event - the bug is that the location chosen for the temporary storage
> is not in the same place as the target table. It should be same as the target
> table (/user/hive/warehouse/grisha.db in the above example) because this is
> where presumably the user running the query would have write permissions to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11427) Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
[
https://issues.apache.org/jira/browse/HIVE-11427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241597#comment-15241597
]
Yongzhi Chen commented on HIVE-11427:
-
I do not know. It should not be related to this jira, this jira just use the
destination database's resource as the tmp storage and the tmp storage will be
deleted after at the end of the CTAS. It does not affect how the final data is
stored.
And I think Insert --- select may not so strict, for example, how do we handle
select with several tables joins, some encrypted, some not? We may create a
different jira for the encryption issue.
> Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
>
>
> Key: HIVE-11427
> URL: https://issues.apache.org/jira/browse/HIVE-11427
> Project: Hive
> Issue Type: Bug
>Reporter: Grisha Trubetskoy
>Assignee: Yongzhi Chen
> Attachments: HIVE-11427.1.patch, HIVE-11427.2.patch
>
>
> If a user _does not_ have HDFS write permissions to the _default_ database,
> and attempts to create a table in a _private_ database to which the user
> _does_ have permissions using CREATE TABLE AS SELECT from a table in the
> default database, the following happens:
> {code}
> use default;
> create table grisha.blahblah as select * from some_table;
> FAILED: SemanticException 0:0 Error creating temporary folder on:
> hdfs://nn.example.com/user/hive/warehouse. Error encountered near token
> 'TOK_TMP_FILE’
> {code}
> I've edited this issue because my initial explanation was completely bogus. A
> more likely explanation is in
> https://github.com/apache/hive/commit/1614314ef7bd0c3b8527ee32a434ababf7711278
> {code}
> -fname = ctx.getExternalTmpPath(
> +fname = ctx.getExtTmpPathRelTo(
> // and then something incorrect happens in getExtTmpPathRelTo()
> {code}
> In any event - the bug is that the location chosen for the temporary storage
> is not in the same place as the target table. It should be same as the target
> table (/user/hive/warehouse/grisha.db in the above example) because this is
> where presumably the user running the query would have write permissions to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11427) Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
[
https://issues.apache.org/jira/browse/HIVE-11427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241345#comment-15241345
]
Sergio Peña commented on HIVE-11427:
What if the destination table is not encrypted? Should we fail running the CTAS
command?
I don't remember if we have that validation with INSERT ... SELECT.
> Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
>
>
> Key: HIVE-11427
> URL: https://issues.apache.org/jira/browse/HIVE-11427
> Project: Hive
> Issue Type: Bug
>Reporter: Grisha Trubetskoy
>Assignee: Yongzhi Chen
> Attachments: HIVE-11427.1.patch, HIVE-11427.2.patch
>
>
> If a user _does not_ have HDFS write permissions to the _default_ database,
> and attempts to create a table in a _private_ database to which the user
> _does_ have permissions using CREATE TABLE AS SELECT from a table in the
> default database, the following happens:
> {code}
> use default;
> create table grisha.blahblah as select * from some_table;
> FAILED: SemanticException 0:0 Error creating temporary folder on:
> hdfs://nn.example.com/user/hive/warehouse. Error encountered near token
> 'TOK_TMP_FILE’
> {code}
> I've edited this issue because my initial explanation was completely bogus. A
> more likely explanation is in
> https://github.com/apache/hive/commit/1614314ef7bd0c3b8527ee32a434ababf7711278
> {code}
> -fname = ctx.getExternalTmpPath(
> +fname = ctx.getExtTmpPathRelTo(
> // and then something incorrect happens in getExtTmpPathRelTo()
> {code}
> In any event - the bug is that the location chosen for the temporary storage
> is not in the same place as the target table. It should be same as the target
> table (/user/hive/warehouse/grisha.db in the above example) because this is
> where presumably the user running the query would have write permissions to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11427) Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
[
https://issues.apache.org/jira/browse/HIVE-11427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15241248#comment-15241248
]
Yongzhi Chen commented on HIVE-11427:
-
CTAS function like create a table then do insert with select statement from
other tables. I think if the data can be returned to select statement, it will
be inserted to the destination table. So user should make sure the destination
table is encrypted too.
> Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
>
>
> Key: HIVE-11427
> URL: https://issues.apache.org/jira/browse/HIVE-11427
> Project: Hive
> Issue Type: Bug
>Reporter: Grisha Trubetskoy
>Assignee: Yongzhi Chen
> Attachments: HIVE-11427.1.patch, HIVE-11427.2.patch
>
>
> If a user _does not_ have HDFS write permissions to the _default_ database,
> and attempts to create a table in a _private_ database to which the user
> _does_ have permissions using CREATE TABLE AS SELECT from a table in the
> default database, the following happens:
> {code}
> use default;
> create table grisha.blahblah as select * from some_table;
> FAILED: SemanticException 0:0 Error creating temporary folder on:
> hdfs://nn.example.com/user/hive/warehouse. Error encountered near token
> 'TOK_TMP_FILE’
> {code}
> I've edited this issue because my initial explanation was completely bogus. A
> more likely explanation is in
> https://github.com/apache/hive/commit/1614314ef7bd0c3b8527ee32a434ababf7711278
> {code}
> -fname = ctx.getExternalTmpPath(
> +fname = ctx.getExtTmpPathRelTo(
> // and then something incorrect happens in getExtTmpPathRelTo()
> {code}
> In any event - the bug is that the location chosen for the temporary storage
> is not in the same place as the target table. It should be same as the target
> table (/user/hive/warehouse/grisha.db in the above example) because this is
> where presumably the user running the query would have write permissions to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11427) Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
[
https://issues.apache.org/jira/browse/HIVE-11427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239350#comment-15239350
]
Sergio Peña commented on HIVE-11427:
The patch looks good and simple.
Could you verify and/or add a test case when HDFS encryption is enabled? There
are a few {{encryption_*.q}} files you can use to see examples about it.
I wonder what we should do if a user is attempting to create a table
(non-encrypted) from an encrypted table. If the encrypted table has sensitive
information, such as SSN, then they will be able to copy the data and stored
without encryption in the target location?
> Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
>
>
> Key: HIVE-11427
> URL: https://issues.apache.org/jira/browse/HIVE-11427
> Project: Hive
> Issue Type: Bug
>Reporter: Grisha Trubetskoy
>Assignee: Yongzhi Chen
> Attachments: HIVE-11427.1.patch
>
>
> If a user _does not_ have HDFS write permissions to the _default_ database,
> and attempts to create a table in a _private_ database to which the user
> _does_ have permissions using CREATE TABLE AS SELECT from a table in the
> default database, the following happens:
> {code}
> use default;
> create table grisha.blahblah as select * from some_table;
> FAILED: SemanticException 0:0 Error creating temporary folder on:
> hdfs://nn.example.com/user/hive/warehouse. Error encountered near token
> 'TOK_TMP_FILE’
> {code}
> I've edited this issue because my initial explanation was completely bogus. A
> more likely explanation is in
> https://github.com/apache/hive/commit/1614314ef7bd0c3b8527ee32a434ababf7711278
> {code}
> -fname = ctx.getExternalTmpPath(
> +fname = ctx.getExtTmpPathRelTo(
> // and then something incorrect happens in getExtTmpPathRelTo()
> {code}
> In any event - the bug is that the location chosen for the temporary storage
> is not in the same place as the target table. It should be same as the target
> table (/user/hive/warehouse/grisha.db in the above example) because this is
> where presumably the user running the query would have write permissions to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11427) Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
[
https://issues.apache.org/jira/browse/HIVE-11427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15239211#comment-15239211
]
Yongzhi Chen commented on HIVE-11427:
-
The failures are not related.
[~spena], could you review the change? Thanks. I manually tested the fix in a
cluster.
> Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
>
>
> Key: HIVE-11427
> URL: https://issues.apache.org/jira/browse/HIVE-11427
> Project: Hive
> Issue Type: Bug
>Reporter: Grisha Trubetskoy
>Assignee: Yongzhi Chen
> Attachments: HIVE-11427.1.patch
>
>
> If a user _does not_ have HDFS write permissions to the _default_ database,
> and attempts to create a table in a _private_ database to which the user
> _does_ have permissions using CREATE TABLE AS SELECT from a table in the
> default database, the following happens:
> {code}
> use default;
> create table grisha.blahblah as select * from some_table;
> FAILED: SemanticException 0:0 Error creating temporary folder on:
> hdfs://nn.example.com/user/hive/warehouse. Error encountered near token
> 'TOK_TMP_FILE’
> {code}
> I've edited this issue because my initial explanation was completely bogus. A
> more likely explanation is in
> https://github.com/apache/hive/commit/1614314ef7bd0c3b8527ee32a434ababf7711278
> {code}
> -fname = ctx.getExternalTmpPath(
> +fname = ctx.getExtTmpPathRelTo(
> // and then something incorrect happens in getExtTmpPathRelTo()
> {code}
> In any event - the bug is that the location chosen for the temporary storage
> is not in the same place as the target table. It should be same as the target
> table (/user/hive/warehouse/grisha.db in the above example) because this is
> where presumably the user running the query would have write permissions to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11427) Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
[
https://issues.apache.org/jira/browse/HIVE-11427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15238114#comment-15238114
]
Hive QA commented on HIVE-11427:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798319/HIVE-11427.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9974 tests executed
*Failed tests:*
{noformat}
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7567/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7567/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7567/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12798319 - PreCommit-HIVE-TRUNK-Build
> Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
>
>
> Key: HIVE-11427
> URL: https://issues.apache.org/jira/browse/HIVE-11427
> Project: Hive
> Issue Type: Bug
>Reporter: Grisha Trubetskoy
>Assignee: Yongzhi Chen
> Attachments: HIVE-11427.1.patch
>
>
> If a user _does not_ have HDFS write permissions to the _default_ database,
> and attempts to create a table in a _private_ database to which the user
> _does_ have permissions using CREATE TABLE AS SELECT from a table in the
> default database, the following happens:
> {code}
> use default;
> create table grisha.blahblah as select * from some_table;
> FAILED: SemanticException 0:0 Error creating temporary folder on:
> hdfs://nn.example.com/user/hive/warehouse. Error encountered near token
> 'TOK_TMP_FILE’
> {code}
> I've edited this issue because my initial explanation was completely bogus. A
> more likely explanation is in
> https://github.com/apache/hive/commit/1614314ef7bd0c3b8527ee32a434ababf7711278
> {code}
> -fname = ctx.getExternalTmpPath(
> +fname = ctx.getExtTmpPathRelTo(
> // and then something incorrect happens in getExtTmpPathRelTo()
> {code}
> In any event - the bug is that the location chosen for the temporary storage
> is not in the same place as the target table. It should be same as the target
> table (/user/hive/warehouse/grisha.db in the above example) because this is
> where presumably the user running the query would have write permissions to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11427) Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
[
https://issues.apache.org/jira/browse/HIVE-11427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15237613#comment-15237613
]
Yongzhi Chen commented on HIVE-11427:
-
TOK_TMP_FILE does not have a database before it, so Utilities.getDbTableName
provide current working db as its db which is not right for CTAS.
Should use the database where the table to be created belongs to.
Attach patch for the fix.
> Location of temporary table for CREATE TABLE SELECT broken by HIVE-7079
>
>
> Key: HIVE-11427
> URL: https://issues.apache.org/jira/browse/HIVE-11427
> Project: Hive
> Issue Type: Bug
>Reporter: Grisha Trubetskoy
>Assignee: Yongzhi Chen
>
> If a user _does not_ have HDFS write permissions to the _default_ database,
> and attempts to create a table in a _private_ database to which the user
> _does_ have permissions using CREATE TABLE AS SELECT from a table in the
> default database, the following happens:
> {code}
> use default;
> create table grisha.blahblah as select * from some_table;
> FAILED: SemanticException 0:0 Error creating temporary folder on:
> hdfs://nn.example.com/user/hive/warehouse. Error encountered near token
> 'TOK_TMP_FILE’
> {code}
> I've edited this issue because my initial explanation was completely bogus. A
> more likely explanation is in
> https://github.com/apache/hive/commit/1614314ef7bd0c3b8527ee32a434ababf7711278
> {code}
> -fname = ctx.getExternalTmpPath(
> +fname = ctx.getExtTmpPathRelTo(
> // and then something incorrect happens in getExtTmpPathRelTo()
> {code}
> In any event - the bug is that the location chosen for the temporary storage
> is not in the same place as the target table. It should be same as the target
> table (/user/hive/warehouse/grisha.db in the above example) because this is
> where presumably the user running the query would have write permissions to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
