[
https://issues.apache.org/jira/browse/HIVE-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730995#comment-15730995
]
Hive QA commented on HIVE-15367:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12842225/HIVE-15367.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10784 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2]
(batchId=134)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_based_fetch_decision]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_1]
(batchId=91)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4]
(batchId=92)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2477/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2477/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2477/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12842225 - PreCommit-HIVE-Build
> CTAS with LOCATION should write temp data under location directory rather
> than database location
> ------------------------------------------------------------------------------------------------
>
> Key: HIVE-15367
> URL: https://issues.apache.org/jira/browse/HIVE-15367
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Attachments: HIVE-15367.1.patch, HIVE-15367.2.patch,
> HIVE-15367.3.patch, HIVE-15367.4.patch
>
>
> For regular CTAS queries, temp data from a SELECT query will be written to to
> a staging directory under the database location. The code to control this is
> in {{SemanticAnalyzer.java}}
> {code}
> // allocate a temporary output dir on the location of the table
> String tableName = getUnescapedName((ASTNode) ast.getChild(0));
> String[] names = Utilities.getDbTableName(tableName);
> Path location;
> try {
> Warehouse wh = new Warehouse(conf);
> //Use destination table's db location.
> String destTableDb = qb.getTableDesc() != null?
> qb.getTableDesc().getDatabaseName(): null;
> if (destTableDb == null) {
> destTableDb = names[0];
> }
> location = wh.getDatabasePath(db.getDatabase(destTableDb));
> } catch (MetaException e) {
> throw new SemanticException(e);
> }
> {code}
> However, CTAS queries allow specifying a {{LOCATION}} for the new table. Its
> possible for this location to be on a different filesystem than the database
> location. If this happens temp data will be written to the database
> filesystem and will be copied to the table filesystem in {{MoveTask}}.
> This extra copying of data can drastically affect performance. Rather than
> always use the database location as the staging dir for CTAS queries, Hive
> should first check if there is an explicit {{LOCATION}} specified in the CTAS
> query. If there is, staging data should be stored under the {{LOCATION}}
> directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
