[jira] [Commented] (HIVE-6185) DDLTask is inconsistent in creating a table and adding a partition when dealing with location
[
https://issues.apache.org/jira/browse/HIVE-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869669#comment-13869669
]
Ashutosh Chauhan commented on HIVE-6185:
+1, left a minor comment on RB.
DDLTask is inconsistent in creating a table and adding a partition when
dealing with location
-
Key: HIVE-6185
URL: https://issues.apache.org/jira/browse/HIVE-6185
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.12.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Attachments: HIVE-6185.1.patch, HIVE-6185.2.patch, HIVE-6185.patch,
HIVE-6185.patch
When creating a table, Hive uses URI to represent location:
{code}
if (crtTbl.getLocation() != null) {
tbl.setDataLocation(new Path(crtTbl.getLocation()).toUri());
}
{code}
When adding a partition, Hive uses Path to represent location:
{code}
// set partition path relative to table
db.createPartition(tbl, addPartitionDesc.getPartSpec(), new Path(tbl
.getPath(), addPartitionDesc.getLocation()),
addPartitionDesc.getPartParams(),
addPartitionDesc.getInputFormat(),
addPartitionDesc.getOutputFormat(),
addPartitionDesc.getNumBuckets(),
addPartitionDesc.getCols(),
addPartitionDesc.getSerializationLib(),
addPartitionDesc.getSerdeParams(),
addPartitionDesc.getBucketCols(),
addPartitionDesc.getSortCols());
{code}
This disparity makes the values stored in metastore be encoded differently,
causing problems w.r.t. special character as demonstrated in HIVE-5446. As a
result, the code dealing with location for table is different for partition,
creating maintenance burden.
We need to standardize it to Path to be in line with other Path related
cleanup effort.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-6185) DDLTask is inconsistent in creating a table and adding a partition when dealing with location
[
https://issues.apache.org/jira/browse/HIVE-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869898#comment-13869898
]
Xuefu Zhang commented on HIVE-6185:
---
Patch #3 incorporated the review feedback.
DDLTask is inconsistent in creating a table and adding a partition when
dealing with location
-
Key: HIVE-6185
URL: https://issues.apache.org/jira/browse/HIVE-6185
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.12.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Attachments: HIVE-6185.1.patch, HIVE-6185.2.patch, HIVE-6185.3.patch,
HIVE-6185.patch, HIVE-6185.patch
When creating a table, Hive uses URI to represent location:
{code}
if (crtTbl.getLocation() != null) {
tbl.setDataLocation(new Path(crtTbl.getLocation()).toUri());
}
{code}
When adding a partition, Hive uses Path to represent location:
{code}
// set partition path relative to table
db.createPartition(tbl, addPartitionDesc.getPartSpec(), new Path(tbl
.getPath(), addPartitionDesc.getLocation()),
addPartitionDesc.getPartParams(),
addPartitionDesc.getInputFormat(),
addPartitionDesc.getOutputFormat(),
addPartitionDesc.getNumBuckets(),
addPartitionDesc.getCols(),
addPartitionDesc.getSerializationLib(),
addPartitionDesc.getSerdeParams(),
addPartitionDesc.getBucketCols(),
addPartitionDesc.getSortCols());
{code}
This disparity makes the values stored in metastore be encoded differently,
causing problems w.r.t. special character as demonstrated in HIVE-5446. As a
result, the code dealing with location for table is different for partition,
creating maintenance burden.
We need to standardize it to Path to be in line with other Path related
cleanup effort.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-6185) DDLTask is inconsistent in creating a table and adding a partition when dealing with location
[
https://issues.apache.org/jira/browse/HIVE-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13870120#comment-13870120
]
Hive QA commented on HIVE-6185:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12622690/HIVE-6185.3.patch
{color:green}SUCCESS:{color} +1 4924 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/889/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/889/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12622690
DDLTask is inconsistent in creating a table and adding a partition when
dealing with location
-
Key: HIVE-6185
URL: https://issues.apache.org/jira/browse/HIVE-6185
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.12.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Attachments: HIVE-6185.1.patch, HIVE-6185.2.patch, HIVE-6185.3.patch,
HIVE-6185.patch, HIVE-6185.patch
When creating a table, Hive uses URI to represent location:
{code}
if (crtTbl.getLocation() != null) {
tbl.setDataLocation(new Path(crtTbl.getLocation()).toUri());
}
{code}
When adding a partition, Hive uses Path to represent location:
{code}
// set partition path relative to table
db.createPartition(tbl, addPartitionDesc.getPartSpec(), new Path(tbl
.getPath(), addPartitionDesc.getLocation()),
addPartitionDesc.getPartParams(),
addPartitionDesc.getInputFormat(),
addPartitionDesc.getOutputFormat(),
addPartitionDesc.getNumBuckets(),
addPartitionDesc.getCols(),
addPartitionDesc.getSerializationLib(),
addPartitionDesc.getSerdeParams(),
addPartitionDesc.getBucketCols(),
addPartitionDesc.getSortCols());
{code}
This disparity makes the values stored in metastore be encoded differently,
causing problems w.r.t. special character as demonstrated in HIVE-5446. As a
result, the code dealing with location for table is different for partition,
creating maintenance burden.
We need to standardize it to Path to be in line with other Path related
cleanup effort.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-6185) DDLTask is inconsistent in creating a table and adding a partition when dealing with location
[
https://issues.apache.org/jira/browse/HIVE-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869170#comment-13869170
]
Mohammad Kamrul Islam commented on HIVE-6185:
-
Patch looks good!
Few comments:
1. In Partition::setBucketCount(),
FileSystem fs = FileSystem.get(getDataLocation().toUri(), Hive.get().getConf())
can be rewritten as (to make it consistent for other places):
FileSystem fs = getDataLocation().getFileSystem(Hive.get().getConf());
2. Same thing in SamplePruner:: limitPrune()
FileSystem fs = FileSystem.get(part.getDataLocation().toUri(), Hive.get()
.getConf());
can be rewritten as
FileSystem fs = part.getDataLocation().getFileSystem(Hive.get().getConf());
3. In Partition.java
A new method public Path getDataLocation() is introduced. Is it replacing
public Path getPartitionPath() or final public URI getDataLocation()? If
it is the later one, do we need to keep the final modifier?
DDLTask is inconsistent in creating a table and adding a partition when
dealing with location
-
Key: HIVE-6185
URL: https://issues.apache.org/jira/browse/HIVE-6185
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.12.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Attachments: HIVE-6185.1.patch, HIVE-6185.2.patch, HIVE-6185.patch,
HIVE-6185.patch
When creating a table, Hive uses URI to represent location:
{code}
if (crtTbl.getLocation() != null) {
tbl.setDataLocation(new Path(crtTbl.getLocation()).toUri());
}
{code}
When adding a partition, Hive uses Path to represent location:
{code}
// set partition path relative to table
db.createPartition(tbl, addPartitionDesc.getPartSpec(), new Path(tbl
.getPath(), addPartitionDesc.getLocation()),
addPartitionDesc.getPartParams(),
addPartitionDesc.getInputFormat(),
addPartitionDesc.getOutputFormat(),
addPartitionDesc.getNumBuckets(),
addPartitionDesc.getCols(),
addPartitionDesc.getSerializationLib(),
addPartitionDesc.getSerdeParams(),
addPartitionDesc.getBucketCols(),
addPartitionDesc.getSortCols());
{code}
This disparity makes the values stored in metastore be encoded differently,
causing problems w.r.t. special character as demonstrated in HIVE-5446. As a
result, the code dealing with location for table is different for partition,
creating maintenance burden.
We need to standardize it to Path to be in line with other Path related
cleanup effort.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-6185) DDLTask is inconsistent in creating a table and adding a partition when dealing with location
[
https://issues.apache.org/jira/browse/HIVE-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13868799#comment-13868799
]
Hive QA commented on HIVE-6185:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12622492/HIVE-6185.patch
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/863/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/863/console
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] Deleting /data/hive-ptest/working/apache-svn-trunk-source/itests
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @ hive-it ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-it ---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/target/tmp/conf
[copy] Copying 5 files to
/data/hive-ptest/working/apache-svn-trunk-source/itests/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-it ---
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/itests/pom.xml to
/data/hive-ptest/working/maven/org/apache/hive/hive-it/0.13.0-SNAPSHOT/hive-it-0.13.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Integration - Custom Serde 0.13.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-it-custom-serde
---
[INFO] Deleting
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde (includes
= [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @
hive-it-custom-serde ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/src/main/resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @
hive-it-custom-serde ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
hive-it-custom-serde ---
[INFO] Compiling 8 source files to
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/classes
[INFO]
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @
hive-it-custom-serde ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/src/test/resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-it-custom-serde
---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/tmp/conf
[copy] Copying 5 files to
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-it-custom-serde ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @
hive-it-custom-serde ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-it-custom-serde ---
[INFO] Building jar:
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/hive-it-custom-serde-0.13.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @
hive-it-custom-serde ---
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/hive-it-custom-serde-0.13.0-SNAPSHOT.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hive-it-custom-serde/0.13.0-SNAPSHOT/hive-it-custom-serde-0.13.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/pom.xml to
/data/hive-ptest/working/maven/org/apache/hive/hive-it-custom-serde/0.13.0-SNAPSHOT/hive-it-custom-serde-0.13.0-SNAPSHOT.pom
[INFO]
[jira] [Commented] (HIVE-6185) DDLTask is inconsistent in creating a table and adding a partition when dealing with location
[
https://issues.apache.org/jira/browse/HIVE-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13868886#comment-13868886
]
Hive QA commented on HIVE-6185:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12622513/HIVE-6185.1.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 4917 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_14_managed_location_over_existing
{noformat}
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/866/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/866/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12622513
DDLTask is inconsistent in creating a table and adding a partition when
dealing with location
-
Key: HIVE-6185
URL: https://issues.apache.org/jira/browse/HIVE-6185
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.12.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Attachments: HIVE-6185.1.patch, HIVE-6185.patch, HIVE-6185.patch
When creating a table, Hive uses URI to represent location:
{code}
if (crtTbl.getLocation() != null) {
tbl.setDataLocation(new Path(crtTbl.getLocation()).toUri());
}
{code}
When adding a partition, Hive uses Path to represent location:
{code}
// set partition path relative to table
db.createPartition(tbl, addPartitionDesc.getPartSpec(), new Path(tbl
.getPath(), addPartitionDesc.getLocation()),
addPartitionDesc.getPartParams(),
addPartitionDesc.getInputFormat(),
addPartitionDesc.getOutputFormat(),
addPartitionDesc.getNumBuckets(),
addPartitionDesc.getCols(),
addPartitionDesc.getSerializationLib(),
addPartitionDesc.getSerdeParams(),
addPartitionDesc.getBucketCols(),
addPartitionDesc.getSortCols());
{code}
This disparity makes the values stored in metastore be encoded differently,
causing problems w.r.t. special character as demonstrated in HIVE-5446. As a
result, the code dealing with location for table is different for partition,
creating maintenance burden.
We need to standardize it to Path to be in line with other Path related
cleanup effort.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-6185) DDLTask is inconsistent in creating a table and adding a partition when dealing with location
[
https://issues.apache.org/jira/browse/HIVE-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13868943#comment-13868943
]
Hive QA commented on HIVE-6185:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12622525/HIVE-6185.2.patch
{color:green}SUCCESS:{color} +1 4917 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/871/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/871/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12622525
DDLTask is inconsistent in creating a table and adding a partition when
dealing with location
-
Key: HIVE-6185
URL: https://issues.apache.org/jira/browse/HIVE-6185
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.12.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang
Attachments: HIVE-6185.1.patch, HIVE-6185.2.patch, HIVE-6185.patch,
HIVE-6185.patch
When creating a table, Hive uses URI to represent location:
{code}
if (crtTbl.getLocation() != null) {
tbl.setDataLocation(new Path(crtTbl.getLocation()).toUri());
}
{code}
When adding a partition, Hive uses Path to represent location:
{code}
// set partition path relative to table
db.createPartition(tbl, addPartitionDesc.getPartSpec(), new Path(tbl
.getPath(), addPartitionDesc.getLocation()),
addPartitionDesc.getPartParams(),
addPartitionDesc.getInputFormat(),
addPartitionDesc.getOutputFormat(),
addPartitionDesc.getNumBuckets(),
addPartitionDesc.getCols(),
addPartitionDesc.getSerializationLib(),
addPartitionDesc.getSerdeParams(),
addPartitionDesc.getBucketCols(),
addPartitionDesc.getSortCols());
{code}
This disparity makes the values stored in metastore be encoded differently,
causing problems w.r.t. special character as demonstrated in HIVE-5446. As a
result, the code dealing with location for table is different for partition,
creating maintenance burden.
We need to standardize it to Path to be in line with other Path related
cleanup effort.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
