[jira] [Commented] (HIVE-7618) TestDDLWithRemoteMetastoreSecondNamenode unit test failure
[
https://issues.apache.org/jira/browse/HIVE-7618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091126#comment-14091126
]
Sushanth Sowmyan commented on HIVE-7618:
+1 on the patch as it currently stands.
While I am, in theory, in favour of adding this to the SH interface, I think we
should hold off on that for now. I would rather open discussion with the hive
group at large in re-architecting StorageHandlers in general, trying to do the
following:
a) Deprecation/removal of HiveOutputFormat/HiveRecordWriter in general, in
favour of using M/R definitions of the same, and having Committer semantics
included.
b) Rearchitecting/refactoring native Hive storage in a way that makes
everything go through the SH interface, rather than having special-casing for
SH and native
c) Support for notion of SH per partition, rather than SH per table
d) Notion of possible plan modifications by SH for any add-on tasks that are
required.
And if we're making that many changes, it's likely that we will break SHs
significantly at that time, and I'd rather do it once rather than have a
constant stream of breaking.
I'd like to see us pursue that as a major initiative in the 0.15 timeframe, if
possible. I'll shoot out a mail to the list on that regard.
TestDDLWithRemoteMetastoreSecondNamenode unit test failure
--
Key: HIVE-7618
URL: https://issues.apache.org/jira/browse/HIVE-7618
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-7618.1.patch, HIVE-7618.2.patch
Looks like TestDDLWithRemoteMetastoreSecondNamenode started failing after
HIVE-6584 was committed.
{noformat}
TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode:272-createTableAndCheck:201-createTableAndCheck:219
Table should be located in the second filesystem expected:[hdfs] but
was:[pfile]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7618) TestDDLWithRemoteMetastoreSecondNamenode unit test failure
[
https://issues.apache.org/jira/browse/HIVE-7618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088923#comment-14088923
]
Hive QA commented on HIVE-7618:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12660244/HIVE-7618.2.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 5883 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/201/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/201/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-201/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12660244
TestDDLWithRemoteMetastoreSecondNamenode unit test failure
--
Key: HIVE-7618
URL: https://issues.apache.org/jira/browse/HIVE-7618
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-7618.1.patch, HIVE-7618.2.patch
Looks like TestDDLWithRemoteMetastoreSecondNamenode started failing after
HIVE-6584 was committed.
{noformat}
TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode:272-createTableAndCheck:201-createTableAndCheck:219
Table should be located in the second filesystem expected:[hdfs] but
was:[pfile]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7618) TestDDLWithRemoteMetastoreSecondNamenode unit test failure
[
https://issues.apache.org/jira/browse/HIVE-7618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087698#comment-14087698
]
Hive QA commented on HIVE-7618:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12659945/HIVE-7618.1.patch
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 5877 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_mixed_case
org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/186/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/186/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-186/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12659945
TestDDLWithRemoteMetastoreSecondNamenode unit test failure
--
Key: HIVE-7618
URL: https://issues.apache.org/jira/browse/HIVE-7618
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-7618.1.patch
Looks like TestDDLWithRemoteMetastoreSecondNamenode started failing after
HIVE-6584 was committed.
{noformat}
TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode:272-createTableAndCheck:201-createTableAndCheck:219
Table should be located in the second filesystem expected:[hdfs] but
was:[pfile]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7618) TestDDLWithRemoteMetastoreSecondNamenode unit test failure
[
https://issues.apache.org/jira/browse/HIVE-7618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088295#comment-14088295
]
Jason Dere commented on HIVE-7618:
--
I think the point of the test is that the test case should work properly
without explicitly setting the table location. What I think we need here is a
way to determine if the StorageHandler needs a location or not. We can either
1. Add a new method to the StorageHandler interface (needsLocation()?), which
we use in place of StorageDescriptor.isSetLocation(). The downside to this
would be that this may break any existing 3rd party StorageHandlers, as
mentioned in HIVE-6853.
2. DDLTask already has some HBaseStorageHandler-specific logic in
showCreateTable(). We could make this logic into a needsLocation() routine
within DDLTask itself, and use it in a similar way as described in (1).
TestDDLWithRemoteMetastoreSecondNamenode unit test failure
--
Key: HIVE-7618
URL: https://issues.apache.org/jira/browse/HIVE-7618
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-7618.1.patch
Looks like TestDDLWithRemoteMetastoreSecondNamenode started failing after
HIVE-6584 was committed.
{noformat}
TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode:272-createTableAndCheck:201-createTableAndCheck:219
Table should be located in the second filesystem expected:[hdfs] but
was:[pfile]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7618) TestDDLWithRemoteMetastoreSecondNamenode unit test failure
[
https://issues.apache.org/jira/browse/HIVE-7618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088450#comment-14088450
]
Nick Dimiduk commented on HIVE-7618:
Option (1) would pave the way for migrating all these baked-in storage systems
over to the StorageHandler interface -- a healthy thing for the long-term
success of Hive, IMHO. That's a larger conversation though.
TestDDLWithRemoteMetastoreSecondNamenode unit test failure
--
Key: HIVE-7618
URL: https://issues.apache.org/jira/browse/HIVE-7618
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-7618.1.patch, HIVE-7618.2.patch
Looks like TestDDLWithRemoteMetastoreSecondNamenode started failing after
HIVE-6584 was committed.
{noformat}
TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode:272-createTableAndCheck:201-createTableAndCheck:219
Table should be located in the second filesystem expected:[hdfs] but
was:[pfile]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7618) TestDDLWithRemoteMetastoreSecondNamenode unit test failure
[
https://issues.apache.org/jira/browse/HIVE-7618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088453#comment-14088453
]
Nick Dimiduk commented on HIVE-7618:
Anyway, +1 for your patch v2. It's 80% of what we'd want for option (1) anyway.
Would be best if DDLTask didn't need that nasty hack.
TestDDLWithRemoteMetastoreSecondNamenode unit test failure
--
Key: HIVE-7618
URL: https://issues.apache.org/jira/browse/HIVE-7618
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-7618.1.patch, HIVE-7618.2.patch
Looks like TestDDLWithRemoteMetastoreSecondNamenode started failing after
HIVE-6584 was committed.
{noformat}
TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode:272-createTableAndCheck:201-createTableAndCheck:219
Table should be located in the second filesystem expected:[hdfs] but
was:[pfile]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7618) TestDDLWithRemoteMetastoreSecondNamenode unit test failure
[
https://issues.apache.org/jira/browse/HIVE-7618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088544#comment-14088544
]
Jason Dere commented on HIVE-7618:
--
If anyone else wants to chime in on if it's ok to break compatibility, I'd be
willing to try (1).
TestDDLWithRemoteMetastoreSecondNamenode unit test failure
--
Key: HIVE-7618
URL: https://issues.apache.org/jira/browse/HIVE-7618
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-7618.1.patch, HIVE-7618.2.patch
Looks like TestDDLWithRemoteMetastoreSecondNamenode started failing after
HIVE-6584 was committed.
{noformat}
TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode:272-createTableAndCheck:201-createTableAndCheck:219
Table should be located in the second filesystem expected:[hdfs] but
was:[pfile]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7618) TestDDLWithRemoteMetastoreSecondNamenode unit test failure
[
https://issues.apache.org/jira/browse/HIVE-7618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087080#comment-14087080
]
Nick Dimiduk commented on HIVE-7618:
This will cause failure in HIVE-6584. Reason being, StorageHandler tables don't
have a location, so this makeLocationQualified constructs locations that don't
exist. IIRC, this resulted in an HBase table being assigned a location in the
warehouse, which then confuses other pieces.
TestDDLWithRemoteMetastoreSecondNamenode unit test failure
--
Key: HIVE-7618
URL: https://issues.apache.org/jira/browse/HIVE-7618
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-7618.1.patch
Looks like TestDDLWithRemoteMetastoreSecondNamenode started failing after
HIVE-6584 was committed.
{noformat}
TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode:272-createTableAndCheck:201-createTableAndCheck:219
Table should be located in the second filesystem expected:[hdfs] but
was:[pfile]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HIVE-7618) TestDDLWithRemoteMetastoreSecondNamenode unit test failure
[
https://issues.apache.org/jira/browse/HIVE-7618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087082#comment-14087082
]
Nick Dimiduk commented on HIVE-7618:
Can the test be updated to explicitly set a foreign hdfs for its location?
TestDDLWithRemoteMetastoreSecondNamenode unit test failure
--
Key: HIVE-7618
URL: https://issues.apache.org/jira/browse/HIVE-7618
Project: Hive
Issue Type: Bug
Components: Tests
Reporter: Jason Dere
Assignee: Jason Dere
Attachments: HIVE-7618.1.patch
Looks like TestDDLWithRemoteMetastoreSecondNamenode started failing after
HIVE-6584 was committed.
{noformat}
TestDDLWithRemoteMetastoreSecondNamenode.testCreateTableWithIndexAndPartitionsNonDefaultNameNode:272-createTableAndCheck:201-createTableAndCheck:219
Table should be located in the second filesystem expected:[hdfs] but
was:[pfile]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
