[jira] [Commented] (HIVE-12239) Constants in hive.common.metrics.common.MetricsConstant are not final
[ https://issues.apache.org/jira/browse/HIVE-12239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972739#comment-14972739 ] Ashutosh Chauhan commented on HIVE-12239: - +1 > Constants in hive.common.metrics.common.MetricsConstant are not final > - > > Key: HIVE-12239 > URL: https://issues.apache.org/jira/browse/HIVE-12239 > Project: Hive > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Aleksei Statkevich >Assignee: Aleksei Statkevich >Priority: Trivial > Attachments: HIVE-12239.1.patch, HIVE-12239.patch > > > Constants defined in > org.apache.hadoop.hive.common.metrics.common.MetricsConstant are not marked > as final. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11582) Remove conf variable hive.mapred.supports.subdirectories
[ https://issues.apache.org/jira/browse/HIVE-11582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972768#comment-14972768 ] Ashutosh Chauhan commented on HIVE-11582: - +1 > Remove conf variable hive.mapred.supports.subdirectories > > > Key: HIVE-11582 > URL: https://issues.apache.org/jira/browse/HIVE-11582 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Chetna Chaudhari > Attachments: HIVE-11582.1.patch, HIVE-11582.2.patch > > > This configuration is redundant since MAPREDUCE-1501 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9013) Hive set command exposes metastore db password
[
https://issues.apache.org/jira/browse/HIVE-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972627#comment-14972627
]
Hive QA commented on HIVE-9013:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768485/HIVE-9013.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9688 tests executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-auto_join30.q-vector_data_types.q-filter_join_breaktask.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5776/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5776/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5776/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768485 - PreCommit-HIVE-TRUNK-Build
> Hive set command exposes metastore db password
> --
>
> Key: HIVE-9013
> URL: https://issues.apache.org/jira/browse/HIVE-9013
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.1
>Reporter: Binglin Chang
>Assignee: Binglin Chang
> Attachments: HIVE-9013.1.patch, HIVE-9013.2.patch, HIVE-9013.3.patch,
> HIVE-9013.4.patch, HIVE-9013.5.patch, HIVE-9013.5.patch
>
>
> When auth is enabled, we still need set command to set some variables(e.g.
> mapreduce.job.queuename), but set command alone also list all
> information(including vars in restrict list), this exposes like
> "javax.jdo.option.ConnectionPassword"
> I think conf var in the restrict list should also excluded from dump vars
> command.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11582) Remove conf variable hive.mapred.supports.subdirectories
[ https://issues.apache.org/jira/browse/HIVE-11582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetna Chaudhari updated HIVE-11582: Attachment: HIVE-11582.1.patch Removed conf variable. [~ashutoshc] Please review. > Remove conf variable hive.mapred.supports.subdirectories > > > Key: HIVE-11582 > URL: https://issues.apache.org/jira/browse/HIVE-11582 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan > Attachments: HIVE-11582.1.patch > > > This configuration is redundant since MAPREDUCE-1501 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-11582) Remove conf variable hive.mapred.supports.subdirectories
[ https://issues.apache.org/jira/browse/HIVE-11582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetna Chaudhari reassigned HIVE-11582: --- Assignee: Chetna Chaudhari > Remove conf variable hive.mapred.supports.subdirectories > > > Key: HIVE-11582 > URL: https://issues.apache.org/jira/browse/HIVE-11582 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Chetna Chaudhari > Attachments: HIVE-11582.1.patch > > > This configuration is redundant since MAPREDUCE-1501 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12239) Constants in hive.common.metrics.common.MetricsConstant are not final
[
https://issues.apache.org/jira/browse/HIVE-12239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972670#comment-14972670
]
Hive QA commented on HIVE-12239:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768493/HIVE-12239.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9696 tests executed
*Failed tests:*
{noformat}
TestWebHCatE2e - did not produce a TEST-*.xml file
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarDataNucleusUnCaching
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5777/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5777/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5777/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768493 - PreCommit-HIVE-TRUNK-Build
> Constants in hive.common.metrics.common.MetricsConstant are not final
> -
>
> Key: HIVE-12239
> URL: https://issues.apache.org/jira/browse/HIVE-12239
> Project: Hive
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Aleksei Statkevich
>Assignee: Aleksei Statkevich
>Priority: Trivial
> Attachments: HIVE-12239.1.patch, HIVE-12239.patch
>
>
> Constants defined in
> org.apache.hadoop.hive.common.metrics.common.MetricsConstant are not marked
> as final.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12245) Support column comments for an HBase backed table
[
https://issues.apache.org/jira/browse/HIVE-12245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972702#comment-14972702
]
Chaoyu Tang commented on HIVE-12245:
Patch has been posted on RB https://reviews.apache.org/r/39624/ , thanks in
advanced for review.
> Support column comments for an HBase backed table
> -
>
> Key: HIVE-12245
> URL: https://issues.apache.org/jira/browse/HIVE-12245
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
>Priority: Minor
> Attachments: HIVE-12245.patch
>
>
> Currently the column comments of an HBase backed table are always returned as
> "from deserializer". For example,
> {code}
> CREATE TABLE hbasetbl
> (key string comment 'It is key',
> state string comment 'It is state',
> country string comment 'It is country',
> country_id int comment 'It is country_id')
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> "hbase.columns.mapping" = "info:state,info:country,info:country_id"
> );
> hive> describe hbasetbl;
> key string from deserializer
> state string from deserializer
> country string from deserializer
> country_idint from deserializer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11582) Remove conf variable hive.mapred.supports.subdirectories
[
https://issues.apache.org/jira/browse/HIVE-11582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972718#comment-14972718
]
Ashutosh Chauhan commented on HIVE-11582:
-
* You can get rid of Driver::validateConfVariables() Its entirely redundant now.
* In Hive::checkPaths() you can get rid of entire {{if}} since now config
variable is always true, there is no way that if condition can ever become true.
> Remove conf variable hive.mapred.supports.subdirectories
>
>
> Key: HIVE-11582
> URL: https://issues.apache.org/jira/browse/HIVE-11582
> Project: Hive
> Issue Type: Task
> Components: Configuration
>Reporter: Ashutosh Chauhan
>Assignee: Chetna Chaudhari
> Attachments: HIVE-11582.1.patch
>
>
> This configuration is redundant since MAPREDUCE-1501
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11582) Remove conf variable hive.mapred.supports.subdirectories
[ https://issues.apache.org/jira/browse/HIVE-11582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetna Chaudhari updated HIVE-11582: Attachment: HIVE-11582.2.patch [~ashutoshc] Thanks for the review. I have worked on mentioned comments and updated patch. Please review. > Remove conf variable hive.mapred.supports.subdirectories > > > Key: HIVE-11582 > URL: https://issues.apache.org/jira/browse/HIVE-11582 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Chetna Chaudhari > Attachments: HIVE-11582.1.patch, HIVE-11582.2.patch > > > This configuration is redundant since MAPREDUCE-1501 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7575) GetTables thrift call is very slow
[ https://issues.apache.org/jira/browse/HIVE-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-7575: Attachment: HIVE-7575.2.patch.txt > GetTables thrift call is very slow > -- > > Key: HIVE-7575 > URL: https://issues.apache.org/jira/browse/HIVE-7575 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0 >Reporter: Ashu Pachauri >Assignee: Navis > Attachments: HIVE-7575.1.patch.txt, HIVE-7575.2.patch.txt > > > The GetTables thrift call takes a long time when the number of table is large. > With around 5000 tables, the call takes around 80 seconds compared to a "Show > Tables" query on the same HiveServer2 instance which takes 3-7 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12245) Support column comments for an HBase backed table
[
https://issues.apache.org/jira/browse/HIVE-12245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-12245:
---
Attachment: HIVE-12245.patch
> Support column comments for an HBase backed table
> -
>
> Key: HIVE-12245
> URL: https://issues.apache.org/jira/browse/HIVE-12245
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
>Priority: Minor
> Attachments: HIVE-12245.patch
>
>
> Currently the column comments of an HBase backed table are always returned as
> "from deserializer". For example,
> {code}
> CREATE TABLE hbasetbl
> (key string comment 'It is key',
> state string comment 'It is state',
> country string comment 'It is country',
> country_id int comment 'It is country_id')
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> "hbase.columns.mapping" = "info:state,info:country,info:country_id"
> );
> hive> describe hbasetbl;
> key string from deserializer
> state string from deserializer
> country string from deserializer
> country_idint from deserializer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12248) The rawStore used in DBTokenStore should be thread-safe
[
https://issues.apache.org/jira/browse/HIVE-12248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-12248:
---
Attachment: HIVE-12245.patch
> The rawStore used in DBTokenStore should be thread-safe
> ---
>
> Key: HIVE-12248
> URL: https://issues.apache.org/jira/browse/HIVE-12248
> Project: Hive
> Issue Type: Bug
> Components: Authentication
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
>
> A non-thread-safe implementation of RawStore, particularly ObjectStore, set
> in DBTokenStore is being shared by multi-threads, which causes the race
> condition in DataNuclues to access the backend DB.
> The DN PersistenceManager(PM) in ObjectStore is not thread safe, so
> DBTokenStore should use a ThreadLocal ObjectStore.
> Following errors might be root caused by the race condition in DN PM.
> {code}
> Object of type "org.apache.hadoop.hive.metastore.model.MDelegationToken" is
> detached. Detached objects cannot be used with this operation.
> org.datanucleus.exceptions.ObjectDetachedException: Object of type
> "org.apache.hadoop.hive.metastore.model.MDelegationToken" is detached.
> Detached objects cannot be used with this operation.
> at
> org.datanucleus.ExecutionContextImpl.assertNotDetached(ExecutionContextImpl.java:5728)
> at
> org.datanucleus.ExecutionContextImpl.retrieveObject(ExecutionContextImpl.java:1859)
> at
> org.datanucleus.ExecutionContextThreadedImpl.retrieveObject(ExecutionContextThreadedImpl.java:203)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:605)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:693)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:713)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getAllTokenIdentifiers(ObjectStore.java:6517)
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12248) The rawStore used in DBTokenStore should be thread-safe
[
https://issues.apache.org/jira/browse/HIVE-12248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-12248:
---
Attachment: (was: HIVE-12245.patch)
> The rawStore used in DBTokenStore should be thread-safe
> ---
>
> Key: HIVE-12248
> URL: https://issues.apache.org/jira/browse/HIVE-12248
> Project: Hive
> Issue Type: Bug
> Components: Authentication
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
>
> A non-thread-safe implementation of RawStore, particularly ObjectStore, set
> in DBTokenStore is being shared by multi-threads, which causes the race
> condition in DataNuclues to access the backend DB.
> The DN PersistenceManager(PM) in ObjectStore is not thread safe, so
> DBTokenStore should use a ThreadLocal ObjectStore.
> Following errors might be root caused by the race condition in DN PM.
> {code}
> Object of type "org.apache.hadoop.hive.metastore.model.MDelegationToken" is
> detached. Detached objects cannot be used with this operation.
> org.datanucleus.exceptions.ObjectDetachedException: Object of type
> "org.apache.hadoop.hive.metastore.model.MDelegationToken" is detached.
> Detached objects cannot be used with this operation.
> at
> org.datanucleus.ExecutionContextImpl.assertNotDetached(ExecutionContextImpl.java:5728)
> at
> org.datanucleus.ExecutionContextImpl.retrieveObject(ExecutionContextImpl.java:1859)
> at
> org.datanucleus.ExecutionContextThreadedImpl.retrieveObject(ExecutionContextThreadedImpl.java:203)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.jdoRetrieve(JDOPersistenceManager.java:605)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:693)
> at
> org.datanucleus.api.jdo.JDOPersistenceManager.retrieveAll(JDOPersistenceManager.java:713)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getAllTokenIdentifiers(ObjectStore.java:6517)
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12213) Investigating the test failure TestHCatClient.testTableSchemaPropagation
[
https://issues.apache.org/jira/browse/HIVE-12213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972777#comment-14972777
]
Hive QA commented on HIVE-12213:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768491/HIVE-12213.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9687 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.initializationError
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5778/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5778/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5778/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768491 - PreCommit-HIVE-TRUNK-Build
> Investigating the test failure TestHCatClient.testTableSchemaPropagation
>
>
> Key: HIVE-12213
> URL: https://issues.apache.org/jira/browse/HIVE-12213
> Project: Hive
> Issue Type: Test
> Components: Test
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aleksei Statkevich
>Priority: Minor
> Attachments: HIVE-12213.2.patch, HIVE-12213.3.patch,
> HIVE-12213.4.patch, HIVE-12213.patch, HIVE-12231.1.patch
>
>
> The test has been failing for some time with following error.
> {noformat}
> Error Message
> Table after deserialization should have been identical to sourceTable.
> expected:<[TABLE_PROPERTIES]> but was:<[]>
> Stacktrace
> java.lang.AssertionError: Table after deserialization should have been
> identical to sourceTable. expected:<[TABLE_PROPERTIES]> but was:<[]>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation(TestHCatClient.java:1065)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11540) Too many delta files during Compaction - OOM
[
https://issues.apache.org/jira/browse/HIVE-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-11540:
--
Attachment: HIVE-11540.6.patch
> Too many delta files during Compaction - OOM
>
>
> Key: HIVE-11540
> URL: https://issues.apache.org/jira/browse/HIVE-11540
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Nivin Mathew
>Assignee: Eugene Koifman
> Attachments: HIVE-11540.3.patch, HIVE-11540.4.patch,
> HIVE-11540.6.patch, HIVE-11540.patch
>
>
> Hello,
> I am streaming weblogs to Kafka and then to Flume 1.6 using a Hive sink, with
> an average of 20 million records a day. I have 5 compactors running at
> various times (30m/5m/5s), no matter what time I give, the compactors seem to
> run out of memory cleaning up a couple thousand delta files and ultimately
> falls behind compacting/cleaning delta files. Any suggestions on what I can
> do to improve performance? Or can Hive streaming not handle this kind of load?
> I used this post as reference:
> http://henning.kropponline.de/2015/05/19/hivesink-for-flume/
> {noformat}
> 2015-08-12 15:05:01,197 FATAL [main] org.apache.hadoop.mapred.YarnChild:
> Error running child : java.lang.OutOfMemoryError: Direct buffer memory
> Max block location exceeded for split: CompactorInputSplit{base:
> hdfs://Dev01HWNameService/user/hive/warehouse/weblogs.db/dt=15-08-12/base_1056406,
> bucket: 0, length: 6493042, deltas: [delta_1056407_1056408,
> delta_1056409_1056410, delta_1056411_1056412, delta_1056413_1056414,
> delta_1056415_1056416, delta_1056417_1056418,…
> , delta_1074039_1074040, delta_1074041_1074042, delta_1074043_1074044,
> delta_1074045_1074046, delta_1074047_1074048, delta_1074049_1074050,
> delta_1074051_1074052]} splitsize: 8772 maxsize: 10
> 2015-08-12 15:34:25,271 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(198)) - number of
> splits:3
> 2015-08-12 15:34:25,367 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.JobSubmitter (JobSubmitter.java:printTokens(287)) - Submitting
> tokens for job: job_1439397150426_0068
> 2015-08-12 15:34:25,603 INFO [upladevhwd04v.researchnow.com-18]:
> impl.YarnClientImpl (YarnClientImpl.java:submitApplication(274)) - Submitted
> application application_1439397150426_0068
> 2015-08-12 15:34:25,610 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:submit(1294)) - The url to track the job:
> http://upladevhwd02v.researchnow.com:8088/proxy/application_1439397150426_0068/
> 2015-08-12 15:34:25,611 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1339)) - Running job:
> job_1439397150426_0068
> 2015-08-12 15:34:30,170 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:33,756 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1360)) - Job
> job_1439397150426_0068 running in uber mode : false
> 2015-08-12 15:34:33,757 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 0% reduce 0%
> 2015-08-12 15:34:35,147 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:40,155 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:45,184 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:50,201 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:55,256 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:35:00,205 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:35:02,975 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 33% reduce 0%
> 2015-08-12 15:35:02,982 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:printTaskEvents(1406)) - Task Id :
> attempt_1439397150426_0068_m_00_0, Status : FAILED
> 2015-08-12 15:35:03,000 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:printTaskEvents(1406)) - Task Id :
> attempt_1439397150426_0068_m_01_0, Status : FAILED
> 2015-08-12 15:35:04,008 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job
[jira] [Updated] (HIVE-12259) Command containing semicolon is broken in Beeline
[
https://issues.apache.org/jira/browse/HIVE-12259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-12259:
---
Attachment: HIVE-12259.patch
1. The SQLLine commands (!cmd) in Beeline containing ";" are broken.
2. The connection URLs having ";" included do not work either for the same
reason.
> Command containing semicolon is broken in Beeline
> -
>
> Key: HIVE-12259
> URL: https://issues.apache.org/jira/browse/HIVE-12259
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-12259.patch
>
>
> The Beeline command (!cmd) containing semicolon is broken.
> For example:
> !connect jdbc:hive2://localhost:10001/default;principal=hive/xyz@realm.com
> is broken because the included ";" makes it not to run with
> execCommandWithPrefix as a whole command.
> {code}
> if (line.startsWith(COMMAND_PREFIX) && !line.contains(";")) {
> // handle the case "!cmd" for beeline
> return execCommandWithPrefix(line);
> } else {
> return commands.sql(line, getOpts().getEntireLineAsCommand());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7575) GetTables thrift call is very slow
[
https://issues.apache.org/jira/browse/HIVE-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972869#comment-14972869
]
Hive QA commented on HIVE-7575:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768525/HIVE-7575.2.patch.txt
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 9701 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testDataDeletion
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat.testOutputFormat
org.apache.hive.jdbc.TestJdbcWithMiniMr.testTempTable
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.jdbc.authorization.TestJdbcMetadataApiAuth.testMetaApiAllowed
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5780/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5780/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5780/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768525 - PreCommit-HIVE-TRUNK-Build
> GetTables thrift call is very slow
> --
>
> Key: HIVE-7575
> URL: https://issues.apache.org/jira/browse/HIVE-7575
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Ashu Pachauri
>Assignee: Navis
> Attachments: HIVE-7575.1.patch.txt, HIVE-7575.2.patch.txt
>
>
> The GetTables thrift call takes a long time when the number of table is large.
> With around 5000 tables, the call takes around 80 seconds compared to a "Show
> Tables" query on the same HiveServer2 instance which takes 3-7 seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11755) Incorrect method called with Kerberos enabled in AccumuloStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-11755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-11755:
Component/s: Accumulo Storage Handler
> Incorrect method called with Kerberos enabled in AccumuloStorageHandler
> ---
>
> Key: HIVE-11755
> URL: https://issues.apache.org/jira/browse/HIVE-11755
> Project: Hive
> Issue Type: Bug
> Components: Accumulo Storage Handler
>Affects Versions: 1.2.1
>Reporter: Josh Elser
>Assignee: Josh Elser
> Fix For: 2.0.0
>
> Attachments: HIVE-11755.001.patch, HIVE-11755.002.patch,
> HIVE-11755.003.patch
>
>
> The following exception was noticed in testing out the
> AccumuloStorageHandler's OutputFormat:
> {noformat}
> java.lang.IllegalStateException: Connector info for AccumuloOutputFormat can
> only be set once per job
> at
> org.apache.accumulo.core.client.mapreduce.lib.impl.ConfiguratorBase.setConnectorInfo(ConfiguratorBase.java:146)
> at
> org.apache.accumulo.core.client.mapred.AccumuloOutputFormat.setConnectorInfo(AccumuloOutputFormat.java:125)
> at
> org.apache.hadoop.hive.accumulo.mr.HiveAccumuloTableOutputFormat.configureAccumuloOutputFormat(HiveAccumuloTableOutputFormat.java:95)
> at
> org.apache.hadoop.hive.accumulo.mr.HiveAccumuloTableOutputFormat.checkOutputSpecs(HiveAccumuloTableOutputFormat.java:51)
> at
> org.apache.hadoop.hive.ql.io.HivePassThroughOutputFormat.checkOutputSpecs(HivePassThroughOutputFormat.java:46)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.checkOutputSpecs(FileSinkOperator.java:1124)
> at
> org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl.checkOutputSpecs(HiveOutputFormatImpl.java:67)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
> at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:431)
> at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
> at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:409)
> at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:425)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:714)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Job Submission failed with exception
> 'java.lang.IllegalStateException(Connector info for AccumuloOutputFormat can
> only be set once per job)'
> {noformat}
> The OutputFormat implementation already had a method in place to account for
> this exception but the
[jira] [Commented] (HIVE-12259) Command containing semicolon is broken in Beeline
[
https://issues.apache.org/jira/browse/HIVE-12259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972909#comment-14972909
]
Chaoyu Tang commented on HIVE-12259:
Patch has been posted on https://reviews.apache.org/r/39626/ . [~szehon] and
[~Ferd] could you review that? Thanks
> Command containing semicolon is broken in Beeline
> -
>
> Key: HIVE-12259
> URL: https://issues.apache.org/jira/browse/HIVE-12259
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-12259.patch
>
>
> The Beeline command (!cmd) containing semicolon is broken.
> For example:
> !connect jdbc:hive2://localhost:10001/default;principal=hive/xyz@realm.com
> is broken because the included ";" makes it not to run with
> execCommandWithPrefix as a whole command.
> {code}
> if (line.startsWith(COMMAND_PREFIX) && !line.contains(";")) {
> // handle the case "!cmd" for beeline
> return execCommandWithPrefix(line);
> } else {
> return commands.sql(line, getOpts().getEntireLineAsCommand());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11100) Beeline should escape semi-colon in queries
[ https://issues.apache.org/jira/browse/HIVE-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972930#comment-14972930 ] Lefty Leverenz commented on HIVE-11100: --- Okay, thanks [~ctang.ma]. > Beeline should escape semi-colon in queries > --- > > Key: HIVE-11100 > URL: https://issues.apache.org/jira/browse/HIVE-11100 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.2.0, 1.1.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11100.patch > > > Beeline should escape the semicolon in queries. for example, the query like > followings: > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY ';' LINES TERMINATED BY '\n'; > or > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY '\;' LINES TERMINATED BY '\n'; > both failed. > But the 2nd query with semicolon escaped with "\" works in CLI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11981) ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)
[ https://issues.apache.org/jira/browse/HIVE-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972835#comment-14972835 ] Matt McCline commented on HIVE-11981: - [~prasanth_j] I removed some of the intrusive changes. Now, no changes to ReaderImpl and one line change to RecordReaderImpl. Common code (new class SchemaEvolution) for determining "schema on read" is called from two places: RecordReaderFactory and OrcRawRecordMerger. TreeReaderFactory still has code for determining which columns to null since it seems to be in the best position to know how to do it instead of RecordReaderImpl... Latest Hive QA run appears to be successful. > ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized) > -- > > Key: HIVE-11981 > URL: https://issues.apache.org/jira/browse/HIVE-11981 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-11981.01.patch, HIVE-11981.02.patch, > HIVE-11981.03.patch, HIVE-11981.05.patch, HIVE-11981.06.patch, > HIVE-11981.07.patch, HIVE-11981.08.patch, HIVE-11981.09.patch, > HIVE-11981.091.patch, ORC Schema Evolution Issues.docx > > > High priority issues with schema evolution for the ORC file format. > Schema evolution here is limited to adding new columns and a few cases of > column type-widening (e.g. int to bigint). > Renaming columns, deleting column, moving columns and other schema evolution > were not pursued due to lack of importance and lack of time. Also, it > appears a much more sophisticated metadata would be needed to support them. > The biggest issues for users have been adding new columns for ACID table > (HIVE-11421 Support Schema evolution for ACID tables) and vectorization > (HIVE-10598 Vectorization borks when column is added to table). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11100) Beeline should escape semi-colon in queries
[ https://issues.apache.org/jira/browse/HIVE-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972826#comment-14972826 ] Chaoyu Tang commented on HIVE-11100: I think that it is most likely a regression from recent HIVE-11640. > Beeline should escape semi-colon in queries > --- > > Key: HIVE-11100 > URL: https://issues.apache.org/jira/browse/HIVE-11100 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.2.0, 1.1.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11100.patch > > > Beeline should escape the semicolon in queries. for example, the query like > followings: > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY ';' LINES TERMINATED BY '\n'; > or > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY '\;' LINES TERMINATED BY '\n'; > both failed. > But the 2nd query with semicolon escaped with "\" works in CLI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11981) ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)
[
https://issues.apache.org/jira/browse/HIVE-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972831#comment-14972831
]
Hive QA commented on HIVE-11981:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768497/HIVE-11981.091.patch
{color:green}SUCCESS:{color} +1 due to 16 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9722 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5779/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5779/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5779/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768497 - PreCommit-HIVE-TRUNK-Build
> ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)
> --
>
> Key: HIVE-11981
> URL: https://issues.apache.org/jira/browse/HIVE-11981
> Project: Hive
> Issue Type: Bug
> Components: Hive, Transactions
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-11981.01.patch, HIVE-11981.02.patch,
> HIVE-11981.03.patch, HIVE-11981.05.patch, HIVE-11981.06.patch,
> HIVE-11981.07.patch, HIVE-11981.08.patch, HIVE-11981.09.patch,
> HIVE-11981.091.patch, ORC Schema Evolution Issues.docx
>
>
> High priority issues with schema evolution for the ORC file format.
> Schema evolution here is limited to adding new columns and a few cases of
> column type-widening (e.g. int to bigint).
> Renaming columns, deleting column, moving columns and other schema evolution
> were not pursued due to lack of importance and lack of time. Also, it
> appears a much more sophisticated metadata would be needed to support them.
> The biggest issues for users have been adding new columns for ACID table
> (HIVE-11421 Support Schema evolution for ACID tables) and vectorization
> (HIVE-10598 Vectorization borks when column is added to table).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12039) Fix TestSSL#testSSLVersion
[ https://issues.apache.org/jira/browse/HIVE-12039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972867#comment-14972867 ] Ashutosh Chauhan commented on HIVE-12039: - +1 > Fix TestSSL#testSSLVersion > --- > > Key: HIVE-12039 > URL: https://issues.apache.org/jira/browse/HIVE-12039 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.3.0, 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-12039.1.patch > > > Looks like it's only run on Linux and failing after HIVE-11720. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11755) Incorrect method called with Kerberos enabled in AccumuloStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-11755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972880#comment-14972880
]
Josh Elser commented on HIVE-11755:
---
Ooo, thanks [~ashutoshc]. This had totally fallen of my radar again :)
> Incorrect method called with Kerberos enabled in AccumuloStorageHandler
> ---
>
> Key: HIVE-11755
> URL: https://issues.apache.org/jira/browse/HIVE-11755
> Project: Hive
> Issue Type: Bug
> Components: Accumulo Storage Handler
>Affects Versions: 1.2.1
>Reporter: Josh Elser
>Assignee: Josh Elser
> Fix For: 2.0.0
>
> Attachments: HIVE-11755.001.patch, HIVE-11755.002.patch,
> HIVE-11755.003.patch
>
>
> The following exception was noticed in testing out the
> AccumuloStorageHandler's OutputFormat:
> {noformat}
> java.lang.IllegalStateException: Connector info for AccumuloOutputFormat can
> only be set once per job
> at
> org.apache.accumulo.core.client.mapreduce.lib.impl.ConfiguratorBase.setConnectorInfo(ConfiguratorBase.java:146)
> at
> org.apache.accumulo.core.client.mapred.AccumuloOutputFormat.setConnectorInfo(AccumuloOutputFormat.java:125)
> at
> org.apache.hadoop.hive.accumulo.mr.HiveAccumuloTableOutputFormat.configureAccumuloOutputFormat(HiveAccumuloTableOutputFormat.java:95)
> at
> org.apache.hadoop.hive.accumulo.mr.HiveAccumuloTableOutputFormat.checkOutputSpecs(HiveAccumuloTableOutputFormat.java:51)
> at
> org.apache.hadoop.hive.ql.io.HivePassThroughOutputFormat.checkOutputSpecs(HivePassThroughOutputFormat.java:46)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.checkOutputSpecs(FileSinkOperator.java:1124)
> at
> org.apache.hadoop.hive.ql.io.HiveOutputFormatImpl.checkOutputSpecs(HiveOutputFormatImpl.java:67)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:268)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:575)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:570)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:570)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:561)
> at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:431)
> at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:311)
> at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:409)
> at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:425)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:714)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Job Submission failed with exception
> 'java.lang.IllegalStateException(Connector info for AccumuloOutputFormat can
> only be set once per job)'
> {noformat}
> The OutputFormat implementation already had a
[jira] [Commented] (HIVE-11582) Remove conf variable hive.mapred.supports.subdirectories
[
https://issues.apache.org/jira/browse/HIVE-11582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972902#comment-14972902
]
Hive QA commented on HIVE-11582:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768542/HIVE-11582.2.patch
{color:green}SUCCESS:{color} +1 due to 96 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9701 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_invalid_config1
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_invalid_config2
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5781/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5781/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5781/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768542 - PreCommit-HIVE-TRUNK-Build
> Remove conf variable hive.mapred.supports.subdirectories
>
>
> Key: HIVE-11582
> URL: https://issues.apache.org/jira/browse/HIVE-11582
> Project: Hive
> Issue Type: Task
> Components: Configuration
>Reporter: Ashutosh Chauhan
>Assignee: Chetna Chaudhari
> Attachments: HIVE-11582.1.patch, HIVE-11582.2.patch
>
>
> This configuration is redundant since MAPREDUCE-1501
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11100) Beeline should escape semi-colon in queries
[ https://issues.apache.org/jira/browse/HIVE-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972913#comment-14972913 ] Chaoyu Tang commented on HIVE-11100: The fix to the issue has been provided in HIVE-12259. [~leftylev], since HIVE-11640 was only committed to 2.0.0 dev branch, we do not need to document the issue for other branches. Thanks > Beeline should escape semi-colon in queries > --- > > Key: HIVE-11100 > URL: https://issues.apache.org/jira/browse/HIVE-11100 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 1.2.0, 1.1.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11100.patch > > > Beeline should escape the semicolon in queries. for example, the query like > followings: > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY ';' LINES TERMINATED BY '\n'; > or > CREATE TABLE beeline_tb (c1 int, c2 string) ROW FORMAT DELIMITED FIELDS > TERMINATED BY '\;' LINES TERMINATED BY '\n'; > both failed. > But the 2nd query with semicolon escaped with "\" works in CLI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972820#comment-14972820 ] Ashutosh Chauhan commented on HIVE-12244: - Thanks, [~alina.abramova] Would you like to make status Patch Available, so that Hive QA can run on it. > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Alina Abramova >Priority: Minor > Labels: patch > Attachments: HIVE-12244.1.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12059) Clean up reference to deprecated constants in AvroSerdeUtils
[ https://issues.apache.org/jira/browse/HIVE-12059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972865#comment-14972865 ] Ashutosh Chauhan commented on HIVE-12059: - +1 > Clean up reference to deprecated constants in AvroSerdeUtils > > > Key: HIVE-12059 > URL: https://issues.apache.org/jira/browse/HIVE-12059 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Aaron Dossett >Assignee: Aaron Dossett >Priority: Minor > Attachments: HIVE-12059.patch > > > AvroSerdeUtils contains several deprecated String constants that are used by > other Hive modules. Those should be cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11540) Too many delta files during Compaction - OOM
[
https://issues.apache.org/jira/browse/HIVE-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-11540:
--
Labels: TODOC2.0 (was: )
> Too many delta files during Compaction - OOM
>
>
> Key: HIVE-11540
> URL: https://issues.apache.org/jira/browse/HIVE-11540
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Nivin Mathew
>Assignee: Eugene Koifman
> Labels: TODOC2.0
> Attachments: HIVE-11540.3.patch, HIVE-11540.4.patch,
> HIVE-11540.6.patch, HIVE-11540.patch
>
>
> Hello,
> I am streaming weblogs to Kafka and then to Flume 1.6 using a Hive sink, with
> an average of 20 million records a day. I have 5 compactors running at
> various times (30m/5m/5s), no matter what time I give, the compactors seem to
> run out of memory cleaning up a couple thousand delta files and ultimately
> falls behind compacting/cleaning delta files. Any suggestions on what I can
> do to improve performance? Or can Hive streaming not handle this kind of load?
> I used this post as reference:
> http://henning.kropponline.de/2015/05/19/hivesink-for-flume/
> {noformat}
> 2015-08-12 15:05:01,197 FATAL [main] org.apache.hadoop.mapred.YarnChild:
> Error running child : java.lang.OutOfMemoryError: Direct buffer memory
> Max block location exceeded for split: CompactorInputSplit{base:
> hdfs://Dev01HWNameService/user/hive/warehouse/weblogs.db/dt=15-08-12/base_1056406,
> bucket: 0, length: 6493042, deltas: [delta_1056407_1056408,
> delta_1056409_1056410, delta_1056411_1056412, delta_1056413_1056414,
> delta_1056415_1056416, delta_1056417_1056418,…
> , delta_1074039_1074040, delta_1074041_1074042, delta_1074043_1074044,
> delta_1074045_1074046, delta_1074047_1074048, delta_1074049_1074050,
> delta_1074051_1074052]} splitsize: 8772 maxsize: 10
> 2015-08-12 15:34:25,271 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(198)) - number of
> splits:3
> 2015-08-12 15:34:25,367 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.JobSubmitter (JobSubmitter.java:printTokens(287)) - Submitting
> tokens for job: job_1439397150426_0068
> 2015-08-12 15:34:25,603 INFO [upladevhwd04v.researchnow.com-18]:
> impl.YarnClientImpl (YarnClientImpl.java:submitApplication(274)) - Submitted
> application application_1439397150426_0068
> 2015-08-12 15:34:25,610 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:submit(1294)) - The url to track the job:
> http://upladevhwd02v.researchnow.com:8088/proxy/application_1439397150426_0068/
> 2015-08-12 15:34:25,611 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1339)) - Running job:
> job_1439397150426_0068
> 2015-08-12 15:34:30,170 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:33,756 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1360)) - Job
> job_1439397150426_0068 running in uber mode : false
> 2015-08-12 15:34:33,757 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 0% reduce 0%
> 2015-08-12 15:34:35,147 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:40,155 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:45,184 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:50,201 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:34:55,256 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:35:00,205 INFO [Thread-7]: compactor.Initiator
> (Initiator.java:run(88)) - Checking to see if we should compact
> weblogs.vop_hs.dt=15-08-12
> 2015-08-12 15:35:02,975 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 33% reduce 0%
> 2015-08-12 15:35:02,982 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:printTaskEvents(1406)) - Task Id :
> attempt_1439397150426_0068_m_00_0, Status : FAILED
> 2015-08-12 15:35:03,000 INFO [upladevhwd04v.researchnow.com-18]:
> mapreduce.Job (Job.java:printTaskEvents(1406)) - Task Id :
> attempt_1439397150426_0068_m_01_0, Status : FAILED
> 2015-08-12 15:35:04,008 INFO
[jira] [Updated] (HIVE-7575) GetTables thrift call is very slow
[ https://issues.apache.org/jira/browse/HIVE-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-7575: Attachment: HIVE-7575.3.patch.txt Fixed test fails > GetTables thrift call is very slow > -- > > Key: HIVE-7575 > URL: https://issues.apache.org/jira/browse/HIVE-7575 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0 >Reporter: Ashu Pachauri >Assignee: Navis > Attachments: HIVE-7575.1.patch.txt, HIVE-7575.2.patch.txt, > HIVE-7575.3.patch.txt > > > The GetTables thrift call takes a long time when the number of table is large. > With around 5000 tables, the call takes around 80 seconds compared to a "Show > Tables" query on the same HiveServer2 instance which takes 3-7 seconds. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11927) Implement/Enable constant related optimization rules in Calcite: enable HiveReduceExpressionsRule to fold constants
[ https://issues.apache.org/jira/browse/HIVE-11927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-11927: --- Attachment: HIVE-11927.06.patch > Implement/Enable constant related optimization rules in Calcite: enable > HiveReduceExpressionsRule to fold constants > --- > > Key: HIVE-11927 > URL: https://issues.apache.org/jira/browse/HIVE-11927 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-11927.01.patch, HIVE-11927.02.patch, > HIVE-11927.03.patch, HIVE-11927.04.patch, HIVE-11927.05.patch, > HIVE-11927.06.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12259) Command containing semicolon is broken in Beeline
[
https://issues.apache.org/jira/browse/HIVE-12259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972968#comment-14972968
]
Chaoyu Tang commented on HIVE-12259:
The four failed tests seems not related to this patch.
> Command containing semicolon is broken in Beeline
> -
>
> Key: HIVE-12259
> URL: https://issues.apache.org/jira/browse/HIVE-12259
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
> Attachments: HIVE-12259.patch
>
>
> The Beeline command (!cmd) containing semicolon is broken.
> For example:
> !connect jdbc:hive2://localhost:10001/default;principal=hive/xyz@realm.com
> is broken because the included ";" makes it not to run with
> execCommandWithPrefix as a whole command.
> {code}
> if (line.startsWith(COMMAND_PREFIX) && !line.contains(";")) {
> // handle the case "!cmd" for beeline
> return execCommandWithPrefix(line);
> } else {
> return commands.sql(line, getOpts().getEntireLineAsCommand());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12245) Support column comments for an HBase backed table
[
https://issues.apache.org/jira/browse/HIVE-12245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972969#comment-14972969
]
Chaoyu Tang commented on HIVE-12245:
The precommit build did not run. Reattach the patch to kick off the build.
> Support column comments for an HBase backed table
> -
>
> Key: HIVE-12245
> URL: https://issues.apache.org/jira/browse/HIVE-12245
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
>Priority: Minor
> Attachments: HIVE-12245.patch, HIVE-12245.patch
>
>
> Currently the column comments of an HBase backed table are always returned as
> "from deserializer". For example,
> {code}
> CREATE TABLE hbasetbl
> (key string comment 'It is key',
> state string comment 'It is state',
> country string comment 'It is country',
> country_id int comment 'It is country_id')
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> "hbase.columns.mapping" = "info:state,info:country,info:country_id"
> );
> hive> describe hbasetbl;
> key string from deserializer
> state string from deserializer
> country string from deserializer
> country_idint from deserializer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12245) Support column comments for an HBase backed table
[
https://issues.apache.org/jira/browse/HIVE-12245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chaoyu Tang updated HIVE-12245:
---
Attachment: HIVE-12245.patch
> Support column comments for an HBase backed table
> -
>
> Key: HIVE-12245
> URL: https://issues.apache.org/jira/browse/HIVE-12245
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler
>Reporter: Chaoyu Tang
>Assignee: Chaoyu Tang
>Priority: Minor
> Attachments: HIVE-12245.patch, HIVE-12245.patch
>
>
> Currently the column comments of an HBase backed table are always returned as
> "from deserializer". For example,
> {code}
> CREATE TABLE hbasetbl
> (key string comment 'It is key',
> state string comment 'It is state',
> country string comment 'It is country',
> country_id int comment 'It is country_id')
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> "hbase.columns.mapping" = "info:state,info:country,info:country_id"
> );
> hive> describe hbasetbl;
> key string from deserializer
> state string from deserializer
> country string from deserializer
> country_idint from deserializer
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11981) ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)
[ https://issues.apache.org/jira/browse/HIVE-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-11981: Attachment: HIVE-11981.091.patch > ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized) > -- > > Key: HIVE-11981 > URL: https://issues.apache.org/jira/browse/HIVE-11981 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-11981.01.patch, HIVE-11981.02.patch, > HIVE-11981.03.patch, HIVE-11981.05.patch, HIVE-11981.06.patch, > HIVE-11981.07.patch, HIVE-11981.08.patch, HIVE-11981.09.patch, > HIVE-11981.091.patch, ORC Schema Evolution Issues.docx > > > High priority issues with schema evolution for the ORC file format. > Schema evolution here is limited to adding new columns and a few cases of > column type-widening (e.g. int to bigint). > Renaming columns, deleting column, moving columns and other schema evolution > were not pursued due to lack of importance and lack of time. Also, it > appears a much more sophisticated metadata would be needed to support them. > The biggest issues for users have been adding new columns for ACID table > (HIVE-11421 Support Schema evolution for ACID tables) and vectorization > (HIVE-10598 Vectorization borks when column is added to table). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12078) LLAP: document config settings
[ https://issues.apache.org/jira/browse/HIVE-12078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972446#comment-14972446 ] Lefty Leverenz commented on HIVE-12078: --- HIVE-12178 revises the description for *hive.llap.io.lrfu.lambda*. All of these configuration parameters need to be documented in the wiki for the 2.0.0 release: * [Hive Configuration Properties (needs a new LLAP section) | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties] > LLAP: document config settings > -- > > Key: HIVE-12078 > URL: https://issues.apache.org/jira/browse/HIVE-12078 > Project: Hive > Issue Type: Bug >Reporter: Lefty Leverenz >Assignee: Sergey Shelukhin > Labels: TODOC2.0 > Fix For: llap > > Attachments: HIVE-12078.patch > > > From HIVE-12060: > There's a typo in the description of > hive.tez.input.generate.consistent.splits: "Whether to generate consisten > split" – need "t" for consistent. > Several of the new hive.llap.* configs don't have descriptions. Are they for > internal use only? > Please add newlines (\n) in the description of > hive.llap.queue.metrics.percentiles.intervals and keep the indentation > identical for all three lines of the description. (And to pick a nit, a few > config description indentations are off by one character, including that one.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12258) read/write into same partitioned table + concurrency = deadlock
[ https://issues.apache.org/jira/browse/HIVE-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Furcy Pin updated HIVE-12258: - Description: When hive.support.concurrency is enabled if you launch a query that reads data from a partition and writes data into another partition of the same table, it creates a deadlock. The worse part is that once the deadlock is active, you can't query the table until it times out. * How to reproduce : ``` CREATE TABLE test_table (id INT) PARTITIONED BY (part STRING) ; INSERT INTO TABLE test_table PARTITION (part="test") VALUES (1), (2), (3), (4) ; INSERT OVERWRITE TABLE test_table PARTITION (part="test2") SELECT id FROM test_table WHERE part="test1"; ``` Nothing happens, and when doing a SHOW LOCKS in another terminal we get : ``` SHOW LOCKS ; +--+---+++-+--+-+-++ | lockid | database | table | partition | lock_state | lock_type| transaction_id | last_heartbeat | acquired_at | +--+---+++-+--+-+-++ | 3765 | default | test_table | NULL | WAITING | SHARED_READ | NULL| 1440603633148 | NULL | | 3765 | default | test_table | part=test2 | WAITING | EXCLUSIVE| NULL| 1440603633148 | NULL | +--+---+++-+--+-+-++ ``` This was tested on Hive 1.1.0-cdh5.4.2 but I believe the bug is still presents in 1.2.1. I could not reproduce it easily locally because it requires a pseudo-distributed setup with zookeeper to have concurrency enabled. >From looking at the code I believe the problem comes from the >EmbeddedLockManager method `public List lock(List objs, int numRetriesForLock, long sleepTime)` that keeps trying to acquire two incompatible locks, and ends up failing after hive.lock.numretries*hive.lock.sleep.between.retries which by defaut is 100*60s = 100 minutes. was: When hive.support.concurrency is enabled if you launch a query that reads data from a partition and writes data into another partition of the same table, it creates a deadlock. The worse part is that once the deadlock is active, you can't query the table until it times out. ## How to reproduce : ```sql CREATE TABLE test_table (id INT) PARTITIONED BY (part STRING) ; INSERT INTO TABLE test_table PARTITION (part="test") VALUES (1), (2), (3), (4) ; INSERT OVERWRITE TABLE test_table PARTITION (part="test2") SELECT id FROM test_table WHERE part="test1"; ``` Nothing happens, and when doing a SHOW LOCKS in another terminal we get : ``` SHOW LOCKS ; +--+---+++-+--+-+-++ | lockid | database | table | partition | lock_state | lock_type| transaction_id | last_heartbeat | acquired_at | +--+---+++-+--+-+-++ | 3765 | default | test_table | NULL | WAITING | SHARED_READ | NULL| 1440603633148 | NULL | | 3765 | default | test_table | part=test2 | WAITING | EXCLUSIVE| NULL| 1440603633148 | NULL | +--+---+++-+--+-+-++ ``` This was tested on Hive 1.1.0-cdh5.4.2 but I believe the bug is still presents in 1.2.1. I could not reproduce it easily locally because it requires a pseudo-distributed setup with zookeeper to have concurrency enabled. >From looking at the code I believe the problem comes from the >EmbeddedLockManager method `public List lock(List objs, int numRetriesForLock, long sleepTime)` that keeps trying to acquire two incompatible locks, and ends up failing after hive.lock.numretries*hive.lock.sleep.between.retries which by defaut is 100*60s = 100 minutes. > read/write into same partitioned table + concurrency = deadlock > --- > > Key: HIVE-12258 > URL: https://issues.apache.org/jira/browse/HIVE-12258 > Project: Hive > Issue Type: Bug >Reporter: Furcy Pin > > When hive.support.concurrency is enabled if you launch a query that reads > data from a partition and writes data into another partition of the same > table, > it creates a deadlock. > The worse part is that once the deadlock is active, you can't query the table > until it times out. > * How to reproduce : > ``` > CREATE TABLE test_table (id INT) > PARTITIONED BY (part STRING) > ; >
[jira] [Updated] (HIVE-12258) read/write into same partitioned table + concurrency = deadlock
[ https://issues.apache.org/jira/browse/HIVE-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Furcy Pin updated HIVE-12258: - Description: When hive.support.concurrency is enabled if you launch a query that reads data from a partition and writes data into another partition of the same table, it creates a deadlock. The worse part is that once the deadlock is active, you can't query the table until it times out. * How to reproduce : CREATE TABLE test_table (id INT) PARTITIONED BY (part STRING) ; INSERT INTO TABLE test_table PARTITION (part="test") VALUES (1), (2), (3), (4) ; INSERT OVERWRITE TABLE test_table PARTITION (part="test2") SELECT id FROM test_table WHERE part="test1"; Nothing happens, and when doing a SHOW LOCKS in another terminal we get : +--+---+++-+--+-+-++ | lockid | database | table | partition | lock_state | lock_type| transaction_id | last_heartbeat | acquired_at | +--+---+++-+--+-+-++ | 3765 | default | test_table | NULL | WAITING | SHARED_READ | NULL| 1440603633148 | NULL | | 3765 | default | test_table | part=test2 | WAITING | EXCLUSIVE| NULL| 1440603633148 | NULL | +--+---+++-+--+-+-++ This was tested on Hive 1.1.0-cdh5.4.2 but I believe the bug is still presents in 1.2.1. I could not reproduce it easily locally because it requires a pseudo-distributed setup with zookeeper to have concurrency enabled. >From looking at the code I believe the problem comes from the >EmbeddedLockManager method `public List lock(List objs, int numRetriesForLock, long sleepTime)` that keeps trying to acquire two incompatible locks, and ends up failing after hive.lock.numretries*hive.lock.sleep.between.retries which by defaut is 100*60s = 100 minutes. was: When hive.support.concurrency is enabled if you launch a query that reads data from a partition and writes data into another partition of the same table, it creates a deadlock. The worse part is that once the deadlock is active, you can't query the table until it times out. * How to reproduce : ``` CREATE TABLE test_table (id INT) PARTITIONED BY (part STRING) ; INSERT INTO TABLE test_table PARTITION (part="test") VALUES (1), (2), (3), (4) ; INSERT OVERWRITE TABLE test_table PARTITION (part="test2") SELECT id FROM test_table WHERE part="test1"; ``` Nothing happens, and when doing a SHOW LOCKS in another terminal we get : ``` SHOW LOCKS ; +--+---+++-+--+-+-++ | lockid | database | table | partition | lock_state | lock_type| transaction_id | last_heartbeat | acquired_at | +--+---+++-+--+-+-++ | 3765 | default | test_table | NULL | WAITING | SHARED_READ | NULL| 1440603633148 | NULL | | 3765 | default | test_table | part=test2 | WAITING | EXCLUSIVE| NULL| 1440603633148 | NULL | +--+---+++-+--+-+-++ ``` This was tested on Hive 1.1.0-cdh5.4.2 but I believe the bug is still presents in 1.2.1. I could not reproduce it easily locally because it requires a pseudo-distributed setup with zookeeper to have concurrency enabled. >From looking at the code I believe the problem comes from the >EmbeddedLockManager method `public List lock(List objs, int numRetriesForLock, long sleepTime)` that keeps trying to acquire two incompatible locks, and ends up failing after hive.lock.numretries*hive.lock.sleep.between.retries which by defaut is 100*60s = 100 minutes. > read/write into same partitioned table + concurrency = deadlock > --- > > Key: HIVE-12258 > URL: https://issues.apache.org/jira/browse/HIVE-12258 > Project: Hive > Issue Type: Bug >Reporter: Furcy Pin > > When hive.support.concurrency is enabled if you launch a query that reads > data from a partition and writes data into another partition of the same > table, > it creates a deadlock. > The worse part is that once the deadlock is active, you can't query the table > until it times out. > * How to reproduce : > CREATE TABLE test_table (id INT) > PARTITIONED BY (part STRING) > ; > INSERT INTO TABLE test_table PARTITION
[jira] [Updated] (HIVE-12258) read/write into same partitioned table + concurrency = deadlock
[ https://issues.apache.org/jira/browse/HIVE-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Furcy Pin updated HIVE-12258: - Description: When hive.support.concurrency is enabled if you launch a query that reads data from a partition and writes data into another partition of the same table, it creates a deadlock. The worse part is that once the deadlock is active, you can't query the table until it times out. * How to reproduce : CREATE TABLE test_table (id INT) PARTITIONED BY (part STRING) ; INSERT INTO TABLE test_table PARTITION (part="test") VALUES (1), (2), (3), (4) ; INSERT OVERWRITE TABLE test_table PARTITION (part="test2") SELECT id FROM test_table WHERE part="test1"; Nothing happens, and when doing a SHOW LOCKS in another terminal we get : | lockid | database | table | partition | lock_state | lock_type| transaction_id | last_heartbeat | acquired_at | | 3765 | default | test_table | NULL | WAITING | SHARED_READ | NULL| 1440603633148 | NULL | | 3765 | default | test_table | part=test2 | WAITING | EXCLUSIVE| NULL| 1440603633148 | NULL | This was tested on Hive 1.1.0-cdh5.4.2 but I believe the bug is still presents in 1.2.1. I could not reproduce it easily locally because it requires a pseudo-distributed setup with zookeeper to have concurrency enabled. >From looking at the code I believe the problem comes from the >EmbeddedLockManager method `public List lock(List objs, int numRetriesForLock, long sleepTime)` that keeps trying to acquire two incompatible locks, and ends up failing after hive.lock.numretries*hive.lock.sleep.between.retries which by defaut is 100*60s = 100 minutes. was: When hive.support.concurrency is enabled if you launch a query that reads data from a partition and writes data into another partition of the same table, it creates a deadlock. The worse part is that once the deadlock is active, you can't query the table until it times out. * How to reproduce : CREATE TABLE test_table (id INT) PARTITIONED BY (part STRING) ; INSERT INTO TABLE test_table PARTITION (part="test") VALUES (1), (2), (3), (4) ; INSERT OVERWRITE TABLE test_table PARTITION (part="test2") SELECT id FROM test_table WHERE part="test1"; Nothing happens, and when doing a SHOW LOCKS in another terminal we get : +--+---+++-+--+-+-++ | lockid | database | table | partition | lock_state | lock_type| transaction_id | last_heartbeat | acquired_at | +--+---+++-+--+-+-++ | 3765 | default | test_table | NULL | WAITING | SHARED_READ | NULL| 1440603633148 | NULL | | 3765 | default | test_table | part=test2 | WAITING | EXCLUSIVE| NULL| 1440603633148 | NULL | +--+---+++-+--+-+-++ This was tested on Hive 1.1.0-cdh5.4.2 but I believe the bug is still presents in 1.2.1. I could not reproduce it easily locally because it requires a pseudo-distributed setup with zookeeper to have concurrency enabled. >From looking at the code I believe the problem comes from the >EmbeddedLockManager method `public List lock(List objs, int numRetriesForLock, long sleepTime)` that keeps trying to acquire two incompatible locks, and ends up failing after hive.lock.numretries*hive.lock.sleep.between.retries which by defaut is 100*60s = 100 minutes. > read/write into same partitioned table + concurrency = deadlock > --- > > Key: HIVE-12258 > URL: https://issues.apache.org/jira/browse/HIVE-12258 > Project: Hive > Issue Type: Bug >Reporter: Furcy Pin > > When hive.support.concurrency is enabled if you launch a query that reads > data from a partition and writes data into another partition of the same > table, > it creates a deadlock. > The worse part is that once the deadlock is active, you can't query the table > until it times out. > * How to reproduce : > CREATE TABLE test_table (id INT) > PARTITIONED BY (part STRING) > ; > INSERT INTO TABLE test_table PARTITION (part="test") > VALUES (1), (2), (3), (4) > ; > INSERT OVERWRITE TABLE test_table PARTITION (part="test2") > SELECT id FROM test_table WHERE part="test1"; > Nothing happens, and when doing a SHOW LOCKS in another terminal we get : > | lockid | database | table | partition | lock_state | lock_type > | transaction_id | last_heartbeat | acquired_at | > | 3765 | default | test_table | NULL |
[jira] [Commented] (HIVE-11523) org.apache.hadoop.hive.ql.io.orc.FileDump should handle errors
[
https://issues.apache.org/jira/browse/HIVE-11523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972512#comment-14972512
]
Hive QA commented on HIVE-11523:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768472/HIVE-11523.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9687 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.initializationError
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestFileDump.testDataDumpThrowsIOException
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5774/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5774/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5774/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768472 - PreCommit-HIVE-TRUNK-Build
> org.apache.hadoop.hive.ql.io.orc.FileDump should handle errors
> --
>
> Key: HIVE-11523
> URL: https://issues.apache.org/jira/browse/HIVE-11523
> Project: Hive
> Issue Type: Bug
> Components: File Formats
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-11523-branch-1.patch, HIVE-11523.patch,
> HIVE-11523.patch
>
>
> this utility can take N files as arguments. Currently if it fails to read
> some file, it fails and bails out. It should instead log the file name and
> error and proceed with other files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12178) LLAP: NPE in LRFU policy
[
https://issues.apache.org/jira/browse/HIVE-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-12178:
--
Labels: TODOC2.0 (was: )
> LLAP: NPE in LRFU policy
>
>
> Key: HIVE-12178
> URL: https://issues.apache.org/jira/browse/HIVE-12178
> Project: Hive
> Issue Type: Bug
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
> Labels: TODOC2.0
> Fix For: 2.0.0
>
> Attachments: HIVE-12178.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.cache.LowLevelLrfuCachePolicy.removeFromListUnderLock(LowLevelLrfuCachePolicy.java:346)
> at
> org.apache.hadoop.hive.llap.cache.LowLevelLrfuCachePolicy.removeFromListAndUnlock(LowLevelLrfuCachePolicy.java:335)
> at
> org.apache.hadoop.hive.llap.cache.LowLevelLrfuCachePolicy.notifyUnlock(LowLevelLrfuCachePolicy.java:133)
> at
> org.apache.hadoop.hive.llap.cache.LowLevelCacheImpl.unlockBuffer(LowLevelCacheImpl.java:354)
> at
> org.apache.hadoop.hive.llap.cache.LowLevelCacheImpl.releaseBuffers(LowLevelCacheImpl.java:344)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.returnData(OrcEncodedDataReader.java:662)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.returnData(OrcEncodedDataReader.java:74)
> at
> org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.returnSourceData(EncodedDataConsumer.java:131)
> at
> org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.consumeData(EncodedDataConsumer.java:122)
> at
> org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.consumeData(EncodedDataConsumer.java:36)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:405)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:413)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:194)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:191)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1655)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:191)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:74)
> at org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
> ... 4 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12258) read/write into same partitioned table + concurrency = deadlock
[ https://issues.apache.org/jira/browse/HIVE-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Furcy Pin updated HIVE-12258: - Description: When hive.support.concurrency is enabled if you launch a query that reads data from a partition and writes data into another partition of the same table, it creates a deadlock. The worse part is that once the deadlock is active, you can't query the table until it times out. * How to reproduce : CREATE TABLE test_table (id INT) PARTITIONED BY (part STRING) ; INSERT INTO TABLE test_table PARTITION (part="test") VALUES (1), (2), (3), (4) ; INSERT OVERWRITE TABLE test_table PARTITION (part="test2") SELECT id FROM test_table WHERE part="test1"; Nothing happens, and when doing a SHOW LOCKS in another terminal we get : | lockid | database | table | partition | lock_state | lock_type| transaction_id | last_heartbeat | acquired_at | | 3765 | default | test_table | NULL | WAITING | SHARED_READ | NULL| 1440603633148 | NULL | | 3765 | default | test_table | part=test2 | WAITING | EXCLUSIVE| NULL| 1440603633148 | NULL | This was tested on Hive 1.1.0-cdh5.4.2 but I believe the bug is still presents in 1.2.1. I could not reproduce it easily locally because it requires a pseudo-distributed setup with zookeeper to have concurrency enabled. >From looking at the code I believe the problem comes from the >EmbeddedLockManager method public List lock(List objs, int numRetriesForLock, long sleepTime) that keeps trying to acquire two incompatible locks, and ends up failing after hive.lock.numretries*hive.lock.sleep.between.retries which by defaut is 100*60s = 100 minutes. was: When hive.support.concurrency is enabled if you launch a query that reads data from a partition and writes data into another partition of the same table, it creates a deadlock. The worse part is that once the deadlock is active, you can't query the table until it times out. * How to reproduce : CREATE TABLE test_table (id INT) PARTITIONED BY (part STRING) ; INSERT INTO TABLE test_table PARTITION (part="test") VALUES (1), (2), (3), (4) ; INSERT OVERWRITE TABLE test_table PARTITION (part="test2") SELECT id FROM test_table WHERE part="test1"; Nothing happens, and when doing a SHOW LOCKS in another terminal we get : | lockid | database | table | partition | lock_state | lock_type| transaction_id | last_heartbeat | acquired_at | | 3765 | default | test_table | NULL | WAITING | SHARED_READ | NULL| 1440603633148 | NULL | | 3765 | default | test_table | part=test2 | WAITING | EXCLUSIVE| NULL| 1440603633148 | NULL | This was tested on Hive 1.1.0-cdh5.4.2 but I believe the bug is still presents in 1.2.1. I could not reproduce it easily locally because it requires a pseudo-distributed setup with zookeeper to have concurrency enabled. >From looking at the code I believe the problem comes from the >EmbeddedLockManager method `public List lock(List objs, int numRetriesForLock, long sleepTime)` that keeps trying to acquire two incompatible locks, and ends up failing after hive.lock.numretries*hive.lock.sleep.between.retries which by defaut is 100*60s = 100 minutes. > read/write into same partitioned table + concurrency = deadlock > --- > > Key: HIVE-12258 > URL: https://issues.apache.org/jira/browse/HIVE-12258 > Project: Hive > Issue Type: Bug >Reporter: Furcy Pin > > When hive.support.concurrency is enabled if you launch a query that reads > data from a partition and writes data into another partition of the same > table, > it creates a deadlock. > The worse part is that once the deadlock is active, you can't query the table > until it times out. > * How to reproduce : > CREATE TABLE test_table (id INT) > PARTITIONED BY (part STRING) > ; > INSERT INTO TABLE test_table PARTITION (part="test") > VALUES (1), (2), (3), (4) > ; > INSERT OVERWRITE TABLE test_table PARTITION (part="test2") > SELECT id FROM test_table WHERE part="test1"; > Nothing happens, and when doing a SHOW LOCKS in another terminal we get : > | lockid | database | table | partition | lock_state | lock_type > | transaction_id | last_heartbeat | acquired_at | > | 3765 | default | test_table | NULL | WAITING | SHARED_READ > | NULL| 1440603633148 | NULL | > | 3765 | default | test_table | part=test2 | WAITING | EXCLUSIVE > | NULL| 1440603633148 | NULL | > This was tested on Hive 1.1.0-cdh5.4.2 but I believe the bug is still > presents in 1.2.1. > I could not reproduce it easily locally because it requires a > pseudo-distributed
[jira] [Commented] (HIVE-11378) Remove hadoop-1 support from master branch
[
https://issues.apache.org/jira/browse/HIVE-11378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972540#comment-14972540
]
Hive QA commented on HIVE-11378:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768483/HIVE-11378.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 9696 tests executed
*Failed tests:*
{noformat}
TestJdbcWithLocalClusterSpark - did not produce a TEST-*.xml file
org.apache.hadoop.hive.hwi.TestHWISessionManager.testHiveDriver
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5775/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5775/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5775/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768483 - PreCommit-HIVE-TRUNK-Build
> Remove hadoop-1 support from master branch
> --
>
> Key: HIVE-11378
> URL: https://issues.apache.org/jira/browse/HIVE-11378
> Project: Hive
> Issue Type: Task
> Components: Build Infrastructure
>Affects Versions: 2.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Fix For: 2.0.0
>
> Attachments: HIVE-11378.2.patch, HIVE-11378.3.patch,
> HIVE-11378.4.patch, HIVE-11378.5.patch, HIVE-11378.patch
>
>
> When we branched branch-1 one of the goals was the ability to remove hadoop1
> support from master. I propose to do this softly at first by removing it
> from the poms removing the 20S implementation of the shims.
> I am not going to remove the shim layer. That would be much more disruptive.
> Also, I haven't done the homework to see if we could, as there may still be
> incompatibility issues between various versions of hadoop2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12178) LLAP: NPE in LRFU policy
[
https://issues.apache.org/jira/browse/HIVE-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972444#comment-14972444
]
Lefty Leverenz commented on HIVE-12178:
---
Doc note: This revises HIVE-12078's description for
*hive.llap.io.lrfu.lambda*. It needs to be documented in the wiki for the
2.0.0 release.
* [Hive Configuration Properties (needs a new LLAP section) |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties]
> LLAP: NPE in LRFU policy
>
>
> Key: HIVE-12178
> URL: https://issues.apache.org/jira/browse/HIVE-12178
> Project: Hive
> Issue Type: Bug
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
> Labels: TODOC2.0
> Fix For: 2.0.0
>
> Attachments: HIVE-12178.patch
>
>
> {noformat}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.llap.cache.LowLevelLrfuCachePolicy.removeFromListUnderLock(LowLevelLrfuCachePolicy.java:346)
> at
> org.apache.hadoop.hive.llap.cache.LowLevelLrfuCachePolicy.removeFromListAndUnlock(LowLevelLrfuCachePolicy.java:335)
> at
> org.apache.hadoop.hive.llap.cache.LowLevelLrfuCachePolicy.notifyUnlock(LowLevelLrfuCachePolicy.java:133)
> at
> org.apache.hadoop.hive.llap.cache.LowLevelCacheImpl.unlockBuffer(LowLevelCacheImpl.java:354)
> at
> org.apache.hadoop.hive.llap.cache.LowLevelCacheImpl.releaseBuffers(LowLevelCacheImpl.java:344)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.returnData(OrcEncodedDataReader.java:662)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.returnData(OrcEncodedDataReader.java:74)
> at
> org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.returnSourceData(EncodedDataConsumer.java:131)
> at
> org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.consumeData(EncodedDataConsumer.java:122)
> at
> org.apache.hadoop.hive.llap.io.decode.EncodedDataConsumer.consumeData(EncodedDataConsumer.java:36)
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:405)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:413)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:194)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:191)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1655)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:191)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:74)
> at org.apache.hadoop.hive.common.CallableWithNdc.call(CallableWithNdc.java:37)
> ... 4 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11961) Create table displays Result inappropriately in beeline.
[ https://issues.apache.org/jira/browse/HIVE-11961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972465#comment-14972465 ] Chetna Chaudhari commented on HIVE-11961: - [~chetdb] it shows 'No rows affected (0.62 seconds)'. I guess thats the behaviour across beeline cli as compared to hive-cli which returns OK on table creation. > Create table displays Result inappropriately in beeline. > > > Key: HIVE-11961 > URL: https://issues.apache.org/jira/browse/HIVE-11961 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 0.13.0 > Environment: SUSE >Reporter: Chetan Bhat >Priority: Minor > Original Estimate: 2,520h > Remaining Estimate: 2,520h > > In beeline User tries to create a table using the create table DDL query. > create table test( > idint, > dtDontQuery string, > name string > ) > partitioned by (date string); > Actual Result : > The result field is displayed inappropriately instead of confirmation message. > +-+ > | Result | > +-+ > +-+ > Expected Result : > Confirmation message needs to be displayed instead of result field. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12246) Orc FileDump fails with Missing CLI jar
[
https://issues.apache.org/jira/browse/HIVE-12246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14972457#comment-14972457
]
Hive QA commented on HIVE-12246:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12768471/HIVE-12246.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 9701 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.io.orc.TestColumnStatistics.testHasNull
org.apache.hadoop.hive.ql.io.orc.TestJsonFileDump.testJsonDump
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarDataNucleusUnCaching
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5773/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5773/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5773/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12768471 - PreCommit-HIVE-TRUNK-Build
> Orc FileDump fails with Missing CLI jar
> ---
>
> Key: HIVE-12246
> URL: https://issues.apache.org/jira/browse/HIVE-12246
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-12246.patch, HIVE-12246.patch
>
>
> Running hive --orcfiledump fails with "Missing CLI jar"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
