[jira] [Issue Comment Deleted] (HIVE-14818) Reduce number of retries while starting HiveServer for tests
[ https://issues.apache.org/jira/browse/HIVE-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-14818: -- Comment: was deleted (was: Doc note: This creates a new configuration parameter (*hive.server2.sleep.interval.between.start.attempts*) and changes the function and description of another (*hive.server2.max.start.attempts*) so they need to be documented in the wiki. Note that the description change should include version information. * [Configuration Properties -- HiveServer2 | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-HiveServer2] * [Configuration Properties -- HiveServer2 -- hive.server2.max.start.attempts | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.max.start.attempts] Added a TODOC2.2 label.) > Reduce number of retries while starting HiveServer for tests > > > Key: HIVE-14818 > URL: https://issues.apache.org/jira/browse/HIVE-14818 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14818.01.patch, HIVE-14818.02.patch > > > Current is 30 retries, with a 1minute sleep between each one. > The settings are likely bad for a production cluster as well. For tests, this > should be a lot lower. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14818) Reduce number of retries while starting HiveServer for tests
[ https://issues.apache.org/jira/browse/HIVE-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518448#comment-15518448 ] Lefty Leverenz commented on HIVE-14818: --- Doc note: This creates a new configuration parameter (*hive.server2.sleep.interval.between.start.attempts*) and changes the function and description of another (*hive.server2.max.start.attempts*) so they need to be documented in the wiki. Note that the description change should include version information. * [Configuration Properties -- HiveServer2 | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-HiveServer2] * [Configuration Properties -- HiveServer2 -- hive.server2.max.start.attempts | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.max.start.attempts] Added a TODOC2.2 label. > Reduce number of retries while starting HiveServer for tests > > > Key: HIVE-14818 > URL: https://issues.apache.org/jira/browse/HIVE-14818 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14818.01.patch, HIVE-14818.02.patch > > > Current is 30 retries, with a 1minute sleep between each one. > The settings are likely bad for a production cluster as well. For tests, this > should be a lot lower. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14818) Reduce number of retries while starting HiveServer for tests
[ https://issues.apache.org/jira/browse/HIVE-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518446#comment-15518446 ] Lefty Leverenz commented on HIVE-14818: --- Doc note: This creates a new configuration parameter (*hive.server2.sleep.interval.between.start.attempts*) and changes the function and description of another (*hive.server2.max.start.attempts*) so they need to be documented in the wiki. Note that the description change should include version information. * [Configuration Properties -- HiveServer2 | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-HiveServer2] * [Configuration Properties -- HiveServer2 -- hive.server2.max.start.attempts | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.max.start.attempts] Added a TODOC2.2 label. > Reduce number of retries while starting HiveServer for tests > > > Key: HIVE-14818 > URL: https://issues.apache.org/jira/browse/HIVE-14818 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14818.01.patch, HIVE-14818.02.patch > > > Current is 30 retries, with a 1minute sleep between each one. > The settings are likely bad for a production cluster as well. For tests, this > should be a lot lower. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14818) Reduce number of retries while starting HiveServer for tests
[ https://issues.apache.org/jira/browse/HIVE-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-14818: -- Labels: TODOC2.2 (was: ) > Reduce number of retries while starting HiveServer for tests > > > Key: HIVE-14818 > URL: https://issues.apache.org/jira/browse/HIVE-14818 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14818.01.patch, HIVE-14818.02.patch > > > Current is 30 retries, with a 1minute sleep between each one. > The settings are likely bad for a production cluster as well. For tests, this > should be a lot lower. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14751) Add support for date truncation
[
https://issues.apache.org/jira/browse/HIVE-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518417#comment-15518417
]
Lefty Leverenz commented on HIVE-14751:
---
Doc note: floor ( to ) needs to be documented in the wiki for
release 2.2.0.
* [Hive Operators and UDFs -- Date Functions |
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-DateFunctions]
Added a TODOC2.2 label.
> Add support for date truncation
> ---
>
> Key: HIVE-14751
> URL: https://issues.apache.org/jira/browse/HIVE-14751
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-14751.patch
>
>
> Add support for {{floor ( to )}}, which is equivalent to
> {{date_trunc(, )}}.
> https://www.postgresql.org/docs/9.1/static/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14751) Add support for date truncation
[
https://issues.apache.org/jira/browse/HIVE-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-14751:
--
Labels: TODOC2.2 (was: )
> Add support for date truncation
> ---
>
> Key: HIVE-14751
> URL: https://issues.apache.org/jira/browse/HIVE-14751
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-14751.patch
>
>
> Add support for {{floor ( to )}}, which is equivalent to
> {{date_trunc(, )}}.
> https://www.postgresql.org/docs/9.1/static/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14579) Add support for date extract
[ https://issues.apache.org/jira/browse/HIVE-14579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518392#comment-15518392 ] Lefty Leverenz commented on HIVE-14579: --- Doc note: This needs to be documented in the wiki for release 2.2.0. * [Hive Operators and UDFs -- Date Functions | https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-DateFunctions] Added a TODOC2.2 label. > Add support for date extract > > > Key: HIVE-14579 > URL: https://issues.apache.org/jira/browse/HIVE-14579 > Project: Hive > Issue Type: Sub-task > Components: UDF >Reporter: Ashutosh Chauhan >Assignee: Jesus Camacho Rodriguez > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14579.01.patch, HIVE-14579.patch, HIVE-14579.patch > > > https://www.postgresql.org/docs/9.1/static/functions-datetime.html#FUNCTIONS-DATETIME-EXTRACT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14579) Add support for date extract
[ https://issues.apache.org/jira/browse/HIVE-14579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-14579: -- Labels: TODOC2.2 (was: ) > Add support for date extract > > > Key: HIVE-14579 > URL: https://issues.apache.org/jira/browse/HIVE-14579 > Project: Hive > Issue Type: Sub-task > Components: UDF >Reporter: Ashutosh Chauhan >Assignee: Jesus Camacho Rodriguez > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14579.01.patch, HIVE-14579.patch, HIVE-14579.patch > > > https://www.postgresql.org/docs/9.1/static/functions-datetime.html#FUNCTIONS-DATETIME-EXTRACT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14837) JDBC: standalone jar is missing hadoop core dependencies
[
https://issues.apache.org/jira/browse/HIVE-14837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-14837:
---
Component/s: JDBC
> JDBC: standalone jar is missing hadoop core dependencies
>
>
> Key: HIVE-14837
> URL: https://issues.apache.org/jira/browse/HIVE-14837
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Gopal V
>
> {code}
> 2016/09/24 00:31:57 ERROR - jmeter.threads.JMeterThread: Test failed!
> java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
> at
> org.apache.hive.jdbc.HiveConnection.createUnderlyingTransport(HiveConnection.java:418)
> at
> org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:438)
> at
> org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:225)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:182)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14837) JDBC: standalone jar is missing hadoop core dependencies
[
https://issues.apache.org/jira/browse/HIVE-14837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-14837:
---
Affects Version/s: 2.2.0
> JDBC: standalone jar is missing hadoop core dependencies
>
>
> Key: HIVE-14837
> URL: https://issues.apache.org/jira/browse/HIVE-14837
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Gopal V
>
> {code}
> 2016/09/24 00:31:57 ERROR - jmeter.threads.JMeterThread: Test failed!
> java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
> at
> org.apache.hive.jdbc.HiveConnection.createUnderlyingTransport(HiveConnection.java:418)
> at
> org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:438)
> at
> org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:225)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:182)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14826) Support vectorization for Parquet
[ https://issues.apache.org/jira/browse/HIVE-14826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518333#comment-15518333 ] Ferdinand Xu commented on HIVE-14826: - Sure, maybe we can begin reviews for 1st part which is the fundamental patch. Otherwise, we could not work since the lack of the basic functionality > Support vectorization for Parquet > - > > Key: HIVE-14826 > URL: https://issues.apache.org/jira/browse/HIVE-14826 > Project: Hive > Issue Type: New Feature >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > > Parquet vectorized reader can improve both throughput and also leverages > existing Hive vectorization execution engine. This is an umbrella ticket to > track this feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-14029: Attachment: HIVE-14029.6.patch Let's see whether it breaks qtest after removing org.glassfish.jersey related dependencies from Spark_core > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.4.patch, HIVE-14029.5.patch, > HIVE-14029.6.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. > To update Spark version to 2.0.0, the following changes are required: > * Spark API updates: > ** SparkShuffler#call return Iterator instead of Iterable > ** SparkListener -> JavaSparkListener > ** InputMetrics constructor doesn’t accept readMethod > ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics > return long type instead of integer > * Dependency upgrade: > ** Jackson: 2.4.2 -> 2.6.5 > ** Netty version: 4.0.23.Final -> 4.0.29.Final > ** Scala binary version: 2.10 -> 2.11 > ** Scala version: 2.10.4 -> 2.11.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14819) FunctionInfo for permanent functions shows TEMPORARY FunctionType
[
https://issues.apache.org/jira/browse/HIVE-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518296#comment-15518296
]
Hive QA commented on HIVE-14819:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12830125/HIVE-14819.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10633 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1297/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1297/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1297/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12830125 - PreCommit-HIVE-Build
> FunctionInfo for permanent functions shows TEMPORARY FunctionType
> -
>
> Key: HIVE-14819
> URL: https://issues.apache.org/jira/browse/HIVE-14819
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 2.1.0
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-14819.1.patch, HIVE-14819.2.patch
>
>
> The FunctionInfo has a FunctionType field which describes if the function is
> a builtin/persistent/temporary function. But for permanent functions, the
> FunctionInfo being returned by the FunctionRegistry is showing the type to be
> TEMPORARY.
> This affects things which may be depending on function type, for example
> LlapDecider, which will allow builtin/persistent UDFs to be used in LLAP but
> not temporary functions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14821) build q test
[ https://issues.apache.org/jira/browse/HIVE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14821: -- Resolution: Won't Fix Status: Resolved (was: Patch Available) > build q test > > > Key: HIVE-14821 > URL: https://issues.apache.org/jira/browse/HIVE-14821 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14821.1.patch, HIVE-14821.2.patch, > HIVE-14821.3.patch, HIVE-14821.4.patch, HIVE-14821.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518237#comment-15518237 ] Ferdinand Xu edited comment on HIVE-14029 at 9/24/16 3:10 AM: -- Hi [~xuefuz] These two dependencies (Jackson and Netty) are not required in build. It's required for the runtime. If you try to run some HoS job, it will fail to create Spark client since API changes in these two library. You can see failed queries above for the reference. was (Author: ferd): Hi [~xuefuz] These two dependencies are not required in build. It's required for the runtime. If you try to run some HoS job, it will fail to create Spark client since API changes in these two library. You can see failed queries above for the reference. > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.4.patch, HIVE-14029.5.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. > To update Spark version to 2.0.0, the following changes are required: > * Spark API updates: > ** SparkShuffler#call return Iterator instead of Iterable > ** SparkListener -> JavaSparkListener > ** InputMetrics constructor doesn’t accept readMethod > ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics > return long type instead of integer > * Dependency upgrade: > ** Jackson: 2.4.2 -> 2.6.5 > ** Netty version: 4.0.23.Final -> 4.0.29.Final > ** Scala binary version: 2.10 -> 2.11 > ** Scala version: 2.10.4 -> 2.11.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518237#comment-15518237 ] Ferdinand Xu edited comment on HIVE-14029 at 9/24/16 3:07 AM: -- Hi [~xuefuz] These two dependencies are not required in build. It's required for the runtime. If you try to run some HoS job, it will fail to create Spark client since API changes in these two library. You can see failed queries above for the reference. was (Author: ferd): Hi [~xuefuz] If you try to run some HoS job, it will fail to create Spark client since API changes in these two library. I just test some simple queries on my cluster and find these two dependencies are required. > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.4.patch, HIVE-14029.5.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. > To update Spark version to 2.0.0, the following changes are required: > * Spark API updates: > ** SparkShuffler#call return Iterator instead of Iterable > ** SparkListener -> JavaSparkListener > ** InputMetrics constructor doesn’t accept readMethod > ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics > return long type instead of integer > * Dependency upgrade: > ** Jackson: 2.4.2 -> 2.6.5 > ** Netty version: 4.0.23.Final -> 4.0.29.Final > ** Scala binary version: 2.10 -> 2.11 > ** Scala version: 2.10.4 -> 2.11.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518237#comment-15518237 ] Ferdinand Xu commented on HIVE-14029: - Hi [~xuefuz] If you try to run some HoS job, it will fail to create Spark client since API changes in these two library. I just test some simple queries on my cluster and find these two dependencies are required. > Update Spark version to 2.0.0 > - > > Key: HIVE-14029 > URL: https://issues.apache.org/jira/browse/HIVE-14029 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch, > HIVE-14029.3.patch, HIVE-14029.4.patch, HIVE-14029.5.patch, HIVE-14029.patch > > > There are quite some new optimizations in Spark 2.0.0. We need to bump up > Spark to 2.0.0 to benefit those performance improvements. > To update Spark version to 2.0.0, the following changes are required: > * Spark API updates: > ** SparkShuffler#call return Iterator instead of Iterable > ** SparkListener -> JavaSparkListener > ** InputMetrics constructor doesn’t accept readMethod > ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics > return long type instead of integer > * Dependency upgrade: > ** Jackson: 2.4.2 -> 2.6.5 > ** Netty version: 4.0.23.Final -> 4.0.29.Final > ** Scala binary version: 2.10 -> 2.11 > ** Scala version: 2.10.4 -> 2.11.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14821) build q test
[
https://issues.apache.org/jira/browse/HIVE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518192#comment-15518192
]
Hive QA commented on HIVE-14821:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12830139/HIVE-14821.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10638 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdate.testMergeProto
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testMergeProto
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1296/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1296/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1296/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12830139 - PreCommit-HIVE-Build
> build q test
>
>
> Key: HIVE-14821
> URL: https://issues.apache.org/jira/browse/HIVE-14821
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-14821.1.patch, HIVE-14821.2.patch,
> HIVE-14821.3.patch, HIVE-14821.4.patch, HIVE-14821.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14831) Missing Druid dependencies at runtime
[
https://issues.apache.org/jira/browse/HIVE-14831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15518090#comment-15518090
]
Hive QA commented on HIVE-14831:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12830094/HIVE-14831.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10629 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.hcatalog.mapreduce.TestHCatMultiOutputFormat.testOutputFormat
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1295/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1295/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1295/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12830094 - PreCommit-HIVE-Build
> Missing Druid dependencies at runtime
> -
>
> Key: HIVE-14831
> URL: https://issues.apache.org/jira/browse/HIVE-14831
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14831.patch
>
>
> Excluded some packages when shading in the initial patch that should have
> been included.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14821) build q test
[ https://issues.apache.org/jira/browse/HIVE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14821: -- Attachment: HIVE-14821.4.patch > build q test > > > Key: HIVE-14821 > URL: https://issues.apache.org/jira/browse/HIVE-14821 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14821.1.patch, HIVE-14821.2.patch, > HIVE-14821.3.patch, HIVE-14821.4.patch, HIVE-14821.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14821) build q test
[
https://issues.apache.org/jira/browse/HIVE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517884#comment-15517884
]
Hive QA commented on HIVE-14821:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12830075/HIVE-14821.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1294/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1294/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1294/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-09-23 23:36:55.971
+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1294/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-09-23 23:36:55.973
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
990927e..a213115 master -> origin/master
+ git reset --hard HEAD
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
HEAD is now at 990927e HIVE-14713: LDAP Authentication Provider should be
covered with unit tests (Illya Yalovyy, reviewed by Chaoyu Tang, Szehon Ho)
+ git clean -f -d
warning: unable to access '/home/sseth/.config/git/ignore': Permission denied
Removing jdbc/src/test/org/apache/hive/jdbc/TestRunInitialSQL.java
+ git checkout master
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
warning: unable to access '/home/sseth/.config/git/ignore': Permission denied
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
HEAD is now at a213115 HIVE-14818. Reduce number of retries while starting
HiveServer for tests. (Siddharth Seth, reviewed by Prasanth Jayachandran)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-09-23 23:36:57.566
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g:329
error: ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g: patch does not
apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12830075 - PreCommit-HIVE-Build
> build q test
>
>
> Key: HIVE-14821
> URL: https://issues.apache.org/jira/browse/HIVE-14821
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-14821.1.patch, HIVE-14821.2.patch,
> HIVE-14821.3.patch, HIVE-14821.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14835) Improve ptest2 build time
[ https://issues.apache.org/jira/browse/HIVE-14835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14835: - Attachment: HIVE-14835.1.patch This does not require a ptest run. Although this changes has to be made in ptest2 cluster. [~spena]/[~sseth] can someone please review this patch? > Improve ptest2 build time > - > > Key: HIVE-14835 > URL: https://issues.apache.org/jira/browse/HIVE-14835 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran > Attachments: HIVE-14835.1.patch > > > 2 things can be improved > 1) ptest2 always downloads jars for compiling its own directory which takes > about 1m30s which should take only 5s with cache jars. The reason for that is > maven.repo.local is pointing to a path under WORKSPACE which will be cleaned > by jenkins for every run. > 2) For hive build we can make use of parallel build and quite the output of > build which should shave off another 15-30s. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14835) Improve ptest2 build time
[ https://issues.apache.org/jira/browse/HIVE-14835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-14835: Assignee: Prasanth Jayachandran > Improve ptest2 build time > - > > Key: HIVE-14835 > URL: https://issues.apache.org/jira/browse/HIVE-14835 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-14835.1.patch > > > NO PRECOMMIT TESTS > 2 things can be improved > 1) ptest2 always downloads jars for compiling its own directory which takes > about 1m30s which should take only 5s with cache jars. The reason for that is > maven.repo.local is pointing to a path under WORKSPACE which will be cleaned > by jenkins for every run. > 2) For hive build we can make use of parallel build and quite the output of > build which should shave off another 15-30s. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14835) Improve ptest2 build time
[ https://issues.apache.org/jira/browse/HIVE-14835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-14835: - Description: NO PRECOMMIT TESTS 2 things can be improved 1) ptest2 always downloads jars for compiling its own directory which takes about 1m30s which should take only 5s with cache jars. The reason for that is maven.repo.local is pointing to a path under WORKSPACE which will be cleaned by jenkins for every run. 2) For hive build we can make use of parallel build and quite the output of build which should shave off another 15-30s. was: 2 things can be improved 1) ptest2 always downloads jars for compiling its own directory which takes about 1m30s which should take only 5s with cache jars. The reason for that is maven.repo.local is pointing to a path under WORKSPACE which will be cleaned by jenkins for every run. 2) For hive build we can make use of parallel build and quite the output of build which should shave off another 15-30s. > Improve ptest2 build time > - > > Key: HIVE-14835 > URL: https://issues.apache.org/jira/browse/HIVE-14835 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran > Attachments: HIVE-14835.1.patch > > > NO PRECOMMIT TESTS > 2 things can be improved > 1) ptest2 always downloads jars for compiling its own directory which takes > about 1m30s which should take only 5s with cache jars. The reason for that is > maven.repo.local is pointing to a path under WORKSPACE which will be cleaned > by jenkins for every run. > 2) For hive build we can make use of parallel build and quite the output of > build which should shave off another 15-30s. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14558) Add support for listing views similar to "show tables"
[ https://issues.apache.org/jira/browse/HIVE-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-14558: - Status: Patch Available (was: Open) > Add support for listing views similar to "show tables" > -- > > Key: HIVE-14558 > URL: https://issues.apache.org/jira/browse/HIVE-14558 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14558.1.patch, HIVE-14558.2.patch > > > Users have been asking for such feature where they can get a lists of views > separately. > So perhaps a syntax similar to "show tables" command? > show views [in/from ] [] > Does it make sense to add such command? or is it not worth the effort? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14558) Add support for listing views similar to "show tables"
[ https://issues.apache.org/jira/browse/HIVE-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-14558: - Attachment: HIVE-14558.2.patch New patch. 1) merged a few methods to ensure showTables() and showViews() re-use common code. 2) Deleted the new methods I added to for getViews*() I added in the prior patch. The new patch adds a getTablesByType() method that is used by the old getTables*() methods too. 3) Improved java docs comments. 4) Changes some view names to match table name in the qtest file. > Add support for listing views similar to "show tables" > -- > > Key: HIVE-14558 > URL: https://issues.apache.org/jira/browse/HIVE-14558 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14558.1.patch, HIVE-14558.2.patch > > > Users have been asking for such feature where they can get a lists of views > separately. > So perhaps a syntax similar to "show tables" command? > show views [in/from ] [] > Does it make sense to add such command? or is it not worth the effort? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14558) Add support for listing views similar to "show tables"
[ https://issues.apache.org/jira/browse/HIVE-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-14558: - Status: Open (was: Patch Available) Have some feedback from review. Incorporated it. > Add support for listing views similar to "show tables" > -- > > Key: HIVE-14558 > URL: https://issues.apache.org/jira/browse/HIVE-14558 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14558.1.patch > > > Users have been asking for such feature where they can get a lists of views > separately. > So perhaps a syntax similar to "show tables" command? > show views [in/from ] [] > Does it make sense to add such command? or is it not worth the effort? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14819) FunctionInfo for permanent functions shows TEMPORARY FunctionType
[ https://issues.apache.org/jira/browse/HIVE-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14819: -- Attachment: HIVE-14819.2.patch Updating patch to fix the persistent function issue that occurs during Hive initialization. > FunctionInfo for permanent functions shows TEMPORARY FunctionType > - > > Key: HIVE-14819 > URL: https://issues.apache.org/jira/browse/HIVE-14819 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 2.1.0 >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14819.1.patch, HIVE-14819.2.patch > > > The FunctionInfo has a FunctionType field which describes if the function is > a builtin/persistent/temporary function. But for permanent functions, the > FunctionInfo being returned by the FunctionRegistry is showing the type to be > TEMPORARY. > This affects things which may be depending on function type, for example > LlapDecider, which will allow builtin/persistent UDFs to be used in LLAP but > not temporary functions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14819) FunctionInfo for permanent functions shows TEMPORARY FunctionType
[
https://issues.apache.org/jira/browse/HIVE-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517776#comment-15517776
]
Jason Dere commented on HIVE-14819:
---
I see another issue related to being able to determine if a function is a
permanent UDF:
When Hive attempts to register all of the existing permanent UDFs during Hive
initialization, it does not download the JARs and add them to the classloader.
However as you can see from the stack trace, it does try to get the UDF class
in order to add it to this persistent UDFs list. This fails since Hive has not
loaded the resources. As far as I can tell, this is going to fail for any
permanent UDFs which require JARs which are not in Hive's classpath at
initialization time.
While this does not prevent permanent functions from working in most cases
(dynamic lookup works and it's added to the session registry), this does cause
problems with being able to tell if this is a permanent function because the
UDF's class is never added to the system registry's list of persistent
functions. Thus FunctionRegistry.isPermanentFunction() does not work properly.
{noformat}
016-09-23T00:32:47,869 WARN [main]: metadata.Hive
(Hive.java:reloadFunctions(247)) - Failed to register persistent function
sleep2:org.apache.hive.udf.generic.GenericUDFSleep. Ignore and continue.
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.hive.udf.generic.GenericUDFSleep
at
org.apache.hadoop.hive.ql.exec.Registry.getPermanentUdfClass(Registry.java:483)
~[hive-exec-2.1.0.jar:2.1.0]
at
org.apache.hadoop.hive.ql.exec.Registry.addFunction(Registry.java:466)
~[hive-exec-2.1.0.jar:2.1.0]
at
org.apache.hadoop.hive.ql.exec.Registry.registerPermanentFunction(Registry.java:239)
~[hive-exec-2.1.0.jar:2.1.0]
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerPermanentFunction(FunctionRegistry.java:1511)
~[hive-exec-2.1.0.jar:2.1.0]
at
org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:243)
~[hive-exec-2.1.0.jar:2.1.0]
at
org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:224)
~[hive-exec-2.1.0.jar:2.1.0]
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:369)
~[hive-exec-2.1.0.jar:2.1.0]
at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:313)
~[hive-exec-2.1.0.jar:2.1.0]
at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:293)
~[hive-exec-2.1.0.jar:2.1.0]
at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:269)
~[hive-exec-2.1.0.jar:2.1.0]
at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:558)
~[hive-exec-2.1.0.jar:2.1.0]
at
org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:531)
~[hive-exec-2.1.0.jar:2.1.0]
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
~[hive-cli-2.1.0.2.5.1.0-46.jar:2.1.0.2.5.1.0-46]
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
~[hive-cli-2.1.0.2.5.1.0-46.jar:2.1.0.2.5.1.0-46]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_91]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_91]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_91]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_91]
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
~[hadoop-common-2.7.3.2.5.1.0-46.jar:?]
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
~[hadoop-common-2.7.3.2.5.1.0-46.jar:?]
Caused by: java.lang.ClassNotFoundException:
org.apache.hive.udf.generic.GenericUDFSleep
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
~[?:1.8.0_91]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_91]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_91]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_91]
at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_91]
at
org.apache.hadoop.hive.ql.exec.Registry.getPermanentUdfClass(Registry.java:481)
~[hive-exec-2.1.0.jar:2.1.0]
... 19 more
{noformat}
> FunctionInfo for permanent functions shows TEMPORARY FunctionType
> -
>
> Key: HIVE-14819
> URL: https://issues.apache.org/jira/browse/HIVE-14819
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Affects Versions: 2.1.0
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-14819.1.patch
>
>
> The FunctionInfo has a FunctionType field which describes if the function is
> a builtin/persistent/temporary function. But for permanent functions, the
> FunctionInfo being
[jira] [Updated] (HIVE-14818) Reduce number of retries while starting HiveServer for tests
[ https://issues.apache.org/jira/browse/HIVE-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14818: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed. Thanks for the review. > Reduce number of retries while starting HiveServer for tests > > > Key: HIVE-14818 > URL: https://issues.apache.org/jira/browse/HIVE-14818 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.2.0 > > Attachments: HIVE-14818.01.patch, HIVE-14818.02.patch > > > Current is 30 retries, with a 1minute sleep between each one. > The settings are likely bad for a production cluster as well. For tests, this > should be a lot lower. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14817) Shutdown the SessionManager timeoutChecker thread properly upon shutdown
[ https://issues.apache.org/jira/browse/HIVE-14817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14817: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed. Thanks for the review. > Shutdown the SessionManager timeoutChecker thread properly upon shutdown > > > Key: HIVE-14817 > URL: https://issues.apache.org/jira/browse/HIVE-14817 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.2.0 > > Attachments: HIVE-14817.01.patch > > > Shutdown for SessionManager waits 10seconds for all threads on the > threadpoolExecutor to shutdown correctly. > The cleaner thread - with default settings - will take 6 hours to shutdown, > so essentially any shutdown of HS2 is always delayed by 10s. > The cleaner thread should be shutdown properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5867) JDBC driver and beeline should support executing an initial SQL script
[
https://issues.apache.org/jira/browse/HIVE-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517576#comment-15517576
]
Hive QA commented on HIVE-5867:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12830001/HIVE-5867.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10639 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1291/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1291/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1291/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12830001 - PreCommit-HIVE-Build
> JDBC driver and beeline should support executing an initial SQL script
> --
>
> Key: HIVE-5867
> URL: https://issues.apache.org/jira/browse/HIVE-5867
> Project: Hive
> Issue Type: Improvement
> Components: Clients, JDBC
>Reporter: Prasad Mujumdar
>Assignee: Jianguo Tian
> Attachments: HIVE-5867.1.patch, HIVE-5867.2.patch
>
>
> HiveCLI support the .hiverc script that is executed at the start of the
> session. This is helpful for things like registering UDFs, session specific
> configs etc.
> This functionality is missing for beeline and JDBC clients. It would be
> useful for JDBC driver to support an init script with SQL statements that's
> automatically executed after connection. The script path can be specified via
> JDBC connection URL. For example
> {noformat}
> jdbc:hive2://localhost:1/default;initScript=/home/user1/scripts/init.sql
> {noformat}
> This can be added to Beeline's command line option like "-i
> /home/user1/scripts/init.sql"
> To help transition from HiveCLI to Beeline, we can keep the default init
> script as $HOME/.hiverc
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14831) Missing Druid dependencies at runtime
[ https://issues.apache.org/jira/browse/HIVE-14831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517511#comment-15517511 ] Ashutosh Chauhan commented on HIVE-14831: - +1 > Missing Druid dependencies at runtime > - > > Key: HIVE-14831 > URL: https://issues.apache.org/jira/browse/HIVE-14831 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14831.patch > > > Excluded some packages when shading in the initial patch that should have > been included. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14713) LDAP Authentication Provider should be covered with unit tests
[ https://issues.apache.org/jira/browse/HIVE-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14713: --- Resolution: Fixed Fix Version/s: 2.1.1 2.2.0 Status: Resolved (was: Patch Available) Patch has been committed to 2.2.0 and 2.1.1. Thanks [~yalovyyi] for the patch and [~szehon] for review. > LDAP Authentication Provider should be covered with unit tests > -- > > Key: HIVE-14713 > URL: https://issues.apache.org/jira/browse/HIVE-14713 > Project: Hive > Issue Type: Test > Components: Authentication, Tests >Affects Versions: 2.1.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14713.1.patch, HIVE-14713.2.patch, > HIVE-14713.3.patch > > > Currently LdapAuthenticationProviderImpl class is not covered with unit > tests. To make this class testable some minor refactoring will be required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp
[
https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517410#comment-15517410
]
Hive QA commented on HIVE-14412:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12830014/HIVE-14412.8.patch
{color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10567 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1290/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1290/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1290/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12830014 - PreCommit-HIVE-Build
> Add a timezone-aware timestamp
> --
>
> Key: HIVE-14412
> URL: https://issues.apache.org/jira/browse/HIVE-14412
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-14412.1.patch, HIVE-14412.2.patch,
> HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch,
> HIVE-14412.6.patch, HIVE-14412.7.patch, HIVE-14412.8.patch
>
>
> Java's Timestamp stores the time elapsed since the epoch. While it's by
> itself unambiguous, ambiguity comes when we parse a string into timestamp, or
> convert a timestamp to string, causing problems like HIVE-14305.
> To solve the issue, I think we should make timestamp aware of timezone.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517298#comment-15517298
]
Chaoyu Tang commented on HIVE-9423:
---
Yeah, it makes sense. This patch improves the usability though we might have
missed some error case (e.g. TTransportException: java.net.SocketException:
Connection reset") which is also resulted from exceeding the max worker #.
+1
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14826) Support vectorization for Parquet
[ https://issues.apache.org/jira/browse/HIVE-14826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517274#comment-15517274 ] Chao Sun commented on HIVE-14826: - Thanks [~Ferd]! please create more tasks if you feel necessary. I'll may spent some time looking into this. > Support vectorization for Parquet > - > > Key: HIVE-14826 > URL: https://issues.apache.org/jira/browse/HIVE-14826 > Project: Hive > Issue Type: New Feature >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > > Parquet vectorized reader can improve both throughput and also leverages > existing Hive vectorization execution engine. This is an umbrella ticket to > track this feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14829) metastore.sh fails due to classpath conflict with hive-service-rpc
[
https://issues.apache.org/jira/browse/HIVE-14829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14829:
---
Resolution: Invalid
Status: Resolved (was: Patch Available)
This patch is not valid anymore on Hive 2.x
I was testing something incorrect.
> metastore.sh fails due to classpath conflict with hive-service-rpc
> --
>
> Key: HIVE-14829
> URL: https://issues.apache.org/jira/browse/HIVE-14829
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-14829.1.patch
>
>
> NO PRECOMMIT TESTS
> When attempting to run metastore.sh to start a new HMS server, then the
> script fails that the metastore class cannot be found on the classpath. This
> issue happens because a new {{hive-service-rpc}} jar added on Hive 2.x is
> making conflict with {{hive-service}} jar.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14833) Add vectorization support for date truncation UDFs
[ https://issues.apache.org/jira/browse/HIVE-14833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14833: --- Assignee: (was: Jesus Camacho Rodriguez) > Add vectorization support for date truncation UDFs > -- > > Key: HIVE-14833 > URL: https://issues.apache.org/jira/browse/HIVE-14833 > Project: Hive > Issue Type: Improvement > Components: UDF >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14831) Missing Druid dependencies at runtime
[ https://issues.apache.org/jira/browse/HIVE-14831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517241#comment-15517241 ] Jesus Camacho Rodriguez commented on HIVE-14831: [~ashutoshc], could you take a look? Thanks > Missing Druid dependencies at runtime > - > > Key: HIVE-14831 > URL: https://issues.apache.org/jira/browse/HIVE-14831 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14831.patch > > > Excluded some packages when shading in the initial patch that should have > been included. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14831) Missing Druid dependencies at runtime
[ https://issues.apache.org/jira/browse/HIVE-14831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14831: --- Status: Patch Available (was: In Progress) > Missing Druid dependencies at runtime > - > > Key: HIVE-14831 > URL: https://issues.apache.org/jira/browse/HIVE-14831 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14831.patch > > > Excluded some packages when shading in the initial patch that should have > been included. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-14831) Missing Druid dependencies at runtime
[ https://issues.apache.org/jira/browse/HIVE-14831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-14831 started by Jesus Camacho Rodriguez. -- > Missing Druid dependencies at runtime > - > > Key: HIVE-14831 > URL: https://issues.apache.org/jira/browse/HIVE-14831 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14831.patch > > > Excluded some packages when shading in the initial patch that should have > been included. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14831) Missing Druid dependencies at runtime
[ https://issues.apache.org/jira/browse/HIVE-14831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14831: --- Attachment: HIVE-14831.patch > Missing Druid dependencies at runtime > - > > Key: HIVE-14831 > URL: https://issues.apache.org/jira/browse/HIVE-14831 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14831.patch > > > Excluded some packages when shading in the initial patch that should have > been included. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14818) Reduce number of retries while starting HiveServer for tests
[
https://issues.apache.org/jira/browse/HIVE-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517213#comment-15517213
]
Hive QA commented on HIVE-14818:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829972/HIVE-14818.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10559 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1289/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1289/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1289/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829972 - PreCommit-HIVE-Build
> Reduce number of retries while starting HiveServer for tests
>
>
> Key: HIVE-14818
> URL: https://issues.apache.org/jira/browse/HIVE-14818
> Project: Hive
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14818.01.patch, HIVE-14818.02.patch
>
>
> Current is 30 retries, with a 1minute sleep between each one.
> The settings are likely bad for a production cluster as well. For tests, this
> should be a lot lower.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14830) Move a majority of the MiniLlapCliDriver tests to use an inline AM
[ https://issues.apache.org/jira/browse/HIVE-14830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517141#comment-15517141 ] Matt McCline commented on HIVE-14830: - Does this mean we will be able use -Dmaven.surefire.debug? Inquiring minds want to know. > Move a majority of the MiniLlapCliDriver tests to use an inline AM > -- > > Key: HIVE-14830 > URL: https://issues.apache.org/jira/browse/HIVE-14830 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14731) Use Tez cartesian product edge in Hive (unpartitioned case only)
[ https://issues.apache.org/jira/browse/HIVE-14731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517029#comment-15517029 ] Zhiyuan Yang commented on HIVE-14731: - Would you mind taking a review? [~hagleitn] > Use Tez cartesian product edge in Hive (unpartitioned case only) > > > Key: HIVE-14731 > URL: https://issues.apache.org/jira/browse/HIVE-14731 > Project: Hive > Issue Type: Bug >Reporter: Zhiyuan Yang >Assignee: Zhiyuan Yang > Attachments: HIVE-14731.1.patch, HIVE-14731.2.patch, > HIVE-14731.3.patch, HIVE-14731.4.patch, HIVE-14731.5.patch, > HIVE-14731.6.patch, HIVE-14731.7.patch, HIVE-14731.8.patch > > > Given cartesian product edge is available in Tez now (see TEZ-3230), let's > integrate it into Hive on Tez. This allows us to have more than one reducer > in cross product queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14824) Separate fstype from cluster type in QTestUtil
[
https://issues.apache.org/jira/browse/HIVE-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517010#comment-15517010
]
Hive QA commented on HIVE-14824:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829968/HIVE-14824.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 10559 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_ctas]
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_drop_partition]
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_drop_table]
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_drop_view]
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_partition_dynamic]
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_partition_static]
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_values]
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_unencrypted_tbl]
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_load_data_to_encrypted_tables]
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_select_read_only_encrypted_tbl]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1288/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1288/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1288/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829968 - PreCommit-HIVE-Build
> Separate fstype from cluster type in QTestUtil
> --
>
> Key: HIVE-14824
> URL: https://issues.apache.org/jira/browse/HIVE-14824
> Project: Hive
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14824.01.patch
>
>
> QTestUtil cluster type encodes the file system. e.g.
> MiniClusterType.encrypted means mr + encrypted hdfs, spark means file://, mr
> means hdfs etc.
> These can be separated out. e.g. To add tests for tez against encrypted, and
> llap against encrypted - I'd need to introduce 2 new cluster types.
> Instead it's better to separate the storage into it's own types.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-10924) add support for MERGE statement
[ https://issues.apache.org/jira/browse/HIVE-10924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-10924 started by Eugene Koifman. - > add support for MERGE statement > --- > > Key: HIVE-10924 > URL: https://issues.apache.org/jira/browse/HIVE-10924 > Project: Hive > Issue Type: New Feature > Components: Query Planning, Query Processor, Transactions >Affects Versions: 1.2.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > add support for > MERGE INTO tbl USING src ON … WHEN MATCHED THEN ... WHEN NOT MATCHED THEN ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14713) LDAP Authentication Provider should be covered with unit tests
[ https://issues.apache.org/jira/browse/HIVE-14713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516992#comment-15516992 ] Illya Yalovyy commented on HIVE-14713: -- Thank you! > LDAP Authentication Provider should be covered with unit tests > -- > > Key: HIVE-14713 > URL: https://issues.apache.org/jira/browse/HIVE-14713 > Project: Hive > Issue Type: Test > Components: Authentication, Tests >Affects Versions: 2.1.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: HIVE-14713.1.patch, HIVE-14713.2.patch, > HIVE-14713.3.patch > > > Currently LdapAuthenticationProviderImpl class is not covered with unit > tests. To make this class testable some minor refactoring will be required. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516910#comment-15516910
]
Peter Vary commented on HIVE-9423:
--
I will think about this on the weekend, but my first guess is, that if there is
not enough connection in the TThreadPoolServer, then it accepts the
connections, and after the timeout closes it.
{code}
TTransport client = serverTransport_.accept();
[..]
client.close();
{code}
Ideally the executor service will not write anything into it (since it will
throw an RejectedExecutionException) - what is the configuration of the
ThreadPool in Sentry? Does it throw a RejectedExecutionException when the
Theadpool is exhausted?
On the Thrift client side we have this code:
{code}
try {
bytesRead = inputStream_.read(buf, off, len);
} catch (IOException iox) {
throw new TTransportException(TTransportException.UNKNOWN, iox);
}
if (bytesRead < 0) {
throw new TTransportException(TTransportException.END_OF_FILE);
}
{code}
Reading from the InputStream read function documentation:
{code}
If no byte is available because the stream is at end of file, the value -1 is
returned; otherwise, at least one byte is read and stored into b.
{code}
{code}
IOException - If the first byte cannot be read for any reason other than end of
file, or if the input stream has been closed, or if some other I/O error occurs.
{code}
Reading this my guess this can be a timing issue here. Thinking :)
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[
https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516900#comment-15516900
]
Xuefu Zhang commented on HIVE-14029:
Hi [~lirui], Thanks for the investigation. I'm wondering if that dependency can
be excluded in Hive's build? While the latest patch builds, it changes Hive's
existing dependency, which might cause some problem.
Also, we are upgrading the following libraries. I'm not sure If it's absolutely
necessary. From my build alone, it seems not. [~Ferd], any thoughts?
{quote}
** Jackson: 2.4.2 -> 2.6.5
** Netty version: 4.0.23.Final -> 4.0.29.Final
{quote}
> Update Spark version to 2.0.0
> -
>
> Key: HIVE-14029
> URL: https://issues.apache.org/jira/browse/HIVE-14029
> Project: Hive
> Issue Type: Bug
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
> Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch,
> HIVE-14029.3.patch, HIVE-14029.4.patch, HIVE-14029.5.patch, HIVE-14029.patch
>
>
> There are quite some new optimizations in Spark 2.0.0. We need to bump up
> Spark to 2.0.0 to benefit those performance improvements.
> To update Spark version to 2.0.0, the following changes are required:
> * Spark API updates:
> ** SparkShuffler#call return Iterator instead of Iterable
> ** SparkListener -> JavaSparkListener
> ** InputMetrics constructor doesn’t accept readMethod
> ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics
> return long type instead of integer
> * Dependency upgrade:
> ** Jackson: 2.4.2 -> 2.6.5
> ** Netty version: 4.0.23.Final -> 4.0.29.Final
> ** Scala binary version: 2.10 -> 2.11
> ** Scala version: 2.10.4 -> 2.11.8
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-14337) Skewed table stored as DIRECTORIES throws IOException for simple query
[
https://issues.apache.org/jira/browse/HIVE-14337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu resolved HIVE-14337.
-
Resolution: Cannot Reproduce
Assignee: (was: Aihua Xu)
We need to set {{set mapred.input.dir.recursive=true;}} to make it work.
> Skewed table stored as DIRECTORIES throws IOException for simple query
> --
>
> Key: HIVE-14337
> URL: https://issues.apache.org/jira/browse/HIVE-14337
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Aihua Xu
>
> CREATE TABLE list_bucket_single (key STRING, value STRING)
> SKEWED BY (key) ON (1,5,6) STORED AS DIRECTORIES;
> insert into list_bucket_single values('1', '1');
> select * from list_bucket_single;
> It will throw IOException Failed with exception
> java.io.IOException:java.io.IOException: Not a file:
> file:/user/hive/warehouse/list_bucket_single/HIVE_DEFAULT_LIST_BUCKETING_DIR_NAME.
> Seems it's not handling the folder when it's created with 'STORED AS
> DIRECTORIES'.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14821) build q test
[ https://issues.apache.org/jira/browse/HIVE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14821: -- Attachment: HIVE-14821.3.patch > build q test > > > Key: HIVE-14821 > URL: https://issues.apache.org/jira/browse/HIVE-14821 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14821.1.patch, HIVE-14821.2.patch, > HIVE-14821.3.patch, HIVE-14821.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14821) build q test
[ https://issues.apache.org/jira/browse/HIVE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14821: -- Status: Patch Available (was: Open) > build q test > > > Key: HIVE-14821 > URL: https://issues.apache.org/jira/browse/HIVE-14821 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14821.1.patch, HIVE-14821.2.patch, > HIVE-14821.3.patch, HIVE-14821.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14821) build q test
[ https://issues.apache.org/jira/browse/HIVE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14821: -- Status: Open (was: Patch Available) > build q test > > > Key: HIVE-14821 > URL: https://issues.apache.org/jira/browse/HIVE-14821 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14821.1.patch, HIVE-14821.2.patch, HIVE-14821.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[
https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516797#comment-15516797
]
Hive QA commented on HIVE-14029:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829975/HIVE-14029.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10559 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1287/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1287/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1287/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829975 - PreCommit-HIVE-Build
> Update Spark version to 2.0.0
> -
>
> Key: HIVE-14029
> URL: https://issues.apache.org/jira/browse/HIVE-14029
> Project: Hive
> Issue Type: Bug
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
> Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch,
> HIVE-14029.3.patch, HIVE-14029.4.patch, HIVE-14029.5.patch, HIVE-14029.patch
>
>
> There are quite some new optimizations in Spark 2.0.0. We need to bump up
> Spark to 2.0.0 to benefit those performance improvements.
> To update Spark version to 2.0.0, the following changes are required:
> * Spark API updates:
> ** SparkShuffler#call return Iterator instead of Iterable
> ** SparkListener -> JavaSparkListener
> ** InputMetrics constructor doesn’t accept readMethod
> ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics
> return long type instead of integer
> * Dependency upgrade:
> ** Jackson: 2.4.2 -> 2.6.5
> ** Netty version: 4.0.23.Final -> 4.0.29.Final
> ** Scala binary version: 2.10 -> 2.11
> ** Scala version: 2.10.4 -> 2.11.8
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14829) metastore.sh fails due to classpath conflict with hive-service-rpc
[
https://issues.apache.org/jira/browse/HIVE-14829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14829:
---
Attachment: HIVE-14829.1.patch
> metastore.sh fails due to classpath conflict with hive-service-rpc
> --
>
> Key: HIVE-14829
> URL: https://issues.apache.org/jira/browse/HIVE-14829
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-14829.1.patch
>
>
> When attempting to run metastore.sh to start a new HMS server, then the
> script fails that the metastore class cannot be found on the classpath. This
> issue happens because a new {{hive-service-rpc}} jar added on Hive 2.x is
> making conflict with {{hive-service}} jar.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14829) metastore.sh fails due to classpath conflict with hive-service-rpc
[
https://issues.apache.org/jira/browse/HIVE-14829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14829:
---
Target Version/s: 2.2.0, 2.1.1, 2.0.2
> metastore.sh fails due to classpath conflict with hive-service-rpc
> --
>
> Key: HIVE-14829
> URL: https://issues.apache.org/jira/browse/HIVE-14829
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-14829.1.patch
>
>
> NO PRECOMMIT TESTS
> When attempting to run metastore.sh to start a new HMS server, then the
> script fails that the metastore class cannot be found on the classpath. This
> issue happens because a new {{hive-service-rpc}} jar added on Hive 2.x is
> making conflict with {{hive-service}} jar.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14829) metastore.sh fails due to classpath conflict with hive-service-rpc
[
https://issues.apache.org/jira/browse/HIVE-14829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14829:
---
Status: Patch Available (was: Open)
> metastore.sh fails due to classpath conflict with hive-service-rpc
> --
>
> Key: HIVE-14829
> URL: https://issues.apache.org/jira/browse/HIVE-14829
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.0.1, 2.1.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-14829.1.patch
>
>
> NO PRECOMMIT TESTS
> When attempting to run metastore.sh to start a new HMS server, then the
> script fails that the metastore class cannot be found on the classpath. This
> issue happens because a new {{hive-service-rpc}} jar added on Hive 2.x is
> making conflict with {{hive-service}} jar.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14829) metastore.sh fails due to classpath conflict with hive-service-rpc
[
https://issues.apache.org/jira/browse/HIVE-14829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergio Peña updated HIVE-14829:
---
Description:
NO PRECOMMIT TESTS
When attempting to run metastore.sh to start a new HMS server, then the script
fails that the metastore class cannot be found on the classpath. This issue
happens because a new {{hive-service-rpc}} jar added on Hive 2.x is making
conflict with {{hive-service}} jar.
was:When attempting to run metastore.sh to start a new HMS server, then the
script fails that the metastore class cannot be found on the classpath. This
issue happens because a new {{hive-service-rpc}} jar added on Hive 2.x is
making conflict with {{hive-service}} jar.
> metastore.sh fails due to classpath conflict with hive-service-rpc
> --
>
> Key: HIVE-14829
> URL: https://issues.apache.org/jira/browse/HIVE-14829
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-14829.1.patch
>
>
> NO PRECOMMIT TESTS
> When attempting to run metastore.sh to start a new HMS server, then the
> script fails that the metastore class cannot be found on the classpath. This
> issue happens because a new {{hive-service-rpc}} jar added on Hive 2.x is
> making conflict with {{hive-service}} jar.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12222) Define port range in property for RPCServer
[
https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-1:
Attachment: HIVE-1.2.patch
Patch-2: added retry logic if the port is used and also changed to throw
exception if the configuration is incorrect rather than silently uses 0, since
I feel that that could confuse the users more.
> Define port range in property for RPCServer
> ---
>
> Key: HIVE-1
> URL: https://issues.apache.org/jira/browse/HIVE-1
> Project: Hive
> Issue Type: Improvement
> Components: CLI, Spark
>Affects Versions: 1.2.1
> Environment: Apache Hadoop 2.7.0
> Apache Hive 1.2.1
> Apache Spark 1.5.1
>Reporter: Andrew Lee
>Assignee: Aihua Xu
> Attachments: HIVE-1.1.patch, HIVE-1.2.patch
>
>
> Creating this JIRA after discussin with Xuefu on the dev mailing list. Would
> need some help to review and update the fields in this JIRA ticket, thanks.
> I notice that in
> ./spark-client/src/main/java/org/apache/hive/spark/client/rpc/RpcServer.java
> The port number is assigned with 0 which means it will be a random port every
> time when the RPC Server is created to talk to Spark in the same session.
> Because of this, this is causing problems to configure firewall between the
> HiveCLI RPC Server and Spark due to unpredictable port numbers here. In other
> word, users need to open all hive ports range
> from Data Node => HiveCLI (edge node).
> {code}
> this.channel = new ServerBootstrap()
> .group(group)
> .channel(NioServerSocketChannel.class)
> .childHandler(new ChannelInitializer() {
> @Override
> public void initChannel(SocketChannel ch) throws Exception {
> SaslServerHandler saslHandler = new SaslServerHandler(config);
> final Rpc newRpc = Rpc.createServer(saslHandler, config, ch,
> group);
> saslHandler.rpc = newRpc;
> Runnable cancelTask = new Runnable() {
> @Override
> public void run() {
> LOG.warn("Timed out waiting for hello from client.");
> newRpc.close();
> }
> };
> saslHandler.cancelTask = group.schedule(cancelTask,
> RpcServer.this.config.getServerConnectTimeoutMs(),
> TimeUnit.MILLISECONDS);
> }
> })
> {code}
> 2 Main reasons.
> - Most users (what I see and encounter) use HiveCLI as a command line tool,
> and in order to use that, they need to login to the edge node (via SSH). Now,
> here comes the interesting part.
> Could be true or not, but this is what I observe and encounter from time to
> time. Most users will abuse the resource on that edge node (increasing
> HADOOP_HEAPSIZE, dumping output to local disk, running huge python workflow,
> etc), this may cause the HS2 process to run into OOME, choke and die, etc.
> various resource issues including others like login, etc.
> - Analyst connects to Hive via HS2 + ODBC. So HS2 needs to be highly
> available. This makes sense to run it on the gateway node or a service node
> and separated from the HiveCLI.
> The logs are located in different location, monitoring and auditing is easier
> to run HS2 with a daemon user account, etc. so we don't want users to run
> HiveCLI where HS2 is running.
> It's better to isolate the resource this way to avoid any memory, file
> handlers, disk space, issues.
> From a security standpoint,
> - Since users can login to edge node (via SSH), the security on the edge node
> needs to be fortified and enhanced. Therefore, all the FW comes in and
> auditing.
> - Regulation/compliance for auditing is another requirement to monitor all
> traffic, specifying ports and locking down the ports makes it easier since we
> can focus
> on a range to monitor and audit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516513#comment-15516513
]
Chaoyu Tang edited comment on HIVE-9423 at 9/23/16 1:57 PM:
That was run into by the Sentry, I wonder if it could be a similar case to HS2.
It could be quite common that the client could run into
"java.net.SocketException: Connection reset" when it tries to connect to server
and reads data from socket but the socket has been closed at the server site
(TThreadPoolServer closes it due to reaching the max # and timeout).
{code}
Caused by: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Connection reset
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:277)
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport.baseOpen(SentryPolicyServiceClientDefaultImpl.java:126)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport.access$000(SentryPolicyServiceClientDefaultImpl.java:85)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport$1.run(SentryPolicyServiceClientDefaultImpl.java:112)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport$1.run(SentryPolicyServiceClientDefaultImpl.java:110)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1714)
... 23 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
{code}
was (Author: ctang.ma):
That was run into by the Sentry, I wonder if it could be a similar case to HS2:
{code}
Caused by: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Connection reset
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:277)
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport.baseOpen(SentryPolicyServiceClientDefaultImpl.java:126)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport.access$000(SentryPolicyServiceClientDefaultImpl.java:85)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport$1.run(SentryPolicyServiceClientDefaultImpl.java:112)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport$1.run(SentryPolicyServiceClientDefaultImpl.java:110)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1714)
... 23 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
{code}
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
>
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516513#comment-15516513
]
Chaoyu Tang commented on HIVE-9423:
---
That was run into by the Sentry, I wonder if it could be a similar case to HS2:
{code}
Caused by: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Connection reset
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:277)
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport.baseOpen(SentryPolicyServiceClientDefaultImpl.java:126)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport.access$000(SentryPolicyServiceClientDefaultImpl.java:85)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport$1.run(SentryPolicyServiceClientDefaultImpl.java:112)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl$UgiSaslClientTransport$1.run(SentryPolicyServiceClientDefaultImpl.java:110)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1714)
... 23 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
{code}
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14820) RPC server for spark inside HS2 is not getting server address properly
[
https://issues.apache.org/jira/browse/HIVE-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14820:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks Yongzhi for reviewing.
> RPC server for spark inside HS2 is not getting server address properly
> --
>
> Key: HIVE-14820
> URL: https://issues.apache.org/jira/browse/HIVE-14820
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.2.0
>
> Attachments: HIVE-14820.1.patch
>
>
> When hive.spark.client.rpc.server.address is configured, this property is not
> retrieved properly because we are getting the value by {{String hiveHost =
> config.get(HiveConf.ConfVars.SPARK_RPC_SERVER_ADDRESS);}} which always
> returns null in getServerAddress() call of RpcConfiguration.java. Rather it
> should be {{String hiveHost =
> config.get(HiveConf.ConfVars.SPARK_RPC_SERVER_ADDRESS.varname);}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14820) RPC server for spark inside HS2 is not getting server address properly
[
https://issues.apache.org/jira/browse/HIVE-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516405#comment-15516405
]
Aihua Xu commented on HIVE-14820:
-
Those tests don't seem to be related to the change.
> RPC server for spark inside HS2 is not getting server address properly
> --
>
> Key: HIVE-14820
> URL: https://issues.apache.org/jira/browse/HIVE-14820
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14820.1.patch
>
>
> When hive.spark.client.rpc.server.address is configured, this property is not
> retrieved properly because we are getting the value by {{String hiveHost =
> config.get(HiveConf.ConfVars.SPARK_RPC_SERVER_ADDRESS);}} which always
> returns null in getServerAddress() call of RpcConfiguration.java. Rather it
> should be {{String hiveHost =
> config.get(HiveConf.ConfVars.SPARK_RPC_SERVER_ADDRESS.varname);}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14821) build q test
[
https://issues.apache.org/jira/browse/HIVE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516399#comment-15516399
]
Hive QA commented on HIVE-14821:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829962/HIVE-14821.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1286/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1286/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1286/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-09-23 13:11:55.094
+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1286/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-09-23 13:11:55.096
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
da376eb..f284b6d master -> origin/master
+ git reset --hard HEAD
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
HEAD is now at da376eb HIVE-14579: Add support for date extract (Jesus Camacho
Rodriguez, reviewed by Ashutosh Chauhan)
+ git clean -f -d
warning: unable to access '/home/sseth/.config/git/ignore': Permission denied
+ git checkout master
warning: unable to access '/home/sseth/.config/git/ignore': Permission denied
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
HEAD is now at f284b6d HIVE-14805: Subquery inside a view will have the object
in the subquery as the direct input (Aihua Xu, reviewed by Yongzhi Chen)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-09-23 13:11:56.550
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g:329
error: ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g: patch does not
apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829962 - PreCommit-HIVE-Build
> build q test
>
>
> Key: HIVE-14821
> URL: https://issues.apache.org/jira/browse/HIVE-14821
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-14821.1.patch, HIVE-14821.2.patch, HIVE-14821.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14805) Subquery inside a view will have the object in the subquery as the direct input
[
https://issues.apache.org/jira/browse/HIVE-14805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14805:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks Yongzhi for reviewing.
> Subquery inside a view will have the object in the subquery as the direct
> input
>
>
> Key: HIVE-14805
> URL: https://issues.apache.org/jira/browse/HIVE-14805
> Project: Hive
> Issue Type: Bug
> Components: Views
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Fix For: 2.2.0
>
> Attachments: HIVE-14805.1.patch, HIVE-14805.2.patch
>
>
> Here is the repro steps.
> {noformat}
> create table t1(col string);
> create view v1 as select * from t1;
> create view dataview as select * from (select * from v1) v2;
> select * from dataview;
> {noformat}
> If hive is configured with authorization hook like Sentry, it will require
> the access not only for dataview but also for v1, which should not be
> required.
> The subquery seems to not carry insideview property from the parent query.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516381#comment-15516381
]
Peter Vary commented on HIVE-9423:
--
Hi [~ctang.ma],
I had finished my other tasks, and was about to collect the logs for you.
HiveServer2:
{code}
2016-09-23T14:59:23,142 WARN [Thread-7] server.TThreadPoolServer: Task has
been rejected by ExecutorService 10 times till timedout, reason:
java.util.concurrent.RejectedExecutionException: Task
org.apache.thrift.server.TThreadPoolServer$WorkerProcess@798d4da3 rejected from
org.apache.hive.service.cli.thrift.ThreadPoolExecutorWithOomHook@f7ed8ef[Running,
pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 1]
{code}
Beeline without patch, with --verbose=true:
{code}
Connecting to jdbc:hive2://localhost:1
16/09/23 14:59:23 [main]: WARN jdbc.HiveConnection: Failed to connect to
localhost:1
HS2 may be unavailable, check server status
Error: Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:1: null (state=08S01,code=0)
java.sql.SQLException: Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:1: null
at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:209)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at
org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145)
at
org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209)
at org.apache.hive.beeline.Commands.connect(Commands.java:1524)
at org.apache.hive.beeline.Commands.connect(Commands.java:1419)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56)
at
org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1128)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1167)
at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:798)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:886)
at
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:512)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:495)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:307)
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at
org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:227)
at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:182)
... 24 more
Beeline version 2.2.0-SNAPSHOT by Apache Hive
{code}
In my test cases the server side of Thrift closes the connection without
writing any data to the output in case of reaching the max # of thrift threads.
As for your question, I have not tried kerberized Beeline/HS2. If there is
another timeout somewhere, which shorter/equal than the one defined by
hive.server2.thrift.login.timeout which is used by ThriftBinaryCLIService, then
it might throw the TTransportException with an inner "Connection reset"
IOException and with the type of TTransportException.UNKNOWN. It would be good
to find the source of the Connection reset. Could you please post your stack
trace? Meanwhile I am trying to reproduce your output.
Thanks,
Peter
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components:
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516382#comment-15516382
]
Peter Vary commented on HIVE-9423:
--
Hi [~ctang.ma],
I had finished my other tasks, and was about to collect the logs for you.
HiveServer2:
{code}
2016-09-23T14:59:23,142 WARN [Thread-7] server.TThreadPoolServer: Task has
been rejected by ExecutorService 10 times till timedout, reason:
java.util.concurrent.RejectedExecutionException: Task
org.apache.thrift.server.TThreadPoolServer$WorkerProcess@798d4da3 rejected from
org.apache.hive.service.cli.thrift.ThreadPoolExecutorWithOomHook@f7ed8ef[Running,
pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 1]
{code}
Beeline without patch, with --verbose=true:
{code}
Connecting to jdbc:hive2://localhost:1
16/09/23 14:59:23 [main]: WARN jdbc.HiveConnection: Failed to connect to
localhost:1
HS2 may be unavailable, check server status
Error: Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:1: null (state=08S01,code=0)
java.sql.SQLException: Could not open client transport with JDBC Uri:
jdbc:hive2://localhost:1: null
at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:209)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at
org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145)
at
org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209)
at org.apache.hive.beeline.Commands.connect(Commands.java:1524)
at org.apache.hive.beeline.Commands.connect(Commands.java:1419)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56)
at
org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1128)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1167)
at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:798)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:886)
at
org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:512)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:495)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.thrift.transport.TTransportException
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:307)
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at
org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:227)
at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:182)
... 24 more
Beeline version 2.2.0-SNAPSHOT by Apache Hive
{code}
In my test cases the server side of Thrift closes the connection without
writing any data to the output in case of reaching the max # of thrift threads.
As for your question, I have not tried kerberized Beeline/HS2. If there is
another timeout somewhere, which shorter/equal than the one defined by
hive.server2.thrift.login.timeout which is used by ThriftBinaryCLIService, then
it might throw the TTransportException with an inner "Connection reset"
IOException and with the type of TTransportException.UNKNOWN. It would be good
to find the source of the Connection reset. Could you please post your stack
trace? Meanwhile I am trying to reproduce your output.
Thanks,
Peter
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components:
[jira] [Commented] (HIVE-9423) HiveServer2: Provide the user with different error messages depending on the Thrift client exception code
[
https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516278#comment-15516278
]
Chaoyu Tang commented on HIVE-9423:
---
In some case when the configured max # of thrift thread is reached, I think the
client might also run into the TTransportException wrapping the
java.net.SocketException with message "Connection reset". I wonder if you have
run into such a case or not, and have you tried that in kerberized Beeline/HS2?
> HiveServer2: Provide the user with different error messages depending on the
> Thrift client exception code
> -
>
> Key: HIVE-9423
> URL: https://issues.apache.org/jira/browse/HIVE-9423
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>Reporter: Vaibhav Gumashta
>Assignee: Peter Vary
> Attachments: HIVE-9423.2.patch, HIVE-9423.3.patch, HIVE-9423.4.patch,
> HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client
> connections is greater than {{hive.server2.thrift.max.worker.threads}},
> HiveServer2 stops accepting new connections and ends up having to be
> restarted. This should be handled more gracefully by the server and the JDBC
> driver, so that the end user gets aware of the problem and can take
> appropriate steps (either close existing connections or bump of the config
> value or use multiple server instances with dynamic service discovery
> enabled). Similarly, we should also review the behaviour of background thread
> pool to have a well defined behavior on the the pool getting exhausted.
> Ideally implementing some form of general admission control will be a better
> solution, so that we do not accept new work unless sufficient resources are
> available and display graceful degradation under overload.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0
[
https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515964#comment-15515964
]
Rui Li commented on HIVE-14029:
---
The offending jar comes as a dependency of jersey-client:
{noformat}
[INFO] | +- org.glassfish.jersey.core:jersey-client:jar:2.22.2:compile
[INFO] | | +- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:compile
[INFO] | | +- org.glassfish.hk2:hk2-api:jar:2.4.0-b34:compile
[INFO] | | | +- org.glassfish.hk2:hk2-utils:jar:2.4.0-b34:compile
[INFO] | | | \-
org.glassfish.hk2.external:aopalliance-repackaged:jar:2.4.0-b34:compile
[INFO] | | +- org.glassfish.hk2.external:javax.inject:jar:2.4.0-b34:compile
[INFO] | | \- org.glassfish.hk2:hk2-locator:jar:2.4.0-b34:compile
[INFO] | | \- org.javassist:javassist:jar:3.18.1-GA:compile
{noformat}
I think it's related to SPARK-12154. Spark updated to Jersey 2 and replaced
com.sun.jersey with org.glassfish.jersey. Good news is seems we don't pack the
jersey stuff in hive-exec. But not sure if this only affects the compile.
[~xuefuz] what do you think about this?
> Update Spark version to 2.0.0
> -
>
> Key: HIVE-14029
> URL: https://issues.apache.org/jira/browse/HIVE-14029
> Project: Hive
> Issue Type: Bug
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
> Attachments: HIVE-14029.1.patch, HIVE-14029.2.patch,
> HIVE-14029.3.patch, HIVE-14029.4.patch, HIVE-14029.5.patch, HIVE-14029.patch
>
>
> There are quite some new optimizations in Spark 2.0.0. We need to bump up
> Spark to 2.0.0 to benefit those performance improvements.
> To update Spark version to 2.0.0, the following changes are required:
> * Spark API updates:
> ** SparkShuffler#call return Iterator instead of Iterable
> ** SparkListener -> JavaSparkListener
> ** InputMetrics constructor doesn’t accept readMethod
> ** Method remoteBlocksFetched and localBlocksFetched in ShuffleReadMetrics
> return long type instead of integer
> * Dependency upgrade:
> ** Jackson: 2.4.2 -> 2.6.5
> ** Netty version: 4.0.23.Final -> 4.0.29.Final
> ** Scala binary version: 2.10 -> 2.11
> ** Scala version: 2.10.4 -> 2.11.8
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14821) build q test
[
https://issues.apache.org/jira/browse/HIVE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515962#comment-15515962
]
Hive QA commented on HIVE-14821:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829962/HIVE-14821.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1285/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1285/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1285/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-09-23 09:39:31.456
+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-1285/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-09-23 09:39:31.458
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
3237bb2..da376eb master -> origin/master
+ git reset --hard HEAD
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
HEAD is now at 3237bb2 HIVE-14774: Canceling query using Ctrl-C in beeline
might lead to stale locks (Chaoyu Tang, reviewed by Jimmy Xiang, Mohit
Sabharwal)
+ git clean -f -d
warning: unable to access '/home/sseth/.config/git/ignore': Permission denied
+ git checkout master
warning: unable to access '/home/sseth/.config/git/ignore': Permission denied
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
HEAD is now at da376eb HIVE-14579: Add support for date extract (Jesus Camacho
Rodriguez, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-09-23 09:39:33.234
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
warning: unable to access '/home/sseth/.config/git/attributes': Permission
denied
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g:329
error: ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g: patch does not
apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829962 - PreCommit-HIVE-Build
> build q test
>
>
> Key: HIVE-14821
> URL: https://issues.apache.org/jira/browse/HIVE-14821
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-14821.1.patch, HIVE-14821.2.patch, HIVE-14821.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14579) Add support for date extract
[ https://issues.apache.org/jira/browse/HIVE-14579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-14579: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Thanks! :) Pushed to master. > Add support for date extract > > > Key: HIVE-14579 > URL: https://issues.apache.org/jira/browse/HIVE-14579 > Project: Hive > Issue Type: Sub-task > Components: UDF >Reporter: Ashutosh Chauhan >Assignee: Jesus Camacho Rodriguez > Fix For: 2.2.0 > > Attachments: HIVE-14579.01.patch, HIVE-14579.patch, HIVE-14579.patch > > > https://www.postgresql.org/docs/9.1/static/functions-datetime.html#FUNCTIONS-DATETIME-EXTRACT -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14751) Add support for date truncation
[
https://issues.apache.org/jira/browse/HIVE-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-14751:
---
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Pushed to master, thanks for the review [~ashutoshc]!
> Add support for date truncation
> ---
>
> Key: HIVE-14751
> URL: https://issues.apache.org/jira/browse/HIVE-14751
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Fix For: 2.2.0
>
> Attachments: HIVE-14751.patch
>
>
> Add support for {{floor ( to )}}, which is equivalent to
> {{date_trunc(, )}}.
> https://www.postgresql.org/docs/9.1/static/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14828) Cloud/S3: Stats publishing should be on HDFS instead of S3
[
https://issues.apache.org/jira/browse/HIVE-14828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515918#comment-15515918
]
Rajesh Balamohan edited comment on HIVE-14828 at 9/23/16 9:20 AM:
--
For master branch, HIVE-14270 fixes for s3,s3a,s3n.
was (Author: rajesh.balamohan):
For master branch, HIVE-14270 fixes for blob stores s3,s3a,s3n.
> Cloud/S3: Stats publishing should be on HDFS instead of S3
> --
>
> Key: HIVE-14828
> URL: https://issues.apache.org/jira/browse/HIVE-14828
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Affects Versions: 1.2.0
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-14828.branch-1.2.001.patch
>
>
> Currently, stats files are created in S3. Later as a part of
> FSStatsAggregator, it reads this file and populates MS again.
> {noformat}
> 2016-09-23 05:57:46,772 INFO [main]: fs.FSStatsPublisher
> (FSStatsPublisher.java:init(49)) - created :
> s3a://BUCKET/test/.hive-staging_hive_2016-09-23_05-57-34_309_2648485988937054815-1/-ext-10001
> 2016-09-23 05:57:46,773 DEBUG [main]: fs.FSStatsAggregator
> (FSStatsAggregator.java:connect(53)) - About to read stats from :
> s3a://BUCKET/test/.hive-staging_hive_2016-09-23_05-57-34_309_2648485988937054815-1/-ext-10001
> {noformat}
> Instead of this, stats can be written directly on to HDFS and read locally
> instead of S3, which would help in reducing couple of calls to S3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14828) Cloud/S3: Stats publishing should be on HDFS instead of S3
[
https://issues.apache.org/jira/browse/HIVE-14828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-14828:
Attachment: HIVE-14828.branch-1.2.001.patch
For master branch, HIVE-14270 fixes for blob stores.
> Cloud/S3: Stats publishing should be on HDFS instead of S3
> --
>
> Key: HIVE-14828
> URL: https://issues.apache.org/jira/browse/HIVE-14828
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Affects Versions: 1.2.0
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-14828.branch-1.2.001.patch
>
>
> Currently, stats files are created in S3. Later as a part of
> FSStatsAggregator, it reads this file and populates MS again.
> {noformat}
> 2016-09-23 05:57:46,772 INFO [main]: fs.FSStatsPublisher
> (FSStatsPublisher.java:init(49)) - created :
> s3a://BUCKET/test/.hive-staging_hive_2016-09-23_05-57-34_309_2648485988937054815-1/-ext-10001
> 2016-09-23 05:57:46,773 DEBUG [main]: fs.FSStatsAggregator
> (FSStatsAggregator.java:connect(53)) - About to read stats from :
> s3a://BUCKET/test/.hive-staging_hive_2016-09-23_05-57-34_309_2648485988937054815-1/-ext-10001
> {noformat}
> Instead of this, stats can be written directly on to HDFS and read locally
> instead of S3, which would help in reducing couple of calls to S3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14828) Cloud/S3: Stats publishing should be on HDFS instead of S3
[
https://issues.apache.org/jira/browse/HIVE-14828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515918#comment-15515918
]
Rajesh Balamohan edited comment on HIVE-14828 at 9/23/16 9:19 AM:
--
For master branch, HIVE-14270 fixes for blob stores s3,s3a,s3n.
was (Author: rajesh.balamohan):
For master branch, HIVE-14270 fixes for blob stores.
> Cloud/S3: Stats publishing should be on HDFS instead of S3
> --
>
> Key: HIVE-14828
> URL: https://issues.apache.org/jira/browse/HIVE-14828
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Affects Versions: 1.2.0
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-14828.branch-1.2.001.patch
>
>
> Currently, stats files are created in S3. Later as a part of
> FSStatsAggregator, it reads this file and populates MS again.
> {noformat}
> 2016-09-23 05:57:46,772 INFO [main]: fs.FSStatsPublisher
> (FSStatsPublisher.java:init(49)) - created :
> s3a://BUCKET/test/.hive-staging_hive_2016-09-23_05-57-34_309_2648485988937054815-1/-ext-10001
> 2016-09-23 05:57:46,773 DEBUG [main]: fs.FSStatsAggregator
> (FSStatsAggregator.java:connect(53)) - About to read stats from :
> s3a://BUCKET/test/.hive-staging_hive_2016-09-23_05-57-34_309_2648485988937054815-1/-ext-10001
> {noformat}
> Instead of this, stats can be written directly on to HDFS and read locally
> instead of S3, which would help in reducing couple of calls to S3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14828) Cloud/S3: Stats publishing should be on HDFS instead of S3
[
https://issues.apache.org/jira/browse/HIVE-14828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-14828:
Affects Version/s: 1.2.0
> Cloud/S3: Stats publishing should be on HDFS instead of S3
> --
>
> Key: HIVE-14828
> URL: https://issues.apache.org/jira/browse/HIVE-14828
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
>Affects Versions: 1.2.0
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
>
> Currently, stats files are created in S3. Later as a part of
> FSStatsAggregator, it reads this file and populates MS again.
> {noformat}
> 2016-09-23 05:57:46,772 INFO [main]: fs.FSStatsPublisher
> (FSStatsPublisher.java:init(49)) - created :
> s3a://BUCKET/test/.hive-staging_hive_2016-09-23_05-57-34_309_2648485988937054815-1/-ext-10001
> 2016-09-23 05:57:46,773 DEBUG [main]: fs.FSStatsAggregator
> (FSStatsAggregator.java:connect(53)) - About to read stats from :
> s3a://BUCKET/test/.hive-staging_hive_2016-09-23_05-57-34_309_2648485988937054815-1/-ext-10001
> {noformat}
> Instead of this, stats can be written directly on to HDFS and read locally
> instead of S3, which would help in reducing couple of calls to S3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14751) Add support for date truncation
[
https://issues.apache.org/jira/browse/HIVE-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515909#comment-15515909
]
Jesus Camacho Rodriguez commented on HIVE-14751:
You are right, only timestamp is supported as the functions are taken directly
from Druid and they work with that type. But actually, we should create a
follow-up to extend them; I will do that. In addition, we should create the
vectorized counterparts too.
> Add support for date truncation
> ---
>
> Key: HIVE-14751
> URL: https://issues.apache.org/jira/browse/HIVE-14751
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14751.patch
>
>
> Add support for {{floor ( to )}}, which is equivalent to
> {{date_trunc(, )}}.
> https://www.postgresql.org/docs/9.1/static/functions-datetime.html#FUNCTIONS-DATETIME-TRUNC
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14412) Add a timezone-aware timestamp
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14412: -- Attachment: HIVE-14412.8.patch Thanks [~jdere] for your comments! Update v8 patch accordingly. > Add a timezone-aware timestamp > -- > > Key: HIVE-14412 > URL: https://issues.apache.org/jira/browse/HIVE-14412 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-14412.1.patch, HIVE-14412.2.patch, > HIVE-14412.3.patch, HIVE-14412.4.patch, HIVE-14412.5.patch, > HIVE-14412.6.patch, HIVE-14412.7.patch, HIVE-14412.8.patch > > > Java's Timestamp stores the time elapsed since the epoch. While it's by > itself unambiguous, ambiguity comes when we parse a string into timestamp, or > convert a timestamp to string, causing problems like HIVE-14305. > To solve the issue, I think we should make timestamp aware of timezone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14817) Shutdown the SessionManager timeoutChecker thread properly upon shutdown
[ https://issues.apache.org/jira/browse/HIVE-14817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515774#comment-15515774 ] Thejas M Nair commented on HIVE-14817: -- +1 > Shutdown the SessionManager timeoutChecker thread properly upon shutdown > > > Key: HIVE-14817 > URL: https://issues.apache.org/jira/browse/HIVE-14817 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-14817.01.patch > > > Shutdown for SessionManager waits 10seconds for all threads on the > threadpoolExecutor to shutdown correctly. > The cleaner thread - with default settings - will take 6 hours to shutdown, > so essentially any shutdown of HS2 is always delayed by 10s. > The cleaner thread should be shutdown properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14821) build q test
[
https://issues.apache.org/jira/browse/HIVE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515765#comment-15515765
]
Hive QA commented on HIVE-14821:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12829962/HIVE-14821.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10564 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testMetaDataCounts
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdate.testMergeProto
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testMergeProto
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1284/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1284/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-Build-1284/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12829962 - PreCommit-HIVE-Build
> build q test
>
>
> Key: HIVE-14821
> URL: https://issues.apache.org/jira/browse/HIVE-14821
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-14821.1.patch, HIVE-14821.2.patch, HIVE-14821.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14797) reducer number estimating may lead to data skew
[
https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511843#comment-15511843
]
roncenzhao edited comment on HIVE-14797 at 9/23/16 8:04 AM:
I don't think they are related to my patch. The failure testcases have run
successfully in my own machine.
was (Author: roncenzhao):
I think they are not related to my patch. The failure testcases have run
successfully in my own machine.
> reducer number estimating may lead to data skew
> ---
>
> Key: HIVE-14797
> URL: https://issues.apache.org/jira/browse/HIVE-14797
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: roncenzhao
>Assignee: roncenzhao
> Attachments: HIVE-14797.2.patch, HIVE-14797.3.patch, HIVE-14797.patch
>
>
> HiveKey's hash code is generated by multipling by 31 key by key which is
> implemented in method `ObjectInspectorUtils.getBucketHashCode()`:
> for (int i = 0; i < bucketFields.length; i++) {
> int fieldHash = ObjectInspectorUtils.hashCode(bucketFields[i],
> bucketFieldInspectors[i]);
> hashCode = 31 * hashCode + fieldHash;
> }
> The follow example will lead to data skew:
> I hava two table called tbl1 and tbl2 and they have the same column: a int, b
> string. The values of column 'a' in both two tables are not skew, but values
> of column 'b' in both two tables are skew.
> When my sql is "select * from tbl1 join tbl2 on tbl1.a=tbl2.a and
> tbl1.b=tbl2.b" and the estimated reducer number is 31, it will lead to data
> skew.
> As we know, the HiveKey's hash code is generated by `hash(a)*31 + hash(b)`.
> When reducer number is 31 the reducer No. of each row is `hash(b)%31`. In the
> result, the job will be skew.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14426) Extensive logging on info level in WebHCat
[ https://issues.apache.org/jira/browse/HIVE-14426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515731#comment-15515731 ] Peter Vary commented on HIVE-14426: --- Thanks for the info [~ekoifman]! > Extensive logging on info level in WebHCat > -- > > Key: HIVE-14426 > URL: https://issues.apache.org/jira/browse/HIVE-14426 > Project: Hive > Issue Type: Bug >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14426.2.patch, HIVE-14426.3.patch, > HIVE-14426.4.patch, HIVE-14426.5.patch, HIVE-14426.6.patch, > HIVE-14426.7.patch, HIVE-14426.8.patch, HIVE-14426.9-branch-2.1.patch, > HIVE-14426.9.patch, HIVE-14426.patch > > > There is an extensive logging in WebHCat at info level, and even some > sensitive information could be logged -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13903) getFunctionInfo is downloading jar on every call
[ https://issues.apache.org/jira/browse/HIVE-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515728#comment-15515728 ] Rajat Khandelwal commented on HIVE-13903: - Last I checked, it was getting downloaded more than 30 times for a single query containing udf. The query was being issued through an external thrift client. > getFunctionInfo is downloading jar on every call > > > Key: HIVE-13903 > URL: https://issues.apache.org/jira/browse/HIVE-13903 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Fix For: 2.1.0 > > Attachments: HIVE-13903.01.patch, HIVE-13903.01.patch, > HIVE-13903.02.patch > > > on queries using permanent udfs, the jar file of the udf is downloaded > multiple times. Each call originating from Registry.getFunctionInfo. This > increases time for the query, especially if that query is just an explain > query. The jar should be downloaded once, and not downloaded again if the udf > class is accessible in the current thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14815) Implement Parquet vectorization reader
[ https://issues.apache.org/jira/browse/HIVE-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-14815: Summary: Implement Parquet vectorization reader (was: Support vectorization for Parquet) > Implement Parquet vectorization reader > --- > > Key: HIVE-14815 > URL: https://issues.apache.org/jira/browse/HIVE-14815 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > > Parquet doesn't provide a vectorized reader which can be used by Hive > directly. Also for Decimal Column batch, it consists of a batch of > HiveDecimal which is a Hive type which is unknown for Parquet. To support > Hive vectorization execution engine in Hive, we have to implement the > vectorized Parquet reader in Hive side. To limit the performance impacts, we > need to implement a page level vectorized reader. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14826) Support vectorization for Parquet
[ https://issues.apache.org/jira/browse/HIVE-14826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515625#comment-15515625 ] Ferdinand Xu commented on HIVE-14826: - Hi [~csun], as discussed offline, this ticket is created as an umbrella ticket. > Support vectorization for Parquet > - > > Key: HIVE-14826 > URL: https://issues.apache.org/jira/browse/HIVE-14826 > Project: Hive > Issue Type: New Feature >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > > Parquet vectorized reader can improve both throughput and also leverages > existing Hive vectorization execution engine. This is an umbrella ticket to > track this feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14815) Support vectorization for Parquet
[ https://issues.apache.org/jira/browse/HIVE-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-14815: Issue Type: Sub-task (was: Bug) Parent: HIVE-14826 > Support vectorization for Parquet > - > > Key: HIVE-14815 > URL: https://issues.apache.org/jira/browse/HIVE-14815 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > > Parquet doesn't provide a vectorized reader which can be used by Hive > directly. Also for Decimal Column batch, it consists of a batch of > HiveDecimal which is a Hive type which is unknown for Parquet. To support > Hive vectorization execution engine in Hive, we have to implement the > vectorized Parquet reader in Hive side. To limit the performance impacts, we > need to implement a page level vectorized reader. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-5867) JDBC driver and beeline should support executing an initial SQL script
[
https://issues.apache.org/jira/browse/HIVE-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianguo Tian updated HIVE-5867:
---
Attachment: HIVE-5867.2.patch
> JDBC driver and beeline should support executing an initial SQL script
> --
>
> Key: HIVE-5867
> URL: https://issues.apache.org/jira/browse/HIVE-5867
> Project: Hive
> Issue Type: Improvement
> Components: Clients, JDBC
>Reporter: Prasad Mujumdar
>Assignee: Jianguo Tian
> Attachments: HIVE-5867.1.patch, HIVE-5867.2.patch
>
>
> HiveCLI support the .hiverc script that is executed at the start of the
> session. This is helpful for things like registering UDFs, session specific
> configs etc.
> This functionality is missing for beeline and JDBC clients. It would be
> useful for JDBC driver to support an init script with SQL statements that's
> automatically executed after connection. The script path can be specified via
> JDBC connection URL. For example
> {noformat}
> jdbc:hive2://localhost:1/default;initScript=/home/user1/scripts/init.sql
> {noformat}
> This can be added to Beeline's command line option like "-i
> /home/user1/scripts/init.sql"
> To help transition from HiveCLI to Beeline, we can keep the default init
> script as $HOME/.hiverc
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
