[jira] [Commented] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[
https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701479#comment-16701479
]
Hive QA commented on HIVE-20915:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} ql: The patch generated 0 new + 236 unchanged - 1
fixed = 236 total (was 237) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hive-blobstore passed checkstyle {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15090/dev-support/hive-personality.sh
|
| git revision | master / de7b3a5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15090/yetus/whitespace-tabs.txt
|
| modules | C: ql itests itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15090/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make dynamic sort partition optimization available to HoS and MR
>
>
> Key: HIVE-20915
> URL: https://issues.apache.org/jira/browse/HIVE-20915
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch,
> HIVE-20915.3.patch
>
>
> HIVE-20703 put dynamic sort partition optimization under cost based decision,
> but it also makes the optimizer only available to tez.
> hive.optimize.sort.dynamic.partition works with other execution engines for a
> long time, we should keep the optimizer available to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20976) JDBC queries containing joins gives wrong results
[
https://issues.apache.org/jira/browse/HIVE-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701459#comment-16701459
]
Hive QA commented on HIVE-20976:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949775/HIVE-20976.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 78 failed/errored test(s), 15636 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterPartition
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterTable
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterTableCascade
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testAlterViewParititon
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testColumnStatistics
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testComplexTable
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testComplexTypeApi
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testConcurrentMetastores
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testCreateAndGetTableWithDriver
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testCreateTableSettingId
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDBLocationChange
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDBOwner
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDBOwnerChange
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabase
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabaseLocation
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDatabaseLocationWithPermissionProblems
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDropDatabaseCascadeMVMultiDB
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testDropTable
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testFilterLastPartition
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testFilterSinglePartition
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testFunctionWithResources
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetConfigValue
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetMetastoreUuid
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetPartitionsWithSpec
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetSchemaWithNoClassDefFoundError
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetTableObjects
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testGetUUIDInParallel
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testJDOPersistanceManagerCleanup
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testListPartitionNames
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testListPartitions
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testListPartitionsWihtLimitEnabled
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testNameMethods
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testPartition
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testPartitionFilter
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testRenamePartition
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testRetriableClientWithConnLifetime
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSimpleFunction
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSimpleTable
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSimpleTypeApi
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testStatsFastTrivial
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testSynchronized
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testTableDatabase
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testTableFilter
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testUpdatePartitionStat_doesNotUpdateStats
(batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testValidateTableCols
(batchId=230)
org.apache.hadoop.hive.metastore.client.TestDatabases.createDatabaseInNonExistentCatalog[Remote]
[jira] [Updated] (HIVE-20978) "hive.jdbc.*" should add to sqlStdAuthSafeVarNameRegexes
[ https://issues.apache.org/jira/browse/HIVE-20978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20978: -- Attachment: HIVE-20978.1.patch > "hive.jdbc.*" should add to sqlStdAuthSafeVarNameRegexes > - > > Key: HIVE-20978 > URL: https://issues.apache.org/jira/browse/HIVE-20978 > Project: Hive > Issue Type: Bug > Components: Configuration >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20978.1.patch > > > User should be able to change hive.jdbc settings, include > "hive.jdbc.pushdown.enable". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20978) "hive.jdbc.*" should add to sqlStdAuthSafeVarNameRegexes
[ https://issues.apache.org/jira/browse/HIVE-20978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai reassigned HIVE-20978: - > "hive.jdbc.*" should add to sqlStdAuthSafeVarNameRegexes > - > > Key: HIVE-20978 > URL: https://issues.apache.org/jira/browse/HIVE-20978 > Project: Hive > Issue Type: Bug > Components: Configuration >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20978.1.patch > > > User should be able to change hive.jdbc settings, include > "hive.jdbc.pushdown.enable". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20976) JDBC queries containing joins gives wrong results
[
https://issues.apache.org/jira/browse/HIVE-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701433#comment-16701433
]
Hive QA commented on HIVE-20976:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
15s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
38s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} jdbc-handler in master has 12 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} jdbc-handler: The patch generated 1 new + 13 unchanged
- 0 fixed = 14 total (was 13) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15089/dev-support/hive-personality.sh
|
| git revision | master / de7b3a5 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15089/yetus/diff-checkstyle-jdbc-handler.txt
|
| modules | C: standalone-metastore/metastore-common ql jdbc-handler U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15089/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> JDBC queries containing joins gives wrong results
> -
>
> Key: HIVE-20976
> URL: https://issues.apache.org/jira/browse/HIVE-20976
> Project: Hive
> Issue Type: Bug
> Components: JDBC, StorageHandler
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20976.patch
>
>
> JDBC queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> jdbc_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM jdbc_table_with_nulls
> GROUP BY `username`
> ORDER BY
[jira] [Updated] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[ https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-20915: Attachment: HIVE-20915.3.patch > Make dynamic sort partition optimization available to HoS and MR > > > Key: HIVE-20915 > URL: https://issues.apache.org/jira/browse/HIVE-20915 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch, > HIVE-20915.3.patch > > > HIVE-20703 put dynamic sort partition optimization under cost based decision, > but it also makes the optimizer only available to tez. > hive.optimize.sort.dynamic.partition works with other execution engines for a > long time, we should keep the optimizer available to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20794) Use Zookeeper for metastore service discovery
[ https://issues.apache.org/jira/browse/HIVE-20794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek updated HIVE-20794: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Use Zookeeper for metastore service discovery > - > > Key: HIVE-20794 > URL: https://issues.apache.org/jira/browse/HIVE-20794 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20794.01, HIVE-20794.02, HIVE-20794.03, > HIVE-20794.03, HIVE-20794.04, HIVE-20794.05, HIVE-20794.06, HIVE-20794.07, > HIVE-20794.07, HIVE-20794.08, HIVE-20794.08 > > > Right now, multiple metastore services can be specified in > hive.metastore.uris configuration, but that list is static and can not be > modified dynamically. Use Zookeeper for dynamic service discovery of > metastore. > h3. Improve ZooKeeperHiveHelper class (suggestions for name welcome) > The Zookeeper related code (for service discovery) accesses Zookeeper > parameters directly from HiveConf. The class is changed so that it could be > used for both HiveServer2 and Metastore server and works with both the > configurations. Following methods from HiveServer2 are now moved into > ZooKeeperHiveHelper. # startZookeeperClient # addServerInstanceToZooKeeper # > removeServerInstanceFromZooKeeper > h3. HiveMetaStore conf changes > # THRIFT_URIS (hive.metastore.uris) can also be used to specify ZooKeeper > quorum. When THRIFT_SERVICE_DISCOVERY_MODE > (hive.metastore.service.discovery.mode) is set to "zookeeper" the URIs are > used as ZooKeeper quorum. When it's set to be empty, the URIs are used to > locate the metastore directly. > # Here's list of Hiveserver2's parameters and their proposed metastore conf > counterparts. It looks odd that the Metastore related configurations do not > have their macros start with METASTORE, but start with THRIFT. I have just > followed naming convention used for other parameters. > ** HIVE_SERVER2_ZOOKEEPER_NAMESPACE - THRIFT_ZOOKEEPER_NAMESPACE > (hive.metastore.zookeeper.namespace) > ** HIVE_ZOOKEEPER_CLIENT_PORT - THRIFT_ZOOKEEPER_CLIENT_PORT > (hive.metastore.zookeeper.client.port) > ** HIVE_ZOOKEEPER_CONNECTION_TIMEOUT - THRIFT_ZOOKEEPER_CONNECTION_TIMEOUT - > (hive.metastore.zookeeper.connection.timeout) > ** HIVE_ZOOKEEPER_CONNECTION_MAX_RETRIES - > THRIFT_ZOOKEEPER_CONNECTION_MAX_RETRIES > (hive.metastore.zookeeper.connection.max.retries) > ** HIVE_ZOOKEEPER_CONNECTION_BASESLEEPTIME - > THRIFT_ZOOKEEPER_CONNECTION_BASESLEEPTIME > (hive.metastore.zookeeper.connection.basesleeptime) > # Additional configuration THRIFT_BIND_HOST is used to specify the host > address to bind Metastore service to. Right now Metastore binds to *, i.e all > addresses. Metastore doesn't then know which of those addresses it should add > to the ZooKeeper. THRIFT_BIND_HOST solves that problem. When this > configuration is specified the metastore server binds to that address and > also adds it to the ZooKeeper if dynamic service discovery mode is ZooKeeper. > Following Hive ZK configurations seem to be related to managing locks and > seem irrelevant for MS ZK. > # HIVE_ZOOKEEPER_SESSION_TIMEOUT > # HIVE_ZOOKEEPER_CLEAN_EXTRA_NODES > Since there is no configuration to be published, > HIVE_ZOOKEEPER_PUBLISH_CONFIGS does not have a THRIFT counterpart. > h3. HiveMetaStore class changes > # startMetaStore should also register the instance with Zookeeper, when > configured. > # When shutting a metastore server down it should deregister itself from > Zookeeper, when configured. > # These changes use the refactored code described above. > h3. HiveMetaStoreClient class changes > When service discovery mode is zookeeper, we fetch the metatstore URIs from > the specified ZooKeeper and treat those as if they were specified in > THRIFT_URIS i.e. use the existing mechanisms to choose a metastore server to > connect to and establish a connection. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20975) Fix PreUpgradeTool Orc namespace
[
https://issues.apache.org/jira/browse/HIVE-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701407#comment-16701407
]
Hive QA commented on HIVE-20975:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949764/HIVE-20975.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15546 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15088/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15088/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15088/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949764 - PreCommit-HIVE-Build
> Fix PreUpgradeTool Orc namespace
>
>
> Key: HIVE-20975
> URL: https://issues.apache.org/jira/browse/HIVE-20975
> Project: Hive
> Issue Type: Bug
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Blocker
> Attachments: HIVE-20975.01.patch, HIVE-20975.02.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20794) Use Zookeeper for metastore service discovery
[ https://issues.apache.org/jira/browse/HIVE-20794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek updated HIVE-20794: --- Fix Version/s: 4.0.0 > Use Zookeeper for metastore service discovery > - > > Key: HIVE-20794 > URL: https://issues.apache.org/jira/browse/HIVE-20794 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20794.01, HIVE-20794.02, HIVE-20794.03, > HIVE-20794.03, HIVE-20794.04, HIVE-20794.05, HIVE-20794.06, HIVE-20794.07, > HIVE-20794.07, HIVE-20794.08, HIVE-20794.08 > > > Right now, multiple metastore services can be specified in > hive.metastore.uris configuration, but that list is static and can not be > modified dynamically. Use Zookeeper for dynamic service discovery of > metastore. > h3. Improve ZooKeeperHiveHelper class (suggestions for name welcome) > The Zookeeper related code (for service discovery) accesses Zookeeper > parameters directly from HiveConf. The class is changed so that it could be > used for both HiveServer2 and Metastore server and works with both the > configurations. Following methods from HiveServer2 are now moved into > ZooKeeperHiveHelper. # startZookeeperClient # addServerInstanceToZooKeeper # > removeServerInstanceFromZooKeeper > h3. HiveMetaStore conf changes > # THRIFT_URIS (hive.metastore.uris) can also be used to specify ZooKeeper > quorum. When THRIFT_SERVICE_DISCOVERY_MODE > (hive.metastore.service.discovery.mode) is set to "zookeeper" the URIs are > used as ZooKeeper quorum. When it's set to be empty, the URIs are used to > locate the metastore directly. > # Here's list of Hiveserver2's parameters and their proposed metastore conf > counterparts. It looks odd that the Metastore related configurations do not > have their macros start with METASTORE, but start with THRIFT. I have just > followed naming convention used for other parameters. > ** HIVE_SERVER2_ZOOKEEPER_NAMESPACE - THRIFT_ZOOKEEPER_NAMESPACE > (hive.metastore.zookeeper.namespace) > ** HIVE_ZOOKEEPER_CLIENT_PORT - THRIFT_ZOOKEEPER_CLIENT_PORT > (hive.metastore.zookeeper.client.port) > ** HIVE_ZOOKEEPER_CONNECTION_TIMEOUT - THRIFT_ZOOKEEPER_CONNECTION_TIMEOUT - > (hive.metastore.zookeeper.connection.timeout) > ** HIVE_ZOOKEEPER_CONNECTION_MAX_RETRIES - > THRIFT_ZOOKEEPER_CONNECTION_MAX_RETRIES > (hive.metastore.zookeeper.connection.max.retries) > ** HIVE_ZOOKEEPER_CONNECTION_BASESLEEPTIME - > THRIFT_ZOOKEEPER_CONNECTION_BASESLEEPTIME > (hive.metastore.zookeeper.connection.basesleeptime) > # Additional configuration THRIFT_BIND_HOST is used to specify the host > address to bind Metastore service to. Right now Metastore binds to *, i.e all > addresses. Metastore doesn't then know which of those addresses it should add > to the ZooKeeper. THRIFT_BIND_HOST solves that problem. When this > configuration is specified the metastore server binds to that address and > also adds it to the ZooKeeper if dynamic service discovery mode is ZooKeeper. > Following Hive ZK configurations seem to be related to managing locks and > seem irrelevant for MS ZK. > # HIVE_ZOOKEEPER_SESSION_TIMEOUT > # HIVE_ZOOKEEPER_CLEAN_EXTRA_NODES > Since there is no configuration to be published, > HIVE_ZOOKEEPER_PUBLISH_CONFIGS does not have a THRIFT counterpart. > h3. HiveMetaStore class changes > # startMetaStore should also register the instance with Zookeeper, when > configured. > # When shutting a metastore server down it should deregister itself from > Zookeeper, when configured. > # These changes use the refactored code described above. > h3. HiveMetaStoreClient class changes > When service discovery mode is zookeeper, we fetch the metatstore URIs from > the specified ZooKeeper and treat those as if they were specified in > THRIFT_URIS i.e. use the existing mechanisms to choose a metastore server to > connect to and establish a connection. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20794) Use Zookeeper for metastore service discovery
[ https://issues.apache.org/jira/browse/HIVE-20794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701406#comment-16701406 ] anishek commented on HIVE-20794: +1 , Committed to master, Thanks [~ashutosh.bapat] > Use Zookeeper for metastore service discovery > - > > Key: HIVE-20794 > URL: https://issues.apache.org/jira/browse/HIVE-20794 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20794.01, HIVE-20794.02, HIVE-20794.03, > HIVE-20794.03, HIVE-20794.04, HIVE-20794.05, HIVE-20794.06, HIVE-20794.07, > HIVE-20794.07, HIVE-20794.08, HIVE-20794.08 > > > Right now, multiple metastore services can be specified in > hive.metastore.uris configuration, but that list is static and can not be > modified dynamically. Use Zookeeper for dynamic service discovery of > metastore. > h3. Improve ZooKeeperHiveHelper class (suggestions for name welcome) > The Zookeeper related code (for service discovery) accesses Zookeeper > parameters directly from HiveConf. The class is changed so that it could be > used for both HiveServer2 and Metastore server and works with both the > configurations. Following methods from HiveServer2 are now moved into > ZooKeeperHiveHelper. # startZookeeperClient # addServerInstanceToZooKeeper # > removeServerInstanceFromZooKeeper > h3. HiveMetaStore conf changes > # THRIFT_URIS (hive.metastore.uris) can also be used to specify ZooKeeper > quorum. When THRIFT_SERVICE_DISCOVERY_MODE > (hive.metastore.service.discovery.mode) is set to "zookeeper" the URIs are > used as ZooKeeper quorum. When it's set to be empty, the URIs are used to > locate the metastore directly. > # Here's list of Hiveserver2's parameters and their proposed metastore conf > counterparts. It looks odd that the Metastore related configurations do not > have their macros start with METASTORE, but start with THRIFT. I have just > followed naming convention used for other parameters. > ** HIVE_SERVER2_ZOOKEEPER_NAMESPACE - THRIFT_ZOOKEEPER_NAMESPACE > (hive.metastore.zookeeper.namespace) > ** HIVE_ZOOKEEPER_CLIENT_PORT - THRIFT_ZOOKEEPER_CLIENT_PORT > (hive.metastore.zookeeper.client.port) > ** HIVE_ZOOKEEPER_CONNECTION_TIMEOUT - THRIFT_ZOOKEEPER_CONNECTION_TIMEOUT - > (hive.metastore.zookeeper.connection.timeout) > ** HIVE_ZOOKEEPER_CONNECTION_MAX_RETRIES - > THRIFT_ZOOKEEPER_CONNECTION_MAX_RETRIES > (hive.metastore.zookeeper.connection.max.retries) > ** HIVE_ZOOKEEPER_CONNECTION_BASESLEEPTIME - > THRIFT_ZOOKEEPER_CONNECTION_BASESLEEPTIME > (hive.metastore.zookeeper.connection.basesleeptime) > # Additional configuration THRIFT_BIND_HOST is used to specify the host > address to bind Metastore service to. Right now Metastore binds to *, i.e all > addresses. Metastore doesn't then know which of those addresses it should add > to the ZooKeeper. THRIFT_BIND_HOST solves that problem. When this > configuration is specified the metastore server binds to that address and > also adds it to the ZooKeeper if dynamic service discovery mode is ZooKeeper. > Following Hive ZK configurations seem to be related to managing locks and > seem irrelevant for MS ZK. > # HIVE_ZOOKEEPER_SESSION_TIMEOUT > # HIVE_ZOOKEEPER_CLEAN_EXTRA_NODES > Since there is no configuration to be published, > HIVE_ZOOKEEPER_PUBLISH_CONFIGS does not have a THRIFT counterpart. > h3. HiveMetaStore class changes > # startMetaStore should also register the instance with Zookeeper, when > configured. > # When shutting a metastore server down it should deregister itself from > Zookeeper, when configured. > # These changes use the refactored code described above. > h3. HiveMetaStoreClient class changes > When service discovery mode is zookeeper, we fetch the metatstore URIs from > the specified ZooKeeper and treat those as if they were specified in > THRIFT_URIS i.e. use the existing mechanisms to choose a metastore server to > connect to and establish a connection. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20975) Fix PreUpgradeTool Orc namespace
[
https://issues.apache.org/jira/browse/HIVE-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701388#comment-16701388
]
Hive QA commented on HIVE-20975:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} upgrade-acid/pre-upgrade in master has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} upgrade-acid/pre-upgrade: The patch generated 7 new +
268 unchanged - 23 fixed = 275 total (was 291) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15088/dev-support/hive-personality.sh
|
| git revision | master / 56926af |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15088/yetus/diff-checkstyle-upgrade-acid_pre-upgrade.txt
|
| modules | C: upgrade-acid/pre-upgrade U: upgrade-acid/pre-upgrade |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15088/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix PreUpgradeTool Orc namespace
>
>
> Key: HIVE-20975
> URL: https://issues.apache.org/jira/browse/HIVE-20975
> Project: Hive
> Issue Type: Bug
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Blocker
> Attachments: HIVE-20975.01.patch, HIVE-20975.02.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20828) Upgrade to Spark 2.4.0
[
https://issues.apache.org/jira/browse/HIVE-20828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701380#comment-16701380
]
Hive QA commented on HIVE-20828:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949763/HIVE-20828.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 15546 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestLocalSparkCliDriver.testCliDriver[spark_local_queries]
(batchId=275)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.testCliDriver[spark_submit_negative_executor_cores]
(batchId=275)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.testCliDriver[spark_submit_negative_executor_memory]
(batchId=275)
org.apache.hadoop.hive.ql.exec.spark.TestSparkInvalidFileFormat.readTextFileAsParquet
(batchId=317)
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSessionSparkSessionTimeout
(batchId=253)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15087/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15087/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15087/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949763 - PreCommit-HIVE-Build
> Upgrade to Spark 2.4.0
> --
>
> Key: HIVE-20828
> URL: https://issues.apache.org/jira/browse/HIVE-20828
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-20828.1.patch, HIVE-20828.2.patch
>
>
> The Spark community is in the process of releasing Spark 2.4.0. We should do
> some testing with the RC candidates and then upgrade once the release is
> finalized.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20828) Upgrade to Spark 2.4.0
[
https://issues.apache.org/jira/browse/HIVE-20828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701373#comment-16701373
]
Hive QA commented on HIVE-20828:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15087/dev-support/hive-personality.sh
|
| git revision | master / 56926af |
| Default Java | 1.8.0_111 |
| modules | C: . itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15087/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade to Spark 2.4.0
> --
>
> Key: HIVE-20828
> URL: https://issues.apache.org/jira/browse/HIVE-20828
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-20828.1.patch, HIVE-20828.2.patch
>
>
> The Spark community is in the process of releasing Spark 2.4.0. We should do
> some testing with the RC candidates and then upgrade once the release is
> finalized.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[
https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701346#comment-16701346
]
Hive QA commented on HIVE-20915:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949752/HIVE-20915.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15547 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[orc_merge1]
(batchId=189)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15086/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15086/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15086/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949752 - PreCommit-HIVE-Build
> Make dynamic sort partition optimization available to HoS and MR
>
>
> Key: HIVE-20915
> URL: https://issues.apache.org/jira/browse/HIVE-20915
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch
>
>
> HIVE-20703 put dynamic sort partition optimization under cost based decision,
> but it also makes the optimizer only available to tez.
> hive.optimize.sort.dynamic.partition works with other execution engines for a
> long time, we should keep the optimizer available to them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[
https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701315#comment-16701315
]
Hive QA commented on HIVE-20915:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 236 unchanged - 1
fixed = 237 total (was 237) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 3 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15086/dev-support/hive-personality.sh
|
| git revision | master / 56926af |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15086/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15086/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15086/yetus/whitespace-tabs.txt
|
| modules | C: ql itests itests/hive-blobstore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15086/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make dynamic sort partition optimization available to HoS and MR
>
>
> Key: HIVE-20915
> URL: https://issues.apache.org/jira/browse/HIVE-20915
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
>Priority: Major
> Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch
>
>
> HIVE-20703 put dynamic sort partition optimization under cost based decision,
> but it also makes the optimizer only available to tez.
> hive.optimize.sort.dynamic.partition works with other execution engines for a
> long time, we should keep the optimizer available to them.
--
This
[jira] [Commented] (HIVE-20936) Allow the Worker thread in the metastore to run outside of it
[
https://issues.apache.org/jira/browse/HIVE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701283#comment-16701283
]
Hive QA commented on HIVE-20936:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949749/HIVE-20936.7.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15546 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15085/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15085/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15085/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949749 - PreCommit-HIVE-Build
> Allow the Worker thread in the metastore to run outside of it
> -
>
> Key: HIVE-20936
> URL: https://issues.apache.org/jira/browse/HIVE-20936
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20936.1.patch, HIVE-20936.2.patch,
> HIVE-20936.3.patch, HIVE-20936.4.patch, HIVE-20936.5.patch,
> HIVE-20936.6.patch, HIVE-20936.7.patch
>
>
> Currently the Worker thread in the metastore in bounded to the metastore,
> mainly because of the TxnHandler that it has. This thread runs some map
> reduce jobs which may not only be an option wherever the metastore is
> running. A solution for this can be to run this thread in HS2 depending on a
> flag.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20936) Allow the Worker thread in the metastore to run outside of it
[
https://issues.apache.org/jira/browse/HIVE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701268#comment-16701268
]
Hive QA commented on HIVE-20936:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
14s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
4s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 2 new + 639 unchanged - 5
fixed = 641 total (was 644) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 173
unchanged - 0 fixed = 174 total (was 173) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 105 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
58s{color} | {color:red} ql generated 3 new + 2311 unchanged - 1 fixed = 2314
total (was 2312) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
0s{color} | {color:red} standalone-metastore_metastore-common generated 1 new +
16 unchanged - 0 fixed = 17 total (was 16) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Field MetaStoreCompactorThread.threadId masks field in superclass
org.apache.hadoop.hive.ql.txn.compactor.CompactorThread In
MetaStoreCompactorThread.java:superclass
org.apache.hadoop.hive.ql.txn.compactor.CompactorThread In
MetaStoreCompactorThread.java |
| | Field MetaStoreCompactorThread.rs masks field in superclass
org.apache.hadoop.hive.ql.txn.compactor.CompactorThread In
MetaStoreCompactorThread.java:superclass
org.apache.hadoop.hive.ql.txn.compactor.CompactorThread In
MetaStoreCompactorThread.java |
| |
[jira] [Commented] (HIVE-20976) JDBC queries containing joins gives wrong results
[
https://issues.apache.org/jira/browse/HIVE-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701245#comment-16701245
]
Ashutosh Chauhan commented on HIVE-20976:
-
+1
> JDBC queries containing joins gives wrong results
> -
>
> Key: HIVE-20976
> URL: https://issues.apache.org/jira/browse/HIVE-20976
> Project: Hive
> Issue Type: Bug
> Components: JDBC, StorageHandler
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20976.patch
>
>
> JDBC queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> jdbc_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM jdbc_table_with_nulls
> GROUP BY `username`
> ORDER BY `sum_double2`
> DESC LIMIT 10
> )
> `tbl2`
> ON (`tbl1`.`username` = `tbl2`.`username`)
> GROUP BY `tbl1`.`username`;
> {code}
> Similar to HIVE-19762.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20976) JDBC queries containing joins gives wrong results
[
https://issues.apache.org/jira/browse/HIVE-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701229#comment-16701229
]
Jesus Camacho Rodriguez commented on HIVE-20976:
[~ashutoshc], could you take a look? Thanks
> JDBC queries containing joins gives wrong results
> -
>
> Key: HIVE-20976
> URL: https://issues.apache.org/jira/browse/HIVE-20976
> Project: Hive
> Issue Type: Bug
> Components: JDBC, StorageHandler
>Reporter: Nishant Bangarwa
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20976.patch
>
>
> JDBC queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> jdbc_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM jdbc_table_with_nulls
> GROUP BY `username`
> ORDER BY `sum_double2`
> DESC LIMIT 10
> )
> `tbl2`
> ON (`tbl1`.`username` = `tbl2`.`username`)
> GROUP BY `tbl1`.`username`;
> {code}
> Similar to HIVE-19762.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20975) Fix PreUpgradeTool Orc namespace
[ https://issues.apache.org/jira/browse/HIVE-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701231#comment-16701231 ] Ashutosh Chauhan commented on HIVE-20975: - +1 > Fix PreUpgradeTool Orc namespace > > > Key: HIVE-20975 > URL: https://issues.apache.org/jira/browse/HIVE-20975 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Blocker > Attachments: HIVE-20975.01.patch, HIVE-20975.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20976) JDBC queries containing joins gives wrong results
[
https://issues.apache.org/jira/browse/HIVE-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20976:
---
Reporter: Jesus Camacho Rodriguez (was: Nishant Bangarwa)
> JDBC queries containing joins gives wrong results
> -
>
> Key: HIVE-20976
> URL: https://issues.apache.org/jira/browse/HIVE-20976
> Project: Hive
> Issue Type: Bug
> Components: JDBC, StorageHandler
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20976.patch
>
>
> JDBC queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> jdbc_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM jdbc_table_with_nulls
> GROUP BY `username`
> ORDER BY `sum_double2`
> DESC LIMIT 10
> )
> `tbl2`
> ON (`tbl1`.`username` = `tbl2`.`username`)
> GROUP BY `tbl1`.`username`;
> {code}
> Similar to HIVE-19762.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20976) JDBC queries containing joins gives wrong results
[
https://issues.apache.org/jira/browse/HIVE-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20976:
---
Attachment: HIVE-20976.patch
> JDBC queries containing joins gives wrong results
> -
>
> Key: HIVE-20976
> URL: https://issues.apache.org/jira/browse/HIVE-20976
> Project: Hive
> Issue Type: Bug
> Components: JDBC, StorageHandler
>Reporter: Nishant Bangarwa
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20976.patch
>
>
> JDBC queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> jdbc_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM jdbc_table_with_nulls
> GROUP BY `username`
> ORDER BY `sum_double2`
> DESC LIMIT 10
> )
> `tbl2`
> ON (`tbl1`.`username` = `tbl2`.`username`)
> GROUP BY `tbl1`.`username`;
> {code}
> Similar to HIVE-19762.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20976) JDBC queries containing joins gives wrong results
[
https://issues.apache.org/jira/browse/HIVE-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20976:
---
Status: Patch Available (was: In Progress)
> JDBC queries containing joins gives wrong results
> -
>
> Key: HIVE-20976
> URL: https://issues.apache.org/jira/browse/HIVE-20976
> Project: Hive
> Issue Type: Bug
> Components: JDBC, StorageHandler
>Reporter: Nishant Bangarwa
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> JDBC queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> jdbc_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM jdbc_table_with_nulls
> GROUP BY `username`
> ORDER BY `sum_double2`
> DESC LIMIT 10
> )
> `tbl2`
> ON (`tbl1`.`username` = `tbl2`.`username`)
> GROUP BY `tbl1`.`username`;
> {code}
> Similar to HIVE-19762.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701218#comment-16701218
]
Hive QA commented on HIVE-20330:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949747/HIVE-20330.6.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15553 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15084/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15084/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15084/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949747 - PreCommit-HIVE-Build
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch,
> HIVE-20330.5.patch, HIVE-20330.6.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
> performance drop in a workload that has multiple inputs from HCatLoader.
> The reason is that for a particular MR job with multiple Hive tables as
> input, Pig calls {{setLocation}} on each {{LoaderFunc (HCatLoader)}} instance
> but only one table's information (InputJobInfo instance) gets tracked in the
> JobConf. (This is under config key {{HCatConstants.HCAT_KEY_JOB_INFO}}).
> Any such call overwrites preexisting values, and thus only the last table's
> information will be considered when Pig calls {{getStatistics}} to calculate
> and estimate required reducer count.
> In cases when there are 2 input tables, 256GB and 1MB in size respectively,
> Pig will query the size information from HCat for both of them, but it will
> either see 1MB+1MB=2MB or 256GB+256GB=0.5TB depending on input order in the
> execution plan's DAG.
> It should of course see 256.00097GB in total and use 257 reducers by default
> accordingly.
> In unlucky cases this will be seen as 2MB and 1 reducer will have to struggle
> with the actual 256.00097GB...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20977) Lazy evaluate the table object in PreReadTableEvent to improve get_partition performance
[
https://issues.apache.org/jira/browse/HIVE-20977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Manamcheri reassigned HIVE-20977:
-
> Lazy evaluate the table object in PreReadTableEvent to improve get_partition
> performance
>
>
> Key: HIVE-20977
> URL: https://issues.apache.org/jira/browse/HIVE-20977
> Project: Hive
> Issue Type: Improvement
>Reporter: Karthik Manamcheri
>Assignee: Karthik Manamcheri
>Priority: Minor
>
> The PreReadTableEvent is generated for non-table operations (such as
> get_partitions), but only if there is an event listener attached. However,
> this is also not necessary if the event listener is not interested in the
> read table event.
> For example, the TransactionalValidationListener's onEvent looks like this
> {code:java}
> @Override
> public void onEvent(PreEventContext context) throws MetaException,
> NoSuchObjectException,
> InvalidOperationException {
> switch (context.getEventType()) {
> case CREATE_TABLE:
> handle((PreCreateTableEvent) context);
> break;
> case ALTER_TABLE:
> handle((PreAlterTableEvent) context);
> break;
> default:
> //no validation required..
> }
> }{code}
>
> Note that for read table events it is a no-op. The problem is that the
> get_table is evaluated when creating the PreReadTableEvent finally to be just
> ignored!
> Look at the code below.. {{getMS().getTable(..)}} is evaluated irrespective
> of if the listener uses it or not.
> {code:java}
> private void fireReadTablePreEvent(String catName, String dbName, String
> tblName)
> throws MetaException, NoSuchObjectException {
> if(preListeners.size() > 0) {
> // do this only if there is a pre event listener registered (avoid
> unnecessary
> // metastore api call)
> Table t = getMS().getTable(catName, dbName, tblName);
> if (t == null) {
> throw new NoSuchObjectException(TableName.getQualified(catName, dbName,
> tblName)
> + " table not found");
> }
> firePreEvent(new PreReadTableEvent(t, this));
> }
> }
> {code}
> This can be improved by using a {{Supplier}} and lazily evaluating the table
> when needed (once when the first time it is called, memorized after that).
> *Motivation*
> Whenever a partition call occurs (get_partition, etc.), we fire the
> PreReadTableEvent. This affects performance since it fetches the table even
> if it is not being used. This change will improve performance on the
> get_partition calls.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20546) Upgrade to Druid 0.13.0
[ https://issues.apache.org/jira/browse/HIVE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701186#comment-16701186 ] slim bouguerra commented on HIVE-20546: --- [~nishantbangarwa] please make sure to fix the style violation issues. > Upgrade to Druid 0.13.0 > --- > > Key: HIVE-20546 > URL: https://issues.apache.org/jira/browse/HIVE-20546 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20546.1.patch, HIVE-20546.patch > > > This task is to upgrade to druid 0.13.0 when it is released. Note that it > will hopefully be first apache release for Druid. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701180#comment-16701180
]
Hive QA commented on HIVE-20330:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} hcatalog/core in master has 31 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} hcatalog/hcatalog-pig-adapter in master has 2 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} hcatalog/core: The patch generated 0 new + 113
unchanged - 10 fixed = 113 total (was 123) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} hcatalog/hcatalog-pig-adapter: The patch generated 0
new + 102 unchanged - 1 fixed = 102 total (was 103) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} hcatalog/core generated 0 new + 30 unchanged - 1
fixed = 30 total (was 31) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} hcatalog-pig-adapter in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15084/dev-support/hive-personality.sh
|
| git revision | master / 56926af |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: hcatalog/core hcatalog/hcatalog-pig-adapter U: hcatalog |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15084/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch,
> HIVE-20330.5.patch, HIVE-20330.6.patch
>
>
> While running performance tests
[jira] [Updated] (HIVE-20976) JDBC queries containing joins gives wrong results
[
https://issues.apache.org/jira/browse/HIVE-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20976:
---
Description:
JDBC queries that have joins against self table gives wrong results.
e.g.
{code}
SELECT
username AS `username`,
SUM(double1) AS `sum_double1`
FROM
jdbc_table_with_nulls `tbl1`
JOIN (
SELECT
username AS `username`,
SUM(double1) AS `sum_double2`
FROM jdbc_table_with_nulls
GROUP BY `username`
ORDER BY `sum_double2`
DESC LIMIT 10
)
`tbl2`
ON (`tbl1`.`username` = `tbl2`.`username`)
GROUP BY `tbl1`.`username`;
{code}
was:
Druid queries that have joins against self table gives wrong results.
e.g.
{code}
SELECT
username AS `username`,
SUM(double1) AS `sum_double1`
FROM
druid_table_with_nulls `tbl1`
JOIN (
SELECT
username AS `username`,
SUM(double1) AS `sum_double2`
FROM druid_table_with_nulls
GROUP BY `username`
ORDER BY `sum_double2`
DESC LIMIT 10
)
`tbl2`
ON (`tbl1`.`username` = `tbl2`.`username`)
GROUP BY `tbl1`.`username`;
{code}
In this case one of the queries is a druid scan query and other is groupBy
query.
During planning, the properties of these queries are set to the tableDesc and
serdeInfo, while setting the map work, we overwrite the properties from the
properties present in serdeInfo, this causes the scan query results to be
deserialized using wrong column names and results in Null values.
> JDBC queries containing joins gives wrong results
> -
>
> Key: HIVE-20976
> URL: https://issues.apache.org/jira/browse/HIVE-20976
> Project: Hive
> Issue Type: Bug
> Components: JDBC, StorageHandler
>Reporter: Nishant Bangarwa
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> JDBC queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> jdbc_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM jdbc_table_with_nulls
> GROUP BY `username`
> ORDER BY `sum_double2`
> DESC LIMIT 10
> )
> `tbl2`
> ON (`tbl1`.`username` = `tbl2`.`username`)
> GROUP BY `tbl1`.`username`;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20546) Upgrade to Druid 0.13.0
[
https://issues.apache.org/jira/browse/HIVE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701163#comment-16701163
]
Hive QA commented on HIVE-20546:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949723/HIVE-20546.1.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15541 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=196)
[druidmini_dynamic_partition.q,druidmini_test1.q,druidkafkamini_basic.q,druidmini_test_insert.q,druidkafkamini_delimited.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_timeseries]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_topn] (batchId=3)
org.apache.hadoop.hive.druid.TestHiveDruidQueryBasedInputFormat.testTimeZone
(batchId=281)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15083/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15083/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15083/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949723 - PreCommit-HIVE-Build
> Upgrade to Druid 0.13.0
> ---
>
> Key: HIVE-20546
> URL: https://issues.apache.org/jira/browse/HIVE-20546
> Project: Hive
> Issue Type: Task
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20546.1.patch, HIVE-20546.patch
>
>
> This task is to upgrade to druid 0.13.0 when it is released. Note that it
> will hopefully be first apache release for Druid.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20546) Upgrade to Druid 0.13.0
[
https://issues.apache.org/jira/browse/HIVE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701157#comment-16701157
]
Hive QA commented on HIVE-20546:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} druid-handler in master has 4 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
18s{color} | {color:blue} itests/qtest-druid in master has 6 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} druid-handler: The patch generated 28 new + 0
unchanged - 0 fixed = 28 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red} druid-handler generated 1 new + 4 unchanged - 0 fixed
= 5 total (was 4) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:druid-handler |
| | org.apache.hadoop.hive.druid.json.KafkaSupervisorReport doesn't override
org.apache.druid.indexing.overlord.supervisor.SupervisorReport.equals(Object)
At KafkaSupervisorReport.java:At KafkaSupervisorReport.java:[line 1] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15083/dev-support/hive-personality.sh
|
| git revision | master / 56926af |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15083/yetus/diff-checkstyle-druid-handler.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15083/yetus/new-findbugs-druid-handler.html
|
| modules | C: ql druid-handler . itests/qtest-druid U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15083/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade to Druid 0.13.0
> ---
>
> Key: HIVE-20546
> URL: https://issues.apache.org/jira/browse/HIVE-20546
> Project: Hive
> Issue Type: Task
>Reporter: Nishant Bangarwa
>
[jira] [Commented] (HIVE-20838) Timestamps with timezone are set to null when using the streaming API
[
https://issues.apache.org/jira/browse/HIVE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701148#comment-16701148
]
Gopal V commented on HIVE-20838:
LGTM - +1
The string to timestamp goes via the date route during failure - "2013-02-12
21:04:XX" -> "2013-02-12 00:00:00" instead of NULL.
> Timestamps with timezone are set to null when using the streaming API
> -
>
> Key: HIVE-20838
> URL: https://issues.apache.org/jira/browse/HIVE-20838
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Jaume M
>Assignee: Jaume M
>Priority: Major
> Attachments: HIVE-20838.1.patch, HIVE-20838.10.patch,
> HIVE-20838.2.patch, HIVE-20838.3.patch, HIVE-20838.3.patch,
> HIVE-20838.4.patch, HIVE-20838.5.patch, HIVE-20838.6.patch,
> HIVE-20838.7.patch, HIVE-20838.8.patch, HIVE-20838.9.patch
>
>
> For example:
> {code}
> beeline> create table default.timest (a TIMESTAMP) stored as orc " +
> "TBLPROPERTIES('transactional'='true')
> # And then:
> connection.write("2018-10-19 10:35:00 America/Los_Angeles".getBytes());
> {code}
> inserts NULL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20975) Fix PreUpgradeTool Orc namespace
[ https://issues.apache.org/jira/browse/HIVE-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20975: -- Status: Open (was: Patch Available) > Fix PreUpgradeTool Orc namespace > > > Key: HIVE-20975 > URL: https://issues.apache.org/jira/browse/HIVE-20975 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Blocker > Attachments: HIVE-20975.01.patch, HIVE-20975.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20976) JDB Queries containing Joins gives wrong results.
[
https://issues.apache.org/jira/browse/HIVE-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-20976:
--
> JDB Queries containing Joins gives wrong results.
> --
>
> Key: HIVE-20976
> URL: https://issues.apache.org/jira/browse/HIVE-20976
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Fix For: 3.1.0
>
>
> Druid queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> druid_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM druid_table_with_nulls
> GROUP BY `username`
> ORDER BY `sum_double2`
> DESC LIMIT 10
> )
> `tbl2`
> ON (`tbl1`.`username` = `tbl2`.`username`)
> GROUP BY `tbl1`.`username`;
> {code}
> In this case one of the queries is a druid scan query and other is groupBy
> query.
> During planning, the properties of these queries are set to the tableDesc and
> serdeInfo, while setting the map work, we overwrite the properties from the
> properties present in serdeInfo, this causes the scan query results to be
> deserialized using wrong column names and results in Null values.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-20976) JDBC queries containing joins gives wrong results
[
https://issues.apache.org/jira/browse/HIVE-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-20976 started by Jesus Camacho Rodriguez.
--
> JDBC queries containing joins gives wrong results
> -
>
> Key: HIVE-20976
> URL: https://issues.apache.org/jira/browse/HIVE-20976
> Project: Hive
> Issue Type: Bug
> Components: JDBC, StorageHandler
>Reporter: Nishant Bangarwa
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> JDBC queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> jdbc_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM jdbc_table_with_nulls
> GROUP BY `username`
> ORDER BY `sum_double2`
> DESC LIMIT 10
> )
> `tbl2`
> ON (`tbl1`.`username` = `tbl2`.`username`)
> GROUP BY `tbl1`.`username`;
> {code}
> Similar to HIVE-19762.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20976) JDBC queries containing joins gives wrong results
[
https://issues.apache.org/jira/browse/HIVE-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20976:
---
Description:
JDBC queries that have joins against self table gives wrong results.
e.g.
{code}
SELECT
username AS `username`,
SUM(double1) AS `sum_double1`
FROM
jdbc_table_with_nulls `tbl1`
JOIN (
SELECT
username AS `username`,
SUM(double1) AS `sum_double2`
FROM jdbc_table_with_nulls
GROUP BY `username`
ORDER BY `sum_double2`
DESC LIMIT 10
)
`tbl2`
ON (`tbl1`.`username` = `tbl2`.`username`)
GROUP BY `tbl1`.`username`;
{code}
Similar to HIVE-19762.
was:
JDBC queries that have joins against self table gives wrong results.
e.g.
{code}
SELECT
username AS `username`,
SUM(double1) AS `sum_double1`
FROM
jdbc_table_with_nulls `tbl1`
JOIN (
SELECT
username AS `username`,
SUM(double1) AS `sum_double2`
FROM jdbc_table_with_nulls
GROUP BY `username`
ORDER BY `sum_double2`
DESC LIMIT 10
)
`tbl2`
ON (`tbl1`.`username` = `tbl2`.`username`)
GROUP BY `tbl1`.`username`;
{code}
> JDBC queries containing joins gives wrong results
> -
>
> Key: HIVE-20976
> URL: https://issues.apache.org/jira/browse/HIVE-20976
> Project: Hive
> Issue Type: Bug
> Components: JDBC, StorageHandler
>Reporter: Nishant Bangarwa
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> JDBC queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> jdbc_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM jdbc_table_with_nulls
> GROUP BY `username`
> ORDER BY `sum_double2`
> DESC LIMIT 10
> )
> `tbl2`
> ON (`tbl1`.`username` = `tbl2`.`username`)
> GROUP BY `tbl1`.`username`;
> {code}
> Similar to HIVE-19762.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20976) JDBC queries containing joins gives wrong results
[
https://issues.apache.org/jira/browse/HIVE-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20976:
---
Fix Version/s: (was: 3.1.0)
> JDBC queries containing joins gives wrong results
> -
>
> Key: HIVE-20976
> URL: https://issues.apache.org/jira/browse/HIVE-20976
> Project: Hive
> Issue Type: Bug
> Components: JDBC, StorageHandler
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
>
> Druid queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> druid_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM druid_table_with_nulls
> GROUP BY `username`
> ORDER BY `sum_double2`
> DESC LIMIT 10
> )
> `tbl2`
> ON (`tbl1`.`username` = `tbl2`.`username`)
> GROUP BY `tbl1`.`username`;
> {code}
> In this case one of the queries is a druid scan query and other is groupBy
> query.
> During planning, the properties of these queries are set to the tableDesc and
> serdeInfo, while setting the map work, we overwrite the properties from the
> properties present in serdeInfo, this causes the scan query results to be
> deserialized using wrong column names and results in Null values.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20976) JDBC queries containing joins gives wrong results
[
https://issues.apache.org/jira/browse/HIVE-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20976:
---
Component/s: (was: Druid integration)
StorageHandler
JDBC
> JDBC queries containing joins gives wrong results
> -
>
> Key: HIVE-20976
> URL: https://issues.apache.org/jira/browse/HIVE-20976
> Project: Hive
> Issue Type: Bug
> Components: JDBC, StorageHandler
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
>
> Druid queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> druid_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM druid_table_with_nulls
> GROUP BY `username`
> ORDER BY `sum_double2`
> DESC LIMIT 10
> )
> `tbl2`
> ON (`tbl1`.`username` = `tbl2`.`username`)
> GROUP BY `tbl1`.`username`;
> {code}
> In this case one of the queries is a druid scan query and other is groupBy
> query.
> During planning, the properties of these queries are set to the tableDesc and
> serdeInfo, while setting the map work, we overwrite the properties from the
> properties present in serdeInfo, this causes the scan query results to be
> deserialized using wrong column names and results in Null values.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20976) JDBC queries containing joins gives wrong results
[
https://issues.apache.org/jira/browse/HIVE-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez reassigned HIVE-20976:
--
Assignee: Jesus Camacho Rodriguez (was: Nishant Bangarwa)
> JDBC queries containing joins gives wrong results
> -
>
> Key: HIVE-20976
> URL: https://issues.apache.org/jira/browse/HIVE-20976
> Project: Hive
> Issue Type: Bug
> Components: JDBC, StorageHandler
>Reporter: Nishant Bangarwa
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> Druid queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> druid_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM druid_table_with_nulls
> GROUP BY `username`
> ORDER BY `sum_double2`
> DESC LIMIT 10
> )
> `tbl2`
> ON (`tbl1`.`username` = `tbl2`.`username`)
> GROUP BY `tbl1`.`username`;
> {code}
> In this case one of the queries is a druid scan query and other is groupBy
> query.
> During planning, the properties of these queries are set to the tableDesc and
> serdeInfo, while setting the map work, we overwrite the properties from the
> properties present in serdeInfo, this causes the scan query results to be
> deserialized using wrong column names and results in Null values.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20976) JDBC queries containing joins gives wrong results
[
https://issues.apache.org/jira/browse/HIVE-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-20976:
---
Summary: JDBC queries containing joins gives wrong results (was: JDB
Queries containing Joins gives wrong results. )
> JDBC queries containing joins gives wrong results
> -
>
> Key: HIVE-20976
> URL: https://issues.apache.org/jira/browse/HIVE-20976
> Project: Hive
> Issue Type: Bug
> Components: JDBC, StorageHandler
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
>
> Druid queries that have joins against self table gives wrong results.
> e.g.
> {code}
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double1`
> FROM
> druid_table_with_nulls `tbl1`
> JOIN (
> SELECT
> username AS `username`,
> SUM(double1) AS `sum_double2`
> FROM druid_table_with_nulls
> GROUP BY `username`
> ORDER BY `sum_double2`
> DESC LIMIT 10
> )
> `tbl2`
> ON (`tbl1`.`username` = `tbl2`.`username`)
> GROUP BY `tbl1`.`username`;
> {code}
> In this case one of the queries is a druid scan query and other is groupBy
> query.
> During planning, the properties of these queries are set to the tableDesc and
> serdeInfo, while setting the map work, we overwrite the properties from the
> properties present in serdeInfo, this causes the scan query results to be
> deserialized using wrong column names and results in Null values.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20975) Fix PreUpgradeTool Orc namespace
[ https://issues.apache.org/jira/browse/HIVE-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20975: -- Status: Patch Available (was: Open) > Fix PreUpgradeTool Orc namespace > > > Key: HIVE-20975 > URL: https://issues.apache.org/jira/browse/HIVE-20975 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Blocker > Attachments: HIVE-20975.01.patch, HIVE-20975.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20975) Fix PreUpgradeTool Orc namespace
[ https://issues.apache.org/jira/browse/HIVE-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20975: -- Attachment: HIVE-20975.02.patch > Fix PreUpgradeTool Orc namespace > > > Key: HIVE-20975 > URL: https://issues.apache.org/jira/browse/HIVE-20975 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Blocker > Attachments: HIVE-20975.01.patch, HIVE-20975.02.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20828) Upgrade to Spark 2.4.0
[ https://issues.apache.org/jira/browse/HIVE-20828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-20828: Attachment: HIVE-20828.2.patch > Upgrade to Spark 2.4.0 > -- > > Key: HIVE-20828 > URL: https://issues.apache.org/jira/browse/HIVE-20828 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-20828.1.patch, HIVE-20828.2.patch > > > The Spark community is in the process of releasing Spark 2.4.0. We should do > some testing with the RC candidates and then upgrade once the release is > finalized. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20794) Use Zookeeper for metastore service discovery
[
https://issues.apache.org/jira/browse/HIVE-20794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701109#comment-16701109
]
Hive QA commented on HIVE-20794:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949721/HIVE-20794.08
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15636 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15082/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15082/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15082/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949721 - PreCommit-HIVE-Build
> Use Zookeeper for metastore service discovery
> -
>
> Key: HIVE-20794
> URL: https://issues.apache.org/jira/browse/HIVE-20794
> Project: Hive
> Issue Type: Improvement
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20794.01, HIVE-20794.02, HIVE-20794.03,
> HIVE-20794.03, HIVE-20794.04, HIVE-20794.05, HIVE-20794.06, HIVE-20794.07,
> HIVE-20794.07, HIVE-20794.08, HIVE-20794.08
>
>
> Right now, multiple metastore services can be specified in
> hive.metastore.uris configuration, but that list is static and can not be
> modified dynamically. Use Zookeeper for dynamic service discovery of
> metastore.
> h3. Improve ZooKeeperHiveHelper class (suggestions for name welcome)
> The Zookeeper related code (for service discovery) accesses Zookeeper
> parameters directly from HiveConf. The class is changed so that it could be
> used for both HiveServer2 and Metastore server and works with both the
> configurations. Following methods from HiveServer2 are now moved into
> ZooKeeperHiveHelper. # startZookeeperClient # addServerInstanceToZooKeeper #
> removeServerInstanceFromZooKeeper
> h3. HiveMetaStore conf changes
> # THRIFT_URIS (hive.metastore.uris) can also be used to specify ZooKeeper
> quorum. When THRIFT_SERVICE_DISCOVERY_MODE
> (hive.metastore.service.discovery.mode) is set to "zookeeper" the URIs are
> used as ZooKeeper quorum. When it's set to be empty, the URIs are used to
> locate the metastore directly.
> # Here's list of Hiveserver2's parameters and their proposed metastore conf
> counterparts. It looks odd that the Metastore related configurations do not
> have their macros start with METASTORE, but start with THRIFT. I have just
> followed naming convention used for other parameters.
> ** HIVE_SERVER2_ZOOKEEPER_NAMESPACE - THRIFT_ZOOKEEPER_NAMESPACE
> (hive.metastore.zookeeper.namespace)
> ** HIVE_ZOOKEEPER_CLIENT_PORT - THRIFT_ZOOKEEPER_CLIENT_PORT
> (hive.metastore.zookeeper.client.port)
> ** HIVE_ZOOKEEPER_CONNECTION_TIMEOUT - THRIFT_ZOOKEEPER_CONNECTION_TIMEOUT -
> (hive.metastore.zookeeper.connection.timeout)
> ** HIVE_ZOOKEEPER_CONNECTION_MAX_RETRIES -
> THRIFT_ZOOKEEPER_CONNECTION_MAX_RETRIES
> (hive.metastore.zookeeper.connection.max.retries)
> ** HIVE_ZOOKEEPER_CONNECTION_BASESLEEPTIME -
> THRIFT_ZOOKEEPER_CONNECTION_BASESLEEPTIME
> (hive.metastore.zookeeper.connection.basesleeptime)
> # Additional configuration THRIFT_BIND_HOST is used to specify the host
> address to bind Metastore service to. Right now Metastore binds to *, i.e all
> addresses. Metastore doesn't then know which of those addresses it should add
> to the ZooKeeper. THRIFT_BIND_HOST solves that problem. When this
> configuration is specified the metastore server binds to that address and
> also adds it to the ZooKeeper if dynamic service discovery mode is ZooKeeper.
> Following Hive ZK configurations seem to be related to managing locks and
> seem irrelevant for MS ZK.
> # HIVE_ZOOKEEPER_SESSION_TIMEOUT
> # HIVE_ZOOKEEPER_CLEAN_EXTRA_NODES
> Since there is no configuration to be published,
> HIVE_ZOOKEEPER_PUBLISH_CONFIGS does not have a THRIFT counterpart.
> h3. HiveMetaStore class changes
> # startMetaStore should also register the instance with Zookeeper, when
> configured.
> # When shutting a metastore server down it should deregister itself from
> Zookeeper, when configured.
> # These changes use the refactored code described above.
> h3. HiveMetaStoreClient class changes
> When service discovery mode is zookeeper, we fetch the metatstore URIs from
> the specified ZooKeeper and treat those as if they were specified in
> THRIFT_URIS i.e. use the
[jira] [Commented] (HIVE-20794) Use Zookeeper for metastore service discovery
[
https://issues.apache.org/jira/browse/HIVE-20794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701094#comment-16701094
]
Hive QA commented on HIVE-20794:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
14s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
5s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
42s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/util in master has 48 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch standalone-metastore passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} ql: The patch generated 0 new + 17 unchanged - 4
fixed = 17 total (was 21) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch service passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch util passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
|
[jira] [Commented] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[ https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701025#comment-16701025 ] Yongzhi Chen commented on HIVE-20915: - Attach patch 2 to fix tests failures. > Make dynamic sort partition optimization available to HoS and MR > > > Key: HIVE-20915 > URL: https://issues.apache.org/jira/browse/HIVE-20915 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch > > > HIVE-20703 put dynamic sort partition optimization under cost based decision, > but it also makes the optimizer only available to tez. > hive.optimize.sort.dynamic.partition works with other execution engines for a > long time, we should keep the optimizer available to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
[ https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen updated HIVE-20915: Attachment: HIVE-20915.2.patch > Make dynamic sort partition optimization available to HoS and MR > > > Key: HIVE-20915 > URL: https://issues.apache.org/jira/browse/HIVE-20915 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch > > > HIVE-20703 put dynamic sort partition optimization under cost based decision, > but it also makes the optimizer only available to tez. > hive.optimize.sort.dynamic.partition works with other execution engines for a > long time, we should keep the optimizer available to them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20897) TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error
[
https://issues.apache.org/jira/browse/HIVE-20897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701013#comment-16701013
]
Hive QA commented on HIVE-20897:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949717/HIVE-20897.05.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15546 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=197)
[druidmini_masking.q,druidmini_joins.q,druid_timestamptz.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=61)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15081/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15081/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15081/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949717 - PreCommit-HIVE-Build
> TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error
>
>
> Key: HIVE-20897
> URL: https://issues.apache.org/jira/browse/HIVE-20897
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20897.01.patch, HIVE-20897.02.patch,
> HIVE-20897.03.patch, HIVE-20897.04.patch, HIVE-20897.05.patch
>
>
> if async prepare is enabled, control will be returned to the client before
> driver could set of the query has a result set or not. But in current code,
> while generating the response for the query, it is not checked if the result
> set field is set or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20936) Allow the Worker thread in the metastore to run outside of it
[ https://issues.apache.org/jira/browse/HIVE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20936: --- Attachment: HIVE-20936.7.patch Status: Patch Available (was: Open) > Allow the Worker thread in the metastore to run outside of it > - > > Key: HIVE-20936 > URL: https://issues.apache.org/jira/browse/HIVE-20936 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20936.1.patch, HIVE-20936.2.patch, > HIVE-20936.3.patch, HIVE-20936.4.patch, HIVE-20936.5.patch, > HIVE-20936.6.patch, HIVE-20936.7.patch > > > Currently the Worker thread in the metastore in bounded to the metastore, > mainly because of the TxnHandler that it has. This thread runs some map > reduce jobs which may not only be an option wherever the metastore is > running. A solution for this can be to run this thread in HS2 depending on a > flag. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-20330:
--
Attachment: HIVE-20330.6.patch
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch,
> HIVE-20330.5.patch, HIVE-20330.6.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
> performance drop in a workload that has multiple inputs from HCatLoader.
> The reason is that for a particular MR job with multiple Hive tables as
> input, Pig calls {{setLocation}} on each {{LoaderFunc (HCatLoader)}} instance
> but only one table's information (InputJobInfo instance) gets tracked in the
> JobConf. (This is under config key {{HCatConstants.HCAT_KEY_JOB_INFO}}).
> Any such call overwrites preexisting values, and thus only the last table's
> information will be considered when Pig calls {{getStatistics}} to calculate
> and estimate required reducer count.
> In cases when there are 2 input tables, 256GB and 1MB in size respectively,
> Pig will query the size information from HCat for both of them, but it will
> either see 1MB+1MB=2MB or 256GB+256GB=0.5TB depending on input order in the
> execution plan's DAG.
> It should of course see 256.00097GB in total and use 257 reducers by default
> accordingly.
> In unlucky cases this will be seen as 2MB and 1 reducer will have to struggle
> with the actual 256.00097GB...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20936) Allow the Worker thread in the metastore to run outside of it
[ https://issues.apache.org/jira/browse/HIVE-20936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaume M updated HIVE-20936: --- Status: Open (was: Patch Available) > Allow the Worker thread in the metastore to run outside of it > - > > Key: HIVE-20936 > URL: https://issues.apache.org/jira/browse/HIVE-20936 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20936.1.patch, HIVE-20936.2.patch, > HIVE-20936.3.patch, HIVE-20936.4.patch, HIVE-20936.5.patch, HIVE-20936.6.patch > > > Currently the Worker thread in the metastore in bounded to the metastore, > mainly because of the TxnHandler that it has. This thread runs some map > reduce jobs which may not only be an option wherever the metastore is > running. A solution for this can be to run this thread in HS2 depending on a > flag. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-20330:
--
Status: Patch Available (was: In Progress)
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch,
> HIVE-20330.5.patch, HIVE-20330.6.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
> performance drop in a workload that has multiple inputs from HCatLoader.
> The reason is that for a particular MR job with multiple Hive tables as
> input, Pig calls {{setLocation}} on each {{LoaderFunc (HCatLoader)}} instance
> but only one table's information (InputJobInfo instance) gets tracked in the
> JobConf. (This is under config key {{HCatConstants.HCAT_KEY_JOB_INFO}}).
> Any such call overwrites preexisting values, and thus only the last table's
> information will be considered when Pig calls {{getStatistics}} to calculate
> and estimate required reducer count.
> In cases when there are 2 input tables, 256GB and 1MB in size respectively,
> Pig will query the size information from HCat for both of them, but it will
> either see 1MB+1MB=2MB or 256GB+256GB=0.5TB depending on input order in the
> execution plan's DAG.
> It should of course see 256.00097GB in total and use 257 reducers by default
> accordingly.
> In unlucky cases this will be seen as 2MB and 1 reducer will have to struggle
> with the actual 256.00097GB...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-20330:
--
Status: In Progress (was: Patch Available)
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch, HIVE-20330.5.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
> performance drop in a workload that has multiple inputs from HCatLoader.
> The reason is that for a particular MR job with multiple Hive tables as
> input, Pig calls {{setLocation}} on each {{LoaderFunc (HCatLoader)}} instance
> but only one table's information (InputJobInfo instance) gets tracked in the
> JobConf. (This is under config key {{HCatConstants.HCAT_KEY_JOB_INFO}}).
> Any such call overwrites preexisting values, and thus only the last table's
> information will be considered when Pig calls {{getStatistics}} to calculate
> and estimate required reducer count.
> In cases when there are 2 input tables, 256GB and 1MB in size respectively,
> Pig will query the size information from HCat for both of them, but it will
> either see 1MB+1MB=2MB or 256GB+256GB=0.5TB depending on input order in the
> execution plan's DAG.
> It should of course see 256.00097GB in total and use 257 reducers by default
> accordingly.
> In unlucky cases this will be seen as 2MB and 1 reducer will have to struggle
> with the actual 256.00097GB...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20897) TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error
[
https://issues.apache.org/jira/browse/HIVE-20897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700931#comment-16700931
]
Hive QA commented on HIVE-20897:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
5s{color} | {color:green} The patch service-rpc passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} service: The patch generated 0 new + 60 unchanged -
1 fixed = 60 total (was 61) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15081/dev-support/hive-personality.sh
|
| git revision | master / 56926af |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15081/yetus/whitespace-eol.txt
|
| modules | C: service-rpc service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15081/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error
>
>
> Key: HIVE-20897
> URL: https://issues.apache.org/jira/browse/HIVE-20897
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20897.01.patch, HIVE-20897.02.patch,
> HIVE-20897.03.patch, HIVE-20897.04.patch, HIVE-20897.05.patch
>
>
> if async prepare is enabled, control will be returned to the client before
> driver could set of the query has a result set or not. But in current code,
> while generating the response for the query, it is not checked if the result
> set field is set or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700912#comment-16700912
]
Hive QA commented on HIVE-20330:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949711/HIVE-20330.5.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15080/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15080/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15080/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12949711/HIVE-20330.5.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949711 - PreCommit-HIVE-Build
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch, HIVE-20330.5.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
> performance drop in a workload that has multiple inputs from HCatLoader.
> The reason is that for a particular MR job with multiple Hive tables as
> input, Pig calls {{setLocation}} on each {{LoaderFunc (HCatLoader)}} instance
> but only one table's information (InputJobInfo instance) gets tracked in the
> JobConf. (This is under config key {{HCatConstants.HCAT_KEY_JOB_INFO}}).
> Any such call overwrites preexisting values, and thus only the last table's
> information will be considered when Pig calls {{getStatistics}} to calculate
> and estimate required reducer count.
> In cases when there are 2 input tables, 256GB and 1MB in size respectively,
> Pig will query the size information from HCat for both of them, but it will
> either see 1MB+1MB=2MB or 256GB+256GB=0.5TB depending on input order in the
> execution plan's DAG.
> It should of course see 256.00097GB in total and use 257 reducers by default
> accordingly.
> In unlucky cases this will be seen as 2MB and 1 reducer will have to struggle
> with the actual 256.00097GB...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20975) Fix PreUpgradeTool Orc namespace
[
https://issues.apache.org/jira/browse/HIVE-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700908#comment-16700908
]
Hive QA commented on HIVE-20975:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949675/HIVE-20975.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15079/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15079/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15079/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-11-27 19:38:30.559
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-15079/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-11-27 19:38:30.563
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 56926af HIVE-20971: TestJdbcWithDBTokenStore[*] should both use
MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb (Peter Vary, reviewed by
Vihang Karajgaonkar)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 56926af HIVE-20971: TestJdbcWithDBTokenStore[*] should both use
MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb (Peter Vary, reviewed by
Vihang Karajgaonkar)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-11-27 19:38:31.239
+ rm -rf ../yetus_PreCommit-HIVE-Build-15079
+ mkdir ../yetus_PreCommit-HIVE-Build-15079
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-15079
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15079/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java:
does not exist in index
error:
a/upgrade-acid/pre-upgrade/src/test/java/org/apache/hadoop/hive/upgrade/acid/TestPreUpgradeTool.java:
does not exist in index
Going to apply patch with: git apply -p1
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] COMPILATION ERROR :
[ERROR]
/data/hiveptest/working/apache-github-source-source/upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java:[196,39]
cannot find symbol
symbol: class HiveMetaHookLoader
location: class org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool
[ERROR]
/data/hiveptest/working/apache-github-source-source/upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java:[197,30]
cannot find symbol
symbol: method getHookLoader()
location: class org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool
[ERROR]
/data/hiveptest/working/apache-github-source-source/upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java:[197,68]
cannot find symbol
symbol: class HiveMetaStoreClient
location: class org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool
[ERROR]
/data/hiveptest/working/apache-github-source-source/upgrade-acid/pre-upgrade/src/main/java/org/apache/hadoop/hive/upgrade/acid/PreUpgradeTool.java:[460,26]
package HadoopShims does not exist
protoc-jar: executing: [/tmp/protoc3994231616503101642.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc3994231616503101642.exe,
[jira] [Commented] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700905#comment-16700905
]
Hive QA commented on HIVE-20330:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949711/HIVE-20330.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15553 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar.org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar
(batchId=242)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15078/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15078/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15078/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949711 - PreCommit-HIVE-Build
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch, HIVE-20330.5.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
> performance drop in a workload that has multiple inputs from HCatLoader.
> The reason is that for a particular MR job with multiple Hive tables as
> input, Pig calls {{setLocation}} on each {{LoaderFunc (HCatLoader)}} instance
> but only one table's information (InputJobInfo instance) gets tracked in the
> JobConf. (This is under config key {{HCatConstants.HCAT_KEY_JOB_INFO}}).
> Any such call overwrites preexisting values, and thus only the last table's
> information will be considered when Pig calls {{getStatistics}} to calculate
> and estimate required reducer count.
> In cases when there are 2 input tables, 256GB and 1MB in size respectively,
> Pig will query the size information from HCat for both of them, but it will
> either see 1MB+1MB=2MB or 256GB+256GB=0.5TB depending on input order in the
> execution plan's DAG.
> It should of course see 256.00097GB in total and use 257 reducers by default
> accordingly.
> In unlucky cases this will be seen as 2MB and 1 reducer will have to struggle
> with the actual 256.00097GB...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20546) Upgrade to Druid 0.13.0
[ https://issues.apache.org/jira/browse/HIVE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-20546: Status: Patch Available (was: Open) > Upgrade to Druid 0.13.0 > --- > > Key: HIVE-20546 > URL: https://issues.apache.org/jira/browse/HIVE-20546 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20546.1.patch, HIVE-20546.patch > > > This task is to upgrade to druid 0.13.0 when it is released. Note that it > will hopefully be first apache release for Druid. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700855#comment-16700855
]
Hive QA commented on HIVE-20330:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} hcatalog/core in master has 31 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} hcatalog/hcatalog-pig-adapter in master has 2 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} hcatalog/core: The patch generated 0 new + 113
unchanged - 10 fixed = 113 total (was 123) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} hcatalog/hcatalog-pig-adapter: The patch generated 0
new + 102 unchanged - 1 fixed = 102 total (was 103) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} hcatalog/core generated 0 new + 30 unchanged - 1
fixed = 30 total (was 31) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} hcatalog-pig-adapter in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15078/dev-support/hive-personality.sh
|
| git revision | master / 56926af |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: hcatalog/core hcatalog/hcatalog-pig-adapter U: hcatalog |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15078/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch, HIVE-20330.5.patch
>
>
> While running performance tests on Pig (0.12 and 0.17)
[jira] [Commented] (HIVE-20974) TezTask should set task exception on failures
[
https://issues.apache.org/jira/browse/HIVE-20974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700832#comment-16700832
]
Hive QA commented on HIVE-20974:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949673/HIVE-20974.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 15546 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=232)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15077/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15077/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15077/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949673 - PreCommit-HIVE-Build
> TezTask should set task exception on failures
> -
>
> Key: HIVE-20974
> URL: https://issues.apache.org/jira/browse/HIVE-20974
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Trivial
> Attachments: HIVE-20974.1.patch
>
>
> TezTask logs the error as "Failed to execute tez graph" and proceeds further.
> "TaskRunner.runSequentail()" code would not be able to get these exceptions
> for TezTask. If there are any failure hooks configured, these exceptions
> wouldn't show up.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20546) Upgrade to Druid 0.13.0
[ https://issues.apache.org/jira/browse/HIVE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700784#comment-16700784 ] slim bouguerra commented on HIVE-20546: --- maybe you want to make this patch available and re upload the patch to run checkstyle ? > Upgrade to Druid 0.13.0 > --- > > Key: HIVE-20546 > URL: https://issues.apache.org/jira/browse/HIVE-20546 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20546.1.patch, HIVE-20546.patch > > > This task is to upgrade to druid 0.13.0 when it is released. Note that it > will hopefully be first apache release for Druid. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20974) TezTask should set task exception on failures
[
https://issues.apache.org/jira/browse/HIVE-20974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700769#comment-16700769
]
Hive QA commented on HIVE-20974:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15077/dev-support/hive-personality.sh
|
| git revision | master / dffa3a1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15077/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TezTask should set task exception on failures
> -
>
> Key: HIVE-20974
> URL: https://issues.apache.org/jira/browse/HIVE-20974
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Trivial
> Attachments: HIVE-20974.1.patch
>
>
> TezTask logs the error as "Failed to execute tez graph" and proceeds further.
> "TaskRunner.runSequentail()" code would not be able to get these exceptions
> for TezTask. If there are any failure hooks configured, these exceptions
> wouldn't show up.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20932) Vectorize Druid Storage Handler Reader
[ https://issues.apache.org/jira/browse/HIVE-20932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-20932: -- Resolution: Fixed Status: Resolved (was: Patch Available) pushed to master commit 0a7cc714fe9872c9dddb3601d4b9fc8ddbacb062 > Vectorize Druid Storage Handler Reader > -- > > Key: HIVE-20932 > URL: https://issues.apache.org/jira/browse/HIVE-20932 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20932.3.patch, HIVE-20932.4.patch, > HIVE-20932.5.patch, HIVE-20932.6.patch, HIVE-20932.7.patch, > HIVE-20932.8.patch, HIVE-20932.8.patch, HIVE-20932.8.patch, HIVE-20932.patch > > > This patch aims at adding support for vectorize read of data from Druid to > Hive. > [~t3rmin4t0r] suggested that this will improve the performance of the top > level operators that supports vectorization. > As a first cut am just adding a wrapper around the existing Record Reader to > read up to 1024 row at a time. > Future work will be to avoid going via old reader and convert straight the > Json (smile format) to Vector primitive types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20971) TestJdbcWithDBTokenStore[*] should both use MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb
[ https://issues.apache.org/jira/browse/HIVE-20971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-20971: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks for the review [~vihangk1]! > TestJdbcWithDBTokenStore[*] should both use > MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb > --- > > Key: HIVE-20971 > URL: https://issues.apache.org/jira/browse/HIVE-20971 > Project: Hive > Issue Type: Bug > Components: Test >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20971.2.patch, HIVE-20971.3.patch, HIVE-20971.patch > > > The original intent was to use > MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb in both cases -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20546) Upgrade to Druid 0.13.0
[ https://issues.apache.org/jira/browse/HIVE-20546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-20546: Attachment: HIVE-20546.1.patch > Upgrade to Druid 0.13.0 > --- > > Key: HIVE-20546 > URL: https://issues.apache.org/jira/browse/HIVE-20546 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-20546.1.patch, HIVE-20546.patch > > > This task is to upgrade to druid 0.13.0 when it is released. Note that it > will hopefully be first apache release for Druid. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20971) TestJdbcWithDBTokenStore[*] should both use MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb
[
https://issues.apache.org/jira/browse/HIVE-20971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700723#comment-16700723
]
Hive QA commented on HIVE-20971:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949659/HIVE-20971.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 15546 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15076/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15076/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15076/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949659 - PreCommit-HIVE-Build
> TestJdbcWithDBTokenStore[*] should both use
> MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb
> ---
>
> Key: HIVE-20971
> URL: https://issues.apache.org/jira/browse/HIVE-20971
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20971.2.patch, HIVE-20971.3.patch, HIVE-20971.patch
>
>
> The original intent was to use
> MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb in both cases
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700701#comment-16700701
]
Andrew Sherman commented on HIVE-20740:
---
+1 LGTM
> Remove global lock in ObjectStore.setConf method
>
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch,
> HIVE-20740.04.patch, HIVE-20740.05.patch, HIVE-20740.06.patch,
> HIVE-20740.08.patch, HIVE-20740.09.patch, HIVE-20740.10.patch,
> HIVE-20740.11.patch, HIVE-20740.12.patch, HIVE-20740.13.patch,
> HIVE-20740.14.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other
> clients in concurrent workloads.
> {code}
> @Override
> @SuppressWarnings("nls")
> public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
> isInitialized = false;
> this.conf = conf;
> this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf,
> ConfVars.HIVE_TXN_STATS_ENABLED);
> configureSSL(conf);
> Properties propsFromConf = getDataSourceProps(conf);
> boolean propsChanged = !propsFromConf.equals(prop);
> if (propsChanged) {
> if (pmf != null){
> clearOutPmfClassLoaderCache(pmf);
> if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
> }
> }
> pmf = null;
> prop = null;
> }
> assert(!isActiveTransaction());
> shutdown();
> // Always want to re-create pm as we don't know if it were created by
> the
> // most recent instance of the pmf
> pm = null;
> directSql = null;
> expressionProxy = null;
> openTrasactionCalls = 0;
> currentTransaction = null;
> transactionStatus = TXN_STATUS.NO_STATE;
> initialize(propsFromConf);
> String partitionValidationRegex =
> MetastoreConf.getVar(this.conf,
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
> if (partitionValidationRegex != null &&
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern =
> Pattern.compile(partitionValidationRegex);
> } else {
> partitionValidationPattern = null;
> }
> // Note, if metrics have not been initialized this will return null,
> which means we aren't
> // using metrics. Thus we should always check whether this is non-null
> before using.
> MetricRegistry registry = Metrics.getRegistry();
> if (registry != null) {
> directSqlErrors =
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
> }
> this.batchSize = MetastoreConf.getIntVar(conf,
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
> if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
> } else {
> LOG.debug("Initialized ObjectStore");
> }
> } finally {
> pmfPropLock.unlock();
> }
> }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new
> connection to HMS which is trying to instantiate ObjectStore. We should
> either remove the lock or reduce the scope of the lock so that it is held for
> a very small amount of time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20794) Use Zookeeper for metastore service discovery
[ https://issues.apache.org/jira/browse/HIVE-20794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-20794: -- Attachment: HIVE-20794.08 Status: Patch Available (was: In Progress) Re-attaching the last patch to trigger ptests. Out of the three failures [TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]|https://builds.apache.org/job/PreCommit-HIVE-Build/15073/testReport/junit/org.apache.hadoop.hive.cli/TestMiniLlapLocalCliDriver/testCliDriver_cbo_limit_/] is failing intermittently for last few builds and the failure is unrelated to the changes in the patch. The other test passed when run alone on my laptop. > Use Zookeeper for metastore service discovery > - > > Key: HIVE-20794 > URL: https://issues.apache.org/jira/browse/HIVE-20794 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20794.01, HIVE-20794.02, HIVE-20794.03, > HIVE-20794.03, HIVE-20794.04, HIVE-20794.05, HIVE-20794.06, HIVE-20794.07, > HIVE-20794.07, HIVE-20794.08, HIVE-20794.08 > > > Right now, multiple metastore services can be specified in > hive.metastore.uris configuration, but that list is static and can not be > modified dynamically. Use Zookeeper for dynamic service discovery of > metastore. > h3. Improve ZooKeeperHiveHelper class (suggestions for name welcome) > The Zookeeper related code (for service discovery) accesses Zookeeper > parameters directly from HiveConf. The class is changed so that it could be > used for both HiveServer2 and Metastore server and works with both the > configurations. Following methods from HiveServer2 are now moved into > ZooKeeperHiveHelper. # startZookeeperClient # addServerInstanceToZooKeeper # > removeServerInstanceFromZooKeeper > h3. HiveMetaStore conf changes > # THRIFT_URIS (hive.metastore.uris) can also be used to specify ZooKeeper > quorum. When THRIFT_SERVICE_DISCOVERY_MODE > (hive.metastore.service.discovery.mode) is set to "zookeeper" the URIs are > used as ZooKeeper quorum. When it's set to be empty, the URIs are used to > locate the metastore directly. > # Here's list of Hiveserver2's parameters and their proposed metastore conf > counterparts. It looks odd that the Metastore related configurations do not > have their macros start with METASTORE, but start with THRIFT. I have just > followed naming convention used for other parameters. > ** HIVE_SERVER2_ZOOKEEPER_NAMESPACE - THRIFT_ZOOKEEPER_NAMESPACE > (hive.metastore.zookeeper.namespace) > ** HIVE_ZOOKEEPER_CLIENT_PORT - THRIFT_ZOOKEEPER_CLIENT_PORT > (hive.metastore.zookeeper.client.port) > ** HIVE_ZOOKEEPER_CONNECTION_TIMEOUT - THRIFT_ZOOKEEPER_CONNECTION_TIMEOUT - > (hive.metastore.zookeeper.connection.timeout) > ** HIVE_ZOOKEEPER_CONNECTION_MAX_RETRIES - > THRIFT_ZOOKEEPER_CONNECTION_MAX_RETRIES > (hive.metastore.zookeeper.connection.max.retries) > ** HIVE_ZOOKEEPER_CONNECTION_BASESLEEPTIME - > THRIFT_ZOOKEEPER_CONNECTION_BASESLEEPTIME > (hive.metastore.zookeeper.connection.basesleeptime) > # Additional configuration THRIFT_BIND_HOST is used to specify the host > address to bind Metastore service to. Right now Metastore binds to *, i.e all > addresses. Metastore doesn't then know which of those addresses it should add > to the ZooKeeper. THRIFT_BIND_HOST solves that problem. When this > configuration is specified the metastore server binds to that address and > also adds it to the ZooKeeper if dynamic service discovery mode is ZooKeeper. > Following Hive ZK configurations seem to be related to managing locks and > seem irrelevant for MS ZK. > # HIVE_ZOOKEEPER_SESSION_TIMEOUT > # HIVE_ZOOKEEPER_CLEAN_EXTRA_NODES > Since there is no configuration to be published, > HIVE_ZOOKEEPER_PUBLISH_CONFIGS does not have a THRIFT counterpart. > h3. HiveMetaStore class changes > # startMetaStore should also register the instance with Zookeeper, when > configured. > # When shutting a metastore server down it should deregister itself from > Zookeeper, when configured. > # These changes use the refactored code described above. > h3. HiveMetaStoreClient class changes > When service discovery mode is zookeeper, we fetch the metatstore URIs from > the specified ZooKeeper and treat those as if they were specified in > THRIFT_URIS i.e. use the existing mechanisms to choose a metastore server to > connect to and establish a connection. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20794) Use Zookeeper for metastore service discovery
[ https://issues.apache.org/jira/browse/HIVE-20794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-20794: -- Status: In Progress (was: Patch Available) > Use Zookeeper for metastore service discovery > - > > Key: HIVE-20794 > URL: https://issues.apache.org/jira/browse/HIVE-20794 > Project: Hive > Issue Type: Improvement >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20794.01, HIVE-20794.02, HIVE-20794.03, > HIVE-20794.03, HIVE-20794.04, HIVE-20794.05, HIVE-20794.06, HIVE-20794.07, > HIVE-20794.07, HIVE-20794.08 > > > Right now, multiple metastore services can be specified in > hive.metastore.uris configuration, but that list is static and can not be > modified dynamically. Use Zookeeper for dynamic service discovery of > metastore. > h3. Improve ZooKeeperHiveHelper class (suggestions for name welcome) > The Zookeeper related code (for service discovery) accesses Zookeeper > parameters directly from HiveConf. The class is changed so that it could be > used for both HiveServer2 and Metastore server and works with both the > configurations. Following methods from HiveServer2 are now moved into > ZooKeeperHiveHelper. # startZookeeperClient # addServerInstanceToZooKeeper # > removeServerInstanceFromZooKeeper > h3. HiveMetaStore conf changes > # THRIFT_URIS (hive.metastore.uris) can also be used to specify ZooKeeper > quorum. When THRIFT_SERVICE_DISCOVERY_MODE > (hive.metastore.service.discovery.mode) is set to "zookeeper" the URIs are > used as ZooKeeper quorum. When it's set to be empty, the URIs are used to > locate the metastore directly. > # Here's list of Hiveserver2's parameters and their proposed metastore conf > counterparts. It looks odd that the Metastore related configurations do not > have their macros start with METASTORE, but start with THRIFT. I have just > followed naming convention used for other parameters. > ** HIVE_SERVER2_ZOOKEEPER_NAMESPACE - THRIFT_ZOOKEEPER_NAMESPACE > (hive.metastore.zookeeper.namespace) > ** HIVE_ZOOKEEPER_CLIENT_PORT - THRIFT_ZOOKEEPER_CLIENT_PORT > (hive.metastore.zookeeper.client.port) > ** HIVE_ZOOKEEPER_CONNECTION_TIMEOUT - THRIFT_ZOOKEEPER_CONNECTION_TIMEOUT - > (hive.metastore.zookeeper.connection.timeout) > ** HIVE_ZOOKEEPER_CONNECTION_MAX_RETRIES - > THRIFT_ZOOKEEPER_CONNECTION_MAX_RETRIES > (hive.metastore.zookeeper.connection.max.retries) > ** HIVE_ZOOKEEPER_CONNECTION_BASESLEEPTIME - > THRIFT_ZOOKEEPER_CONNECTION_BASESLEEPTIME > (hive.metastore.zookeeper.connection.basesleeptime) > # Additional configuration THRIFT_BIND_HOST is used to specify the host > address to bind Metastore service to. Right now Metastore binds to *, i.e all > addresses. Metastore doesn't then know which of those addresses it should add > to the ZooKeeper. THRIFT_BIND_HOST solves that problem. When this > configuration is specified the metastore server binds to that address and > also adds it to the ZooKeeper if dynamic service discovery mode is ZooKeeper. > Following Hive ZK configurations seem to be related to managing locks and > seem irrelevant for MS ZK. > # HIVE_ZOOKEEPER_SESSION_TIMEOUT > # HIVE_ZOOKEEPER_CLEAN_EXTRA_NODES > Since there is no configuration to be published, > HIVE_ZOOKEEPER_PUBLISH_CONFIGS does not have a THRIFT counterpart. > h3. HiveMetaStore class changes > # startMetaStore should also register the instance with Zookeeper, when > configured. > # When shutting a metastore server down it should deregister itself from > Zookeeper, when configured. > # These changes use the refactored code described above. > h3. HiveMetaStoreClient class changes > When service discovery mode is zookeeper, we fetch the metatstore URIs from > the specified ZooKeeper and treat those as if they were specified in > THRIFT_URIS i.e. use the existing mechanisms to choose a metastore server to > connect to and establish a connection. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20897) TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error
[ https://issues.apache.org/jira/browse/HIVE-20897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20897: --- Status: Open (was: Patch Available) > TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error > > > Key: HIVE-20897 > URL: https://issues.apache.org/jira/browse/HIVE-20897 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20897.01.patch, HIVE-20897.02.patch, > HIVE-20897.03.patch, HIVE-20897.04.patch > > > if async prepare is enabled, control will be returned to the client before > driver could set of the query has a result set or not. But in current code, > while generating the response for the query, it is not checked if the result > set field is set or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20897) TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error
[ https://issues.apache.org/jira/browse/HIVE-20897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20897: --- Attachment: HIVE-20897.05.patch > TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error > > > Key: HIVE-20897 > URL: https://issues.apache.org/jira/browse/HIVE-20897 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20897.01.patch, HIVE-20897.02.patch, > HIVE-20897.03.patch, HIVE-20897.04.patch, HIVE-20897.05.patch > > > if async prepare is enabled, control will be returned to the client before > driver could set of the query has a result set or not. But in current code, > while generating the response for the query, it is not checked if the result > set field is set or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20897) TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error
[ https://issues.apache.org/jira/browse/HIVE-20897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-20897: --- Status: Patch Available (was: Open) > TestJdbcDriver2#testSelectExecAsync2 fails with result set not present error > > > Key: HIVE-20897 > URL: https://issues.apache.org/jira/browse/HIVE-20897 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20897.01.patch, HIVE-20897.02.patch, > HIVE-20897.03.patch, HIVE-20897.04.patch, HIVE-20897.05.patch > > > if async prepare is enabled, control will be returned to the client before > driver could set of the query has a result set or not. But in current code, > while generating the response for the query, it is not checked if the result > set field is set or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20971) TestJdbcWithDBTokenStore[*] should both use MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb
[
https://issues.apache.org/jira/browse/HIVE-20971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700633#comment-16700633
]
Hive QA commented on HIVE-20971:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15076/dev-support/hive-personality.sh
|
| git revision | master / dffa3a1 |
| Default Java | 1.8.0_111 |
| modules | C: itests/hive-minikdc U: itests/hive-minikdc |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15076/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestJdbcWithDBTokenStore[*] should both use
> MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb
> ---
>
> Key: HIVE-20971
> URL: https://issues.apache.org/jira/browse/HIVE-20971
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20971.2.patch, HIVE-20971.3.patch, HIVE-20971.patch
>
>
> The original intent was to use
> MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb in both cases
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-20330:
--
Status: In Progress (was: Patch Available)
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch, HIVE-20330.5.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
> performance drop in a workload that has multiple inputs from HCatLoader.
> The reason is that for a particular MR job with multiple Hive tables as
> input, Pig calls {{setLocation}} on each {{LoaderFunc (HCatLoader)}} instance
> but only one table's information (InputJobInfo instance) gets tracked in the
> JobConf. (This is under config key {{HCatConstants.HCAT_KEY_JOB_INFO}}).
> Any such call overwrites preexisting values, and thus only the last table's
> information will be considered when Pig calls {{getStatistics}} to calculate
> and estimate required reducer count.
> In cases when there are 2 input tables, 256GB and 1MB in size respectively,
> Pig will query the size information from HCat for both of them, but it will
> either see 1MB+1MB=2MB or 256GB+256GB=0.5TB depending on input order in the
> execution plan's DAG.
> It should of course see 256.00097GB in total and use 257 reducers by default
> accordingly.
> In unlucky cases this will be seen as 2MB and 1 reducer will have to struggle
> with the actual 256.00097GB...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-20330:
--
Status: Patch Available (was: In Progress)
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch, HIVE-20330.5.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
> performance drop in a workload that has multiple inputs from HCatLoader.
> The reason is that for a particular MR job with multiple Hive tables as
> input, Pig calls {{setLocation}} on each {{LoaderFunc (HCatLoader)}} instance
> but only one table's information (InputJobInfo instance) gets tracked in the
> JobConf. (This is under config key {{HCatConstants.HCAT_KEY_JOB_INFO}}).
> Any such call overwrites preexisting values, and thus only the last table's
> information will be considered when Pig calls {{getStatistics}} to calculate
> and estimate required reducer count.
> In cases when there are 2 input tables, 256GB and 1MB in size respectively,
> Pig will query the size information from HCat for both of them, but it will
> either see 1MB+1MB=2MB or 256GB+256GB=0.5TB depending on input order in the
> execution plan's DAG.
> It should of course see 256.00097GB in total and use 257 reducers by default
> accordingly.
> In unlucky cases this will be seen as 2MB and 1 reducer will have to struggle
> with the actual 256.00097GB...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-20330:
--
Attachment: HIVE-20330.5.patch
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch, HIVE-20330.5.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
> performance drop in a workload that has multiple inputs from HCatLoader.
> The reason is that for a particular MR job with multiple Hive tables as
> input, Pig calls {{setLocation}} on each {{LoaderFunc (HCatLoader)}} instance
> but only one table's information (InputJobInfo instance) gets tracked in the
> JobConf. (This is under config key {{HCatConstants.HCAT_KEY_JOB_INFO}}).
> Any such call overwrites preexisting values, and thus only the last table's
> information will be considered when Pig calls {{getStatistics}} to calculate
> and estimate required reducer count.
> In cases when there are 2 input tables, 256GB and 1MB in size respectively,
> Pig will query the size information from HCat for both of them, but it will
> either see 1MB+1MB=2MB or 256GB+256GB=0.5TB depending on input order in the
> execution plan's DAG.
> It should of course see 256.00097GB in total and use 257 reducers by default
> accordingly.
> In unlucky cases this will be seen as 2MB and 1 reducer will have to struggle
> with the actual 256.00097GB...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20776) Run HMS filterHooks on server-side in addition to client-side
[ https://issues.apache.org/jira/browse/HIVE-20776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Manamcheri updated HIVE-20776: -- Summary: Run HMS filterHooks on server-side in addition to client-side (was: Move HMS filterHooks from client-side to server-side) > Run HMS filterHooks on server-side in addition to client-side > - > > Key: HIVE-20776 > URL: https://issues.apache.org/jira/browse/HIVE-20776 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Karthik Manamcheri >Assignee: Karthik Manamcheri >Priority: Major > > In HMS, I noticed that all the filter hooks are applied on the client side > (in HiveMetaStoreClient.java). Is there any reason why we can't apply the > filters on the server-side? > Motivation: Some newer apache projects such as Kudu use HMS for metadata > storage. Kudu is not completely Java-based and there are interaction points > where they have C++ clients. In such cases, it would be ideal to have > consistent behavior from HMS side as far as filters, etc are concerned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700611#comment-16700611
]
Hive QA commented on HIVE-20330:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949674/HIVE-20330.4.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15075/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15075/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15075/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12949674/HIVE-20330.4.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949674 - PreCommit-HIVE-Build
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
> performance drop in a workload that has multiple inputs from HCatLoader.
> The reason is that for a particular MR job with multiple Hive tables as
> input, Pig calls {{setLocation}} on each {{LoaderFunc (HCatLoader)}} instance
> but only one table's information (InputJobInfo instance) gets tracked in the
> JobConf. (This is under config key {{HCatConstants.HCAT_KEY_JOB_INFO}}).
> Any such call overwrites preexisting values, and thus only the last table's
> information will be considered when Pig calls {{getStatistics}} to calculate
> and estimate required reducer count.
> In cases when there are 2 input tables, 256GB and 1MB in size respectively,
> Pig will query the size information from HCat for both of them, but it will
> either see 1MB+1MB=2MB or 256GB+256GB=0.5TB depending on input order in the
> execution plan's DAG.
> It should of course see 256.00097GB in total and use 257 reducers by default
> accordingly.
> In unlucky cases this will be seen as 2MB and 1 reducer will have to struggle
> with the actual 256.00097GB...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700609#comment-16700609
]
Hive QA commented on HIVE-20330:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949674/HIVE-20330.4.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15548 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=195)
[druidmini_test_ts.q,druidmini_expressions.q,druid_timestamptz2.q,druidmini_test_alter.q,druidkafkamini_csv.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15074/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15074/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15074/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949674 - PreCommit-HIVE-Build
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
> performance drop in a workload that has multiple inputs from HCatLoader.
> The reason is that for a particular MR job with multiple Hive tables as
> input, Pig calls {{setLocation}} on each {{LoaderFunc (HCatLoader)}} instance
> but only one table's information (InputJobInfo instance) gets tracked in the
> JobConf. (This is under config key {{HCatConstants.HCAT_KEY_JOB_INFO}}).
> Any such call overwrites preexisting values, and thus only the last table's
> information will be considered when Pig calls {{getStatistics}} to calculate
> and estimate required reducer count.
> In cases when there are 2 input tables, 256GB and 1MB in size respectively,
> Pig will query the size information from HCat for both of them, but it will
> either see 1MB+1MB=2MB or 256GB+256GB=0.5TB depending on input order in the
> execution plan's DAG.
> It should of course see 256.00097GB in total and use 257 reducers by default
> accordingly.
> In unlucky cases this will be seen as 2MB and 1 reducer will have to struggle
> with the actual 256.00097GB...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700537#comment-16700537
]
Hive QA commented on HIVE-20330:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} hcatalog/core in master has 31 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} hcatalog/hcatalog-pig-adapter in master has 2 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} hcatalog/core: The patch generated 0 new + 113
unchanged - 10 fixed = 113 total (was 123) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} hcatalog/hcatalog-pig-adapter: The patch generated 0
new + 102 unchanged - 1 fixed = 102 total (was 103) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} hcatalog/core generated 0 new + 30 unchanged - 1
fixed = 30 total (was 31) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} hcatalog-pig-adapter in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15074/dev-support/hive-personality.sh
|
| git revision | master / dffa3a1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: hcatalog/core hcatalog/hcatalog-pig-adapter U: hcatalog |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15074/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
[jira] [Commented] (HIVE-20794) Use Zookeeper for metastore service discovery
[
https://issues.apache.org/jira/browse/HIVE-20794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700505#comment-16700505
]
Hive QA commented on HIVE-20794:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949652/HIVE-20794.08
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15636 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testAlterPartitionSetLocationNonDefaultNameNode
(batchId=245)
org.apache.hadoop.hive.ql.TestDDLWithRemoteMetastoreSecondNamenode.testCreateDatabaseWithTableNonDefaultNameNode
(batchId=245)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15073/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15073/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15073/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949652 - PreCommit-HIVE-Build
> Use Zookeeper for metastore service discovery
> -
>
> Key: HIVE-20794
> URL: https://issues.apache.org/jira/browse/HIVE-20794
> Project: Hive
> Issue Type: Improvement
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20794.01, HIVE-20794.02, HIVE-20794.03,
> HIVE-20794.03, HIVE-20794.04, HIVE-20794.05, HIVE-20794.06, HIVE-20794.07,
> HIVE-20794.07, HIVE-20794.08
>
>
> Right now, multiple metastore services can be specified in
> hive.metastore.uris configuration, but that list is static and can not be
> modified dynamically. Use Zookeeper for dynamic service discovery of
> metastore.
> h3. Improve ZooKeeperHiveHelper class (suggestions for name welcome)
> The Zookeeper related code (for service discovery) accesses Zookeeper
> parameters directly from HiveConf. The class is changed so that it could be
> used for both HiveServer2 and Metastore server and works with both the
> configurations. Following methods from HiveServer2 are now moved into
> ZooKeeperHiveHelper. # startZookeeperClient # addServerInstanceToZooKeeper #
> removeServerInstanceFromZooKeeper
> h3. HiveMetaStore conf changes
> # THRIFT_URIS (hive.metastore.uris) can also be used to specify ZooKeeper
> quorum. When THRIFT_SERVICE_DISCOVERY_MODE
> (hive.metastore.service.discovery.mode) is set to "zookeeper" the URIs are
> used as ZooKeeper quorum. When it's set to be empty, the URIs are used to
> locate the metastore directly.
> # Here's list of Hiveserver2's parameters and their proposed metastore conf
> counterparts. It looks odd that the Metastore related configurations do not
> have their macros start with METASTORE, but start with THRIFT. I have just
> followed naming convention used for other parameters.
> ** HIVE_SERVER2_ZOOKEEPER_NAMESPACE - THRIFT_ZOOKEEPER_NAMESPACE
> (hive.metastore.zookeeper.namespace)
> ** HIVE_ZOOKEEPER_CLIENT_PORT - THRIFT_ZOOKEEPER_CLIENT_PORT
> (hive.metastore.zookeeper.client.port)
> ** HIVE_ZOOKEEPER_CONNECTION_TIMEOUT - THRIFT_ZOOKEEPER_CONNECTION_TIMEOUT -
> (hive.metastore.zookeeper.connection.timeout)
> ** HIVE_ZOOKEEPER_CONNECTION_MAX_RETRIES -
> THRIFT_ZOOKEEPER_CONNECTION_MAX_RETRIES
> (hive.metastore.zookeeper.connection.max.retries)
> ** HIVE_ZOOKEEPER_CONNECTION_BASESLEEPTIME -
> THRIFT_ZOOKEEPER_CONNECTION_BASESLEEPTIME
> (hive.metastore.zookeeper.connection.basesleeptime)
> # Additional configuration THRIFT_BIND_HOST is used to specify the host
> address to bind Metastore service to. Right now Metastore binds to *, i.e all
> addresses. Metastore doesn't then know which of those addresses it should add
> to the ZooKeeper. THRIFT_BIND_HOST solves that problem. When this
> configuration is specified the metastore server binds to that address and
> also adds it to the ZooKeeper if dynamic service discovery mode is ZooKeeper.
> Following Hive ZK configurations seem to be related to managing locks and
> seem irrelevant for MS ZK.
> # HIVE_ZOOKEEPER_SESSION_TIMEOUT
> # HIVE_ZOOKEEPER_CLEAN_EXTRA_NODES
> Since there is no configuration to be published,
> HIVE_ZOOKEEPER_PUBLISH_CONFIGS does not have a THRIFT counterpart.
> h3. HiveMetaStore class changes
> # startMetaStore should
[jira] [Commented] (HIVE-20794) Use Zookeeper for metastore service discovery
[
https://issues.apache.org/jira/browse/HIVE-20794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700489#comment-16700489
]
Hive QA commented on HIVE-20794:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
16s{color} | {color:blue} standalone-metastore/metastore-common in master has
29 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
0s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
41s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/util in master has 48 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch standalone-metastore passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} The patch metastore-server passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} ql: The patch generated 0 new + 17 unchanged - 4
fixed = 17 total (was 21) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch service passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch util passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
|
[jira] [Commented] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[ https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700472#comment-16700472 ] Antal Sinkovits commented on HIVE-20440: Finally a green one. [~stakiar] I've added the e2e test. Can you please check this? > Create better cache eviction policy for SmallTableCache > --- > > Key: HIVE-20440 > URL: https://issues.apache.org/jira/browse/HIVE-20440 > Project: Hive > Issue Type: Improvement > Components: Spark >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch, > HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch, > HIVE-20440.06.patch, HIVE-20440.07.patch, HIVE-20440.08.patch, > HIVE-20440.09.patch, HIVE-20440.10.patch, HIVE-20440.11.patch, > HIVE-20440.12.patch, HIVE-20440.13.patch, HIVE-20440.14.patch.txt, > HIVE-20440.15.patch > > > Enhance the SmallTableCache, to use guava cache with soft references, so that > we evict when there is memory pressure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[
https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700398#comment-16700398
]
Hive QA commented on HIVE-20440:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949648/HIVE-20440.15.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 1 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15072/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15072/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15072/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949648 - PreCommit-HIVE-Build
> Create better cache eviction policy for SmallTableCache
> ---
>
> Key: HIVE-20440
> URL: https://issues.apache.org/jira/browse/HIVE-20440
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch,
> HIVE-20440.03.patch, HIVE-20440.04.patch, HIVE-20440.05.patch,
> HIVE-20440.06.patch, HIVE-20440.07.patch, HIVE-20440.08.patch,
> HIVE-20440.09.patch, HIVE-20440.10.patch, HIVE-20440.11.patch,
> HIVE-20440.12.patch, HIVE-20440.13.patch, HIVE-20440.14.patch.txt,
> HIVE-20440.15.patch
>
>
> Enhance the SmallTableCache, to use guava cache with soft references, so that
> we evict when there is memory pressure.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20975) Fix PreUpgradeTool Orc namespace
[ https://issues.apache.org/jira/browse/HIVE-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-20975: - > Fix PreUpgradeTool Orc namespace > > > Key: HIVE-20975 > URL: https://issues.apache.org/jira/browse/HIVE-20975 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Blocker > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20440) Create better cache eviction policy for SmallTableCache
[
https://issues.apache.org/jira/browse/HIVE-20440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700364#comment-16700364
]
Hive QA commented on HIVE-20440:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 2312 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} ql: The patch generated 0 new + 54 unchanged - 2
fixed = 54 total (was 56) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} ql generated 0 new + 2311 unchanged - 1 fixed = 2311
total (was 2312) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15072/dev-support/hive-personality.sh
|
| git revision | master / dffa3a1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15072/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15072/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Create better cache eviction policy for SmallTableCache
> ---
>
> Key: HIVE-20440
> URL: https://issues.apache.org/jira/browse/HIVE-20440
> Project: Hive
> Issue Type: Improvement
> Components: Spark
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Attachments: HIVE-20440.01.patch, HIVE-20440.02.patch,
> HIVE-20440.03.patch,
[jira] [Updated] (HIVE-20975) Fix PreUpgradeTool Orc namespace
[ https://issues.apache.org/jira/browse/HIVE-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20975: -- Status: Patch Available (was: Open) > Fix PreUpgradeTool Orc namespace > > > Key: HIVE-20975 > URL: https://issues.apache.org/jira/browse/HIVE-20975 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Blocker > Attachments: HIVE-20975.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20975) Fix PreUpgradeTool Orc namespace
[ https://issues.apache.org/jira/browse/HIVE-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700354#comment-16700354 ] Miklos Gergely commented on HIVE-20975: --- Also removed some unused code. > Fix PreUpgradeTool Orc namespace > > > Key: HIVE-20975 > URL: https://issues.apache.org/jira/browse/HIVE-20975 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Blocker > Attachments: HIVE-20975.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20975) Fix PreUpgradeTool Orc namespace
[ https://issues.apache.org/jira/browse/HIVE-20975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-20975: -- Attachment: HIVE-20975.01.patch > Fix PreUpgradeTool Orc namespace > > > Key: HIVE-20975 > URL: https://issues.apache.org/jira/browse/HIVE-20975 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Blocker > Attachments: HIVE-20975.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-20330:
--
Status: Patch Available (was: In Progress)
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
> performance drop in a workload that has multiple inputs from HCatLoader.
> The reason is that for a particular MR job with multiple Hive tables as
> input, Pig calls {{setLocation}} on each {{LoaderFunc (HCatLoader)}} instance
> but only one table's information (InputJobInfo instance) gets tracked in the
> JobConf. (This is under config key {{HCatConstants.HCAT_KEY_JOB_INFO}}).
> Any such call overwrites preexisting values, and thus only the last table's
> information will be considered when Pig calls {{getStatistics}} to calculate
> and estimate required reducer count.
> In cases when there are 2 input tables, 256GB and 1MB in size respectively,
> Pig will query the size information from HCat for both of them, but it will
> either see 1MB+1MB=2MB or 256GB+256GB=0.5TB depending on input order in the
> execution plan's DAG.
> It should of course see 256.00097GB in total and use 257 reducers by default
> accordingly.
> In unlucky cases this will be seen as 2MB and 1 reducer will have to struggle
> with the actual 256.00097GB...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-20330:
--
Attachment: HIVE-20330.4.patch
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
> performance drop in a workload that has multiple inputs from HCatLoader.
> The reason is that for a particular MR job with multiple Hive tables as
> input, Pig calls {{setLocation}} on each {{LoaderFunc (HCatLoader)}} instance
> but only one table's information (InputJobInfo instance) gets tracked in the
> JobConf. (This is under config key {{HCatConstants.HCAT_KEY_JOB_INFO}}).
> Any such call overwrites preexisting values, and thus only the last table's
> information will be considered when Pig calls {{getStatistics}} to calculate
> and estimate required reducer count.
> In cases when there are 2 input tables, 256GB and 1MB in size respectively,
> Pig will query the size information from HCat for both of them, but it will
> either see 1MB+1MB=2MB or 256GB+256GB=0.5TB depending on input order in the
> execution plan's DAG.
> It should of course see 256.00097GB in total and use 257 reducers by default
> accordingly.
> In unlucky cases this will be seen as 2MB and 1 reducer will have to struggle
> with the actual 256.00097GB...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-20330:
--
Status: In Progress (was: Patch Available)
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch, HIVE-20330.4.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
> performance drop in a workload that has multiple inputs from HCatLoader.
> The reason is that for a particular MR job with multiple Hive tables as
> input, Pig calls {{setLocation}} on each {{LoaderFunc (HCatLoader)}} instance
> but only one table's information (InputJobInfo instance) gets tracked in the
> JobConf. (This is under config key {{HCatConstants.HCAT_KEY_JOB_INFO}}).
> Any such call overwrites preexisting values, and thus only the last table's
> information will be considered when Pig calls {{getStatistics}} to calculate
> and estimate required reducer count.
> In cases when there are 2 input tables, 256GB and 1MB in size respectively,
> Pig will query the size information from HCat for both of them, but it will
> either see 1MB+1MB=2MB or 256GB+256GB=0.5TB depending on input order in the
> execution plan's DAG.
> It should of course see 256.00097GB in total and use 257 reducers by default
> accordingly.
> In unlucky cases this will be seen as 2MB and 1 reducer will have to struggle
> with the actual 256.00097GB...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20974) TezTask should set task exception on failures
[ https://issues.apache.org/jira/browse/HIVE-20974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-20974: Status: Patch Available (was: Open) > TezTask should set task exception on failures > - > > Key: HIVE-20974 > URL: https://issues.apache.org/jira/browse/HIVE-20974 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Trivial > Attachments: HIVE-20974.1.patch > > > TezTask logs the error as "Failed to execute tez graph" and proceeds further. > "TaskRunner.runSequentail()" code would not be able to get these exceptions > for TezTask. If there are any failure hooks configured, these exceptions > wouldn't show up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20974) TezTask should set task exception on failures
[ https://issues.apache.org/jira/browse/HIVE-20974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-20974: Attachment: HIVE-20974.1.patch > TezTask should set task exception on failures > - > > Key: HIVE-20974 > URL: https://issues.apache.org/jira/browse/HIVE-20974 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Trivial > Attachments: HIVE-20974.1.patch > > > TezTask logs the error as "Failed to execute tez graph" and proceeds further. > "TaskRunner.runSequentail()" code would not be able to get these exceptions > for TezTask. If there are any failure hooks configured, these exceptions > wouldn't show up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700312#comment-16700312
]
Hive QA commented on HIVE-20330:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949654/HIVE-20330.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15549 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=197)
[druidmini_masking.q,druidmini_joins.q,druid_timestamptz.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15071/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15071/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15071/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949654 - PreCommit-HIVE-Build
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
> performance drop in a workload that has multiple inputs from HCatLoader.
> The reason is that for a particular MR job with multiple Hive tables as
> input, Pig calls {{setLocation}} on each {{LoaderFunc (HCatLoader)}} instance
> but only one table's information (InputJobInfo instance) gets tracked in the
> JobConf. (This is under config key {{HCatConstants.HCAT_KEY_JOB_INFO}}).
> Any such call overwrites preexisting values, and thus only the last table's
> information will be considered when Pig calls {{getStatistics}} to calculate
> and estimate required reducer count.
> In cases when there are 2 input tables, 256GB and 1MB in size respectively,
> Pig will query the size information from HCat for both of them, but it will
> either see 1MB+1MB=2MB or 256GB+256GB=0.5TB depending on input order in the
> execution plan's DAG.
> It should of course see 256.00097GB in total and use 257 reducers by default
> accordingly.
> In unlucky cases this will be seen as 2MB and 1 reducer will have to struggle
> with the actual 256.00097GB...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20330) HCatLoader cannot handle multiple InputJobInfo objects for a job with multiple inputs
[
https://issues.apache.org/jira/browse/HIVE-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700259#comment-16700259
]
Hive QA commented on HIVE-20330:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} hcatalog/core in master has 31 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} hcatalog/hcatalog-pig-adapter in master has 2 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} hcatalog/core: The patch generated 0 new + 113
unchanged - 10 fixed = 113 total (was 123) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} hcatalog/hcatalog-pig-adapter: The patch generated 0
new + 102 unchanged - 1 fixed = 102 total (was 103) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} hcatalog/core generated 0 new + 30 unchanged - 1
fixed = 30 total (was 31) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} hcatalog-pig-adapter in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-15071/dev-support/hive-personality.sh
|
| git revision | master / 7c5b99b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: hcatalog/core hcatalog/hcatalog-pig-adapter U: hcatalog |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-15071/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HCatLoader cannot handle multiple InputJobInfo objects for a job with
> multiple inputs
> -
>
> Key: HIVE-20330
> URL: https://issues.apache.org/jira/browse/HIVE-20330
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Reporter: Adam Szita
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-20330.0.patch, HIVE-20330.1.patch,
> HIVE-20330.2.patch, HIVE-20330.3.patch
>
>
> While running performance tests on Pig (0.12 and 0.17) we've observed a huge
> performance drop
[jira] [Updated] (HIVE-20873) Use Murmur hash for VectorHashKeyWrapperTwoLong to reduce hash collision
[ https://issues.apache.org/jira/browse/HIVE-20873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20873: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) > Use Murmur hash for VectorHashKeyWrapperTwoLong to reduce hash collision > > > Key: HIVE-20873 > URL: https://issues.apache.org/jira/browse/HIVE-20873 > Project: Hive > Issue Type: Improvement >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20873.1.patch, HIVE-20873.2.patch, > HIVE-20873.3.patch > > > VectorHashKeyWrapperTwoLong is implemented with few bit shift operators and > XOR operators for short computation time, but more hash collision. Group by > operations become very slow on large data sets. It needs Murmur hash or a > better hash function for less hash collision. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20873) Use Murmur hash for VectorHashKeyWrapperTwoLong to reduce hash collision
[ https://issues.apache.org/jira/browse/HIVE-20873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700254#comment-16700254 ] Teddy Choi commented on HIVE-20873: --- Pushed to master. Thanks, [~bslim] and [~gopalv]! > Use Murmur hash for VectorHashKeyWrapperTwoLong to reduce hash collision > > > Key: HIVE-20873 > URL: https://issues.apache.org/jira/browse/HIVE-20873 > Project: Hive > Issue Type: Improvement >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20873.1.patch, HIVE-20873.2.patch, > HIVE-20873.3.patch > > > VectorHashKeyWrapperTwoLong is implemented with few bit shift operators and > XOR operators for short computation time, but more hash collision. Group by > operations become very slow on large data sets. It needs Murmur hash or a > better hash function for less hash collision. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20930) VectorCoalesce in FILTER mode doesn't take effect
[ https://issues.apache.org/jira/browse/HIVE-20930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20930: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) > VectorCoalesce in FILTER mode doesn't take effect > - > > Key: HIVE-20930 > URL: https://issues.apache.org/jira/browse/HIVE-20930 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20930.1.patch, HIVE-20930.2.patch, > HIVE-20930.3.patch > > > HIVE-20277 fixed vectorized case expressions for FILTER, but VectorCoalesce > is still not fixed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20971) TestJdbcWithDBTokenStore[*] should both use MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb
[ https://issues.apache.org/jira/browse/HIVE-20971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-20971: -- Attachment: HIVE-20971.3.patch > TestJdbcWithDBTokenStore[*] should both use > MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb > --- > > Key: HIVE-20971 > URL: https://issues.apache.org/jira/browse/HIVE-20971 > Project: Hive > Issue Type: Bug > Components: Test >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-20971.2.patch, HIVE-20971.3.patch, HIVE-20971.patch > > > The original intent was to use > MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb in both cases -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20971) TestJdbcWithDBTokenStore[*] should both use MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb
[
https://issues.apache.org/jira/browse/HIVE-20971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700191#comment-16700191
]
Hive QA commented on HIVE-20971:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12949636/HIVE-20971.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15542 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit]
(batchId=182)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/15070/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15070/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15070/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12949636 - PreCommit-HIVE-Build
> TestJdbcWithDBTokenStore[*] should both use
> MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb
> ---
>
> Key: HIVE-20971
> URL: https://issues.apache.org/jira/browse/HIVE-20971
> Project: Hive
> Issue Type: Bug
> Components: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20971.2.patch, HIVE-20971.patch
>
>
> The original intent was to use
> MiniHiveKdc.getMiniHS2WithKerbWithRemoteHMSWithKerb in both cases
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
