[jira] [Updated] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
[ https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15305: --- Attachment: HIVE-15305.1.patch > Add tests for METASTORE_EVENT_LISTENERS > --- > > Key: HIVE-15305 > URL: https://issues.apache.org/jira/browse/HIVE-15305 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15305.1.patch, HIVE-15305.1.patch, > HIVE-15305.1.patch, HIVE-15305.1.patch, HIVE-15305.patch > > > HIVE-15232 reused TestDbNotificationListener to test > METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of > METASTORE_EVENT_LISTENERS config. We should test both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[ https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115511#comment-16115511 ] Eric Yang commented on HIVE-17126: -- >From the log output, it appears that SQL statement generated by DataNucleus is >double quoting table names and variable names. The syntax is invalid for >MariaDB. It appears that DataNucleus failed to detect dialect properly for >this specific combination of OS and MariaDB. I agree that HIVE-14870 can >solve this problem in the long run. If there is something that can be done in >detection of SQL dialect, maybe we can have a solution sooner for this issue. > Hive Metastore is incompatible with MariaDB 10.x > > > Key: HIVE-17126 > URL: https://issues.apache.org/jira/browse/HIVE-17126 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.0, 1.1.0, 2.0.0 >Reporter: Eric Yang > > MariaDB 10.x is commonly used for cheap RDBMS high availability. Hive usage > of Datanucleus is currently preventing Hive Metastore to use MariaDB 10.x as > highly available metastore. Datanucleus generate SQL statements that are not > parsable by MariaDB 10.x when dropping Hive table or database schema. > Without MariaDB HA setup, the SQL statement problem also exists for metastore > interaction with MariaDB 10.x. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[
https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115508#comment-16115508
]
Eric Yang commented on HIVE-17126:
--
Output from hive shell:
{code}
hive> drop table employee;
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Exception thrown
when executing query)
hive>
{code}
> Hive Metastore is incompatible with MariaDB 10.x
>
>
> Key: HIVE-17126
> URL: https://issues.apache.org/jira/browse/HIVE-17126
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 1.2.0, 1.1.0, 2.0.0
>Reporter: Eric Yang
>
> MariaDB 10.x is commonly used for cheap RDBMS high availability. Hive usage
> of Datanucleus is currently preventing Hive Metastore to use MariaDB 10.x as
> highly available metastore. Datanucleus generate SQL statements that are not
> parsable by MariaDB 10.x when dropping Hive table or database schema.
> Without MariaDB HA setup, the SQL statement problem also exists for metastore
> interaction with MariaDB 10.x.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[
https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115507#comment-16115507
]
Eric Yang commented on HIVE-17126:
--
Stack trace for the failure:
{code}
2017-08-05 13:42:25,984 ERROR bonecp.BoneCP
(BoneCP.java:destroyConnection(221)) - Error in attempting to close connection
java.sql.SQLException: Unknown system variable 'OPTION'
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
at com.mysql.jdbc.ConnectionImpl.unsetMaxRows(ConnectionImpl.java:5421)
at com.mysql.jdbc.StatementImpl.realClose(StatementImpl.java:2441)
at
com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3079)
at
com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java:1585)
at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4361)
at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1557)
at
com.jolbox.bonecp.ConnectionHandle.internalClose(ConnectionHandle.java:549)
at com.jolbox.bonecp.BoneCP.destroyConnection(BoneCP.java:219)
at
com.jolbox.bonecp.ConnectionHandle.markPossiblyBroken(ConnectionHandle.java:390)
at
com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:183)
at
org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeQuery(ParamLoggingPreparedStatement.java:381)
at
org.datanucleus.store.rdbms.SQLController.executeStatementQuery(SQLController.java:504)
at
org.datanucleus.store.rdbms.query.SQLQuery.performExecute(SQLQuery.java:280)
at org.datanucleus.store.query.Query.executeQuery(Query.java:1786)
at
org.datanucleus.store.query.AbstractSQLQuery.executeWithArray(AbstractSQLQuery.java:339)
at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:312)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.executeWithArray(MetaStoreDirectSql.java:1660)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:483)
at
org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitions(MetaStoreDirectSql.java:403)
at
org.apache.hadoop.hive.metastore.ObjectStore$2.getSqlResult(ObjectStore.java:1735)
at
org.apache.hadoop.hive.metastore.ObjectStore$2.getSqlResult(ObjectStore.java:1731)
at
org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2391)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsInternal(ObjectStore.java:1742)
at
org.apache.hadoop.hive.metastore.ObjectStore.getPartitions(ObjectStore.java:1725)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
at com.sun.proxy.$Proxy2.getPartitions(Unknown Source)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.dropPartitionsAndGetLocations(HiveMetaStore.java:1693)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_core(HiveMetaStore.java:1532)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.drop_table_with_environment_context(HiveMetaStore.java:1737)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy4.drop_table_with_environment_context(Unknown
Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:9256)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$drop_table_with_environment_context.getResult(ThriftHiveMetastore.java:9240)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
at
[jira] [Commented] (HIVE-17126) Hive Metastore is incompatible with MariaDB 10.x
[ https://issues.apache.org/jira/browse/HIVE-17126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115506#comment-16115506 ] Eric Yang commented on HIVE-17126: -- This seems to affect RHEL/CentOS 6.8 with MariaDB 10.x release. It has been reported that CentOS 7.x works fine. > Hive Metastore is incompatible with MariaDB 10.x > > > Key: HIVE-17126 > URL: https://issues.apache.org/jira/browse/HIVE-17126 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.2.0, 1.1.0, 2.0.0 >Reporter: Eric Yang > > MariaDB 10.x is commonly used for cheap RDBMS high availability. Hive usage > of Datanucleus is currently preventing Hive Metastore to use MariaDB 10.x as > highly available metastore. Datanucleus generate SQL statements that are not > parsable by MariaDB 10.x when dropping Hive table or database schema. > Without MariaDB HA setup, the SQL statement problem also exists for metastore > interaction with MariaDB 10.x. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17255) hive_metastoreConstants.TABLE_IS_TRANSACTIONAL vs ConfVars.HIVE_TRANSACTIONAL_TABLE_SCAN
[ https://issues.apache.org/jira/browse/HIVE-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17255: -- Summary: hive_metastoreConstants.TABLE_IS_TRANSACTIONAL vs ConfVars.HIVE_TRANSACTIONAL_TABLE_SCAN (was: hive_metastoreConstants.TABLE_IS_TRANSACTIONAL vs ) > hive_metastoreConstants.TABLE_IS_TRANSACTIONAL vs > ConfVars.HIVE_TRANSACTIONAL_TABLE_SCAN > > > Key: HIVE-17255 > URL: https://issues.apache.org/jira/browse/HIVE-17255 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-17255.01.patch > > > constructor of Context() has > boolean isTableTransactional = > conf.getBoolean(hive_metastoreConstants.TABLE_IS_TRANSACTIONAL, false). > This looks wrong. Everywhere else we use > ConfVars.HIVE_TRANSACTIONAL_TABLE_SCAN. > (yet someone does set it - can't find where) > Utilities.copyTablePropertiesToConf() copies all table props to JobConf > There places in the code setting/expecting > ConfVars.HIVE_TRANSACTIONAL_TABLE_SCAN and other places setting/expecting > hive_metastoreConstants.TABLE_IS_TRANSACTIONAL. This is inconsistent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17255) OrcInputFormat.Context relying on wrong property
[ https://issues.apache.org/jira/browse/HIVE-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17255: -- Description: constructor of Context() has boolean isTableTransactional = conf.getBoolean(hive_metastoreConstants.TABLE_IS_TRANSACTIONAL, false). This looks wrong. Everywhere else we use ConfVars.HIVE_TRANSACTIONAL_TABLE_SCAN. (yet someone does set it - can't find where) Utilities.copyTablePropertiesToConf() copies all table props to JobConf There places in the code setting/expecting ConfVars.HIVE_TRANSACTIONAL_TABLE_SCAN and other places setting/expecting hive_metastoreConstants.TABLE_IS_TRANSACTIONAL. This is inconsistent. was: constructor of Context() has boolean isTableTransactional = conf.getBoolean(hive_metastoreConstants.TABLE_IS_TRANSACTIONAL, false). This looks wrong. Everywhere else we use ConfVars.HIVE_TRANSACTIONAL_TABLE_SCAN. (yet someone does set it - can't find where) > OrcInputFormat.Context relying on wrong property > > > Key: HIVE-17255 > URL: https://issues.apache.org/jira/browse/HIVE-17255 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-17255.01.patch > > > constructor of Context() has > boolean isTableTransactional = > conf.getBoolean(hive_metastoreConstants.TABLE_IS_TRANSACTIONAL, false). > This looks wrong. Everywhere else we use > ConfVars.HIVE_TRANSACTIONAL_TABLE_SCAN. > (yet someone does set it - can't find where) > Utilities.copyTablePropertiesToConf() copies all table props to JobConf > There places in the code setting/expecting > ConfVars.HIVE_TRANSACTIONAL_TABLE_SCAN and other places setting/expecting > hive_metastoreConstants.TABLE_IS_TRANSACTIONAL. This is inconsistent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17255) hive_metastoreConstants.TABLE_IS_TRANSACTIONAL vs
[ https://issues.apache.org/jira/browse/HIVE-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17255: -- Summary: hive_metastoreConstants.TABLE_IS_TRANSACTIONAL vs (was: OrcInputFormat.Context relying on wrong property) > hive_metastoreConstants.TABLE_IS_TRANSACTIONAL vs > -- > > Key: HIVE-17255 > URL: https://issues.apache.org/jira/browse/HIVE-17255 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-17255.01.patch > > > constructor of Context() has > boolean isTableTransactional = > conf.getBoolean(hive_metastoreConstants.TABLE_IS_TRANSACTIONAL, false). > This looks wrong. Everywhere else we use > ConfVars.HIVE_TRANSACTIONAL_TABLE_SCAN. > (yet someone does set it - can't find where) > Utilities.copyTablePropertiesToConf() copies all table props to JobConf > There places in the code setting/expecting > ConfVars.HIVE_TRANSACTIONAL_TABLE_SCAN and other places setting/expecting > hive_metastoreConstants.TABLE_IS_TRANSACTIONAL. This is inconsistent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17254) Skip updating AccessTime of recycled files in ReplChangeManager
[ https://issues.apache.org/jira/browse/HIVE-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-17254: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Test failures are not related. Patch pushed to master. > Skip updating AccessTime of recycled files in ReplChangeManager > --- > > Key: HIVE-17254 > URL: https://issues.apache.org/jira/browse/HIVE-17254 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Daniel Dai >Assignee: Daniel Dai > Fix For: 3.0.0 > > Attachments: HIVE-17254.1.patch > > > For recycled file, we update both ModifyTime and AccessTime: > fs.setTimes(path, now, now); > On some version of hdfs, this is now allowed when > "dfs.namenode.accesstime.precision" is set to 0. Though the issue is solved > in HDFS-9208, we don't use AccessTime in CM and this could be skipped so we > don't have to fail on this scenario. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17181) HCatOutputFormat should expose complete output-schema (including partition-keys) for dynamic-partitioning MR jobs
[
https://issues.apache.org/jira/browse/HIVE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115307#comment-16115307
]
Hive QA commented on HIVE-17181:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12880482/HIVE-17181.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10990 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=56)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=99)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=234)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=179)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=179)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6269/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6269/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6269/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12880482 - PreCommit-HIVE-Build
> HCatOutputFormat should expose complete output-schema (including
> partition-keys) for dynamic-partitioning MR jobs
> -
>
> Key: HIVE-17181
> URL: https://issues.apache.org/jira/browse/HIVE-17181
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 2.2.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-17181.1.patch, HIVE-17181.2.patch,
> HIVE-17181.branch-2.patch
>
>
> Map/Reduce jobs that use HCatalog APIs to write to Hive tables using Dynamic
> partitioning are expected to call the following API methods:
> # {{HCatOutputFormat.setOutput()}} to indicate which table/partitions to
> write to. This call populates the {{OutputJobInfo}} with details fetched from
> the Metastore.
> # {{HCatOutputFormat.setSchema()}} to indicate the output-schema for the data
> being written.
> It is a common mistake to invoke {{HCatOUtputFormat.setSchema()}} as follows:
> {code:java}
> HCatOutputFormat.setSchema(conf, HCatOutputFormat.getTableSchema(conf));
> {code}
> Unfortunately, {{getTableSchema()}} returns only the record-schema, not the
> entire table's schema. We'll need a better API for use in M/R jobs to get the
> complete table-schema.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
