[jira] [Created] (HIVE-23848) TestHiveMetaStoreChecker and TestMiniLlapLocalCliDriver tests are failing
Kishen Das created HIVE-23848: - Summary: TestHiveMetaStoreChecker and TestMiniLlapLocalCliDriver tests are failing Key: HIVE-23848 URL: https://issues.apache.org/jira/browse/HIVE-23848 Project: Hive Issue Type: Test Components: HiveServer2 Reporter: Kishen Das testAddPartitionNormalDeltas – org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker testCliDriver[show_partitions2] – org.apache.hadoop.hive.cli.split21.TestMiniLlapLocalCliDriver testAddPartitionMMBase – org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker testAddPartitionCompactedDeltas – org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker testAddPartitionCompactedBase – org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23847) Extracting hive-parser module broke exec jar upload in tez
Antal Sinkovits created HIVE-23847: -- Summary: Extracting hive-parser module broke exec jar upload in tez Key: HIVE-23847 URL: https://issues.apache.org/jira/browse/HIVE-23847 Project: Hive Issue Type: Bug Reporter: Antal Sinkovits 2020-07-13 16:53:50,551 [INFO] [Dispatcher thread {Central}] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:dag_1594632473849_0001_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=Map 1, taskAttemptId=attempt_1594632473849_0001_1_00_00_0, creationTime=1594652027059, allocationTime=1594652028460, startTime=1594652029356, finishTime=1594652030546, timeTaken=1190, status=FAILED, taskFailureType=NON_FATAL, errorEnum=FRAMEWORK_ERROR, diagnostics=Error: Error while running task ( failure ) : attempt_1594632473849_0001_1_00_00_0:java.lang.RuntimeException: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:340) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266) ... 16 more Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/parse/ParseException at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) at java.lang.Class.getConstructor0(Class.java:3075) at java.lang.Class.getDeclaredConstructor(Class.java:2178) at org.apache.hive.common.util.ReflectionUtil.newInstance(ReflectionUtil.java:79) at org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDTF(Registry.java:225) at org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDTF(Registry.java:217) at org.apache.hadoop.hive.ql.exec.FunctionRegistry.(FunctionRegistry.java:544) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:154) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.isConsistentWithinQuery(ExprNodeEvaluator.java:117) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:102) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:76) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:69) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:359) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:548) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:502) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:368) at org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:506) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:303) ... 17 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.parse.ParseException at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppC
[jira] [Created] (HIVE-23846) Avoid unnecessary serialization and deserialization of bitvectors
Yu-Wen Lai created HIVE-23846: - Summary: Avoid unnecessary serialization and deserialization of bitvectors Key: HIVE-23846 URL: https://issues.apache.org/jira/browse/HIVE-23846 Project: Hive Issue Type: Bug Components: Standalone Metastore Reporter: Yu-Wen Lai Assignee: Yu-Wen Lai In the method *getNdvEstimator* of *ColumnStatsDataInspector*, it will call isSetBitVectors(), in which it serializes the bitvectors again when we already have deserialized bitvectors _ndvEstimator_. For example, we can see this pattern from [LongColumnStatsDataInspector|[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/columnstats/cache/LongColumnStatsDataInspector.java#L106]]. This method could check if the _ndvEstimator_ is set first so that it won't need to serialize and deserialize back. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23845) why the new version removed the parameter hive.enforce.sorting & hive.enforce.bucketing ? Would it have a bad influence on inserting data to a table .
weitianpei created HIVE-23845: - Summary: why the new version removed the parameter hive.enforce.sorting & hive.enforce.bucketing ? Would it have a bad influence on inserting data to a table . Key: HIVE-23845 URL: https://issues.apache.org/jira/browse/HIVE-23845 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 2.2.0 Reporter: weitianpei why the new version removed the parameter hive.enforce.sorting & hive.enforce.bucketing ? Would it have a bad influence on inserting data to a table . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23844) use fastparquet generate parquet format file, import hive, query error
Henry Lu created HIVE-23844: --- Summary: use fastparquet generate parquet format file, import hive, query error Key: HIVE-23844 URL: https://issues.apache.org/jira/browse/HIVE-23844 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 2.1.1 Environment: OS: CentOS Linux release 7.6.1810 JDK: 1.8.0_181 hive: hive-common-2.1.1-cdh6.2.0 fastparquet: 0.4.0 Reporter: Henry Lu Attachments: 1912076_20200330_000334(102_4).parquet i use fastparquet generate parquet format file(Please check the attachment !), message as follows: message schema { optional double timestamps; optional int32 ESC_BrakePressure (UINT_8); optional int32 ESC_BrakePressureValid (UINT_8); optional int32 ESC_EBDWork (UINT_8); optional int32 ESC_ABSWorkLable (UINT_8); optional int32 ESC_EBDAlarm (UINT_8); optional int32 ESC_VehSpdValidFlag (UINT_8); optional int32 ESC_ABSAlarmSignal (UINT_8); optional float ESC_VehSpd; optional float ESC_FrontLeftWHeelSpd; optional int32 ESC_FLWHeelSpdFaultSignal (UINT_8); optional float ESC_FrontRightWHeelSpd; optional int32 ESC_FRWHeelSpdFaultSignal (UINT_8); optional float ESC_RearLeftWheelSpd; optional int32 ESC_RLWHeelSpdFaultSignal (UINT_8); optional float ESC_RearRightWheelSpd; optional int32 ESC_RRWHeelSpdFaultSignal (UINT_8); optional int32 ESC_Longitudinal_Acceleration_flag (UINT_8); optional float ESC_Longitudinal_Acceleration; optional int32 ESC_ESCOFF (UINT_8); optional int32 ESC_ESCWorkStatus (UINT_8); optional int32 ESC_ESCAlarmSig (UINT_8); optional int32 ESC_TCSCFActive (UINT_8); optional int32 ESC_ReqIncreaseTorqueFlag (UINT_8); optional int32 ESC_ReqDecreaseTorqueFlag (UINT_8); optional int32 ESC_ReqIncreaseTorque (UINT_8); optional int32 ESC_ReqDecreaseTorque (UINT_8); optional int32 ESC_ESCValidity (UINT_8); optional int32 ESC_RollingCount_ESC3 (UINT_8); optional int32 ESC_CICkSum_ESC3 (UINT_8); } creator: fastparquet-python version 1.0.0 (build 111) extra: pandas = \{"column_indexes": [{"field_name": null, "metadata": null, "name": null, "numpy_type": "object", "pandas_type": "mixed-integer"}], "columns": [\{"field_name": "timestamps", "metadata": null, "name": "timestamps", "numpy_type": "float64", "pandas_type": "float64"}, \{"field_name": "ESC_BrakePressure", "metadata": null, "name": "ESC_BrakePressure", "numpy_type": "uint8", "pandas_type": "uint8"}, \{"field_name": "ESC_BrakePressureValid", "metadata": null, "name": "ESC_BrakePressureValid", "numpy_type": "uint8", "pandas_type": "uint8"}, \{"field_name": "ESC_EBDWork", "metadata": null, "name": "ESC_EBDWork", "numpy_type": "uint8", "pandas_type": "uint8"}, \{"field_name": "ESC_ABSWorkLable", "metadata": null, "name": "ESC_ABSWorkLable", "numpy_type": "uint8", "pandas_type": "uint8"}, \{"field_name": "ESC_EBDAlarm", "metadata": null, "name": "ESC_EBDAlarm", "numpy_type": "uint8", "pandas_type": "uint8"}, \{"field_name": "ESC_VehSpdValidFlag", "metadata": null, "name": "ESC_VehSpdValidFlag", "numpy_type": "uint8", "pandas_type": "uint8"}, \{"field_name": "ESC_ABSAlarmSignal", "metadata": null, "name": "ESC_ABSAlarmSignal", "numpy_type": "uint8", "pandas_type": "uint8"}, \{"field_name": "ESC_VehSpd", "metadata": null, "name": "ESC_VehSpd", "numpy_type": "float32", "pandas_type": "float32"}, \{"field_name": "ESC_FrontLeftWHeelSpd", "metadata": null, "name": "ESC_FrontLeftWHeelSpd", "numpy_type": "float32", "pandas_type": "float32"}, \{"field_name": "ESC_FLWHeelSpdFaultSignal", "metadata": null, "name": "ESC_FLWHeelSpdFaultSignal", "numpy_type": "uint8", "pandas_type": "uint8"}, \{"field_name": "ESC_FrontRightWHeelSpd", "metadata": null, "name": "ESC_FrontRightWHeelSpd", "numpy_type": "float32", "pandas_type": "float32"}, \{"field_name": "ESC_FRWHeelSpdFaultSignal", "metadata": null, "name": "ESC_FRWHeelSpdFaultSignal", "numpy_type": "uint8", "pandas_type": "uint8"}, \{"field_name": "ESC_RearLeftWheelSpd", "metadata": null, "name": "ESC_RearLeftWheelSpd", "numpy_type": "float32", "pandas_type": "float32"}, \{"field_name": "ESC_RLWHeelSpdFaultSignal", "metadata": null, "name": "ESC_RLWHeelSpdFaultSignal", "numpy_type": "uint8", "pandas_type": "uint8"}, \{"field_name": "ESC_RearRightWheelSpd", "metadata": null, "name": "ESC_RearRightWheelSpd", "numpy_type": "float32", "pandas_type": "float32"}, \{"field_name": "ESC_RRWHeelSpdFaultSignal", "metadata": null, "name": "ESC_RRWHeelSpdFaultSignal", "numpy_type": "uint8", "pandas_type": "uint8"}, \{"field_name": "ESC_Longitudinal_Acceleration_flag", "metadata": null, "name": "ESC_Longitudinal_Acceleration_flag", "numpy_type": "uint8", "pandas_type": "uint8"}, \{"field_name": "ESC_Longitudinal_Acceleration", "metadata": null, "name": "ESC_Longitudinal_Acceleration", "numpy_type": "float32", "pandas_type": "float32"}, \{"field_name": "ESC_ESCOFF", "metadata": null,