[jira] [Created] (HIVE-23848) TestHiveMetaStoreChecker and TestMiniLlapLocalCliDriver tests are failing
Kishen Das created HIVE-23848: - Summary: TestHiveMetaStoreChecker and TestMiniLlapLocalCliDriver tests are failing Key: HIVE-23848 URL: https://issues.apache.org/jira/browse/HIVE-23848 Project: Hive Issue Type: Test Components: HiveServer2 Reporter: Kishen Das testAddPartitionNormalDeltas – org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker testCliDriver[show_partitions2] – org.apache.hadoop.hive.cli.split21.TestMiniLlapLocalCliDriver testAddPartitionMMBase – org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker testAddPartitionCompactedDeltas – org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker testAddPartitionCompactedBase – org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23847) Extracting hive-parser module broke exec jar upload in tez
Antal Sinkovits created HIVE-23847:
--
Summary: Extracting hive-parser module broke exec jar upload in tez
Key: HIVE-23847
URL: https://issues.apache.org/jira/browse/HIVE-23847
Project: Hive
Issue Type: Bug
Reporter: Antal Sinkovits
2020-07-13 16:53:50,551 [INFO] [Dispatcher thread {Central}]
|HistoryEventHandler.criticalEvents|:
[HISTORY][DAG:dag_1594632473849_0001_1][Event:TASK_ATTEMPT_FINISHED]:
vertexName=Map 1, taskAttemptId=attempt_1594632473849_0001_1_00_00_0,
creationTime=1594652027059, allocationTime=1594652028460,
startTime=1594652029356, finishTime=1594652030546, timeTaken=1190,
status=FAILED, taskFailureType=NON_FATAL, errorEnum=FRAMEWORK_ERROR,
diagnostics=Error: Error while running task ( failure ) :
attempt_1594632473849_0001_1_00_00_0:java.lang.RuntimeException:
java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:340)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:266)
... 16 more
Caused by: java.lang.NoClassDefFoundError:
org/apache/hadoop/hive/ql/parse/ParseException
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.getDeclaredConstructor(Class.java:2178)
at
org.apache.hive.common.util.ReflectionUtil.newInstance(ReflectionUtil.java:79)
at
org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDTF(Registry.java:225)
at
org.apache.hadoop.hive.ql.exec.Registry.registerGenericUDTF(Registry.java:217)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.(FunctionRegistry.java:544)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:154)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.isConsistentWithinQuery(ExprNodeEvaluator.java:117)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:102)
at
org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:76)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:69)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:359)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:548)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:502)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:368)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:506)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:303)
... 17 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hive.ql.parse.ParseException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppC
[jira] [Created] (HIVE-23846) Avoid unnecessary serialization and deserialization of bitvectors
Yu-Wen Lai created HIVE-23846: - Summary: Avoid unnecessary serialization and deserialization of bitvectors Key: HIVE-23846 URL: https://issues.apache.org/jira/browse/HIVE-23846 Project: Hive Issue Type: Bug Components: Standalone Metastore Reporter: Yu-Wen Lai Assignee: Yu-Wen Lai In the method *getNdvEstimator* of *ColumnStatsDataInspector*, it will call isSetBitVectors(), in which it serializes the bitvectors again when we already have deserialized bitvectors _ndvEstimator_. For example, we can see this pattern from [LongColumnStatsDataInspector|[https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/columnstats/cache/LongColumnStatsDataInspector.java#L106]]. This method could check if the _ndvEstimator_ is set first so that it won't need to serialize and deserialize back. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23845) why the new version removed the parameter hive.enforce.sorting & hive.enforce.bucketing ? Would it have a bad influence on inserting data to a table .
weitianpei created HIVE-23845: - Summary: why the new version removed the parameter hive.enforce.sorting & hive.enforce.bucketing ? Would it have a bad influence on inserting data to a table . Key: HIVE-23845 URL: https://issues.apache.org/jira/browse/HIVE-23845 Project: Hive Issue Type: Bug Components: HiveServer2 Affects Versions: 2.2.0 Reporter: weitianpei why the new version removed the parameter hive.enforce.sorting & hive.enforce.bucketing ? Would it have a bad influence on inserting data to a table . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23844) use fastparquet generate parquet format file, import hive, query error
Henry Lu created HIVE-23844:
---
Summary: use fastparquet generate parquet format file, import
hive, query error
Key: HIVE-23844
URL: https://issues.apache.org/jira/browse/HIVE-23844
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 2.1.1
Environment: OS: CentOS Linux release 7.6.1810
JDK: 1.8.0_181
hive: hive-common-2.1.1-cdh6.2.0
fastparquet: 0.4.0
Reporter: Henry Lu
Attachments: 1912076_20200330_000334(102_4).parquet
i use fastparquet generate parquet format file(Please check the attachment ！),
message as follows:
message schema {
optional double timestamps;
optional int32 ESC_BrakePressure (UINT_8);
optional int32 ESC_BrakePressureValid (UINT_8);
optional int32 ESC_EBDWork (UINT_8);
optional int32 ESC_ABSWorkLable (UINT_8);
optional int32 ESC_EBDAlarm (UINT_8);
optional int32 ESC_VehSpdValidFlag (UINT_8);
optional int32 ESC_ABSAlarmSignal (UINT_8);
optional float ESC_VehSpd;
optional float ESC_FrontLeftWHeelSpd;
optional int32 ESC_FLWHeelSpdFaultSignal (UINT_8);
optional float ESC_FrontRightWHeelSpd;
optional int32 ESC_FRWHeelSpdFaultSignal (UINT_8);
optional float ESC_RearLeftWheelSpd;
optional int32 ESC_RLWHeelSpdFaultSignal (UINT_8);
optional float ESC_RearRightWheelSpd;
optional int32 ESC_RRWHeelSpdFaultSignal (UINT_8);
optional int32 ESC_Longitudinal_Acceleration_flag (UINT_8);
optional float ESC_Longitudinal_Acceleration;
optional int32 ESC_ESCOFF (UINT_8);
optional int32 ESC_ESCWorkStatus (UINT_8);
optional int32 ESC_ESCAlarmSig (UINT_8);
optional int32 ESC_TCSCFActive (UINT_8);
optional int32 ESC_ReqIncreaseTorqueFlag (UINT_8);
optional int32 ESC_ReqDecreaseTorqueFlag (UINT_8);
optional int32 ESC_ReqIncreaseTorque (UINT_8);
optional int32 ESC_ReqDecreaseTorque (UINT_8);
optional int32 ESC_ESCValidity (UINT_8);
optional int32 ESC_RollingCount_ESC3 (UINT_8);
optional int32 ESC_CICkSum_ESC3 (UINT_8);
}
creator: fastparquet-python version 1.0.0 (build 111)
extra: pandas = \{"column_indexes": [{"field_name": null, "metadata": null,
"name": null, "numpy_type": "object", "pandas_type": "mixed-integer"}],
"columns": [\{"field_name": "timestamps", "metadata": null, "name":
"timestamps", "numpy_type": "float64", "pandas_type": "float64"},
\{"field_name": "ESC_BrakePressure", "metadata": null, "name":
"ESC_BrakePressure", "numpy_type": "uint8", "pandas_type": "uint8"},
\{"field_name": "ESC_BrakePressureValid", "metadata": null, "name":
"ESC_BrakePressureValid", "numpy_type": "uint8", "pandas_type": "uint8"},
\{"field_name": "ESC_EBDWork", "metadata": null, "name": "ESC_EBDWork",
"numpy_type": "uint8", "pandas_type": "uint8"}, \{"field_name":
"ESC_ABSWorkLable", "metadata": null, "name": "ESC_ABSWorkLable", "numpy_type":
"uint8", "pandas_type": "uint8"}, \{"field_name": "ESC_EBDAlarm", "metadata":
null, "name": "ESC_EBDAlarm", "numpy_type": "uint8", "pandas_type": "uint8"},
\{"field_name": "ESC_VehSpdValidFlag", "metadata": null, "name":
"ESC_VehSpdValidFlag", "numpy_type": "uint8", "pandas_type": "uint8"},
\{"field_name": "ESC_ABSAlarmSignal", "metadata": null, "name":
"ESC_ABSAlarmSignal", "numpy_type": "uint8", "pandas_type": "uint8"},
\{"field_name": "ESC_VehSpd", "metadata": null, "name": "ESC_VehSpd",
"numpy_type": "float32", "pandas_type": "float32"}, \{"field_name":
"ESC_FrontLeftWHeelSpd", "metadata": null, "name": "ESC_FrontLeftWHeelSpd",
"numpy_type": "float32", "pandas_type": "float32"}, \{"field_name":
"ESC_FLWHeelSpdFaultSignal", "metadata": null, "name":
"ESC_FLWHeelSpdFaultSignal", "numpy_type": "uint8", "pandas_type": "uint8"},
\{"field_name": "ESC_FrontRightWHeelSpd", "metadata": null, "name":
"ESC_FrontRightWHeelSpd", "numpy_type": "float32", "pandas_type": "float32"},
\{"field_name": "ESC_FRWHeelSpdFaultSignal", "metadata": null, "name":
"ESC_FRWHeelSpdFaultSignal", "numpy_type": "uint8", "pandas_type": "uint8"},
\{"field_name": "ESC_RearLeftWheelSpd", "metadata": null, "name":
"ESC_RearLeftWheelSpd", "numpy_type": "float32", "pandas_type": "float32"},
\{"field_name": "ESC_RLWHeelSpdFaultSignal", "metadata": null, "name":
"ESC_RLWHeelSpdFaultSignal", "numpy_type": "uint8", "pandas_type": "uint8"},
\{"field_name": "ESC_RearRightWheelSpd", "metadata": null, "name":
"ESC_RearRightWheelSpd", "numpy_type": "float32", "pandas_type": "float32"},
\{"field_name": "ESC_RRWHeelSpdFaultSignal", "metadata": null, "name":
"ESC_RRWHeelSpdFaultSignal", "numpy_type": "uint8", "pandas_type": "uint8"},
\{"field_name": "ESC_Longitudinal_Acceleration_flag", "metadata": null, "name":
"ESC_Longitudinal_Acceleration_flag", "numpy_type": "uint8", "pandas_type":
"uint8"}, \{"field_name": "ESC_Longitudinal_Acceleration", "metadata": null,
"name": "ESC_Longitudinal_Acceleration", "numpy_type": "float32",
"pandas_type": "float32"}, \{"field_name": "ESC_ESCOFF", "metadata": null,
