[jira] [Created] (HIVE-21140) hive run mapreduce error: Exception in thread "main" java.lang.IllegalArgumentException: Buffer size too small. size = 262144 needed = 566609
gehaijiang created HIVE-21140: - Summary: hive run mapreduce error: Exception in thread "main" java.lang.IllegalArgumentException: Buffer size too small. size = 262144 needed = 566609 Key: HIVE-21140 URL: https://issues.apache.org/jira/browse/HIVE-21140 Project: Hive Issue Type: Wish Reporter: gehaijiang hadoop 2.7.1 dw_common_site_event_acm_dtl $ hive --orcfiledump /newmogu_cold/apps/hive/warehouse/dw_common_site_event_acm_dtl/visit_date=2018-01-08/platform2_id=62/000801_0 Structure for /newmogu_cold/apps/hive/warehouse/dw_common_site_event_acm_dtl/visit_date=2018-01-08/platform2_id=62/000801_0 File Version: 0.12 with HIVE_8732 Exception in thread "main" java.lang.IllegalArgumentException: Buffer size too small. size = 262144 needed = 566609 at org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.readHeader(InStream.java:193) at org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.read(InStream.java:238) at java.io.InputStream.read(InputStream.java:101) at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:737) at com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701) at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99) at org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeFooter.(OrcProto.java:10661) at org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeFooter.(OrcProto.java:10625) at org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeFooter$1.parsePartialFrom(OrcProto.java:10730) at org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeFooter$1.parsePartialFrom(OrcProto.java:10725) at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49) at org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeFooter.parseFrom(OrcProto.java:10937) at org.apache.hadoop.hive.ql.io.orc.MetadataReader.readStripeFooter(MetadataReader.java:113) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripeFooter(RecordReaderImpl.java:228) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.beginReadStripe(RecordReaderImpl.java:805) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:776) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:986) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1019) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:205) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:549) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rows(ReaderImpl.java:534) at org.apache.hadoop.hive.ql.io.orc.FileDump.printMetaData(FileDump.java:104) at org.apache.hadoop.hive.ql.io.orc.FileDump.main(FileDump.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20253) nativetask cann't working in hive
gehaijiang created HIVE-20253: - Summary: nativetask cann't working in hive Key: HIVE-20253 URL: https://issues.apache.org/jira/browse/HIVE-20253 Project: Hive Issue Type: Wish Affects Versions: 1.2.1 Environment: hadoop 3.0.3 hive sql: set mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator; select count(*) from test_cold; Reporter: gehaijiang hadoop 3.0.3, Support nativetask. mapred-site.xml: mapreduce.job.map.output.collector.class org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator hive sql: set mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator; select count(*) from test_cold; URL: http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1532646043398_0019&tipid=task_1532646043398_0019_m_00 - Diagnostic Messages for this Task: Error: java.io.IOException: Initialization of all the collectors failed. Error in last collector was:java.io.IOException: Cannot find serializer for org.apache.hadoop.hive.ql.io.HiveKey at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:454) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by: java.io.IOException: Cannot find serializer for org.apache.hadoop.hive.ql.io.HiveKey at org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:127) at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:408) ... 7 more 2018-07-27 10:08:25,391 ERROR operation.Operation (SQLOperation.java:run(209)) - Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:316) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156) at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71) at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19185) Orc files can support zstd or lz4 ？
gehaijiang created HIVE-19185: - Summary: Orc files can support zstd or lz4 ？ Key: HIVE-19185 URL: https://issues.apache.org/jira/browse/HIVE-19185 Project: Hive Issue Type: Wish Reporter: gehaijiang orc file ： high level compression (one of NONE, ZLIB, SNAPPY) ， When can we support LZ4 or ZSTD？ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-18943) Hive building with Hadoop 3.0.0 ERROR
gehaijiang created HIVE-18943: - Summary: Hive building with Hadoop 3.0.0 ERROR Key: HIVE-18943 URL: https://issues.apache.org/jira/browse/HIVE-18943 Project: Hive Issue Type: Bug Affects Versions: 2.3.2 Reporter: gehaijiang hive version: 2.3.2 hadoop version: 3.0.0 building error: [WARNING] COMPILATION WARNING : [INFO] - [WARNING] /home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java: /home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java uses or overrides a deprecated API. [WARNING] /home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java: Recompile with -Xlint:deprecation for details. [WARNING] /home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java: /home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java uses unchecked or unsafe operations. [WARNING] /home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java: Recompile with -Xlint:unchecked for details. [INFO] 4 warnings [INFO] - [INFO] - [ERROR] COMPILATION ERROR : [INFO] - [ERROR] /home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[1088,29] constructor DistCpOptions in class org.apache.hadoop.tools.DistCpOptions cannot be applied to given types; required: org.apache.hadoop.tools.DistCpOptions.Builder found: java.util.List,org.apache.hadoop.fs.Path reason: actual and formal argument lists differ in length [ERROR] /home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[1089,12] cannot find symbol symbol: method setSyncFolder(boolean) location: variable options of type org.apache.hadoop.tools.DistCpOptions [ERROR] /home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[1090,12] cannot find symbol symbol: method setSkipCRC(boolean) location: variable options of type org.apache.hadoop.tools.DistCpOptions [ERROR] /home/data/programs/apache-hive-2.3.2-src/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:[1091,12] cannot find symbol symbol: method preserve(org.apache.hadoop.tools.DistCpOptions.FileAttribute) location: variable options of type org.apache.hadoop.tools.DistCpOptions [INFO] 4 errors [INFO] - [INFO] [INFO] Reactor Summary: [INFO] [INFO] Hive ... SUCCESS [ 50.616 s] [INFO] Hive Shims Common .. SUCCESS [07:24 min] [INFO] Hive Shims 0.23 FAILURE [04:17 min] [INFO] Hive Shims Scheduler ... SKIPPED [INFO] Hive Shims . SKIPPED [INFO] Hive Common SKIPPED [INFO] Hive Service RPC ... SKIPPED [INFO] Hive Serde . SKIPPED [INFO] Hive Metastore . SKIPPED [INFO] Hive Vector-Code-Gen Utilities . SKIPPED [INFO] Hive Llap Common ... SKIPPED [INFO] Hive Llap Client ... SKIPPED [INFO] Hive Llap Tez .. SKIPPED [INFO] Spark Remote Client SKIPPED [INFO] Hive Query Language SKIPPED [INFO] Hive Llap Server ... SKIPPED [INFO] Hive Service ... SKIPPED [INFO] Hive Accumulo Handler .. SKIPPED [INFO] Hive JDBC .. SKIPPED [INFO] Hive Beeline ... SKIPPED [INFO] Hive CLI ... SKIPPED [INFO] Hive Contrib ... SKIPPED [INFO] Hive Druid Handler . SKIPPED [INFO] Hive HBase Handler . SKIPPED [INFO] Hive JDBC Handler .. SKIPPED [INFO] Hive HCatalog .. SKIPPED [INFO] Hive HCatalog Core . SKIPPED [INF
[jira] [Created] (HIVE-17462) hive_1.2.1 memory leak
gehaijiang created HIVE-17462: - Summary: hive_1.2.1 memory leak Key: HIVE-17462 URL: https://issues.apache.org/jira/browse/HIVE-17462 Project: Hive Issue Type: Bug Affects Versions: 1.2.1 Environment: hive version 1.2.1 Reporter: gehaijiang hiveserver2 memory leak hive user third UDF （vs-1.0.2-SNAPSHOT.jar ， alogdata-1.0.3-SNAPSHOT-jar-with-dependencies.jar . and so on ) lr-x-- 1 data data 64 Sep 5 18:37 964 -> /tmp/9e38cc04-5693-474b-9c7d-bfdd978bcbb4_resources/vs-1.0.2-SNAPSHOT.jar (deleted) lr-x-- 1 data data 64 Sep 6 10:41 965 -> /tmp/188bbf2a-d8a5-48a7-81fc-b807f9ff201d_resources/alogdata-1.0.3-SNAPSHOT-jar-with-dependencies.jar (deleted) lr-x-- 1 data data 64 Sep 6 17:41 97 -> /home/data/programs/hadoop-2.7.1/share/hadoop/hdfs/lib/jsr305-3.0.0.jar lrwx-- 1 data data 64 Sep 5 18:37 975 -> socket:[1318353317] lr-x-- 1 data data 64 Sep 6 02:38 977 -> /tmp/64e309dc-352f-4ba4-b871-1aa78fe05945_resources/alogdata-1.0.3-SNAPSHOT-jar-with-dependencies.jar (deleted) lr-x-- 1 data data 64 Sep 6 17:41 98 -> /home/data/programs/hadoop-2.7.1/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar lrwx-- 1 data data 64 Sep 6 08:40 983 -> socket:[1299459344] lr-x-- 1 data data 64 Sep 5 19:37 987 -> /tmp/c3054987-c9c6-468a-8b5c-6e20b1972e0b_resources/alogdata-1.0.3-SNAPSHOT-jar-with-dependencies.jar (deleted) lr-x-- 1 data data 64 Sep 6 17:41 99 -> /home/data/programs/hadoop-2.7.1/share/hadoop/hdfs/lib/guava-11.0.2.jar lr-x-- 1 data data 64 Sep 6 08:40 994 -> /tmp/fc5c44b3-9bd8-4a32-a39a-66cd44032fee_resources/alogdata-1.0.3-SNAPSHOT-jar-with-dependencies.jar (deleted) lr-x-- 1 data data 64 Sep 6 06:39 996 -> /tmp/3b3c2bd6-0a0e-4599-b757-4a048a968457_resources/alogdata-1.0.3-SNAPSHOT-jar-with-dependencies.jar (deleted) lr-x-- 1 data data 64 Sep 5 17:36 999 -> /tmp/6ad76494-cdda-430b-b7d0-2213731655a8_resources/alogdata-1.0.3-SNAPSHOT-jar-with-dependencies.jar (deleted) PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 20084 data 20 0 13.6g 11g 533m S 62.3 9.2 6619:16 java /home/data/programs/jdk/jdk-current/bin/java-Djava.net.preferIPv4Stack=true-Dhadoop.log.dir=/home/data/hadoop/logs-Dhadoop.log.file=hadoop.log-Dhadoop.home.dir=/home/data/programs/hadoop-2.7.1-Dhadoop.id.str=data-Dhadoop.root.logger=INFO,DRFA-Djava.library.path=/home/data/programs/hadoop-2.7.1/lib/native-Dhadoop.policy.file=hadoop-policy.xml-Djava.net.preferIPv4Stack=true-XX:+UseConcMarkSweepGC-Xms8g-Xmx8g-Dhadoop.security.logger=INFO,NullAppenderorg.apache.hadoop.util.RunJar/home/data/programs/hive-current/lib/hive-service-1.2.1.jarorg.apache.hive.service.server.HiveServer2--hiveconfhive.log.file=hiveserver2.log -- This message was sent by Atlassian JIRA (v6.4.14#64029)
