Re: Is there a bug in hiveserver2 metastore caused hold huge objects
There are 18908 PM-s. I post it on http://jack-meng.qiniudn.com/hive-heap2.pnghttp://jack-meng.qiniudn.com/hive-heap.png Thanks. 2014-05-20 0:41 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: What is the count of PM-s that you are seeing when it crashes? From that screenshot I don't see what the problem is or I'm not reading it right. On Mon, May 19, 2014 at 1:17 AM, Meng QingPing mqingp...@gmail.com wrote: Thanks. I post it on http://jack-meng.qiniudn.com/hive-heap.png 2014-05-06 23:14 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: I don't think the dev list supports attachment. Can you post to some image sharing service? On Tue, May 6, 2014 at 3:20 AM, Meng QingPing mqingp...@gmail.com wrote: Weird, the picture attached in sent mail. Anyway send again. 2014-05-05 12:17 GMT+08:00 Chandra Reddy chandu...@gmail.com: seems you have missed attachment. -Chandra On Sun, May 4, 2014 at 6:57 PM, Meng QingPing mqingp...@gmail.com wrote: I run hiveserver2 with metastore in mysql. The hiveserver2 OOM and the heap dump show huge objects hold by org.datanucleus.api.jdo. JDOPersistenceManagerFactory as attached. It seems not release org.datanucleus.api.jdo.JDOPersistenceManager. Hive version is 0.13. Thanks, Jack -- Thanks, -Chandra. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping
Re: Is there a bug in hiveserver2 metastore caused hold huge objects
There are 18908 PM-s. I post it on http://jack-meng.qiniudn.com/hive-heap2.png Thanks. 2014-05-20 0:41 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: What is the count of PM-s that you are seeing when it crashes? From that screenshot I don't see what the problem is or I'm not reading it right. On Mon, May 19, 2014 at 1:17 AM, Meng QingPing mqingp...@gmail.com wrote: Thanks. I post it on http://jack-meng.qiniudn.com/hive-heap.png 2014-05-06 23:14 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: I don't think the dev list supports attachment. Can you post to some image sharing service? On Tue, May 6, 2014 at 3:20 AM, Meng QingPing mqingp...@gmail.com wrote: Weird, the picture attached in sent mail. Anyway send again. 2014-05-05 12:17 GMT+08:00 Chandra Reddy chandu...@gmail.com: seems you have missed attachment. -Chandra On Sun, May 4, 2014 at 6:57 PM, Meng QingPing mqingp...@gmail.com wrote: I run hiveserver2 with metastore in mysql. The hiveserver2 OOM and the heap dump show huge objects hold by org.datanucleus.api.jdo. JDOPersistenceManagerFactory as attached. It seems not release org.datanucleus.api.jdo.JDOPersistenceManager. Hive version is 0.13. Thanks, Jack -- Thanks, -Chandra. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping
Re: Is there a bug in hiveserver2 metastore caused hold huge objects
I post the heap bin file on http://jack-meng.qiniudn.com/heap.bin Thanks. 2014-05-21 15:41 GMT+08:00 Meng QingPing mqingp...@gmail.com: There are 18908 PM-s. I post it on http://jack-meng.qiniudn.com/hive-heap2.png Thanks. 2014-05-20 0:41 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: What is the count of PM-s that you are seeing when it crashes? From that screenshot I don't see what the problem is or I'm not reading it right. On Mon, May 19, 2014 at 1:17 AM, Meng QingPing mqingp...@gmail.com wrote: Thanks. I post it on http://jack-meng.qiniudn.com/hive-heap.png 2014-05-06 23:14 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: I don't think the dev list supports attachment. Can you post to some image sharing service? On Tue, May 6, 2014 at 3:20 AM, Meng QingPing mqingp...@gmail.com wrote: Weird, the picture attached in sent mail. Anyway send again. 2014-05-05 12:17 GMT+08:00 Chandra Reddy chandu...@gmail.com: seems you have missed attachment. -Chandra On Sun, May 4, 2014 at 6:57 PM, Meng QingPing mqingp...@gmail.com wrote: I run hiveserver2 with metastore in mysql. The hiveserver2 OOM and the heap dump show huge objects hold by org.datanucleus.api.jdo. JDOPersistenceManagerFactory as attached. It seems not release org.datanucleus.api.jdo.JDOPersistenceManager. Hive version is 0.13. Thanks, Jack -- Thanks, -Chandra. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping -- Thanks, Qingping
Re: Is there a bug in hiveserver2 metastore caused hold huge objects
Vaibhav, thanks for reply. It's a remote metastore, stored in mysql. 2014-05-22 5:31 GMT+08:00 Vaibhav Gumashta vgumas...@hortonworks.com: Meng, are you running a remote metastore or an embedded one (which would be true if you added this config while starting HS2: -hiveconf hive.metastore.uris= )? Thanks, --Vaibhav On Wed, May 21, 2014 at 2:28 PM, Vaibhav Gumashta vgumas...@hortonworks.com wrote: Thanks a lot for the dump Meng! On Wed, May 21, 2014 at 1:41 AM, Meng QingPing mqingp...@gmail.com wrote: I post the heap bin file on http://jack-meng.qiniudn.com/heap.bin Thanks. 2014-05-21 15:41 GMT+08:00 Meng QingPing mqingp...@gmail.com: There are 18908 PM-s. I post it on http://jack-meng.qiniudn.com/hive-heap2.png Thanks. 2014-05-20 0:41 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: What is the count of PM-s that you are seeing when it crashes? From that screenshot I don't see what the problem is or I'm not reading it right. On Mon, May 19, 2014 at 1:17 AM, Meng QingPing mqingp...@gmail.com wrote: Thanks. I post it on http://jack-meng.qiniudn.com/hive-heap.png 2014-05-06 23:14 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com : I don't think the dev list supports attachment. Can you post to some image sharing service? On Tue, May 6, 2014 at 3:20 AM, Meng QingPing mqingp...@gmail.com wrote: Weird, the picture attached in sent mail. Anyway send again. 2014-05-05 12:17 GMT+08:00 Chandra Reddy chandu...@gmail.com : seems you have missed attachment. -Chandra On Sun, May 4, 2014 at 6:57 PM, Meng QingPing mqingp...@gmail.com wrote: I run hiveserver2 with metastore in mysql. The hiveserver2 OOM and the heap dump show huge objects hold by org.datanucleus.api.jdo. JDOPersistenceManagerFactory as attached. It seems not release org.datanucleus.api.jdo.JDOPersistenceManager. Hive version is 0.13. Thanks, Jack -- Thanks, -Chandra. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping
Re: Is there a bug in hiveserver2 metastore caused hold huge objects
Thanks. I post it on http://jack-meng.qiniudn.com/hive-heap.png 2014-05-06 23:14 GMT+08:00 Sergey Shelukhin ser...@hortonworks.com: I don't think the dev list supports attachment. Can you post to some image sharing service? On Tue, May 6, 2014 at 3:20 AM, Meng QingPing mqingp...@gmail.com wrote: Weird, the picture attached in sent mail. Anyway send again. 2014-05-05 12:17 GMT+08:00 Chandra Reddy chandu...@gmail.com: seems you have missed attachment. -Chandra On Sun, May 4, 2014 at 6:57 PM, Meng QingPing mqingp...@gmail.com wrote: I run hiveserver2 with metastore in mysql. The hiveserver2 OOM and the heap dump show huge objects hold by org.datanucleus.api.jdo. JDOPersistenceManagerFactory as attached. It seems not release org.datanucleus.api.jdo.JDOPersistenceManager. Hive version is 0.13. Thanks, Jack -- Thanks, -Chandra. -- Thanks, Qingping -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Thanks, Qingping
Re: Is there a bug in hiveserver2 metastore caused hold huge objects
Weird, the picture attached in sent mail. Anyway send again. 2014-05-05 12:17 GMT+08:00 Chandra Reddy chandu...@gmail.com: seems you have missed attachment. -Chandra On Sun, May 4, 2014 at 6:57 PM, Meng QingPing mqingp...@gmail.com wrote: I run hiveserver2 with metastore in mysql. The hiveserver2 OOM and the heap dump show huge objects hold by org.datanucleus.api.jdo. JDOPersistenceManagerFactory as attached. It seems not release org.datanucleus.api.jdo.JDOPersistenceManager. Hive version is 0.13. Thanks, Jack -- Thanks, -Chandra. -- Thanks, Qingping
Is there a bug in hiveserver2 metastore caused hold huge objects
I run hiveserver2 with metastore in mysql. The hiveserver2 OOM and the heap dump show huge objects hold by org.datanucleus.api.jdo. JDOPersistenceManagerFactory as attached. It seems not release org.datanucleus.api.jdo.JDOPersistenceManager. Hive version is 0.13. Thanks, Jack
Hive 0.13 SNAPSHOT build fail
Hi, I want to inetegrate Hive with HBase 0.96 and Hadoop 2.2. I found Hive 0.13 support them. So I checkout the 0.13 snapshot just now, but get bellow error when build. All sub components built success, but fail at Hive task. Can someone help resolve it? Thanks. [mqingping@LDEV-D042 hive]$ mvn clean -e package assembly:assembly -DskipTests .. [INFO] org/apache/hadoop/hive/shims/Hadoop23Shims$2.class already added, skipping [INFO] org/apache/hadoop/hive/shims/Jetty23Shims$1.class already added, skipping [INFO] org/apache/hadoop/mapred/WebHCatJTShim23.class already added, skipping [INFO] META-INF/maven/org.apache.hive.shims/hive-shims-0.23/pom.xml already added, skipping [INFO] META-INF/maven/org.apache.hive.shims/hive-shims-0.23/pom.properties already added, skipping [INFO] [INFO] Reactor Summary: [INFO] [INFO] Hive .. FAILURE [1.170s] [INFO] Hive Ant Utilities SUCCESS [1.857s] [INFO] Hive Shims Common . SUCCESS [0.577s] [INFO] Hive Shims 0.20 ... SUCCESS [0.353s] [INFO] Hive Shims Secure Common .. SUCCESS [0.582s] [INFO] Hive Shims 0.20S .. SUCCESS [0.700s] [INFO] Hive Shims 0.23 ... SUCCESS [0.615s] [INFO] Hive Shims SUCCESS [1.709s] [INFO] Hive Common ... SUCCESS [3.335s] [INFO] Hive Serde SUCCESS [2.588s] [INFO] Hive Metastore SUCCESS [8.542s] [INFO] Hive Query Language ... SUCCESS [17.326s] [INFO] Hive Service .. SUCCESS [1.511s] [INFO] Hive JDBC . SUCCESS [0.306s] [INFO] Hive Beeline .. SUCCESS [0.283s] [INFO] Hive CLI .. SUCCESS [0.304s] [INFO] Hive Contrib .. SUCCESS [0.303s] [INFO] Hive HBase Handler SUCCESS [0.387s] [INFO] Hive HCatalog . SUCCESS [0.067s] [INFO] Hive HCatalog Core SUCCESS [3.311s] [INFO] Hive HCatalog Pig Adapter . SUCCESS [0.343s] [INFO] Hive HCatalog Server Extensions ... SUCCESS [0.272s] [INFO] Hive HCatalog Webhcat Java Client . SUCCESS [0.265s] [INFO] Hive HCatalog Webhcat . SUCCESS [3.906s] [INFO] Hive HCatalog HBase Storage Handler ... SUCCESS [1.215s] [INFO] Hive HWI .. SUCCESS [0.182s] [INFO] Hive ODBC . SUCCESS [0.043s] [INFO] Hive Shims Aggregator . SUCCESS [0.035s] [INFO] Hive TestUtils SUCCESS [0.067s] [INFO] Hive Packaging SUCCESS [0.080s] [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 53.706s [INFO] Finished at: Mon Nov 18 18:54:13 CST 2013 [INFO] Final Memory: 90M/1423M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.3:single (uberjar) on project hive-shims: Execution uberjar of goal org.apache.maven.plugins:maven-assembly-plugin:2.3:single failed. NullPointerException - [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.3:single (uberjar) on project hive-shims: Execution uberjar of goal org.apache.maven.plugins:maven-assembly-plugin:2.3:single failed. at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:224) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.MojoExecutor.executeForkedExecutions(MojoExecutor.java:364) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:198) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at
Re: Hive 0.13 SNAPSHOT build fail
Hi Jin Jie, Thanks for your reply. mvn install -DskipTests run successfully. Then mvn clean package -DskipTests -Pdist also successfully. Thanks, Jack [INFO] Reactor Summary: [INFO] [INFO] Hive .. SUCCESS [4.293s] [INFO] Hive Ant Utilities SUCCESS [1.981s] [INFO] Hive Shims Common . SUCCESS [1.328s] [INFO] Hive Shims 0.20 ... SUCCESS [1.414s] [INFO] Hive Shims Secure Common .. SUCCESS [0.866s] [INFO] Hive Shims 0.20S .. SUCCESS [0.201s] [INFO] Hive Shims 0.23 ... SUCCESS [0.774s] [INFO] Hive Shims SUCCESS [1.235s] [INFO] Hive Common ... SUCCESS [9.325s] [INFO] Hive Serde SUCCESS [0.530s] [INFO] Hive Metastore SUCCESS [3.954s] [INFO] Hive Query Language ... SUCCESS [6.819s] [INFO] Hive Service .. SUCCESS [0.209s] [INFO] Hive JDBC . SUCCESS [0.105s] [INFO] Hive Beeline .. SUCCESS [0.172s] [INFO] Hive CLI .. SUCCESS [0.325s] [INFO] Hive Contrib .. SUCCESS [0.284s] [INFO] Hive HBase Handler SUCCESS [0.531s] [INFO] Hive HCatalog . SUCCESS [0.170s] [INFO] Hive HCatalog Core SUCCESS [0.187s] [INFO] Hive HCatalog Pig Adapter . SUCCESS [0.140s] [INFO] Hive HCatalog Server Extensions ... SUCCESS [0.191s] [INFO] Hive HCatalog Webhcat Java Client . SUCCESS [0.202s] [INFO] Hive HCatalog Webhcat . SUCCESS [5.702s] [INFO] Hive HCatalog HBase Storage Handler ... SUCCESS [0.237s] [INFO] Hive HWI .. SUCCESS [0.159s] [INFO] Hive ODBC . SUCCESS [0.073s] [INFO] Hive Shims Aggregator . SUCCESS [0.047s] [INFO] Hive TestUtils SUCCESS [0.097s] [INFO] Hive Packaging SUCCESS [0.123s] [INFO] [INFO] BUILD SUCCESS 2013/11/18 Jie Jin hellojin...@gmail.com try mvn install -DskipTests first Best Regards Jin Jie Sent from my mobile device. On Nov 18, 2013 8:08 PM, Meng QingPing mqingp...@gmail.com wrote: Hi, I want to inetegrate Hive with HBase 0.96 and Hadoop 2.2. I found Hive 0.13 support them. So I checkout the 0.13 snapshot just now, but get bellow error when build. All sub components built success, but fail at Hive task. Can someone help resolve it? Thanks. [mqingping@LDEV-D042 hive]$ mvn clean -e package assembly:assembly -DskipTests .. [INFO] org/apache/hadoop/hive/shims/Hadoop23Shims$2.class already added, skipping [INFO] org/apache/hadoop/hive/shims/Jetty23Shims$1.class already added, skipping [INFO] org/apache/hadoop/mapred/WebHCatJTShim23.class already added, skipping [INFO] META-INF/maven/org.apache.hive.shims/hive-shims-0.23/pom.xml already added, skipping [INFO] META-INF/maven/org.apache.hive.shims/hive-shims-0.23/pom.properties already added, skipping [INFO] [INFO] Reactor Summary: [INFO] [INFO] Hive .. FAILURE [1.170s] [INFO] Hive Ant Utilities SUCCESS [1.857s] [INFO] Hive Shims Common . SUCCESS [0.577s] [INFO] Hive Shims 0.20 ... SUCCESS [0.353s] [INFO] Hive Shims Secure Common .. SUCCESS [0.582s] [INFO] Hive Shims 0.20S .. SUCCESS [0.700s] [INFO] Hive Shims 0.23 ... SUCCESS [0.615s] [INFO] Hive Shims SUCCESS [1.709s] [INFO] Hive Common ... SUCCESS [3.335s] [INFO] Hive Serde SUCCESS [2.588s] [INFO] Hive Metastore SUCCESS [8.542s] [INFO] Hive Query Language ... SUCCESS [17.326s] [INFO] Hive Service .. SUCCESS [1.511s] [INFO] Hive JDBC . SUCCESS [0.306s] [INFO] Hive Beeline .. SUCCESS [0.283s] [INFO
