Hi,
I think this could be due to this error:
"org.apache.hive.hcatalog.streaming.TransactionError:
Unable
to acquire lock on {metaStoreUri='thrift://hdp1.local:9083',
database='default', table='test_table', partitionVals=[] }"
Try a telnet on "
thrift://hdp1.local:9083"
and try to check if there is not another process working on table
Cordialement,
TUENO FOTSO Steve Jeffrey
Ingénieur de conception
Génie Informatique
+237 676 57 17 28 <6%2076%2057%2017%2028>
+237 697 86 36 38 <6%2097%2086%2036%2038>
+33 6 23 71 91 52 <+33%206%2023%2071%2091%2052>
https://jobs.jumia.cm/fr/candidats/CVTF1486563.html
____________________________________________________________
______________________
https://play.google.com/store/apps/details?id=com.polytech.remotecomputer
https://play.google.com/store/apps/details?id=com.polytech.
internetaccesschecker
*http://www.traveler.cm/ <http://remotecomputer.traveler.cm/>*
http://remotecomputer.traveler.cm/
https://play.google.com/store/apps/details?id=com.polytech.androidsmssender
https://github.com/stuenofotso/notre-jargon
https://play.google.com/store/apps/details?id=com.polytech.welovecameroon
https://play.google.com/store/apps/details?id=com.polytech.welovefrance
2017-04-03 9:21 GMT+02:00 Marcin Kasiński <[email protected]>:
> Hello.
>
> I have problem with storm hive integration.
>
> I'm using HDP 2.5 and HDF 2.1 (storm 1.0.2 and Hive 1.2.1000)
>
> I'm using sample class
> https://github.com/hortonworks/storm-release/blob/HDP-2.5.0.
> 0-tag/external/storm-hive/src/test/java/org/apache/storm/
> hive/bolt/HiveTopology.java
>
>
> I've created sample table:
>
> create table test_table ( id INT, name STRING, phone STRING, street
> STRING,city STRING, state STRING)
> CLUSTERED BY (name) into 5 buckets
> stored as orc tblproperties ("orc.compress"="NONE");
>
>
> Hive works .
>
> I can insert to table manualy and get records:
>
> INSERT INTO TABLE test_table VALUES ( 1,"MyName",
> "111111","street","city","state");
>
> select * from test_table ;
>
> Problem is that storm sample HiveTopology doesn't work.
>
> When I deploy storm application hive bolt fails:
>
> On metastore I get:
>
> 2017-04-03 08:42:39,682 ERROR [pool-5-thread-188]:
> metastore.RetryingHMSHandler
> (RetryingHMSHandler.java:invokeInternal(195)) -
> java.lang.IllegalStateException: Unexpected DataOperationType: UNSET
> agentInfo=Unknown txnid:74056
> at org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockW
> ithRetry(TxnHandler.java:938)
> at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHand
> ler.java:814)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.
> lock(HiveMetaStore.java:5790)
> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
> thodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeIn
> ternal(RetryingHMSHandler.java:139)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(
> RetryingHMSHandler.java:97)
> at com.sun.proxy.$Proxy12.lock(Unknown Source)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$
> Processor$lock.getResult(ThriftHiveMetastore.java:11860)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$
> Processor$lock.getResult(ThriftHiveMetastore.java:11844)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(
> TUGIBasedProcessor.java:110)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(
> TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
> upInformation.java:1724)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(
> TUGIBasedProcessor.java:118)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.
> run(TThreadPoolServer.java:286)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
> On storm I get:
> org.apache.storm.hive.common.HiveWriter$ConnectFailure: Failed
> connecting to EndPoint {metaStoreUri='thrift://hdp1.local:9083',
> database='default', table='test_table', partitionVals=[] }
> at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:80)
> ~[stormjar.jar:?]
> at org.apache.storm.hive.common.HiveUtils.makeHiveWriter(HiveUt
> ils.java:50)
> ~[stormjar.jar:?]
> at org.apache.storm.hive.bolt.HiveBolt.getOrCreateWriter(HiveBo
> lt.java:259)
> ~[stormjar.jar:?]
> at org.apache.storm.hive.bolt.HiveBolt.execute(HiveBolt.java:112)
> [stormjar.jar:?]
> at org.apache.storm.daemon.executor$fn__6952$tuple_action_fn__
> 6954.invoke(executor.clj:728)
> [storm-core-1.0.2.2.1.2.0-10.jar:1.0.2.2.1.2.0-10]
> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6873.
> invoke(executor.clj:460)
> [storm-core-1.0.2.2.1.2.0-10.jar:1.0.2.2.1.2.0-10]
> at org.apache.storm.disruptor$clojure_handler$reify__6388.onEve
> nt(disruptor.clj:40)
> [storm-core-1.0.2.2.1.2.0-10.jar:1.0.2.2.1.2.0-10]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(D
> isruptorQueue.java:453)
> [storm-core-1.0.2.2.1.2.0-10.jar:1.0.2.2.1.2.0-10]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvaila
> ble(DisruptorQueue.java:432)
> [storm-core-1.0.2.2.1.2.0-10.jar:1.0.2.2.1.2.0-10]
> at org.apache.storm.disruptor$consume_batch_when_available.invo
> ke(disruptor.clj:73)
> [storm-core-1.0.2.2.1.2.0-10.jar:1.0.2.2.1.2.0-10]
> at org.apache.storm.daemon.executor$fn__6952$fn__6965$fn__7018.
> invoke(executor.clj:847)
> [storm-core-1.0.2.2.1.2.0-10.jar:1.0.2.2.1.2.0-10]
> at org.apache.storm.util$async_loop$fn__555.invoke(util.clj:484)
> [storm-core-1.0.2.2.1.2.0-10.jar:1.0.2.2.1.2.0-10]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: org.apache.storm.hive.common.HiveWriter$TxnBatchFailure:
> Failed acquiring Transaction Batch from EndPoint:
> {metaStoreUri='thrift://hdp1.local:9083', database='default',
> table='test_table', partitionVals=[] }
> at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWri
> ter.java:264)
> ~[stormjar.jar:?]
> at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:72)
> ~[stormjar.jar:?]
> ... 13 more
> Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable
> to acquire lock on {metaStoreUri='thrift://hdp1.local:9083',
> database='default', table='test_table', partitionVals=[] }
> at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionB
> atchImpl.beginNextTransactionImpl(HiveEndPoint.java:575)
> ~[stormjar.jar:?]
> at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionB
> atchImpl.beginNextTransaction(HiveEndPoint.java:544)
> ~[stormjar.jar:?]
> at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWri
> ter.java:259)
> ~[stormjar.jar:?]
> at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:72)
> ~[stormjar.jar:?]
> ... 13 more
>
>
>
> I dont know if it is important but during storm deploy I get
> "/usr/bin/storm: line 2: /usr/hdf/2.1.2.0-10/etc/default/hadoop: No
> such file or directory"
>
>
> root@hdf:~# storm jar /root/StormSampleHive-0.0.1-SNAPSHOT.jar
> mk.HiveTopology MKjobarg1XXX
> /usr/bin/storm: line 2: /usr/hdf/2.1.2.0-10/etc/default/hadoop: No
> such file or directory
> Running: /usr/jdk64/jdk1.8.0_77/bin/java -server -Ddaemon.name=
> -Dstorm.options= -Dstorm.home=/usr/hdf/2.1.2.0-10/storm
> -Dstorm.log.dir=/var/log/storm
> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
> -Dstorm.conf.file= -cp /usr/hdf/2.1.2.0-10/storm/lib/objenesis-2.1.j
> ....
>
> It is because I have HDF without hadoop on one cluster and hadoop on
> second cluster.
>
>
> My pom file:
>
>
> <project xmlns="http://maven.apache.org/POM/4.0.0";
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd";>
> <modelVersion>4.0.0</modelVersion>
> <groupId>StormSampleHive</groupId>
> <artifactId>StormSampleHive</artifactId>
> <version>0.0.1-SNAPSHOT</version>
>
> <properties>
> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
> <maven.compiler.source>1.7</maven.compiler.source>
> <maven.compiler.target>1.7</maven.compiler.target>
> <storm.version>1.0.1</storm.version>
> <flux.version>0.3.0</flux.version>
> <kafka_2.10.version>0.8.2.2.3.0.0-2557</kafka_2.10.version>
> <avro.version>1.7.7</avro.version>
> <junit.version>4.11</junit.version>
> </properties>
> <build>
> <sourceDirectory>src</sourceDirectory>
> <plugins>
>
>
> <plugin>
> <artifactId>maven-compiler-plugin</artifactId>
> <version>3.3</version>
> <configuration>
> <source>1.8</source>
> <target>1.8</target>
> </configuration>
> </plugin>
>
>
>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> </dependency>
>
>
> <!-- https://mvnrepository.com/artifact/org.apache.storm/storm-core -->
> <dependency>
> <groupId>org.apache.storm</groupId>
> <artifactId>storm-core</artifactId>
> <version>1.0.2</version>
> <exclusions>
> <exclusion>
> <artifactId>log4j-over-slf4j</artifactId>
> <groupId>org.slf4j</groupId>
> </exclusion>
> </exclusions>
> </dependency>
>
> </dependencies>
>
> <repositories>
> <repository>
> <id>hortonworks</id>
> <url>http://nexus-private.hortonworks.com/nexus/content/grou
> ps/public/</url>
> </repository>
> </repositories>
>
>
>
> pozdrawiam
> Marcin Kasiński
> http://itzone.pl
>
