subject:"Re\: Storm hive bolt"

Re: Storm hive bolt

2017-03-31 Thread Marcin Kasiński

You can dowload my project from  link http://itzone.pl/tmp234der/StormSample.zip

This is simple topology  with Kafka spout (it works ), hbase bolt (it
works), hive bolt (doesnt work)

I've created hive table:

CREATE TABLE stock_prices(
  day DATE,
  open FLOAT,
  high FLOAT,
  low FLOAT,
  close FLOAT,
  volume INT,
  adj_close FLOAT
)
PARTITIONED BY (name STRING)
CLUSTERED BY (day) into 5 buckets
STORED AS ORC
TBLPROPERTIES ('transactional'='true');


I've created hbase table


create 'stock_prices', 'cf'

I've created kafka tpoic:

/usr/hdf/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper
hdf1.local:2181,hdf2.local:2181,hdf3.local:2181 --replication-factor 3
--partition 3 --topic my-topic


I've deployed app to storm.

storm jar /root/StormSample-0.0.1-SNAPSHOT.jar
mk.stormkafka.KafkaSpoutTestTopology  MKjobarg1XXX


When i Deploy and publish to kafka topic sample message I can not save
data to hive table.

I'm 100% sure I have hive configured OK, because I can add something
to this table manually outside storm.


2017-03-30,11,12,13,14,15,16,Marcin2345



Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable
to acquire lock on {metaStoreUri='thrift://hdp1.local:9083',
database='default', table='stock_prices', partitionVals=[Marcin] }
at 
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:575)
~[stormjar.jar:?]
at 
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:544)
~[stormjar.jar:?]
at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:259)
~[stormjar.jar:?]
at org.apache.storm.hive.common.HiveWriter.(HiveWriter.java:72)
~[stormjar.jar:?]
... 13 more


I think it could be pom dependencies problem.

I have no idea how I can fix if.

Can you help me ?
pozdrawiam
Marcin Kasiński
http://itzone.pl


On 31 March 2017 at 12:35, Igor Kuzmenko  wrote:
> Check this example:
> https://github.com/hortonworks/storm-release/blob/HDP-2.5.0.0-tag/external/storm-hive/src/test/java/org/apache/storm/hive/bolt/HiveTopology.java
>
> If you can, please post your topology code. It's strange that you are using
> org.apache.hadoop.hive package directly.
>
>
> On Fri, Mar 31, 2017 at 1:08 PM, Marcin Kasiński 
> wrote:
>>
>> After changin I have lots of errors in eclipse
>> "DescriptionResourcePathLocationType
>> The import org.apache.hadoop.hive cannot be resolved
>> TestHiveBolt.java/StormSample/src/mk/storm/hiveline 26Java
>> Problem
>> "
>>
>> Do you have hello world storm hive project (HDP 1.5 and HDF 2.1) ?
>>
>> Can you send it to me an I will try it ?
>>
>>
>> pozdrawiam
>> Marcin Kasiński
>> http://itzone.pl
>>
>>
>> On 31 March 2017 at 11:30, Igor Kuzmenko  wrote:
>> > I'm using hive streaming bolt with HDP 2.5.0.0.
>> > Try this:
>> >
>> > 
>> > 
>> > hortonworks
>> >
>> >
>> > http://nexus-private.hortonworks.com/nexus/content/groups/public/
>> > 
>> > 
>> >
>> > 
>> > org.apache.storm
>> > storm-hive
>> > 1.0.1.2.5.0.0-1245
>> > 
>> >
>> >
>> > On Fri, Mar 31, 2017 at 11:32 AM, Marcin Kasiński
>> >  wrote:
>> >>
>> >> Hi Eugene.
>> >>
>> >> Below yo have my pom file.
>> >>
>> >> Can you check it and fix it to use repositories in proper way, please ?
>> >>
>> >> I'm working with my problem over 2 weeks and I'm loosing hope.
>> >>
>> >>
>> >> http://maven.apache.org/POM/4.0.0;
>> >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>> >> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
>> >> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
>> >> 4.0.0
>> >> StormSample
>> >> StormSample
>> >> 0.0.1-SNAPSHOT
>> >>
>> >>  
>> >>
>> >> UTF-8
>> >> 1.7
>> >> 1.7
>> >> 1.0.1
>> >> 0.3.0
>> >> 0.8.2.2.3.0.0-2557
>> >> 1.7.7
>> >> 4.11
>> >> 
>> >> 
>> >> src
>> >> 
>> >>
>> >> 
>> >> 
>> >> maven-compiler-plugin
>> >> 3.3
>> >> 
>> >> 1.8
>> >> 1.8
>> >> 
>> >> 
>> >>
>> >> 
>> >> org.apache.maven.plugins
>> >> maven-jar-plugin
>> >> 
>> >> 
>> >> 
>> >> true
>> >> lib/
>> >> mk.StormSample
>> >> 
>> >> 
>> >> 
>> >> 
>> >> 
>> >>  
>> >> org.apache.maven.plugins
>> >> maven-shade-plugin
>> >> 1.4
>> >> 
>> >> true
>> >> 
>> >> 
>> >> 
>> >> package
>> >> 
>> >> shade
>> >> 
>> >> 
>> >>

Re: Storm hive bolt

2017-03-31 Thread Marcin Kasiński

Hi Eugene.

Below yo have my pom file.

Can you check it and fix it to use repositories in proper way, please ?

I'm working with my problem over 2 weeks and I'm loosing hope.


http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
4.0.0
StormSample
StormSample
0.0.1-SNAPSHOT

 
UTF-8
1.7
1.7
1.0.1
0.3.0
0.8.2.2.3.0.0-2557
1.7.7
4.11


src




maven-compiler-plugin
3.3

1.8
1.8




org.apache.maven.plugins
maven-jar-plugin



true
lib/
mk.StormSample





 
org.apache.maven.plugins
maven-shade-plugin
1.4

true



package

shade


 

*:*

META-INF/*.SF
META-INF/*.DSA
META-INF/*.RSA

  defaults.yaml



















 
org.apache.storm
storm-hive
1.0.2



jline
jline







org.apache.storm
storm-hbase
1.0.1





   org.apache.storm
storm-core
1.0.3


 log4j-over-slf4j
org.slf4j








org.apache.kafka
kafka_2.10
0.10.0.0
   

org.apache.zookeeper
zookeeper


org.slf4j
slf4j-log4j12


log4j
log4j




org.slf4j
log4j-over-slf4j
1.7.21





org.apache.storm
storm-kafka
1.0.1
 



 





org.apache.storm
storm-jdbc
1.0.3




org.apache.hadoop
hadoop-hdfs
2.6.0


ch.qos.logback
logback-classic


javax.servlet
servlet-api







com.googlecode.json-simple
json-simple
1.1





log4j
log4j
1.2.17











hortonworks
http://repo.hortonworks.com/content/groups/public/org/apache/storm/storm-hive/1.0.1.2.0.1.0-12/





pozdrawiam
Marcin Kasiński
http://itzone.pl


On 30 March 2017 at 17:51, Eugene Koifman  wrote:
> It maybe because you are mixing artifacts from HDP/F and Apache when 
> compiling the topology.
> Can you try using 
> http://repo.hortonworks.com/content/groups/public/org/apache/storm/storm-hive/1.0.1.2.0.1.0-12/
> Rather than
> 
> org.apache.storm
> storm-hive
> 1.0.3
> 
>
> Eugene
>
> On 3/29/17, 9:47 AM, "Marcin Kasiński"  wrote:
>
> I've upgraded my environment.
>
> I have HIve on HDP 2.5 (environment 1) and storm on HDF 2.1
>
> (environment 2)
>
> I have the same eroor:
>
> On storm (HDF 2.1):
>
> Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable
> to acquire lock on {metaStoreUri='thrift://hdp1.local:9083',
> database='default', table='stock_prices', partitionVals=[Marcin] } at
> 
> org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:575)
> ~[stormjar.jar:?]
>
> On hive metastore (HDP 2.5):
>
> 2017-03-29 11:56:29,926 ERROR [pool-5-thread-17]:
> server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error
> occurred during processing of message.
> java.lang.IllegalStateException: Unexpected DataOperationType: UNSET
> agentInfo=Unknown txnid:54 at
> 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:938)
> at 
> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:814)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaSt
> pozdrawiam
> Marcin Kasiński
> http://itzone.pl
>
>
> On 27 March 2017 at 22:01, Marcin Kasiński  
> wrote:
> > Hello.
> >
> > Thank you for reply.
> >
> > I do really want to solve it.
> >
> > I'm sure i compiled sources again with new jars.
> >
> > I've changed source from

Re: Storm hive bolt

2017-03-30 Thread Eugene Koifman

It maybe because you are mixing artifacts from HDP/F and Apache when compiling 
the topology.
Can you try using 
http://repo.hortonworks.com/content/groups/public/org/apache/storm/storm-hive/1.0.1.2.0.1.0-12/
Rather than

 org.apache.storm
storm-hive
1.0.3


Eugene

On 3/29/17, 9:47 AM, "Marcin Kasiński"  wrote:

I've upgraded my environment.

I have HIve on HDP 2.5 (environment 1) and storm on HDF 2.1

(environment 2)

I have the same eroor:

On storm (HDF 2.1):

Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable
to acquire lock on {metaStoreUri='thrift://hdp1.local:9083',
database='default', table='stock_prices', partitionVals=[Marcin] } at

org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:575)
~[stormjar.jar:?]

On hive metastore (HDP 2.5):

2017-03-29 11:56:29,926 ERROR [pool-5-thread-17]:
server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error
occurred during processing of message.
java.lang.IllegalStateException: Unexpected DataOperationType: UNSET
agentInfo=Unknown txnid:54 at

org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:938)
at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:814)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaSt
pozdrawiam
Marcin Kasiński
http://itzone.pl


On 27 March 2017 at 22:01, Marcin Kasiński  
wrote:
> Hello.
>
> Thank you for reply.
>
> I do really want to solve it.
>
> I'm sure i compiled sources again with new jars.
>
> I've changed source from storm 0.10 ( package backtype.storm.* ) to
> storm 1.0.1 (package org.apache.storm.*) and I've generated jar again
>
> Below you have entire storm worker logs and pom.xml.
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> StormSample
> StormSample
> 0.0.1-SNAPSHOT
>
>  
> UTF-8
> 1.7
> 1.7
> 1.0.1
> 0.3.0
> 0.8.2.2.3.0.0-2557
> 1.7.7
> 4.11
> 
> 
> src
> 
>
> 
> 
> maven-compiler-plugin
> 3.3
> 
> 1.8
> 1.8
> 
> 
>
> 
> org.apache.maven.plugins
> maven-jar-plugin
> 
> 
> 
> true
> lib/
> mk.StormSample
> 
> 
> 
> 
> 
>  
> org.apache.maven.plugins
> maven-shade-plugin
> 1.4
> 
> true
> 
> 
> 
> package
> 
> shade
> 
> 
>  
> 
> *:*
> 
> META-INF/*.SF
> META-INF/*.DSA
> META-INF/*.RSA
> 
>   defaults.yaml
> 
> 
> 
> 
> 
> 
> 
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
> 
> 
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> 
>
>  
> org.apache.storm
> storm-hive
> 1.0.3
> 
> 
> 
> jline
> jline
> 
>
>
> 
> 
> 
> 
> org.apache.storm
> storm-hbase
> 1.0.1
> 
>
> 
> 
> 
>org.apache.storm
> storm-core
> 1.0.1
> 
> 
>  log4j-over-slf4j
> org.slf4j
> 
> 
> 
>
> 
>
>
> 
> org.apache.kafka
> kafka_2.10
> 0.10.0.0
>
> 
> org.apache.zookeeper
>

Re: Storm hive bolt

2017-03-29 Thread Marcin Kasiński

I've upgraded my environment.

I have HIve on HDP 2.5 (environment 1) and storm on HDF 2.1

(environment 2)

I have the same eroor:

On storm (HDF 2.1):

Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable
to acquire lock on {metaStoreUri='thrift://hdp1.local:9083',
database='default', table='stock_prices', partitionVals=[Marcin] } at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:575)
~[stormjar.jar:?]

On hive metastore (HDP 2.5):

2017-03-29 11:56:29,926 ERROR [pool-5-thread-17]:
server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error
occurred during processing of message.
java.lang.IllegalStateException: Unexpected DataOperationType: UNSET
agentInfo=Unknown txnid:54 at
org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:938)
at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:814)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaSt
pozdrawiam
Marcin Kasiński
http://itzone.pl


On 27 March 2017 at 22:01, Marcin Kasiński  wrote:
> Hello.
>
> Thank you for reply.
>
> I do really want to solve it.
>
> I'm sure i compiled sources again with new jars.
>
> I've changed source from storm 0.10 ( package backtype.storm.* ) to
> storm 1.0.1 (package org.apache.storm.*) and I've generated jar again
>
> Below you have entire storm worker logs and pom.xml.
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> StormSample
> StormSample
> 0.0.1-SNAPSHOT
>
>  
> UTF-8
> 1.7
> 1.7
> 1.0.1
> 0.3.0
> 0.8.2.2.3.0.0-2557
> 1.7.7
> 4.11
> 
> 
> src
> 
>
> 
> 
> maven-compiler-plugin
> 3.3
> 
> 1.8
> 1.8
> 
> 
>
> 
> org.apache.maven.plugins
> maven-jar-plugin
> 
> 
> 
> true
> lib/
> mk.StormSample
> 
> 
> 
> 
> 
>  
> org.apache.maven.plugins
> maven-shade-plugin
> 1.4
> 
> true
> 
> 
> 
> package
> 
> shade
> 
> 
>  
> 
> *:*
> 
> META-INF/*.SF
> META-INF/*.DSA
> META-INF/*.RSA
> 
>   defaults.yaml
> 
> 
> 
> 
> 
> 
> implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
> 
> implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> 
>
>  
> org.apache.storm
> storm-hive
> 1.0.3
> 
> 
> 
> jline
> jline
> 
>
>
> 
> 
> 
> 
> org.apache.storm
> storm-hbase
> 1.0.1
> 
>
> 
> 
> 
>org.apache.storm
> storm-core
> 1.0.1
> 
> 
>  log4j-over-slf4j
> org.slf4j
> 
> 
> 
>
> 
>
>
> 
> org.apache.kafka
> kafka_2.10
> 0.10.0.0
>
> 
> org.apache.zookeeper
> zookeeper
> 
> 
> org.slf4j
> slf4j-log4j12
> 
> 
> log4j
> log4j
> 
> 
> 
> 
> org.slf4j
> log4j-over-slf4j
> 1.7.21
> 
>
>
> 
> 
> org.apache.storm
> storm-kafka
> 1.0.1
>  
> 
>
> 
>  
>
> 
>
>
> 
> org.apache.hadoop
> hadoop-hdfs
> 2.6.0
> 
> 
> ch.qos.logback
> logback-classic
> 
> 
> javax.servlet
> servlet-api
> 
> 
> 
>
>
> 
> 
> com.googlecode.json-simple
> json-simple
> 1.1
> 
>
>
> 
> 
> log4j
> log4j
> 1.2.17
> 
>
>
>
> 
> 
> 
>
>
>
>
> 
> 
> clojars.org
> http://clojars.org/repo
> 
> 
> 
>
>
> logs:
>
> 2017-03-27 21:50:36.572 STDERR [INFO] JMXetricAgent instrumented JVM,
> see https://github.com/ganglia/jmxetric
> 2017-03-27 21:50:39.302 STDERR [INFO] Mar 27, 2017 9:50:39 PM
> info.ganglia.gmetric4j.GMonitor start
> 2017-03-27 21:50:39.303

Re: Storm hive bolt

2017-03-27 Thread Marcin Kasiński

Hello.

Thank you for reply.

I do really want to solve it.

I'm sure i compiled sources again with new jars.

I've changed source from storm 0.10 ( package backtype.storm.* ) to
storm 1.0.1 (package org.apache.storm.*) and I've generated jar again

Below you have entire storm worker logs and pom.xml.

http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
4.0.0
StormSample
StormSample
0.0.1-SNAPSHOT

 
UTF-8
1.7
1.7
1.0.1
0.3.0
0.8.2.2.3.0.0-2557
1.7.7
4.11


src




maven-compiler-plugin
3.3

1.8
1.8




org.apache.maven.plugins
maven-jar-plugin



true
lib/
mk.StormSample





 
org.apache.maven.plugins
maven-shade-plugin
1.4

true



package

shade


 

*:*

META-INF/*.SF
META-INF/*.DSA
META-INF/*.RSA

  defaults.yaml



















 
org.apache.storm
storm-hive
1.0.3



jline
jline







org.apache.storm
storm-hbase
1.0.1





   org.apache.storm
storm-core
1.0.1


 log4j-over-slf4j
org.slf4j








org.apache.kafka
kafka_2.10
0.10.0.0
   

org.apache.zookeeper
zookeeper


org.slf4j
slf4j-log4j12


log4j
log4j




org.slf4j
log4j-over-slf4j
1.7.21





org.apache.storm
storm-kafka
1.0.1
 



 





org.apache.hadoop
hadoop-hdfs
2.6.0


ch.qos.logback
logback-classic


javax.servlet
servlet-api







com.googlecode.json-simple
json-simple
1.1





log4j
log4j
1.2.17













clojars.org
http://clojars.org/repo





logs:

2017-03-27 21:50:36.572 STDERR [INFO] JMXetricAgent instrumented JVM,
see https://github.com/ganglia/jmxetric
2017-03-27 21:50:39.302 STDERR [INFO] Mar 27, 2017 9:50:39 PM
info.ganglia.gmetric4j.GMonitor start
2017-03-27 21:50:39.303 STDERR [INFO] INFO: Setting up 1 samplers
2017-03-27 21:50:40.870 STDERR [INFO] SLF4J: Class path contains
multiple SLF4J bindings.
2017-03-27 21:50:40.871 STDERR [INFO] SLF4J: Found binding in
[jar:file:/usr/hdp/2.5.0.0-1245/storm/lib/log4j-slf4j-impl-2.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2017-03-27 21:50:40.872 STDERR [INFO] SLF4J: Found binding in
[jar:file:/hadoop/storm/supervisor/stormdist/kafkatest-3-1490644225/stormjar.jar!/org/slf4j/impl/StaticLoggerBinder.class]
2017-03-27 21:50:40.872 STDERR [INFO] SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2017-03-27 21:50:40.880 STDERR [INFO] SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
2017-03-27 21:50:43.131 o.a.s.d.worker [INFO] Launching worker for
kafkatest-3-1490644225 on db6a91d8-c15a-4b11-84c7-7e5461e02778:6700
with id ae719623-6064-44c0-98d3-ed1614f23bc3 and conf
{"topology.builtin.metrics.bucket.size.secs" 60, "nimbus.childopts"
"-Xmx1024m  
-javaagent:/usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8649,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Nimbus_JVM",
"ui.filter.params" nil, "storm.cluster.mode" "distributed",
"topology.metrics.metric.name.separator" ".",
"storm.messaging.netty.client_worker_threads" 1,
"client.jartransformer.class"
"org.apache.storm.hack.StormShadeTransformer",
"logviewer.max.per.worker.logs.size.mb" 2048,
"supervisor.run.worker.as.user" false, "topology.max.task.parallelism"
nil, "topology.priority" 29, "zmq.threads" 1,
"storm.group.mapping.service"
"org.apache.storm.security.auth.ShellBasedGroupsMapping",
"metrics.reporter.register"

Re: Storm hive bolt

2017-03-27 Thread Eugene Koifman

https://community.hortonworks.com/questions/59681/puthivestreaming-nifi-processor-various-errors.html
 has 
2016-10-03 23:40:24,322 ERROR [pool-5-thread-114]: metastore.RetryingHMSHandler 
(RetryingHMSHandler.java:invokeInternal(195)) - 
java.lang.IllegalStateException: Unexpected DataOperationType: UNSET 
agentInfo=Unknown txnid:98201

I don’t see this in the stack trace below but if you are seeing this, I think  
you need to recompile Storm bolt since it is an uber jar that includes some 
Hive classes.
Based on the error above it is using old classes (from before HDP 2.5).

Eugene



On 3/26/17, 1:20 PM, "Marcin Kasiński"  wrote:

Hello.

I have problem with Storm hive bolt.
When I try to save data to hiev I get error Unable to acquire lock
(storm logs below)


I have very simple application (save simple data to hive ).

It works with HDP 2.4 (Apache Hive 1.2.1 and Apache Storm 0.10.0)

I switched to  HDP 2.5 (Apache Hive 1.2.1 and Apache Storm 1.0.1)

It stopped working.

I saw simillar error here:


https://community.hortonworks.com/questions/59681/puthivestreaming-nifi-processor-various-errors.html

Ther are saying that there is a issue for Hive Streaming between HDF
2.0 and HDP 2.5

I Ilike HDP 2.5 .

My question is:

Do you know how I can solve this problem ?
... or the only way is to switch back to HDP 2.4 ?

Storm logs below:


org.apache.storm.hive.common.HiveWriter$ConnectFailure: Failed
connecting to EndPoint {metaStoreUri='thrift://ambari.local:9083',
database='default', table='stock_prices', partitionVals=[Marcin] }
at org.apache.storm.hive.common.HiveWriter.(HiveWriter.java:80)
~[stormjar.jar:?]
at org.apache.storm.hive.common.HiveUtils.makeHiveWriter(HiveUtils.java:50)
~[stormjar.jar:?]
at org.apache.storm.hive.bolt.HiveBolt.getOrCreateWriter(HiveBolt.java:259)
~[stormjar.jar:?]
at org.apache.storm.hive.bolt.HiveBolt.execute(HiveBolt.java:112)
[stormjar.jar:?]
at 
org.apache.storm.daemon.executor$fn__9362$tuple_action_fn__9364.invoke(executor.clj:734)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at 
org.apache.storm.daemon.executor$mk_task_receiver$fn__9283.invoke(executor.clj:466)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at 
org.apache.storm.disruptor$clojure_handler$reify__8796.onEvent(disruptor.clj:40)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at 
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at 
org.apache.storm.daemon.executor$fn__9362$fn__9375$fn__9428.invoke(executor.clj:853)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at org.apache.storm.util$async_loop$fn__656.invoke(util.clj:484)
[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: org.apache.storm.hive.common.HiveWriter$TxnBatchFailure:
Failed acquiring Transaction Batch from EndPoint:
{metaStoreUri='thrift://ambari.local:9083', database='default',
table='stock_prices', partitionVals=[Marcin] }
at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:264)
~[stormjar.jar:?]
at org.apache.storm.hive.common.HiveWriter.(HiveWriter.java:72)
~[stormjar.jar:?]
... 13 more
Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable
to acquire lock on {metaStoreUri='thrift://ambari.local:9083',
database='default', table='stock_prices', partitionVals=[Marcin] }
at 
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:575)
~[stormjar.jar:?]
at 
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:544)
~[stormjar.jar:?]
at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:259)
~[stormjar.jar:?]
at org.apache.storm.hive.common.HiveWriter.(HiveWriter.java:72)
~[stormjar.jar:?]
... 13 more
Caused by: org.apache.thrift.transport.TTransportException
at 
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
~[stormjar.jar:?]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
~[stormjar.jar:?]
at 
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
~[stormjar.jar:?]
at

Re: Storm hive bolt

Re: Storm hive bolt

Re: Storm hive bolt

Re: Storm hive bolt

Re: Storm hive bolt

Re: Storm hive bolt

6 matches

Site Navigation

Mail list logo

Footer information