subject:"Re\: Storm\-Hive \: lock acquiring problem"

Re: Storm-Hive : lock acquiring problem

2018-06-12 Thread Milind Vaidya

This worked for us. Thanks.

On Tue, Jun 12, 2018 at 3:47 AM, Abhishek Raj 
wrote:

> Circling back to this, I was able to figure out what the problem was.
> storm-hive
> 
>  dependencies
> are compiled with hive version
> 
>   0.14.0 which is very old. We are using hive 2.3.2 so obviously there
> were a lot of differences between the two versions. The fix was to override
> *storm-hive's* hive dependencies in our pom with a newer version. To be
> more verbose, we copied the following into our pom.
>
> 
> org.apache.hive.hcatalog
> hive-hcatalog-streaming
> 2.3.2
> 
> 
> org.slf4j
> slf4j-log4j12
> 
> 
> org.apache.calcite
> calcite-core
> 
> 
> org.apache.calcite
> calcite-avatica
> 
> 
> 
> 
> org.apache.hive.hcatalog
> hive-hcatalog-core
> 2.3.2
> 
> 
> org.slf4j
> slf4j-log4j12
> 
> 
> org.apache.calcite
> calcite-avatica
> 
> 
> org.apache.calcite
> calcite-core
> 
> 
> 
> 
> org.apache.hive
> hive-cli
> 2.3.2
> 
> 
> org.slf4j
> slf4j-log4j12
> 
> 
> org.apache.calcite
> calcite-core
> 
> 
> org.apache.calcite
> calcite-avatica
> 
> 
> 
>
>
> To go a little more deeper about the problem, this is what changed between
> the two hive versions giving us the "Unexpected DataOperationType:
> UNSET". createLockRequest
> 
> in 2.3.2 explicitly passes a operation type "INSERT"
> 
>  while
> acquiring a lock, but in 0.14.0
> 
>  no operation type
> 
>  is
> being passed. So the operation type ends up defaulting to UNSET and
> throwing an error. This is the commit
> 
> where the change occurred.
>
> In light of the fact that there are several different threads with users
> facing the same issue, imho, *storm-hive's* hive dependencies should be
> updated with newer hive releases and there should be a way for users to
> explicitly specify which hive release they want to use storm-hive with. The
> documentation for storm-hive should also be updated to reflect this
> requirement.
>
> Happy to provide prs if that sounds like a good idea.
>
> Thanks.
>
> On Fri, Jun 8, 2018 at 3:21 PM, Abhishek Raj 
> wrote:
>
>> Hi. We faced a similar problem earlier when trying HiveBolt in storm with
>> hive on emr. We were seeing
>>
>> java.lang.IllegalStateException: Unexpected DataOperationType: UNSET
>>> agentInfo=Unknown txnid:130551
>>
>>
>> in hive logs. Any help here would be appreciated.
>>
>>
>> On Fri, Jun 8, 2018 at 10:26 AM, Milind Vaidya  wrote:
>>
>>> Here are some details from the meta store logs:
>>>
>>> 018-06-08T03:34:20,634 ERROR [pool-13-thread-197([])]:
>>> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(204))
>>> - java.lang.IllegalStateException: Unexpected DataOperationType: UNSET
>>> agentInfo=Unknown txnid:130551
>>> at org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockW
>>> ithRetry(TxnHandler.java:1000)
>>> at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHand
>>> ler.java:872)
>>> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lo
>>> ck(HiveMetaStore.java:6366)
>>> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeIn
>>> ternal(RetryingHMSHandler.java:148)
>>> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(R
>>> etryingHMSHandler.java:107)
>>> at com.sun.proxy.$Proxy32.lock(Unknown Source)
>>> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Pro
>>> cessor$lock.getResult(ThriftHiveMetastore.java:14155)
>>> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Pro
>>> cessor$lock.getResult(ThriftHiveMetastore.java:14139)
>>> at org.apache.thrift.ProcessFunction.process(ProcessFunction.ja
>>> va:39)
>>> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TU
>>> GIBasedProcessor.java:110)
>>> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TU
>>> GIBasedProcessor.java:106)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at

Re: Storm-Hive : lock acquiring problem

2018-06-12 Thread Roshan Naik

Great find. I agree with upgrading storm-hive to  newer hive. Maybe even  
overdue.  Would be great if you can provide the PRs. Thanks 
Roshan 

Sent from Yahoo Mail for iPhone


On Tuesday, June 12, 2018, 3:47 AM, Abhishek Raj  
wrote:

Circling back to this, I was able to figure out what the problem was. 
storm-hive dependencies are compiled with hive version  0.14.0 which is very 
old. We are using hive 2.3.2 so obviously there were a lot of differences 
between the two versions. The fix was to override storm-hive's hive 
dependencies in our pom with a newer version. To be more verbose, we copied the 
following into our pom.

|   |
|  |  org.apache.hive.hcatalog |
|  |  hive-hcatalog-streaming |
|  |  2.3.2 |
|  |   |
|  |   |
|  |  org.slf4j |
|  |  slf4j-log4j12 |
|  |   |
|  |   |
|  |  org.apache.calcite |
|  |  calcite-core |
|  |   |
|  |   |
|  |  org.apache.calcite |
|  |  calcite-avatica |
|  |   |
|  |   |
|  |   |
|  |  |
|  |   |
|  |  org.apache.hive.hcatalog |
|  |  hive-hcatalog-core |
|  |  2.3.2 |
|  |   |
|  |   |
|  |  org.slf4j |
|  |  slf4j-log4j12 |
|  |   |
|  |   |
|  |  org.apache.calcite |
|  |  calcite-avatica |
|  |   |
|  |   |
|  |  org.apache.calcite |
|  |  calcite-core |
|  |   |
|  |   |
|  |   |
|  |   |
|  |  org.apache.hive |
|  |  hive-cli |
|  |  2.3.2 |
|  |   |
|  |   |
|  |  org.slf4j |
|  |  slf4j-log4j12 |
|  |   |
|  |   |
|  |  org.apache.calcite |
|  |  calcite-core |
|  |   |
|  |   |
|  |  org.apache.calcite |
|  |  calcite-avatica |
|  |   |
|  |   |
|  |   |

  


To go a little more deeper about the problem, this is what changed between the 
two hive versions giving us the "Unexpected DataOperationType: UNSET". 
createLockRequest in 2.3.2 explicitly passes a operation type "INSERT" while 
acquiring a lock, but in 0.14.0 no operation type is being passed. So the 
operation type ends up defaulting to UNSET and throwing an error. This is the 
commit where the change occurred.
In light of the fact that there are several different threads with users facing 
the same issue, imho, storm-hive's hive dependencies should be updated with 
newer hive releases and there should be a way for users to explicitly specify 
which hive release they want to use storm-hive with. The documentation for 
storm-hive should also be updated to reflect this requirement.
Happy to provide prs if that sounds like a good idea.
Thanks.
On Fri, Jun 8, 2018 at 3:21 PM, Abhishek Raj  wrote:

Hi. We faced a similar problem earlier when trying HiveBolt in storm with hive 
on emr. We were seeing 

java.lang.IllegalStateExceptio n: Unexpected DataOperationType: UNSET 
agentInfo=Unknown txnid:130551

in hive logs. Any help here would be appreciated. 

On Fri, Jun 8, 2018 at 10:26 AM, Milind Vaidya  wrote:

Here are some details from the meta store logs:
018-06-08T03:34:20,634 ERROR [pool-13-thread-197([])]: 
metastore.RetryingHMSHandler (RetryingHMSHandler.java:invok eInternal(204)) - 
java.lang.IllegalStateExceptio n: Unexpected DataOperationType: UNSET 
agentInfo=Unknown txnid:130551    at org.apache.hadoop.hive.metasto 
re.txn.TxnHandler.enqueueLockW ithRetry(TxnHandler.java:1000)    at 
org.apache.hadoop.hive.metasto re.txn.TxnHandler.lock(TxnHand ler.java:872)    
at org.apache.hadoop.hive.metasto re.HiveMetaStore$HMSHandler. 
lock(HiveMetaStore.java:6366)    at sun.reflect.GeneratedMethodAcc 
essor11.invoke(Unknown Source)    at sun.reflect.DelegatingMethodAc 
cessorImpl.invoke(DelegatingMe thodAccessorImpl.java:43)    at 
java.lang.reflect.Method.invok e(Method.java:498)    at 
org.apache.hadoop.hive.metasto re.RetryingHMSHandler.invokeIn 
ternal(RetryingHMSHandler. java:148)    at org.apache.hadoop.hive.metasto 
re.RetryingHMSHandler.invoke( RetryingHMSHandler.java:107)    at 
com.sun.proxy.$Proxy32.lock(Un known Source)    at 
org.apache.hadoop.hive.metasto re.api.ThriftHiveMetastore$ 
Processor$lock.getResult(Thrif tHiveMetastore.java:14155)    at 
org.apache.hadoop.hive.metasto re.api.ThriftHiveMetastore$ 
Processor$lock.getResult(Thrif tHiveMetastore.java:14139)    at 
org.apache.thrift.ProcessFunct ion.process(ProcessFunction. java:39)    at 
org.apache.hadoop.hive.metasto re.TUGIBasedProcessor$1.run( 
TUGIBasedProcessor.java:110)    at org.apache.hadoop.hive.metasto 
re.TUGIBasedProcessor$1.run( TUGIBasedProcessor.java:106)    at 
java.security.AccessController .doPrivileged(Native Method)    at 
javax.security.auth.Subject.do As(Subject.java:422)    at 
org.apache.hadoop.security.Use rGroupInformation.doAs(UserGro 
upInformation.java:1836)    at org.apache.hadoop.hive.metasto 
re.TUGIBasedProcessor.process( TUGIBasedProcessor.java:118)    at 
org.apache.thrift.server.TThre adPoolServer$WorkerProcess. 
run(TThreadPoolServer.java: 286)    at java.util.concurrent.ThreadPoo 
lExecutor.runWorker(ThreadPool Executor.java:1149)    at 
java.util.concurrent.ThreadPoo lExecutor$Worker.run(ThreadPoo 
lExecutor.java:624)    at java.lang.Thread.run(Thread.ja va:748)


Here are some details

Re: Storm-Hive : lock acquiring problem

2018-06-12 Thread Abhishek Raj

Circling back to this, I was able to figure out what the problem was.
storm-hive

dependencies
are compiled with hive version

  0.14.0 which is very old. We are using hive 2.3.2 so obviously there were
a lot of differences between the two versions. The fix was to override
*storm-hive's* hive dependencies in our pom with a newer version. To be
more verbose, we copied the following into our pom.


org.apache.hive.hcatalog
hive-hcatalog-streaming
2.3.2


org.slf4j
slf4j-log4j12


org.apache.calcite
calcite-core


org.apache.calcite
calcite-avatica




org.apache.hive.hcatalog
hive-hcatalog-core
2.3.2


org.slf4j
slf4j-log4j12


org.apache.calcite
calcite-avatica


org.apache.calcite
calcite-core




org.apache.hive
hive-cli
2.3.2


org.slf4j
slf4j-log4j12


org.apache.calcite
calcite-core


org.apache.calcite
calcite-avatica





To go a little more deeper about the problem, this is what changed between
the two hive versions giving us the "Unexpected DataOperationType: UNSET".
createLockRequest

in 2.3.2 explicitly passes a operation type "INSERT"

while
acquiring a lock, but in 0.14.0

 no operation type

is
being passed. So the operation type ends up defaulting to UNSET and
throwing an error. This is the commit

where the change occurred.

In light of the fact that there are several different threads with users
facing the same issue, imho, *storm-hive's* hive dependencies should be
updated with newer hive releases and there should be a way for users to
explicitly specify which hive release they want to use storm-hive with. The
documentation for storm-hive should also be updated to reflect this
requirement.

Happy to provide prs if that sounds like a good idea.

Thanks.

On Fri, Jun 8, 2018 at 3:21 PM, Abhishek Raj 
wrote:

> Hi. We faced a similar problem earlier when trying HiveBolt in storm with
> hive on emr. We were seeing
>
> java.lang.IllegalStateException: Unexpected DataOperationType: UNSET
>> agentInfo=Unknown txnid:130551
>
>
> in hive logs. Any help here would be appreciated.
>
>
> On Fri, Jun 8, 2018 at 10:26 AM, Milind Vaidya  wrote:
>
>> Here are some details from the meta store logs:
>>
>> 018-06-08T03:34:20,634 ERROR [pool-13-thread-197([])]:
>> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(204))
>> - java.lang.IllegalStateException: Unexpected DataOperationType: UNSET
>> agentInfo=Unknown txnid:130551
>> at org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockW
>> ithRetry(TxnHandler.java:1000)
>> at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHand
>> ler.java:872)
>> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.
>> lock(HiveMetaStore.java:6366)
>> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeIn
>> ternal(RetryingHMSHandler.java:148)
>> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(
>> RetryingHMSHandler.java:107)
>> at com.sun.proxy.$Proxy32.lock(Unknown Source)
>> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$
>> Processor$lock.getResult(ThriftHiveMetastore.java:14155)
>> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$
>> Processor$lock.getResult(ThriftHiveMetastore.java:14139)
>> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(
>> TUGIBasedProcessor.java:110)
>> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(
>> TUGIBasedProcessor.java:106)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>> upInformation.java:1836)
>> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(
>> TUGIBasedProcessor.java:118)
>> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.
>>

Re: Storm-Hive : lock acquiring problem

2018-06-08 Thread Abhishek Raj

Hi. We faced a similar problem earlier when trying HiveBolt in storm with
hive on emr. We were seeing

java.lang.IllegalStateException: Unexpected DataOperationType: UNSET
> agentInfo=Unknown txnid:130551


in hive logs. Any help here would be appreciated.


On Fri, Jun 8, 2018 at 10:26 AM, Milind Vaidya  wrote:

> Here are some details from the meta store logs:
>
> 018-06-08T03:34:20,634 ERROR [pool-13-thread-197([])]:
> metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(204))
> - java.lang.IllegalStateException: Unexpected DataOperationType: UNSET
> agentInfo=Unknown txnid:130551
> at org.apache.hadoop.hive.metastore.txn.TxnHandler.
> enqueueLockWithRetry(TxnHandler.java:1000)
> at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(
> TxnHandler.java:872)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$
> HMSHandler.lock(HiveMetaStore.java:6366)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(
> RetryingHMSHandler.java:148)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.
> invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy32.lock(Unknown Source)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$
> lock.getResult(ThriftHiveMetastore.java:14155)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$
> lock.getResult(ThriftHiveMetastore.java:14139)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$
> 1.run(TUGIBasedProcessor.java:110)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$
> 1.run(TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1836)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.
> process(TUGIBasedProcessor.java:118)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(
> TThreadPoolServer.java:286)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
>
>
> Here are some details about the environment
>
> Source :
>
> Storm Topology version : 1.1.1
> storm-hive : version 1.1.1
> mvn dependency plugin show following depedencies
>
> Hive :
>
> *[INFO]org.apache.hive.shims:hive-shims-0.23:jar:0.14.0:runtime*
>
> *[INFO]org.apache.hive:hive-ant:jar:0.14.0:compile*
>
> *[INFO]org.apache.hive:hive-metastore:jar:0.14.0:compile*
>
> *[INFO]org.apache.hive:hive-shims:jar:0.14.0:compile*
>
> *[INFO]org.apache.hive:hive-cli:jar:0.14.0:compile*
>
> *[INFO]org.apache.hive:hive-exec:jar:0.14.0:compile*
>
> *[INFO]
> org.apache.hive.shims:hive-shims-common-secure:jar:0.14.0:compile*
>
> *[INFO]org.apache.hive.shims:hive-shims-common:jar:0.14.0:compile*
>
> *[INFO]org.apache.hive:hive-common:jar:0.14.0:compile*
>
> *[INFO]org.apache.hive.shims:hive-shims-0.20S:jar:0.14.0:runtime*
>
> *[INFO]org.apache.hive.shims:hive-shims-0.20:jar:0.14.0:runtime*
>
> *[INFO]
> org.apache.hive.hcatalog:hive-hcatalog-streaming:jar:0.14.0:compile*
>
> *[INFO]org.apache.hive:hive-serde:jar:0.14.0:compile*
>
> *[INFO]org.apache.storm:storm-hive:jar:1.1.1:compile*
>
> *[INFO]org.apache.hive:hive-service:jar:0.14.0:compile*
>
> *[INFO]org.apache.hive.hcatalog:hive-hcatalog-core:jar:0.14.0:compile*
>
>
> Hadoop :
>
> *org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.1:compile*
>
> *[INFO]org.apache.hadoop:hadoop-yarn-common:jar:2.6.1:compile*
>
> *[INFO]org.apache.hadoop:hadoop-common:jar:2.6.1:compile*
>
> *[INFO]
> org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.6.1:compile*
>
> *[INFO]org.apache.hadoop:hadoop-yarn-api:jar:2.6.1:compile*
>
> *[INFO]org.apache.hadoop:hadoop-client:jar:2.6.1:compile*
>
> *[INFO]org.apache.hadoop:hadoop-auth:jar:2.6.1:compile*
>
> *[INFO]org.apache.hadoop:hadoop-yarn-client:jar:2.6.1:compile*
>
> *[INFO]org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.6.1:compile*
>
> *[INFO]
> org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.6.1:compile*
>
> *[INFO]org.apache.hadoop:hadoop-annotations:jar:2.6.1:compile*
>
> *[INFO]org.apache.hadoop:hadoop-hdfs:jar:2.6.1:compile*
>
> *[INFO]
> org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.6.1:compile*
>
> *[INFO]org.apache.hadoop:hadoop-yarn-server-common:jar:2.6.1:compile*
>
> and  HDFS
>
> *[INFO]org.apache.hadoop:hadoop-hdfs:jar:2.6.1:compile*
>
> Sink:
> Hive EMR
>
>

Re: Storm-Hive : lock acquiring problem

2018-06-07 Thread Milind Vaidya

Here are some details from the meta store logs:

018-06-08T03:34:20,634 ERROR [pool-13-thread-197([])]:
metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(204))
- java.lang.IllegalStateException: Unexpected DataOperationType: UNSET
agentInfo=Unknown txnid:130551
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:1000)
at
org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:872)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:6366)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy32.lock(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:14155)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:14139)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)



Here are some details about the environment

Source :

Storm Topology version : 1.1.1
storm-hive : version 1.1.1
mvn dependency plugin show following depedencies

Hive :

*[INFO]org.apache.hive.shims:hive-shims-0.23:jar:0.14.0:runtime*

*[INFO]org.apache.hive:hive-ant:jar:0.14.0:compile*

*[INFO]org.apache.hive:hive-metastore:jar:0.14.0:compile*

*[INFO]org.apache.hive:hive-shims:jar:0.14.0:compile*

*[INFO]org.apache.hive:hive-cli:jar:0.14.0:compile*

*[INFO]org.apache.hive:hive-exec:jar:0.14.0:compile*

*[INFO]
org.apache.hive.shims:hive-shims-common-secure:jar:0.14.0:compile*

*[INFO]org.apache.hive.shims:hive-shims-common:jar:0.14.0:compile*

*[INFO]org.apache.hive:hive-common:jar:0.14.0:compile*

*[INFO]org.apache.hive.shims:hive-shims-0.20S:jar:0.14.0:runtime*

*[INFO]org.apache.hive.shims:hive-shims-0.20:jar:0.14.0:runtime*

*[INFO]
org.apache.hive.hcatalog:hive-hcatalog-streaming:jar:0.14.0:compile*

*[INFO]org.apache.hive:hive-serde:jar:0.14.0:compile*

*[INFO]org.apache.storm:storm-hive:jar:1.1.1:compile*

*[INFO]org.apache.hive:hive-service:jar:0.14.0:compile*

*[INFO]org.apache.hive.hcatalog:hive-hcatalog-core:jar:0.14.0:compile*


Hadoop :

*org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.1:compile*

*[INFO]org.apache.hadoop:hadoop-yarn-common:jar:2.6.1:compile*

*[INFO]org.apache.hadoop:hadoop-common:jar:2.6.1:compile*

*[INFO]
org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.6.1:compile*

*[INFO]org.apache.hadoop:hadoop-yarn-api:jar:2.6.1:compile*

*[INFO]org.apache.hadoop:hadoop-client:jar:2.6.1:compile*

*[INFO]org.apache.hadoop:hadoop-auth:jar:2.6.1:compile*

*[INFO]org.apache.hadoop:hadoop-yarn-client:jar:2.6.1:compile*

*[INFO]org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.6.1:compile*

*[INFO]
org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.6.1:compile*

*[INFO]org.apache.hadoop:hadoop-annotations:jar:2.6.1:compile*

*[INFO]org.apache.hadoop:hadoop-hdfs:jar:2.6.1:compile*

*[INFO]
org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.6.1:compile*

*[INFO]org.apache.hadoop:hadoop-yarn-server-common:jar:2.6.1:compile*

and  HDFS

*[INFO]org.apache.hadoop:hadoop-hdfs:jar:2.6.1:compile*

Sink:
Hive EMR

Hadoop version :

*hadoop@ip-10-0-6-16 ~]$ hadoop version*

*Hadoop 2.8.3-amzn-0*

Hive version :

*[hadoop@ip-10-0-6-16 ~]$ hive --version*

*Hive 2.3.2-amzn-2*


Any inconsistency leading to such an error ?


On Thu, Jun 7, 2018 at 7:35 PM, Roshan Naik  wrote:

> The lock issue seems to be happening on the Metastore end and surfacing
> via the API.
> Partition creation is working but the API is unable to acquire a TxnBatch
> from the metastore due to the lock issue.
> Check the hive metastore logs and see why the locks are failing.
> Roshan
>
>
>

Re: Storm-Hive : lock acquiring problem

2018-06-07 Thread Roshan Naik

The lock issue seems to be happening on the Metastore end and surfacing via the 
API. Partition creation is working but the API is unable to acquire a TxnBatch 
from the metastore due to the lock issue. Check the hive metastore logs and see 
why the locks are failing. Roshan 


Sent from Yahoo Mail for iPhone


On Thursday, June 7, 2018, 11:08 AM, Milind Vaidya  wrote:

Hi 
I am using storm and strom-hive version 1.1.1 to store data directly to hive 
cluster.
After using mvn shade plugin and overcoming few other errors I am now stuck at 
this point.
The strange thing observed was few partitions were created but the data was not 
inserted.

dt=17688/platform=site/country=SG/entity_id=abcd

dt=17688/platform=site/country=SG/entity_id=asdlfa

dt=17688/platform=site/country=SG/entity_id=asdq13

dt=17688/platform=site/country=SG/entity_id=123124

             What are my debugging options here ? ( some data from log is 
removed intentionally)


2018-06-07 16:35:22.459 h.metastore 
Thread-12-users-by-song-hive-bolt-executor[5 5] [INFO] Connected to metastore.
2018-06-07 16:35:22.545 o.a.s.h.b.HiveBolt 
Thread-12-users-by-song-hive-bolt-executor[5 5] [ERROR] Failed to create 
HiveWriter for endpoint: { }
org.apache.storm.hive.common.HiveWriter$ConnectFailure: Failed connecting to 
EndPoint {metaStoreUri='', database='', table='', partitionVals=[] }
at org.apache.storm.hive.common.HiveWriter.(HiveWriter.java:80) 
~[stormjar.jar:?]
at 
org.apache.storm.hive.common.HiveUtils.makeHiveWriter(HiveUtils.java:50) 
~[stormjar.jar:?]
at 
org.apache.storm.hive.bolt.HiveBolt.getOrCreateWriter(HiveBolt.java:262) 
[stormjar.jar:?]
at org.apache.storm.hive.bolt.HiveBolt.execute(HiveBolt.java:112) 
[stormjar.jar:?]
at 
org.apache.storm.daemon.executor$fn__5030$tuple_action_fn__5032.invoke(executor.clj:729)
 [storm-core-1.1.1.jar:1.1.1]
at 
org.apache.storm.daemon.executor$mk_task_receiver$fn__4951.invoke(executor.clj:461)
 [storm-core-1.1.1.jar:1.1.1]
at 
org.apache.storm.disruptor$clojure_handler$reify__4465.onEvent(disruptor.clj:40)
 [storm-core-1.1.1.jar:1.1.1]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:482)
 [storm-core-1.1.1.jar:1.1.1]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:460)
 [storm-core-1.1.1.jar:1.1.1]
at 
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
 [storm-core-1.1.1.jar:1.1.1]
at 
org.apache.storm.daemon.executor$fn__5030$fn__5043$fn__5096.invoke(executor.clj:848)
 [storm-core-1.1.1.jar:1.1.1]
at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484) 
[storm-core-1.1.1.jar:1.1.1]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_131]
Caused by: org.apache.storm.hive.common.HiveWriter$TxnBatchFailure: Failed 
acquiring Transaction Batch from EndPoint: {metaStoreUri='', database='', 
table='', partitionVals=[, , , ] }
at 
org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:264) 
~[stormjar.jar:?]
at org.apache.storm.hive.common.HiveWriter.(HiveWriter.java:72) 
~[stormjar.jar:?]
... 13 more
Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable to 
acquire lock on { }
at 
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:575)
 ~[stormjar.jar:?]
at 
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:544)
 ~[stormjar.jar:?]
at 
org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:259) 
~[stormjar.jar:?]
at org.apache.storm.hive.common.HiveWriter.(HiveWriter.java:72) 
~[stormjar.jar:?]
... 13 more
Caused by: org.apache.thrift.transport.TTransportException
at 
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
 ~[stormjar.jar:?]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) 
~[stormjar.jar:?]
at 
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) 
~[stormjar.jar:?]
at 
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) 
~[stormjar.jar:?]
at 
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
 ~[stormjar.jar:?]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) 
~[stormjar.jar:?]
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3781)
 ~[stormjar.jar:?]
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3768)
 ~[stormjar.jar:?]
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1736)
 ~[stormjar.jar:?]
at

Re: Storm-Hive : lock acquiring problem

Re: Storm-Hive : lock acquiring problem

Re: Storm-Hive : lock acquiring problem

Re: Storm-Hive : lock acquiring problem

Re: Storm-Hive : lock acquiring problem

Re: Storm-Hive : lock acquiring problem

6 matches

Site Navigation

Mail list logo

Footer information