date:20170403

Re: Storm files and folders permissions on Linux

2017-04-03 Thread Antoine Tran

Have you modified in Storm.yaml settings related to ACL? The defaults 
one disabled ACL for blob



On 03/04/2017 18:27, I PVP wrote:

Thanks for everyone that is answering and helping to look into this issue.



I tried from /tmp. Same error.
I tried submitting the jar from the ec2-user $HOME directory. Same error
I created a new directory to which storm:storm is the owner and tried 
submitting the jar  from there. Same error.



The error log is not really clear on what is going on.

Where/what is setup-path that is mentioned on the log? is it the 
/opt/storm where  storm was “installed”  ?


What is the exact "No such file or directory” that  is missing or 
inaccessible ?



First it says that :
—
2017-04-03 13:13:00.148 pool-15-thread-30 o.a.s.d.nimbus [INFO] 
Finished uploading file from client: 
/opt/storm/storm-local/nimbus/inbox/stormjar-f90ff743-e9ec-4e59-b945-878a47fd9057.j

ar
2017-04-03 13:13:00.173 pool-15-thread-32 o.a.s.d.nimbus [INFO] 
Received topology submission for Topology with conf 
{"topology.max.task.parallelism" nil, "topology
.submitter.principal" "", "topology.acker.executors" nil, 
"topology.eventlogger.executors" 0, "topology.workers" 1, 
"topology.debug" false, "storm.zookeeper.superACL" nil, "topology.us
ers" (), "topology.submitter.user" "storm", "topology.kryo.register" 
nil, "topology.kryo.decorators" (), "storm.id" 
"Topology-3-1491235980", "topology.name" "Emplo

yeePasswordResetTopology"}
2017-04-03 13:13:00.189 pool-15-thread-32 o.a.s.d.nimbus [INFO] 
uploadedJar 
/opt/storm/storm-local/nimbus/inbox/stormjar-f90ff743-e9ec-4e59-b945-878a47fd9057.jar
2017-04-03 13:13:00.189 pool-15-thread-32 
o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting

..



Before trowing the exception it says :
——
2017-04-03 13:13:00.207 pool-15-thread-32 o.a.s.cluster [INFO] 
setup-path/blobstore/Topology-3-1491235980-stormjar.jar/ip-172-31-5-153.sa-east-1.compute.internal:6

627-1
2017-04-03 13:13:00.207 pool-15-thread-32-EventThread 
o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down
2017-04-03 13:13:00.217 pool-15-thread-32 o.a.s.d.nimbus [WARN] 
Topology submission exception. (topology name='Topology') 
#error {

 :cause No such file or directory
 :via
 [{:type java.lang.RuntimeException
   :message java.io.IOException: No such file or directory
   :at [org.apache.storm.blobstore.LocalFsBlobStore createBlob 
LocalFsBlobStore.java 117]}

...
-



best,
IPVP

On April 3, 2017 at 12:12:02 PM, Santosh Pingale 
(pingalesant...@gmail.com ) wrote:


Can you try to submit the jar from a directory which is writable by 
storm user.  Try from /tmp maybe.


On Mon, Apr 3, 2017 at 3:56 PM, I PVP > wrote:


that is the error:


2017-04-03 10:31:44.535 pool-15-thread-56 o.a.s.d.nimbus [INFO]
Uploading file from client to

/opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar
2017-04-03 10:31:44.820 pool-15-thread-17 o.a.s.d.nimbus [INFO]
Finished uploading file from client:

/opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar
2017-04-03 10:31:44.842 pool-15-thread-20 o.a.s.d.nimbus [INFO]
Received topology submission for Topology with conf
{"topology.max.task.parallelism" nil,
"topology.submitter.principal" "", "topology.acker.executors"
nil, "topology.eventlogger.executors" 0, "topology.workers" 1,
"topology.debug" false, "storm.zookeeper.superACL" nil,
"topology.users" (), "topology.submitter.user" "storm",
"topology.kryo.register" nil, "topology.kryo.decorators" (),
"storm.id "
"Topology-2-1491226304", "topology.name
" "Topology"}
2017-04-03 10:31:44.846 pool-15-thread-20 o.a.s.d.nimbus [INFO]
uploadedJar

/opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar
2017-04-03 10:31:44.847 pool-15-thread-20
o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2017-04-03 10:31:44.848 pool-15-thread-20 o.a.s.s.o.a.z.ZooKeeper
[INFO] Initiating client connection,
connectString=172.31.5.153:2181
,172.31.3.29:2181
,172.31.0.218:2181/storm
 sessionTimeout=2
watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@13bf52ea
2017-04-03 10:31:44.851
pool-15-thread-20-SendThread(172.31.0.218:2181
) o.a.s.s.o.a.z.ClientCnxn [INFO]
Opening socket connection to server
172.31.0.218/172.31.0.218:2181
. Will not attempt to
authenticate using SASL (unknown error)
2017-04-03 10:31:44.852
pool-15-thread-20-SendThread(172.31.0.218:2181
) o.a.s.s.o.a.z.ClientCnxn [INFO]
Socket connection established to 172.31.0.218/172.31.0.218:2181

Re: Storm files and folders permissions on Linux

2017-04-03 Thread Santosh Pingale

Can you try to submit the jar from a directory which is writable by storm
user.  Try from /tmp maybe.

On Mon, Apr 3, 2017 at 3:56 PM, I PVP  wrote:

> that is the error:
>
>
> 2017-04-03 10:31:44.535 pool-15-thread-56 o.a.s.d.nimbus [INFO] Uploading
> file from client to /opt/storm/storm-local/nimbus/
> inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar
> 2017-04-03 10:31:44.820 pool-15-thread-17 o.a.s.d.nimbus [INFO] Finished
> uploading file from client: /opt/storm/storm-local/nimbus/
> inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar
> 2017-04-03 10:31:44.842 pool-15-thread-20 o.a.s.d.nimbus [INFO] Received
> topology submission for Topology with conf 
> {"topology.max.task.parallelism"
> nil, "topology.submitter.principal" "", "topology.acker.executors" nil,
> "topology.eventlogger.executors" 0, "topology.workers" 1,
> "topology.debug" false, "storm.zookeeper.superACL" nil, "topology.users"
> (), "topology.submitter.user" "storm", "topology.kryo.register" nil,
> "topology.kryo.decorators" (), "storm.id" 
> "Topology-2-1491226304",
> "topology.name" "Topology"}
> 2017-04-03 10:31:44.846 pool-15-thread-20 o.a.s.d.nimbus [INFO]
> uploadedJar /opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-
> 415d-803d-ab73c0d2fafe.jar
> 2017-04-03 10:31:44.847 pool-15-thread-20 
> o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl
> [INFO] Starting
> 2017-04-03 10:31:44.848 pool-15-thread-20 o.a.s.s.o.a.z.ZooKeeper [INFO]
> Initiating client connection, connectString=172.31.5.153:2181,
> 172.31.3.29:2181,172.31.0.218:2181/storm sessionTimeout=2
> watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@13bf52ea
> 2017-04-03 10:31:44.851 pool-15-thread-20-SendThread(172.31.0.218:2181)
> o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server
> 172.31.0.218/172.31.0.218:2181. Will not attempt to authenticate using
> SASL (unknown error)
> 2017-04-03 10:31:44.852 pool-15-thread-20-SendThread(172.31.0.218:2181)
> o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to
> 172.31.0.218/172.31.0.218:2181, initiating session
> 2017-04-03 10:31:44.855 pool-15-thread-20-SendThread(172.31.0.218:2181)
> o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server
> 172.31.0.218/172.31.0.218:2181, sessionid = 0x35b33b21dad0005, negotiated
> timeout = 2
> 2017-04-03 10:31:44.855 pool-15-thread-20-EventThread 
> o.a.s.s.o.a.c.f.s.ConnectionStateManager
> [INFO] State change: CONNECTED
> 2017-04-03 10:31:44.860 Curator-Framework-0 
> o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl
> [INFO] backgroundOperationsLoop exiting
> 2017-04-03 10:31:44.862 pool-15-thread-20 o.a.s.s.o.a.z.ZooKeeper [INFO]
> Session: 0x35b33b21dad0005 closed
> 2017-04-03 10:31:44.862 pool-15-thread-20 o.a.s.cluster [INFO]
> setup-path/blobstore/Topology-2-1491226304-stormjar.jar/ip-
> 172-31-5-153.sa-east-1.compute.internal:6627-1
> 2017-04-03 10:31:44.862 pool-15-thread-20-EventThread
> o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down
> 2017-04-03 10:31:44.877 pool-15-thread-20 o.a.s.d.nimbus [WARN] Topology
> submission exception. (topology name='Topology') #error {
>  :cause No such file or directory
>  :via
>  [{:type java.lang.RuntimeException
>:message java.io.IOException: No such file or directory
>:at [org.apache.storm.blobstore.LocalFsBlobStore createBlob
> LocalFsBlobStore.java 117]}
>   {:type java.io.IOException
>:message No such file or directory
>:at [java.io.UnixFileSystem createFileExclusively UnixFileSystem.java
> -2]}]
>  :trace
>  [[java.io.UnixFileSystem createFileExclusively UnixFileSystem.java -2]
>   [java.io.File createNewFile File.java 1012]
>   [org.apache.storm.blobstore.LocalFsBlobStoreFile getOutputStream
> LocalFsBlobStoreFile.java 109]
>   [org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream 
> BlobStore.java 316]
>   [org.apache.storm.blobstore.LocalFsBlobStore createBlob
> LocalFsBlobStore.java 111]
>   [org.apache.storm.blobstore.BlobStore createBlob BlobStore.java 217]
>   [sun.reflect.NativeMethodAccessorImpl invoke0
> NativeMethodAccessorImpl.java -2]
>   [sun.reflect.NativeMethodAccessorImpl invoke
> NativeMethodAccessorImpl.java 62]
>   [sun.reflect.DelegatingMethodAccessorImpl invoke
> DelegatingMethodAccessorImpl.java 43]
>   [java.lang.reflect.Method invoke Method.java 498]
>   [clojure.lang.Reflector invokeMatchingMethod Reflector.java 93]
>   [clojure.lang.Reflector invokeInstanceMethod Reflector.java 28]
>   [org.apache.storm.daemon.nimbus$setup_storm_code invoke nimbus.clj 474]
>   [org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138
> submitTopologyWithOpts nimbus.clj 1525]
>   [org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138
> submitTopology nimbus.clj 1541]
>   [org.apache.storm.generated.Nimbus$Processor$submitTopology getResult
> Nimbus.java 2912]
>   [org.apache.storm.generated.Nimbus$Processor$submitTopology getResult
> Nimbus.java 2896]
>

Re: Where to run logviewer?

2017-04-03 Thread Santosh Pingale

You will need to run it on each worker node.

On Sun, Apr 2, 2017 at 6:50 PM, Dmitry Semenov  wrote:

> hi Guys,
>
> In a cluster environment with multiple storm workers - where do I need to
> run storm logviewer? On each worker instance or only @ nimbus instance? Or
> everywhere?
>
> thanks,
> Dmitry
>
>

Re: Storm files and folders permissions on Linux

2017-04-03 Thread I PVP

that is the error:


2017-04-03 10:31:44.535 pool-15-thread-56 o.a.s.d.nimbus [INFO] Uploading file 
from client to 
/opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar
2017-04-03 10:31:44.820 pool-15-thread-17 o.a.s.d.nimbus [INFO] Finished 
uploading file from client: 
/opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar
2017-04-03 10:31:44.842 pool-15-thread-20 o.a.s.d.nimbus [INFO] Received 
topology submission for Topology with conf 
{"topology.max.task.parallelism" nil, "topology.submitter.principal" "", 
"topology.acker.executors" nil, "topology.eventlogger.executors" 0, 
"topology.workers" 1, "topology.debug" false, "storm.zookeeper.superACL" nil, 
"topology.users" (), "topology.submitter.user" "storm", 
"topology.kryo.register" nil, "topology.kryo.decorators" (), "storm.id" 
"Topology-2-1491226304", "topology.name" "Topology"}
2017-04-03 10:31:44.846 pool-15-thread-20 o.a.s.d.nimbus [INFO] uploadedJar 
/opt/storm/storm-local/nimbus/inbox/stormjar-fceb1c76-b88e-415d-803d-ab73c0d2fafe.jar
2017-04-03 10:31:44.847 pool-15-thread-20 
o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2017-04-03 10:31:44.848 pool-15-thread-20 o.a.s.s.o.a.z.ZooKeeper [INFO] 
Initiating client connection, 
connectString=172.31.5.153:2181,172.31.3.29:2181,172.31.0.218:2181/storm 
sessionTimeout=2 
watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@13bf52ea
2017-04-03 10:31:44.851 pool-15-thread-20-SendThread(172.31.0.218:2181) 
o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server 
172.31.0.218/172.31.0.218:2181. Will not attempt to authenticate using SASL 
(unknown error)
2017-04-03 10:31:44.852 pool-15-thread-20-SendThread(172.31.0.218:2181) 
o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to 
172.31.0.218/172.31.0.218:2181, initiating session
2017-04-03 10:31:44.855 pool-15-thread-20-SendThread(172.31.0.218:2181) 
o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server 
172.31.0.218/172.31.0.218:2181, sessionid = 0x35b33b21dad0005, negotiated 
timeout = 2
2017-04-03 10:31:44.855 pool-15-thread-20-EventThread 
o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2017-04-03 10:31:44.860 Curator-Framework-0 
o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] backgroundOperationsLoop exiting
2017-04-03 10:31:44.862 pool-15-thread-20 o.a.s.s.o.a.z.ZooKeeper [INFO] 
Session: 0x35b33b21dad0005 closed
2017-04-03 10:31:44.862 pool-15-thread-20 o.a.s.cluster [INFO] 
setup-path/blobstore/Topology-2-1491226304-stormjar.jar/ip-172-31-5-153.sa-east-1.compute.internal:6627-1
2017-04-03 10:31:44.862 pool-15-thread-20-EventThread o.a.s.s.o.a.z.ClientCnxn 
[INFO] EventThread shut down
2017-04-03 10:31:44.877 pool-15-thread-20 o.a.s.d.nimbus [WARN] Topology 
submission exception. (topology name='Topology') #error {
 :cause No such file or directory
 :via
 [{:type java.lang.RuntimeException
   :message java.io.IOException: No such file or directory
   :at [org.apache.storm.blobstore.LocalFsBlobStore createBlob 
LocalFsBlobStore.java 117]}
  {:type java.io.IOException
   :message No such file or directory
   :at [java.io.UnixFileSystem createFileExclusively UnixFileSystem.java -2]}]
 :trace
 [[java.io.UnixFileSystem createFileExclusively UnixFileSystem.java -2]
  [java.io.File createNewFile File.java 1012]
  [org.apache.storm.blobstore.LocalFsBlobStoreFile getOutputStream 
LocalFsBlobStoreFile.java 109]
  [org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream  
BlobStore.java 316]
  [org.apache.storm.blobstore.LocalFsBlobStore createBlob LocalFsBlobStore.java 
111]
  [org.apache.storm.blobstore.BlobStore createBlob BlobStore.java 217]
  [sun.reflect.NativeMethodAccessorImpl invoke0 NativeMethodAccessorImpl.java 
-2]
  [sun.reflect.NativeMethodAccessorImpl invoke NativeMethodAccessorImpl.java 62]
  [sun.reflect.DelegatingMethodAccessorImpl invoke 
DelegatingMethodAccessorImpl.java 43]
  [java.lang.reflect.Method invoke Method.java 498]
  [clojure.lang.Reflector invokeMatchingMethod Reflector.java 93]
  [clojure.lang.Reflector invokeInstanceMethod Reflector.java 28]
  [org.apache.storm.daemon.nimbus$setup_storm_code invoke nimbus.clj 474]
  [org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138 
submitTopologyWithOpts nimbus.clj 1525]
  [org.apache.storm.daemon.nimbus$fn__9109$exec_fn__1371__auto__$reify__9138 
submitTopology nimbus.clj 1541]
  [org.apache.storm.generated.Nimbus$Processor$submitTopology getResult 
Nimbus.java 2912]
  [org.apache.storm.generated.Nimbus$Processor$submitTopology getResult 
Nimbus.java 2896]
  [org.apache.storm.thrift.ProcessFunction process ProcessFunction.java 39]
  [org.apache.storm.thrift.TBaseProcessor process TBaseProcessor.java 39]
  [org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor 
process SimpleTransportPlugin.java 162]
  [org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer invoke

Re: Storm hive integration sample app doesn't work.

2017-04-03 Thread Marcin Kasiński

Hi thank you for you reply.

I'm pretty sure it is due to "Unableto acquire lock on " error.

My question is "why ?"

This is my DEV environent. Onlu I can do something there.

I have opened hive view in ambari admin console only.

telnet works fine:


In storm logs I can find:

2017-04-03 09:46:48.355 h.metastore [INFO] Connected to metastore.
2017-04-03 09:46:48.480 h.metastore [INFO] Trying to connect to
metastore with URI thrift://hdp1.local:9083
2017-04-03 09:46:48.483 h.metastore [INFO] Connected to metastore.
2017-04-03 09:46:50.809 o.a.s.h.b.HiveBolt [ERROR] Failed to create
HiveWriter for endpoint: {metaStoreUri='thrift://hdp1.local:9083',
database='default', table='test_table', partitionVals=[] }
o

In my opinion storm is connected to hive, but there is some internal
hive problem.

In hive metastore logs there is:

2017-04-03 09:46:50,824 ERROR [pool-5-thread-195]:
metastore.RetryingHMSHandler
(RetryingHMSHandler.java:invokeInternal(195)) -
java.lang.IllegalStateException: Unexpected DataOperationType: UNSET
agentInfo=Unknown txnid:106346
at 
org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:938)
at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:814)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:5790)

I think there is problem with request sent from storm to hive.

Storm sends wrong unexpected data to hive and hive can not process that request.

My question is why?

In my pom file I have:



org.apache.storm
storm-hive
1.0.2





   org.apache.storm
storm-core
1.0.2


 log4j-over-slf4j
org.slf4j





I think versions are OK.



pozdrawiam
Marcin Kasiński
http://itzone.pl


On 3 April 2017 at 09:31, steve tueno  wrote:
> Hi,
>
> I think this could be due to this error:
> "org.apache.hive.hcatalog.streaming.TransactionError: Unable
> to acquire lock on {metaStoreUri='thrift://hdp1.local:9083',
> database='default', table='test_table', partitionVals=[] }"
>
> Try a telnet on "
> thrift://hdp1.local:9083"
> and try to check if there is not another process working on table
>
>
>
>
> Cordialement,
>
> TUENO FOTSO Steve Jeffrey
> Ingénieur de conception
> Génie Informatique
> +237 676 57 17 28
> +237 697 86 36 38
>
> +33 6 23 71 91 52
>
>
> https://jobs.jumia.cm/fr/candidats/CVTF1486563.html
> __
>
> https://play.google.com/store/apps/details?id=com.polytech.remotecomputer
> https://play.google.com/store/apps/details?id=com.polytech.internetaccesschecker
> http://www.traveler.cm/
> http://remotecomputer.traveler.cm/
> https://play.google.com/store/apps/details?id=com.polytech.androidsmssender
> https://github.com/stuenofotso/notre-jargon
> https://play.google.com/store/apps/details?id=com.polytech.welovecameroon
> https://play.google.com/store/apps/details?id=com.polytech.welovefrance
>
> 2017-04-03 9:21 GMT+02:00 Marcin Kasiński :
>>
>> Hello.
>>
>> I have problem with storm hive integration.
>>
>> I'm using HDP 2.5 and HDF 2.1 (storm 1.0.2 and Hive 1.2.1000)
>>
>> I'm using sample class
>>
>> https://github.com/hortonworks/storm-release/blob/HDP-2.5.0.0-tag/external/storm-hive/src/test/java/org/apache/storm/hive/bolt/HiveTopology.java
>>
>>
>> I've created sample table:
>>
>> create table test_table ( id INT, name STRING, phone STRING, street
>> STRING,city STRING, state STRING)
>> CLUSTERED BY (name) into 5 buckets
>> stored as orc tblproperties ("orc.compress"="NONE");
>>
>>
>> Hive works .
>>
>> I can insert to table manualy and get records:
>>
>> INSERT INTO TABLE test_table  VALUES ( 1,"MyName",
>> "11","street","city","state");
>>
>> select * from test_table ;
>>
>> Problem is that storm sample HiveTopology doesn't work.
>>
>> When I deploy storm application hive bolt fails:
>>
>> On metastore I get:
>>
>> 2017-04-03 08:42:39,682 ERROR [pool-5-thread-188]:
>> metastore.RetryingHMSHandler
>> (RetryingHMSHandler.java:invokeInternal(195)) -
>> java.lang.IllegalStateException: Unexpected DataOperationType: UNSET
>> agentInfo=Unknown txnid:74056
>> at
>> org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:938)
>> at
>> org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:814)
>> at
>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaStore.java:5790)
>> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:139)
>> at
>>

Re: Storm hive integration sample app doesn't work.

2017-04-03 Thread steve tueno

Hi,

I think this could be due to this error:
"org.apache.hive.hcatalog.streaming.TransactionError:
Unable
to acquire lock on {metaStoreUri='thrift://hdp1.local:9083',
database='default', table='test_table', partitionVals=[] }"

Try a telnet on "
thrift://hdp1.local:9083"
 
and try to check if there is not another process working on table




Cordialement,

TUENO FOTSO Steve Jeffrey
Ingénieur de conception
Génie Informatique
+237 676 57 17 28 <6%2076%2057%2017%2028>
+237 697 86 36 38 <6%2097%2086%2036%2038>

+33 6 23 71 91 52 <+33%206%2023%2071%2091%2052>


https://jobs.jumia.cm/fr/candidats/CVTF1486563.html

__

https://play.google.com/store/apps/details?id=com.polytech.remotecomputer
https://play.google.com/store/apps/details?id=com.polytech.
internetaccesschecker
*http://www.traveler.cm/ *
http://remotecomputer.traveler.cm/
https://play.google.com/store/apps/details?id=com.polytech.androidsmssender
https://github.com/stuenofotso/notre-jargon
https://play.google.com/store/apps/details?id=com.polytech.welovecameroon
https://play.google.com/store/apps/details?id=com.polytech.welovefrance

2017-04-03 9:21 GMT+02:00 Marcin Kasiński :

> Hello.
>
> I have problem with storm hive integration.
>
> I'm using HDP 2.5 and HDF 2.1 (storm 1.0.2 and Hive 1.2.1000)
>
> I'm using sample class
> https://github.com/hortonworks/storm-release/blob/HDP-2.5.0.
> 0-tag/external/storm-hive/src/test/java/org/apache/storm/
> hive/bolt/HiveTopology.java
>
>
> I've created sample table:
>
> create table test_table ( id INT, name STRING, phone STRING, street
> STRING,city STRING, state STRING)
> CLUSTERED BY (name) into 5 buckets
> stored as orc tblproperties ("orc.compress"="NONE");
>
>
> Hive works .
>
> I can insert to table manualy and get records:
>
> INSERT INTO TABLE test_table  VALUES ( 1,"MyName",
> "11","street","city","state");
>
> select * from test_table ;
>
> Problem is that storm sample HiveTopology doesn't work.
>
> When I deploy storm application hive bolt fails:
>
> On metastore I get:
>
> 2017-04-03 08:42:39,682 ERROR [pool-5-thread-188]:
> metastore.RetryingHMSHandler
> (RetryingHMSHandler.java:invokeInternal(195)) -
> java.lang.IllegalStateException: Unexpected DataOperationType: UNSET
> agentInfo=Unknown txnid:74056
> at org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockW
> ithRetry(TxnHandler.java:938)
> at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHand
> ler.java:814)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.
> lock(HiveMetaStore.java:5790)
> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
> thodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeIn
> ternal(RetryingHMSHandler.java:139)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(
> RetryingHMSHandler.java:97)
> at com.sun.proxy.$Proxy12.lock(Unknown Source)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$
> Processor$lock.getResult(ThriftHiveMetastore.java:11860)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$
> Processor$lock.getResult(ThriftHiveMetastore.java:11844)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(
> TUGIBasedProcessor.java:110)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(
> TUGIBasedProcessor.java:106)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
> upInformation.java:1724)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(
> TUGIBasedProcessor.java:118)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.
> run(TThreadPoolServer.java:286)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
> On storm I get:
> org.apache.storm.hive.common.HiveWriter$ConnectFailure: Failed
> connecting to EndPoint {metaStoreUri='thrift://hdp1.local:9083',
> database='default', table='test_table', partitionVals=[] }
> at org.apache.storm.hive.common.HiveWriter.(HiveWriter.java:80)
> ~[stormjar.jar:?]
> at org.apache.storm.hive.common.HiveUtils.makeHiveWriter(HiveUt
> ils.java:50)
> ~[stormjar.jar:?]
> at org.apache.storm.hive.bolt.HiveBolt.getOrCreateWriter(HiveBo
> lt.java:259)
> ~[stormjar.jar:?]
> at org.apache.storm.hive.bolt.HiveBolt.execute(HiveBolt.java:112)
> [stormjar.jar:?]
> at

Re: Storm files and folders permissions on Linux

Re: Storm files and folders permissions on Linux

Re: Where to run logviewer?

Re: Storm files and folders permissions on Linux

Re: Storm hive integration sample app doesn't work.

Re: Storm hive integration sample app doesn't work.

6 matches

Site Navigation

Mail list logo

Footer information