Re: Hive SerDe maven dependency

[Edward Capriolo]

You should match your hive versions as close as possible. It makes sense
that both hive and hadoop dependencies use a PROVIDED scope, this way if
you are building an assembly/fat/shaded jar the jar is as thin as possible.

On Wed, Mar 29, 2017 at 3:01 PM, srinu reddy  wrote:

>
>
> Hi
>
> I want to implement custom SerDe. But I confused to select the Hive SerDe
> version for maven dependency and also hadoop-core dependency
>
> Could any one please suggest me
>
>
> Below are the hadoop and hive versions which we are using
>
> HDP : 2.2.0
> Hadoop : 2.6.0
> Hive-Hcatalog : 0.14.0
>
>
> Thanks
> Srinu
>

Hive SerDe maven dependency

[srinu reddy]

Hi

I want to implement custom SerDe. But I confused to select the Hive SerDe
version for maven dependency and also hadoop-core dependency

Could any one please suggest me


Below are the hadoop and hive versions which we are using

HDP : 2.2.0
Hadoop : 2.6.0
Hive-Hcatalog : 0.14.0


Thanks
Srinu

Re: Storm hive bolt

[Marcin Kasiński]

I've upgraded my environment.

I have HIve on HDP 2.5 (environment 1) and storm on HDF 2.1

(environment 2)

I have the same eroor:

On storm (HDF 2.1):

Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable
to acquire lock on {metaStoreUri='thrift://hdp1.local:9083',
database='default', table='stock_prices', partitionVals=[Marcin] } at
org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:575)
~[stormjar.jar:?]

On hive metastore (HDP 2.5):

2017-03-29 11:56:29,926 ERROR [pool-5-thread-17]:
server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error
occurred during processing of message.
java.lang.IllegalStateException: Unexpected DataOperationType: UNSET
agentInfo=Unknown txnid:54 at
org.apache.hadoop.hive.metastore.txn.TxnHandler.enqueueLockWithRetry(TxnHandler.java:938)
at org.apache.hadoop.hive.metastore.txn.TxnHandler.lock(TxnHandler.java:814)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.lock(HiveMetaSt
pozdrawiam
Marcin Kasiński
http://itzone.pl


On 27 March 2017 at 22:01, Marcin Kasiński  wrote:
> Hello.
>
> Thank you for reply.
>
> I do really want to solve it.
>
> I'm sure i compiled sources again with new jars.
>
> I've changed source from storm 0.10 ( package backtype.storm.* ) to
> storm 1.0.1 (package org.apache.storm.*) and I've generated jar again
>
> Below you have entire storm worker logs and pom.xml.
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> StormSample
> StormSample
> 0.0.1-SNAPSHOT
>
>  
> UTF-8
> 1.7
> 1.7
> 1.0.1
> 0.3.0
> 0.8.2.2.3.0.0-2557
> 1.7.7
> 4.11
> 
> 
> src
> 
>
> 
> 
> maven-compiler-plugin
> 3.3
> 
> 1.8
> 1.8
> 
> 
>
> 
> org.apache.maven.plugins
> maven-jar-plugin
> 
> 
> 
> true
> lib/
> mk.StormSample
> 
> 
> 
> 
> 
>  
> org.apache.maven.plugins
> maven-shade-plugin
> 1.4
> 
> true
> 
> 
> 
> package
> 
> shade
> 
> 
>  
> 
> *:*
> 
> META-INF/*.SF
> META-INF/*.DSA
> META-INF/*.RSA
> 
>   defaults.yaml
> 
> 
> 
> 
> 
> 
> implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
> 
> implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> 
>
>  
> org.apache.storm
> storm-hive
> 1.0.3
> 
> 
> 
> jline
> jline
> 
>
>
> 
> 
> 
> 
> org.apache.storm
> storm-hbase
> 1.0.1
> 
>
> 
> 
> 
>org.apache.storm
> storm-core
> 1.0.1
> 
> 
>  log4j-over-slf4j
> org.slf4j
> 
> 
> 
>
> 
>
>
> 
> org.apache.kafka
> kafka_2.10
> 0.10.0.0
>
> 
> org.apache.zookeeper
> zookeeper
> 
> 
> org.slf4j
> slf4j-log4j12
> 
> 
> log4j
> log4j
> 
> 
> 
> 
> org.slf4j
> log4j-over-slf4j
> 1.7.21
> 
>
>
> 
> 
> org.apache.storm
> storm-kafka
> 1.0.1
>  
> 
>
> 
>  
>
> 
>
>
> 
> org.apache.hadoop
> hadoop-hdfs
> 2.6.0
> 
> 
> ch.qos.logback
> logback-classic
> 
> 
> javax.servlet
> servlet-api
> 
> 
> 
>
>
> 
> 
> com.googlecode.json-simple
> json-simple
> 1.1
> 
>
>
> 
> 
> log4j
> log4j
> 1.2.17
> 
>
>
>
> 
> 
> 
>
>
>
>
> 
> 
> clojars.org
> http://clojars.org/repo
> 
> 
> 
>
>
> logs:
>
> 2017-03-27 21:50:36.572 STDERR [INFO] JMXetricAgent instrumented JVM,
> see https://github.com/ganglia/jmxetric
> 2017-03-27 21:50:39.302 STDERR [INFO] Mar 27, 2017 9:50:39 PM
> info.ganglia.gmetric4j.GMonitor start
> 2017-03-27 21:50:39.303 

Compaction - get compacted files

[Riccardo Iacomini]

Hello,
I have some questions about the compaction process. I need to manually
trigger compaction operations on a standard partitioned orc table (not
ACID), and be able to get back the list of compacted files. I could achieve
this via HDFS, getting the directory listing and then triggering the
compaction, but will imply stopping the underlying processing to avoid new
files to be added in between. Here are some questions I could not answer
myself from the material I found online:

   - Is the compaction executed as a MapReduce job?

   - Is there a way to get back the list of compacted files?

   - How can you customize the compaction criteria?

Also, any link to documentation/material is really appreciated.

Thank you all for your time.

Riccardo