[ 
https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15505596#comment-15505596
 ] 

Hemanth Yamijala commented on ATLAS-1145:
-----------------------------------------

[~sarath.ku...@gmail.com], Some time back I had a working version of Kafka to 
Hive storm topology capturing lineage into Atlas etc. Code is here: 
https://github.com/yhemanth/storm-samples/blob/master/src/main/java/com/dsinpractice/storm/samples/WordCountTopology.java
 However, we shifted focus from Hive to HDFS for later testing.

The main issue AFAIK is to setup Hive for streaming accurately. It took some 
time for me to figure that out.  Wish I had documented the steps more clearly 
:-(

These are properties I have in my hive-site which *I think* were used to get 
these to work. Please use these in case they are of help.

{code}
    <property>
        <name>hive.support.concurrency</name>
        <value>true</value>
    </property>
    <property>
        <name>hive.enforce.bucketing</name>
        <value>true</value>
    </property>
    <property>
        <name>hive.exec.dynamic.partition.mode</name>
        <value>nonstrict</value>
    </property>
    <property>
        <name>hive.compactor.initiator.on</name>
        <value>true</value>
    </property>
    <property>
        <name>hive.txn.manager</name>
        <value>org.apache.hadoop.hive.ql.lockmgr.DbTxnManager</value>
    </property>
    <property>
        <name>hive.compactor.initiator.on</name>
        <value>true</value> 
    </property>
    <property>
        <name>hive.compactor.worker.threads</name>
        <value>1</value> 
    </property>
{code}

> Storm hook with kafka topic and hive bolt picks wrong hive table name 
> ----------------------------------------------------------------------
>
>                 Key: ATLAS-1145
>                 URL: https://issues.apache.org/jira/browse/ATLAS-1145
>             Project: Atlas
>          Issue Type: Bug
>            Reporter: Shwetha G S
>            Assignee: Sarath Subramanian
>
> Storm hook with kafka topic and hive bolt picks kafka topic name as hive 
> table name



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to