[jira] [Updated] (HUDI-484) NPE in HiveIncrementalPuller

2020-01-03 Thread lamber-ken (Jira) 


 [ 
https://issues.apache.org/jira/browse/HUDI-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

lamber-ken updated HUDI-484:

Parent: HUDI-498
Issue Type: Sub-task  (was: Bug)

> NPE in HiveIncrementalPuller
> 
>
> Key: HUDI-484
> URL: https://issues.apache.org/jira/browse/HUDI-484
> Project: Apache Hudi (incubating)
>  Issue Type: Sub-task
>  Components: Incremental Pull
>Reporter: Pratyaksh Sharma
>Assignee: lamber-ken
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.1
>
> Attachments: Screenshot 2019-12-30 at 4.43.51 PM.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When we try to use HiveIncrementalPuller class to incrementally pull changes 
> from hive, it throws NPE as it is unable to find IncrementalPull.sqltemplate 
> in the bundled jar. 
> Screenshot attached which shows the exception. 
> The jar contains the template. 
> Steps to reproduce - 
>  # copy hive-jdbc-2.3.1.jar, log4j-1.2.17.jar to docker/demo/config folder
>  # run cd docker && ./setup_demo.sh
>  # cat docker/demo/data/batch_1.json | kafkacat -b kafkabroker -t stock_ticks 
> -P
>  #  {{docker exec -it adhoc-2 /bin/bash}}
>  #  {{spark-submit --class 
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer 
> $HUDI_UTILITIES_BUNDLE --storage-type COPY_ON_WRITE --source-class 
> org.apache.hudi.utilities.sources.JsonKafkaSource --source-ordering-field ts 
> --target-base-path /user/hive/warehouse/stock_ticks_cow --target-table 
> stock_ticks_cow --props /var/demo/config/kafka-source.properties 
> --schemaprovider-class 
> org.apache.hudi.utilities.schema.FilebasedSchemaProvider}}
>  #  {{/var/hoodie/ws/hudi-hive/run_sync_tool.sh --jdbc-url 
> jdbc:hive2://hiveserver:1 --user hive --pass hive --partitioned-by dt 
> --base-path /user/hive/warehouse/stock_ticks_cow --database default --table 
> stock_ticks_cow}}
>  # java -cp 
> /var/hoodie/ws/docker/demo/config/hive-jdbc-2.3.1.jar:/var/hoodie/ws/docker/demo/config/log4j-1.2.17.jar:$HUDI_UTILITIES_BUNDLE
>  org.apache.hudi.utilities.HiveIncrementalPuller --hiveUrl 
> jdbc:hive2://hiveserver:1 --hiveUser hive --hivePass hive 
> --extractSQLFile /var/hoodie/ws/docker/demo/config/incr_pull.txt --sourceDb 
> default --sourceTable stock_ticks_cow --targetDb tmp --targetTable tempTable 
> --fromCommitTime 0 --maxCommits 1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HUDI-484) NPE in HiveIncrementalPuller

2019-12-31 Thread Vinoth Chandar (Jira) 


 [ 
https://issues.apache.org/jira/browse/HUDI-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinoth Chandar updated HUDI-484:

Status: Open  (was: New)

> NPE in HiveIncrementalPuller
> 
>
> Key: HUDI-484
> URL: https://issues.apache.org/jira/browse/HUDI-484
> Project: Apache Hudi (incubating)
>  Issue Type: Bug
>  Components: Incremental Pull
>Reporter: Pratyaksh Sharma
>Assignee: Pratyaksh Sharma
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.1
>
> Attachments: Screenshot 2019-12-30 at 4.43.51 PM.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When we try to use HiveIncrementalPuller class to incrementally pull changes 
> from hive, it throws NPE as it is unable to find IncrementalPull.sqltemplate 
> in the bundled jar. 
> Screenshot attached which shows the exception. 
> The jar contains the template. 
> Steps to reproduce - 
>  # copy hive-jdbc-2.3.1.jar, log4j-1.2.17.jar to docker/demo/config folder
>  # run cd docker && ./setup_demo.sh
>  # cat docker/demo/data/batch_1.json | kafkacat -b kafkabroker -t stock_ticks 
> -P
>  #  {{docker exec -it adhoc-2 /bin/bash}}
>  #  {{spark-submit --class 
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer 
> $HUDI_UTILITIES_BUNDLE --storage-type COPY_ON_WRITE --source-class 
> org.apache.hudi.utilities.sources.JsonKafkaSource --source-ordering-field ts 
> --target-base-path /user/hive/warehouse/stock_ticks_cow --target-table 
> stock_ticks_cow --props /var/demo/config/kafka-source.properties 
> --schemaprovider-class 
> org.apache.hudi.utilities.schema.FilebasedSchemaProvider}}
>  #  {{/var/hoodie/ws/hudi-hive/run_sync_tool.sh --jdbc-url 
> jdbc:hive2://hiveserver:1 --user hive --pass hive --partitioned-by dt 
> --base-path /user/hive/warehouse/stock_ticks_cow --database default --table 
> stock_ticks_cow}}
>  # java -cp 
> /var/hoodie/ws/docker/demo/config/hive-jdbc-2.3.1.jar:/var/hoodie/ws/docker/demo/config/log4j-1.2.17.jar:$HUDI_UTILITIES_BUNDLE
>  org.apache.hudi.utilities.HiveIncrementalPuller --hiveUrl 
> jdbc:hive2://hiveserver:1 --hiveUser hive --hivePass hive 
> --extractSQLFile /var/hoodie/ws/docker/demo/config/incr_pull.txt --sourceDb 
> default --sourceTable stock_ticks_cow --targetDb tmp --targetTable tempTable 
> --fromCommitTime 0 --maxCommits 1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HUDI-484) NPE in HiveIncrementalPuller

2019-12-31 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HUDI-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HUDI-484:

Labels: pull-request-available  (was: )

> NPE in HiveIncrementalPuller
> 
>
> Key: HUDI-484
> URL: https://issues.apache.org/jira/browse/HUDI-484
> Project: Apache Hudi (incubating)
>  Issue Type: Bug
>  Components: Incremental Pull
>Reporter: Pratyaksh Sharma
>Assignee: Pratyaksh Sharma
>Priority: Major
>  Labels: pull-request-available
> Fix For: 0.5.1
>
> Attachments: Screenshot 2019-12-30 at 4.43.51 PM.png
>
>
> When we try to use HiveIncrementalPuller class to incrementally pull changes 
> from hive, it throws NPE as it is unable to find IncrementalPull.sqltemplate 
> in the bundled jar. 
> Screenshot attached which shows the exception. 
> The jar contains the template. 
> Steps to reproduce - 
>  # copy hive-jdbc-2.3.1.jar, log4j-1.2.17.jar to docker/demo/config folder
>  # run cd docker && ./setup_demo.sh
>  # cat docker/demo/data/batch_1.json | kafkacat -b kafkabroker -t stock_ticks 
> -P
>  #  {{docker exec -it adhoc-2 /bin/bash}}
>  #  {{spark-submit --class 
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer 
> $HUDI_UTILITIES_BUNDLE --storage-type COPY_ON_WRITE --source-class 
> org.apache.hudi.utilities.sources.JsonKafkaSource --source-ordering-field ts 
> --target-base-path /user/hive/warehouse/stock_ticks_cow --target-table 
> stock_ticks_cow --props /var/demo/config/kafka-source.properties 
> --schemaprovider-class 
> org.apache.hudi.utilities.schema.FilebasedSchemaProvider}}
>  #  {{/var/hoodie/ws/hudi-hive/run_sync_tool.sh --jdbc-url 
> jdbc:hive2://hiveserver:1 --user hive --pass hive --partitioned-by dt 
> --base-path /user/hive/warehouse/stock_ticks_cow --database default --table 
> stock_ticks_cow}}
>  # java -cp 
> /var/hoodie/ws/docker/demo/config/hive-jdbc-2.3.1.jar:/var/hoodie/ws/docker/demo/config/log4j-1.2.17.jar:$HUDI_UTILITIES_BUNDLE
>  org.apache.hudi.utilities.HiveIncrementalPuller --hiveUrl 
> jdbc:hive2://hiveserver:1 --hiveUser hive --hivePass hive 
> --extractSQLFile /var/hoodie/ws/docker/demo/config/incr_pull.txt --sourceDb 
> default --sourceTable stock_ticks_cow --targetDb tmp --targetTable tempTable 
> --fromCommitTime 0 --maxCommits 1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HUDI-484) NPE in HiveIncrementalPuller

2019-12-30 Thread Pratyaksh Sharma (Jira) 


 [ 
https://issues.apache.org/jira/browse/HUDI-484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pratyaksh Sharma updated HUDI-484:
--
Description: 
When we try to use HiveIncrementalPuller class to incrementally pull changes 
from hive, it throws NPE as it is unable to find IncrementalPull.sqltemplate in 
the bundled jar. 

Screenshot attached which shows the exception. 

The jar contains the template. 

Steps to reproduce - 
 # copy hive-jdbc-2.3.1.jar, log4j-1.2.17.jar to docker/demo/config folder
 # run cd docker && ./setup_demo.sh
 # cat docker/demo/data/batch_1.json | kafkacat -b kafkabroker -t stock_ticks -P
 #  {{docker exec -it adhoc-2 /bin/bash}}
 #  {{spark-submit --class 
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer 
$HUDI_UTILITIES_BUNDLE --storage-type COPY_ON_WRITE --source-class 
org.apache.hudi.utilities.sources.JsonKafkaSource --source-ordering-field ts 
--target-base-path /user/hive/warehouse/stock_ticks_cow --target-table 
stock_ticks_cow --props /var/demo/config/kafka-source.properties 
--schemaprovider-class 
org.apache.hudi.utilities.schema.FilebasedSchemaProvider}}
 #  {{/var/hoodie/ws/hudi-hive/run_sync_tool.sh --jdbc-url 
jdbc:hive2://hiveserver:1 --user hive --pass hive --partitioned-by dt 
--base-path /user/hive/warehouse/stock_ticks_cow --database default --table 
stock_ticks_cow}}
 # java -cp 
/var/hoodie/ws/docker/demo/config/hive-jdbc-2.3.1.jar:/var/hoodie/ws/docker/demo/config/log4j-1.2.17.jar:$HUDI_UTILITIES_BUNDLE
 org.apache.hudi.utilities.HiveIncrementalPuller --hiveUrl 
jdbc:hive2://hiveserver:1 --hiveUser hive --hivePass hive --extractSQLFile 
/var/hoodie/ws/docker/demo/config/incr_pull.txt --sourceDb default 
--sourceTable stock_ticks_cow --targetDb tmp --targetTable tempTable 
--fromCommitTime 0 --maxCommits 1

  was:
When we try to use HiveIncrementalPuller class to incrementally pull changes 
from hive, it throws NPE as it is unable to find IncrementalPull.sqltemplate in 
the bundled jar. 

Screenshot attached which shows the exception. 

The jar contains the template. 

Steps to reproduce - 
 # copy hive-jdbc-2.3.1.jar, log4j-1.2.17.jar to docker/demo/config folder
 # run cd docker && ./setup_demo.sh
 # cat docker/demo/data/batch_1.json | kafkacat -b kafkabroker -t stock_ticks -P
 #  {{docker exec -it adhoc-2 /bin/bash}}
 #  {{spark-submit --class 
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer 
$HUDI_UTILITIES_BUNDLE --storage-type COPY_ON_WRITE --source-class 
org.apache.hudi.utilities.sources.JsonKafkaSource --source-ordering-field ts  
--target-base-path /user/hive/warehouse/stock_ticks_cow --target-table 
stock_ticks_cow --props /var/demo/config/kafka-source.properties 
--schemaprovider-class 
org.apache.hudi.utilities.schema.FilebasedSchemaProvider}}
 #  {{/var/hoodie/ws/hudi-hive/run_sync_tool.sh  --jdbc-url 
jdbc:hive2://hiveserver:1 --user hive --pass hive --partitioned-by dt 
--base-path /user/hive/warehouse/stock_ticks_cow --database default --table 
stock_ticks_cow}}
 # java -cp 
/var/hoodie/ws/docker/demo/config/hive-jdbc-2.3.1.jar:/var/hoodie/ws/docker/demo/config/log4j-1.2.17.jar:$HUDI_UTILITIES_BUNDLE
 org.apache.hudi.utilities.HiveIncrementalPuller --hiveUrl 
jdbc:hive2://hiveserver:1 --hiveUser hive --hivePass hive --extractSQLFile 
/var/hoodie/ws/docker/demo/config/incr_pull.txt --sourceDb default 
--sourceTable stock_ticks_cow --targetDb tmp --targetTable tempTable 
--fromCommitTime 0 --maxCommits 1{{}}


> NPE in HiveIncrementalPuller
> 
>
> Key: HUDI-484
> URL: https://issues.apache.org/jira/browse/HUDI-484
> Project: Apache Hudi (incubating)
>  Issue Type: Bug
>  Components: Incremental Pull
>Reporter: Pratyaksh Sharma
>Priority: Major
> Fix For: 0.5.1
>
> Attachments: Screenshot 2019-12-30 at 4.43.51 PM.png
>
>
> When we try to use HiveIncrementalPuller class to incrementally pull changes 
> from hive, it throws NPE as it is unable to find IncrementalPull.sqltemplate 
> in the bundled jar. 
> Screenshot attached which shows the exception. 
> The jar contains the template. 
> Steps to reproduce - 
>  # copy hive-jdbc-2.3.1.jar, log4j-1.2.17.jar to docker/demo/config folder
>  # run cd docker && ./setup_demo.sh
>  # cat docker/demo/data/batch_1.json | kafkacat -b kafkabroker -t stock_ticks 
> -P
>  #  {{docker exec -it adhoc-2 /bin/bash}}
>  #  {{spark-submit --class 
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer 
> $HUDI_UTILITIES_BUNDLE --storage-type COPY_ON_WRITE --source-class 
> org.apache.hudi.utilities.sources.JsonKafkaSource --source-ordering-field ts 
> --target-base-path /user/hive/warehouse/stock_ticks_cow --target-table 
> stock_ticks_cow --props /var/demo/config/kafka-source.properties 
> --schemaprovider-class 
> org.apache.hudi.utilities.schema.FilebasedSchemaProvider}}
>