Wenning Ding created HUDI-1627:
----------------------------------
Summary: DeltaStreamer does not work with SQLtransformation
Key: HUDI-1627
URL: https://issues.apache.org/jira/browse/HUDI-1627
Project: Apache Hudi
Issue Type: Bug
Reporter: Wenning Ding
Hudi deltastreamer , using SQLtransformation :
21/02/04 17:19:50 INFO SqlQueryBasedTransformer: SQL Query for transformation :
(select order_date from `default`.ordertest)
org.apache.spark.sql.AnalysisException: Table or view not found:
`default`.`ordertest`; line 1 pos 23;
’Project [’order_date]
+- ’UnresolvedRelation `default`.`ordertest`
This is because:
DeltaStreamer does not enable HiveSupport when initiating the SparkSession.
this.sparkSession = SparkSession.builder().config(jssc.getConf()).getOrCreate();
line 526 in
[https://github.com/a0x8o/hudi/blob/b8d0747959bc6f101b5b90b8e3ad323aafa2aa6e/hudi-u[…]rg/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java|https://github.com/a0x8o/hudi/blob/b8d0747959bc6f101b5b90b8e3ad323aafa2aa6e/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
