[
https://issues.apache.org/jira/browse/HUDI-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931066#comment-16931066
]
Vinoth Chandar edited comment on HUDI-254 at 9/17/19 3:57 AM:
--------------------------------------------------------------
h3. Spark 2.4.4 on master
Also seems to pass all cases
{code:java}
root@adhoc-2:/opt# export
SPARK_INSTALL=/var/hoodie/ws/docker/spark-2.4.4-bin-hadoop2.7
root@adhoc-2:/opt# export
SPARK_HOME=/var/hoodie/ws/docker/spark-2.4.4-bin-hadoop2.7
root@adhoc-2:/opt# cp $HUDI_SPARK_BUNDLE $SPARK_INSTALL/jars/
root@adhoc-2:/opt# $SPARK_INSTALL/bin/spark-shell --master local[2]
--driver-class-path $HADOOP_CONF_DIR --conf
spark.sql.hive.convertMetastoreParquet=false --deploy-mode client
--driver-memory 1G --executor-memory 3G --num-executors 1
19/09/17 03:55:43 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Spark context Web UI available at http://adhoc-2:4040
Spark context available as 'sc' (master = local[2], app id =
local-1568692554280).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.scala> :paste
// Entering paste mode (ctrl-D to finish)
val jsonDF =
spark.read.json("file:////var/hoodie/ws/docker/demo/data/batch_1.json")import
org.apache.hudi.DataSourceReadOptions;
import org.apache.hudi.DataSourceWriteOptions;
import org.apache.spark.sql.SaveMode;
import org.apache.hudi.config.HoodieWriteConfig;
import org.apache.hudi.HoodieDataSourceHelpers;
import org.apache.hadoop.fs.FileSystem;jsonDF.write.format("org.apache.hudi").
option("hoodie.insert.shuffle.parallelism", "2").
option("hoodie.upsert.shuffle.parallelism","2").
option(DataSourceWriteOptions.STORAGE_TYPE_OPT_KEY,
DataSourceWriteOptions.MOR_STORAGE_TYPE_OPT_VAL).
option(DataSourceWriteOptions.OPERATION_OPT_KEY,
DataSourceWriteOptions.UPSERT_OPERATION_OPT_VAL).
option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "key").
option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY, "date").
option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "ts").
option(HoodieWriteConfig.TABLE_NAME, "stock_ticks_derived_mor").
option(DataSourceWriteOptions.HIVE_TABLE_OPT_KEY,
"stock_ticks_derived_mor").
option(DataSourceWriteOptions.HIVE_DATABASE_OPT_KEY, "default").
option(DataSourceWriteOptions.HIVE_URL_OPT_KEY,
"jdbc:hive2://hiveserver:10000").
option(DataSourceWriteOptions.HIVE_USER_OPT_KEY, "hive").
option(DataSourceWriteOptions.HIVE_PASS_OPT_KEY, "hive").
option(DataSourceWriteOptions.HIVE_SYNC_ENABLED_OPT_KEY, "true").
option(DataSourceWriteOptions.HIVE_PARTITION_FIELDS_OPT_KEY, "datestr").
mode(SaveMode.Append).
save("file:///tmp/stock_ticks_derived_mor");spark.read.format("org.apache.hudi").load("file:///tmp/stock_ticks_derived_mor/*/*/*/*.parquet").showspark.sql("select
symbol, max(ts) from stock_ticks_derived_mor group by symbol HAVING symbol =
'GOOG'").show(100, false)// Exiting paste mode, now interpreting.19/09/17
03:56:17 WARN compact.HoodieRealtimeTableCompactor: After filtering, Nothing to
compact for file:///tmp/stock_ticks_derived_mor
+-------------------+--------------------+------------------+----------------------+--------------------+------+----------+---+-------+------------------+------+-----+-------+------+-------------------+------+----+
|_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path|
_hoodie_file_name| close| date|day| high| key|
low|month| open|symbol| ts|volume|year|
+-------------------+--------------------+------------------+----------------------+--------------------+------+----------+---+-------+------------------+------+-----+-------+------+-------------------+------+----+
| 20190917035613| 20190917035613_0_1|NIHD_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...| 5.67|2018/08/31| 31| 5.67|NIHD_2018-08-31
10| 5.67| 08| 5.67| NIHD|2018-08-31 10:29:00| 2125|2018|
| 20190917035613| 20190917035613_0_2|CDMO_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 7.07|2018/08/31| 31| 7.07|CDMO_2018-08-31
09| 7.02| 08| 7.02| CDMO|2018-08-31 09:58:00| 8577|2018|
| 20190917035613| 20190917035613_0_3|MRTX_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...| 56.15|2018/08/31| 31| 56.15|MRTX_2018-08-31
10| 56.15| 08| 56.15| MRTX|2018-08-31 10:28:00| 1844|2018|
| 20190917035613| 20190917035613_0_4|NTRA_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 27.45|2018/08/31| 31| 27.45|NTRA_2018-08-31
09| 27.45| 08| 27.45| NTRA|2018-08-31 09:59:00| 1326|2018|
| 20190917035613| 20190917035613_0_5|DRNA_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 16.0|2018/08/31| 31| 16.0|DRNA_2018-08-31
09| 16.0| 08| 16.0| DRNA|2018-08-31 09:51:00| 334|2018|
| 20190917035613| 20190917035613_0_6|STAA_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 47.95|2018/08/31| 31| 47.95|STAA_2018-08-31
09| 47.95| 08| 47.95| STAA|2018-08-31 09:59:00| 1709|2018|
| 20190917035613| 20190917035613_0_7|LGCY_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 5.37|2018/08/31| 31| 5.37|LGCY_2018-08-31
09| 5.37| 08| 5.37| LGCY|2018-08-31 09:54:00| 2830|2018|
| 20190917035613| 20190917035613_0_8| MDB_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...|71.349|2018/08/31| 31| 71.349| MDB_2018-08-31
09|71.349| 08| 71.349| MDB|2018-08-31 09:59:00| 880|2018|
| 20190917035613| 20190917035613_0_9|TRHC_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...| 86.95|2018/08/31| 31| 86.95|TRHC_2018-08-31
10| 86.95| 08| 86.95| TRHC|2018-08-31 10:29:00| 110|2018|
| 20190917035613| 20190917035613_0_10|BAND_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 45.08|2018/08/31| 31| 45.08|BAND_2018-08-31
09| 45.08| 08| 45.08| BAND|2018-08-31 09:53:00| 305|2018|
| 20190917035613| 20190917035613_0_11| MDB_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...| 71.64|2018/08/31| 31| 71.64| MDB_2018-08-31
10| 71.64| 08| 71.64| MDB|2018-08-31 10:26:00| 286|2018|
| 20190917035613| 20190917035613_0_12| GFN_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...| 13.45|2018/08/31| 31| 13.45| GFN_2018-08-31
10| 13.45| 08| 13.45| GFN|2018-08-31 10:21:00| 200|2018|
| 20190917035613| 20190917035613_0_13|NEPT_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 4.0|2018/08/31| 31| 4.0|NEPT_2018-08-31
09| 4.0| 08| 4.0| NEPT|2018-08-31 09:58:00| 2609|2018|
| 20190917035613| 20190917035613_0_14|NIHD_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 5.69|2018/08/31| 31| 5.69|NIHD_2018-08-31
09| 5.69| 08| 5.69| NIHD|2018-08-31 09:59:00| 926|2018|
| 20190917035613| 20190917035613_0_15|SRPT_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...|137.74|2018/08/31| 31| 138.0|SRPT_2018-08-31
10|137.74| 08| 138.0| SRPT|2018-08-31 10:28:00| 2985|2018|
| 20190917035613| 20190917035613_0_16|RGNX_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...| 71.1|2018/08/31| 31| 71.1|RGNX_2018-08-31
10| 71.1| 08| 71.1| RGNX|2018-08-31 10:28:00| 1676|2018|
| 20190917035613| 20190917035613_0_17|NVCR_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...| 44.9|2018/08/31| 31| 44.9|NVCR_2018-08-31
10| 44.9| 08| 44.9| NVCR|2018-08-31 10:29:00| 5157|2018|
| 20190917035613| 20190917035613_0_18|SRDX_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 76.4|2018/08/31| 31| 76.4|SRDX_2018-08-31
09| 76.4| 08| 76.4| SRDX|2018-08-31 09:58:00| 1023|2018|
| 20190917035613| 20190917035613_0_19|STAA_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...| 47.5|2018/08/31| 31| 47.5|STAA_2018-08-31
10| 47.5| 08| 47.5| STAA|2018-08-31 10:28:00| 800|2018|
| 20190917035613| 20190917035613_0_20|EGAN_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 14.5|2018/08/31| 31|14.5999|EGAN_2018-08-31
09| 14.5| 08|14.5999| EGAN|2018-08-31 09:57:00| 2489|2018|
+-------------------+--------------------+------------------+----------------------+--------------------+------+----------+---+-------+------------------+------+-----+-------+------+-------------------+------+----+
only showing top 20 rowsSLF4J: Failed to load class
"org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
19/09/17 03:56:26 WARN hadoop.ParquetRecordReader: Can not initialize counter
due to context is not a instance of TaskInputOutputContext, but is
org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+------+-------------------+
|symbol|max(ts) |
+------+-------------------+
|GOOG |2018-08-31 10:29:00|
+------+-------------------+jsonDF: org.apache.spark.sql.DataFrame = [close:
double, date: string ... 10 more fields]
import org.apache.hudi.DataSourceReadOptions
import org.apache.hudi.DataSourceWriteOptions
import org.apache.spark.sql.SaveMode
import org.apache.hudi.config.HoodieWriteConfig
import org.apache.hudi.HoodieDataSourceHelpers
import org.apache.hadoop.fs.FileSystemscala> {code}
was (Author: vc):
Spark 2.4.4 on master
Also seems to pass all cases
{code:java}
root@adhoc-2:/opt# export
SPARK_INSTALL=/var/hoodie/ws/docker/spark-2.4.4-bin-hadoop2.7
root@adhoc-2:/opt# export
SPARK_HOME=/var/hoodie/ws/docker/spark-2.4.4-bin-hadoop2.7
root@adhoc-2:/opt# cp $HUDI_SPARK_BUNDLE $SPARK_INSTALL/jars/
root@adhoc-2:/opt# $SPARK_INSTALL/bin/spark-shell --master local[2]
--driver-class-path $HADOOP_CONF_DIR --conf
spark.sql.hive.convertMetastoreParquet=false --deploy-mode client
--driver-memory 1G --executor-memory 3G --num-executors 1
19/09/17 03:55:43 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Spark context Web UI available at http://adhoc-2:4040
Spark context available as 'sc' (master = local[2], app id =
local-1568692554280).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.scala> :paste
// Entering paste mode (ctrl-D to finish)
val jsonDF =
spark.read.json("file:////var/hoodie/ws/docker/demo/data/batch_1.json")import
org.apache.hudi.DataSourceReadOptions;
import org.apache.hudi.DataSourceWriteOptions;
import org.apache.spark.sql.SaveMode;
import org.apache.hudi.config.HoodieWriteConfig;
import org.apache.hudi.HoodieDataSourceHelpers;
import org.apache.hadoop.fs.FileSystem;jsonDF.write.format("org.apache.hudi").
option("hoodie.insert.shuffle.parallelism", "2").
option("hoodie.upsert.shuffle.parallelism","2").
option(DataSourceWriteOptions.STORAGE_TYPE_OPT_KEY,
DataSourceWriteOptions.MOR_STORAGE_TYPE_OPT_VAL).
option(DataSourceWriteOptions.OPERATION_OPT_KEY,
DataSourceWriteOptions.UPSERT_OPERATION_OPT_VAL).
option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "key").
option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY, "date").
option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "ts").
option(HoodieWriteConfig.TABLE_NAME, "stock_ticks_derived_mor").
option(DataSourceWriteOptions.HIVE_TABLE_OPT_KEY,
"stock_ticks_derived_mor").
option(DataSourceWriteOptions.HIVE_DATABASE_OPT_KEY, "default").
option(DataSourceWriteOptions.HIVE_URL_OPT_KEY,
"jdbc:hive2://hiveserver:10000").
option(DataSourceWriteOptions.HIVE_USER_OPT_KEY, "hive").
option(DataSourceWriteOptions.HIVE_PASS_OPT_KEY, "hive").
option(DataSourceWriteOptions.HIVE_SYNC_ENABLED_OPT_KEY, "true").
option(DataSourceWriteOptions.HIVE_PARTITION_FIELDS_OPT_KEY, "datestr").
mode(SaveMode.Append).
save("file:///tmp/stock_ticks_derived_mor");spark.read.format("org.apache.hudi").load("file:///tmp/stock_ticks_derived_mor/*/*/*/*.parquet").showspark.sql("select
symbol, max(ts) from stock_ticks_derived_mor group by symbol HAVING symbol =
'GOOG'").show(100, false)// Exiting paste mode, now interpreting.19/09/17
03:56:17 WARN compact.HoodieRealtimeTableCompactor: After filtering, Nothing to
compact for file:///tmp/stock_ticks_derived_mor
+-------------------+--------------------+------------------+----------------------+--------------------+------+----------+---+-------+------------------+------+-----+-------+------+-------------------+------+----+
|_hoodie_commit_time|_hoodie_commit_seqno|_hoodie_record_key|_hoodie_partition_path|
_hoodie_file_name| close| date|day| high| key|
low|month| open|symbol| ts|volume|year|
+-------------------+--------------------+------------------+----------------------+--------------------+------+----------+---+-------+------------------+------+-----+-------+------+-------------------+------+----+
| 20190917035613| 20190917035613_0_1|NIHD_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...| 5.67|2018/08/31| 31| 5.67|NIHD_2018-08-31
10| 5.67| 08| 5.67| NIHD|2018-08-31 10:29:00| 2125|2018|
| 20190917035613| 20190917035613_0_2|CDMO_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 7.07|2018/08/31| 31| 7.07|CDMO_2018-08-31
09| 7.02| 08| 7.02| CDMO|2018-08-31 09:58:00| 8577|2018|
| 20190917035613| 20190917035613_0_3|MRTX_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...| 56.15|2018/08/31| 31| 56.15|MRTX_2018-08-31
10| 56.15| 08| 56.15| MRTX|2018-08-31 10:28:00| 1844|2018|
| 20190917035613| 20190917035613_0_4|NTRA_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 27.45|2018/08/31| 31| 27.45|NTRA_2018-08-31
09| 27.45| 08| 27.45| NTRA|2018-08-31 09:59:00| 1326|2018|
| 20190917035613| 20190917035613_0_5|DRNA_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 16.0|2018/08/31| 31| 16.0|DRNA_2018-08-31
09| 16.0| 08| 16.0| DRNA|2018-08-31 09:51:00| 334|2018|
| 20190917035613| 20190917035613_0_6|STAA_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 47.95|2018/08/31| 31| 47.95|STAA_2018-08-31
09| 47.95| 08| 47.95| STAA|2018-08-31 09:59:00| 1709|2018|
| 20190917035613| 20190917035613_0_7|LGCY_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 5.37|2018/08/31| 31| 5.37|LGCY_2018-08-31
09| 5.37| 08| 5.37| LGCY|2018-08-31 09:54:00| 2830|2018|
| 20190917035613| 20190917035613_0_8| MDB_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...|71.349|2018/08/31| 31| 71.349| MDB_2018-08-31
09|71.349| 08| 71.349| MDB|2018-08-31 09:59:00| 880|2018|
| 20190917035613| 20190917035613_0_9|TRHC_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...| 86.95|2018/08/31| 31| 86.95|TRHC_2018-08-31
10| 86.95| 08| 86.95| TRHC|2018-08-31 10:29:00| 110|2018|
| 20190917035613| 20190917035613_0_10|BAND_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 45.08|2018/08/31| 31| 45.08|BAND_2018-08-31
09| 45.08| 08| 45.08| BAND|2018-08-31 09:53:00| 305|2018|
| 20190917035613| 20190917035613_0_11| MDB_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...| 71.64|2018/08/31| 31| 71.64| MDB_2018-08-31
10| 71.64| 08| 71.64| MDB|2018-08-31 10:26:00| 286|2018|
| 20190917035613| 20190917035613_0_12| GFN_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...| 13.45|2018/08/31| 31| 13.45| GFN_2018-08-31
10| 13.45| 08| 13.45| GFN|2018-08-31 10:21:00| 200|2018|
| 20190917035613| 20190917035613_0_13|NEPT_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 4.0|2018/08/31| 31| 4.0|NEPT_2018-08-31
09| 4.0| 08| 4.0| NEPT|2018-08-31 09:58:00| 2609|2018|
| 20190917035613| 20190917035613_0_14|NIHD_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 5.69|2018/08/31| 31| 5.69|NIHD_2018-08-31
09| 5.69| 08| 5.69| NIHD|2018-08-31 09:59:00| 926|2018|
| 20190917035613| 20190917035613_0_15|SRPT_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...|137.74|2018/08/31| 31| 138.0|SRPT_2018-08-31
10|137.74| 08| 138.0| SRPT|2018-08-31 10:28:00| 2985|2018|
| 20190917035613| 20190917035613_0_16|RGNX_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...| 71.1|2018/08/31| 31| 71.1|RGNX_2018-08-31
10| 71.1| 08| 71.1| RGNX|2018-08-31 10:28:00| 1676|2018|
| 20190917035613| 20190917035613_0_17|NVCR_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...| 44.9|2018/08/31| 31| 44.9|NVCR_2018-08-31
10| 44.9| 08| 44.9| NVCR|2018-08-31 10:29:00| 5157|2018|
| 20190917035613| 20190917035613_0_18|SRDX_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 76.4|2018/08/31| 31| 76.4|SRDX_2018-08-31
09| 76.4| 08| 76.4| SRDX|2018-08-31 09:58:00| 1023|2018|
| 20190917035613| 20190917035613_0_19|STAA_2018-08-31 10|
2018/08/31|36cf66f4-8980-4fd...| 47.5|2018/08/31| 31| 47.5|STAA_2018-08-31
10| 47.5| 08| 47.5| STAA|2018-08-31 10:28:00| 800|2018|
| 20190917035613| 20190917035613_0_20|EGAN_2018-08-31 09|
2018/08/31|36cf66f4-8980-4fd...| 14.5|2018/08/31| 31|14.5999|EGAN_2018-08-31
09| 14.5| 08|14.5999| EGAN|2018-08-31 09:57:00| 2489|2018|
+-------------------+--------------------+------------------+----------------------+--------------------+------+----------+---+-------+------------------+------+-----+-------+------+-------------------+------+----+
only showing top 20 rowsSLF4J: Failed to load class
"org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
19/09/17 03:56:26 WARN hadoop.ParquetRecordReader: Can not initialize counter
due to context is not a instance of TaskInputOutputContext, but is
org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+------+-------------------+
|symbol|max(ts) |
+------+-------------------+
|GOOG |2018-08-31 10:29:00|
+------+-------------------+jsonDF: org.apache.spark.sql.DataFrame = [close:
double, date: string ... 10 more fields]
import org.apache.hudi.DataSourceReadOptions
import org.apache.hudi.DataSourceWriteOptions
import org.apache.spark.sql.SaveMode
import org.apache.hudi.config.HoodieWriteConfig
import org.apache.hudi.HoodieDataSourceHelpers
import org.apache.hadoop.fs.FileSystemscala> {code}
> Provide mechanism for installing hudi-spark-bundle onto an existing spark
> installation
> --------------------------------------------------------------------------------------
>
> Key: HUDI-254
> URL: https://issues.apache.org/jira/browse/HUDI-254
> Project: Apache Hudi (incubating)
> Issue Type: Improvement
> Components: Spark datasource, SparkSQL Support
> Reporter: Vinoth Chandar
> Assignee: Vinoth Chandar
> Priority: Major
>
> A lot of discussions around this kicked off from
> [https://github.com/apache/incubator-hudi/issues/869]
> Breaking down into phases, when we drop the hudi-spark-bundle*.jar onto the
> `jars` folder
>
> a) Writing data via Hudi datasource should work
> b) Spark datasource reads should work
>
> c) a + Hive Sync should work
> d) SparkSQL on Hive synced table works
>
> Start with Spark 2.3 (current demo setup) and then proceed to 2.4 and iron
> out issues.
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
