[
https://issues.apache.org/jira/browse/HUDI-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen resolved HUDI-2880.
------------------------------
> Error reading in properties from dfs in Spark Shell
> ---------------------------------------------------
>
> Key: HUDI-2880
> URL: https://issues.apache.org/jira/browse/HUDI-2880
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Ethan Guo
> Assignee: Wenning Ding
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> I encountered the following warnings and an error when using spark datasource
> in spark shell, the write is successful though:
> {code:java}
> scala> val df = spark.read.json(spark.sparkContext.parallelize(inserts, 2))
> warning: there was one deprecation warning (since 2.12.0)
> warning: there was one deprecation warning (since 2.2.0)
> warning: there were two deprecation warnings in total; for details, enable
> `:setting -deprecation' or `:replay -deprecation'
> df: org.apache.spark.sql.DataFrame = [begin_lat: double, begin_lon: double
> ... 8 more fields]
> scala>
> scala> df.write.format("hudi").
> | option("hoodie.insert.shuffle.parallelism", "2").
> | option("hoodie.upsert.shuffle.parallelism", "2").
> | option("hoodie.bulkinsert.shuffle.parallelism", "2").
> | option("hoodie.delete.shuffle.parallelism", "2").
> | option(PRECOMBINE_FIELD_OPT_KEY, "ts").
> | option(RECORDKEY_FIELD_OPT_KEY, "uuid").
> | option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
> | option(TABLE_NAME, tableName).
> | option("hoodie.parquet.small.file.limit", "0").
> | option("hoodie.clustering.inline", "true").
> | option("hoodie.clustering.inline.max.commits", "2").
> | option("hoodie.clustering.plan.strategy.target.file.max.bytes",
> "1073741824").
> | option("hoodie.clustering.plan.strategy.small.file.limit",
> "629145600").
> | option("hoodie.clustering.plan.strategy.sort.columns",
> "rider,driver").
> | option("hoodie.layout.optimize.enable", "true").
> | mode(Append).
> | save(basePath)
> warning: there was one deprecation warning; for details, enable `:setting
> -deprecation' or `:replay -deprecation'
> 21/11/26 20:53:41 WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR,
> please set it as the dir of hudi-defaults.conf
> 21/11/26 20:53:41 ERROR DFSPropertiesConfiguration: Error reading in
> properties from dfs
> 21/11/26 20:53:41 WARN DFSPropertiesConfiguration: Didn't find config file
> under default conf file dir: file:/etc/hudi/conf {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
