Vinaykumar Bhat created HUDI-7504:
-------------------------------------
Summary: Replace expensive file existance check (in object store)
with spark options
Key: HUDI-7504
URL: https://issues.apache.org/jira/browse/HUDI-7504
Project: Apache Hudi
Issue Type: Improvement
Reporter: Vinaykumar Bhat
The incremental loading from S3 and GCS performs a existence check for an
object. This is expensive. This happens
[here|[http://example.com|https://github.com/apache/hudi/blob/130498708bb1cd5da1d0e725971b3d721eeef231/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/CloudObjectsSelectorCommon.java#L161]]
This can be replaced with spark provided options:
spark.sql.files.ignoreMissingFiles
spark.sql.files.ignoreCorruptFiles
Ref for these options:
[https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html#ignore-missing-files]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
