voonhous commented on PR #14257:
URL: https://github.com/apache/hudi/pull/14257#issuecomment-5313498272
Pushed 2c330b6b8b22 on top of this branch (thanks for the feature). What
changed and why, in one place:
- **Nullable column.** `input_file_name()` is non-nullable, so `sourcePath`
became a required Avro field with no default and enabling the flag on an
existing table failed with `READER_FIELD_MISSING_DEFAULT_VALUE`
(`HoodieSchemaCompatibility.checkValidEvolution`, reached from
`deduceWriterSchema` on the default path). Now `withColumn(...,
when(input_file_name() != '', input_file_name()))`; verified nullable and
`["null","string"]` with `default: null`.
- **Name.** `sourcePath` -> `_hoodie_cloud_source_path` (public
`CloudObjectsSelectorCommon.CLOUD_SOURCE_PATH_COLUMN`), matching
`_hoodie_kafka_source_*` / `_hoodie_kinesis_source_*`.
- **Append + overwrite** via `withColumn` like the path-based partition
columns, instead of `selectExpr` prepend + a hard-fail collision guard. The
guard blocked the only way to keep the column when a schema provider is
configured: `Source.fetchNext` swaps in the configured provider and
`SourceFormatAdapter` rewrites rows to its schema, so the column must be
declared there (a declared-but-absent column is read as null and now
overwritten). Documented, plus a `log.warn` when it is missing.
- **Config.** `sinceVersion("1.3.0")`; doc names the column, the
percent-encoded URI value (`input_file_name()` yields
`s3a://b/dir/file%20name.json` for a key with a space, on every supported
Spark), overwrite semantics, the schema-provider caveat, and that disabling
later needs `hoodie.write.set.null.for.missing.columns`.
- Dropped the unrelated prefix refactor in `CloudSourceConfig` (keys were
byte-identical; 3 keys had been left un-refactored).
- **Tests.** Flag parameterization kept only on `partitionValueAddedToRow`
and `loadDatasetWithSchemaAndRepartition` (the latter covers the alias
`createDataFrame(rdd)` rebuild, repartition and 3 files); the two collision
tests + fixtures replaced by one `@TempDir` test pinning `%20`, the overwrite,
and the nullable-with-default Avro field. This harness starts a SparkContext
per invocation, so the suite goes 11 -> 14 instead of 19.
Could you update the PR description? The new config is user-facing and
changes the table schema, and the "S3 incremental keys reorganized" bullet no
longer applies. Approving once CI is green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
