This is an automated email from the ASF dual-hosted git repository.
danny0405 pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 4960e61f6ca Revert [HUDI-6438] Config parameter
'MAKE_NEW_COLUMNS_NULLABLE' (#10783)
4960e61f6ca is described below
commit 4960e61f6ca25a6f636b9256542bb3594583eb65
Author: Geser Dugarov, PhD <[email protected]>
AuthorDate: Fri Mar 1 07:52:46 2024 +0700
Revert [HUDI-6438] Config parameter 'MAKE_NEW_COLUMNS_NULLABLE' (#10783)
---
website/docs/configurations.md | 2 --
1 file changed, 2 deletions(-)
diff --git a/website/docs/configurations.md b/website/docs/configurations.md
index e52f0a52a75..0f0ba7df7e7 100644
--- a/website/docs/configurations.md
+++ b/website/docs/configurations.md
@@ -161,7 +161,6 @@ Options useful for writing tables via
`write.format.option(...)`
|
[hoodie.datasource.write.insert.drop.duplicates](#hoodiedatasourcewriteinsertdropduplicates)
| false
| If set to true, records from the incoming
dataframe will not overwrite existing records with the same key during the
write operation. <br /> **Note** Just for Insert operation in Spark SQL writing
since 0.14.0, users can switch to the config
`hoodie.datasource.insert.dup.policy` [...]
|
[hoodie.datasource.write.keygenerator.class](#hoodiedatasourcewritekeygeneratorclass)
|
org.apache.hudi.keygen.SimpleKeyGenerator | Key generator
class, that implements `org.apache.hudi.keygen.KeyGenerator`<br />`Config
Param: KEYGENERATOR_CLASS_NAME`
[...]
|
[hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled](#hoodiedatasourcewritekeygeneratorconsistentlogicaltimestampenabled)
| false | When set to
true, consistent value will be generated for a logical timestamp type column,
like timestamp-millis and timestamp-micros, irrespective of whether row-writer
is enabled. Disabled by default so as not to break the pipeline that deploy
either fully row-writer path or non [...]
-|
[hoodie.datasource.write.new.columns.nullable](#hoodiedatasourcewritenewcolumnsnullable)
| false
| When a non-nullable column is added
to datasource during a write operation, the write operation will fail schema
compatibility check. Set this option to true will make the newly added column
nullable to successfully complete the write operation.<br />`Config Param:
MAKE_NEW_COL [...]
|
[hoodie.datasource.write.partitionpath.urlencode](#hoodiedatasourcewritepartitionpathurlencode)
| false
| Should we url encode the partition path
value, before creating the folder structure.<br />`Config Param:
URL_ENCODE_PARTITIONING`
[...]
| [hoodie.datasource.write.payload.class](#hoodiedatasourcewritepayloadclass)
|
org.apache.hudi.common.model.OverwriteWithLatestAvroPayload | Payload class
used. Override this, if you like to roll your own merge logic, when
upserting/inserting. This will render any value set for
PRECOMBINE_FIELD_OPT_VAL in-effective<br />`Config Param: PAYLOAD_CLASS_NAME`
[...]
|
[hoodie.datasource.write.reconcile.schema](#hoodiedatasourcewritereconcileschema)
| false
| This config controls how
writer's schema will be selected based on the incoming batch's schema as well
as existing table's one. When schema reconciliation is DISABLED, incoming
batch's schema will be picked as a writer-schema (therefore updating table's
schema). When schema recon [...]
@@ -359,7 +358,6 @@ The following set of configurations are common across Hudi.
| [hoodie.memory.compaction.max.size](#hoodiememorycompactionmaxsize)
| (N/A) | Maximum amount of memory used in bytes for
compaction operations in bytes , before spilling to local storage.<br />`Config
Param: MAX_MEMORY_FOR_COMPACTION`
[...]
|
[hoodie.common.diskmap.compression.enabled](#hoodiecommondiskmapcompressionenabled)
| true | Turn on compression for BITCASK disk map used by the
External Spillable Map<br />`Config Param:
DISK_MAP_BITCASK_COMPRESSION_ENABLED`
[...]
| [hoodie.common.spillable.diskmap.type](#hoodiecommonspillablediskmaptype)
| BITCASK | When handling input data that cannot be held in
memory, to merge with a file on storage, a spillable diskmap is employed. By
default, we use a persistent hashmap based loosely on bitcask, that offers O(1)
inserts, lookups. Change this to `ROCKS_DB` to prefer using rocksDB, for
handling the spill.<br />`Config Param: SPILLABLE_DISK_MAP_TYPE`
[...]
-|
[hoodie.datasource.write.new.columns.nullable](#hoodiedatasourcewritenewcolumnsnullable)
| false | When a non-nullable column is added to datasource during a write
operation, the write operation will fail schema compatibility check. Set this
option to true will make the newly added column nullable to successfully
complete the write operation.<br />`Config Param: MAKE_NEW_COLUMNS_NULLABLE`<br
/>`Since Version: 0.14.0`
[...]
|
[hoodie.datasource.write.reconcile.schema](#hoodiedatasourcewritereconcileschema)
| false | This config controls how writer's schema will be selected
based on the incoming batch's schema as well as existing table's one. When
schema reconciliation is DISABLED, incoming batch's schema will be picked as a
writer-schema (therefore updating table's schema). When schema reconciliation
is ENABLED, writer-schema will be picked such that table's schema (after txn)
is either kept the [...]
| [hoodie.fs.atomic_creation.support](#hoodiefsatomic_creationsupport)
| | This config is used to specify the file system which
supports atomic file creation . atomic means that an operation either succeeds
and has an effect or has fails and has no effect; now this feature is used by
FileSystemLockProvider to guaranteeing that only one writer can create the lock
file at a time. since some FS does not support atomic file creation (eg: S3),
we decide the FileS [...]
| [hoodie.memory.dfs.buffer.max.size](#hoodiememorydfsbuffermaxsize)
| 16777216 | Property to control the max memory in bytes for dfs
input stream buffer size<br />`Config Param: MAX_DFS_STREAM_BUFFER_SIZE`
[...]
