kumarpritam863 commented on PR #15209:
URL: https://github.com/apache/iceberg/pull/15209#issuecomment-3957460435
@danielcweeks
There are two things:
1. In the iceberg Initial-default once set cannot be changed as there is no
api to update initial-default
2. In KC write-default does not make any sense due to the following:
- Write-default is used when the field is missing after it was added means:
If the table schema is {A, B, C, D} and a record ingested with schema {A, B,
C} then for "D" write default will be used but in KC this will never happen.
Consider this scenario:
1. Table is created on encountering the record with schema {"A", "B", "C"}.
This will also initialised the writers with schema {"A", "B", "C"}.
2. Now a record comes with schema {"A", "B", "C", "D"} with the default
value of "D" as "d". So the KC will evolve the table schema to {"A", "B", "C",
"D"} and set the Initial-Default = "d" and Write-Default = "d". This will also
flush the old writers and create the new writers with schema {"A", "B", "C",
"D"}.
3. Now suppose a record comes with schema {"A", "B", "C"} but since the
writers are at {"A", "B", "C", "D"} and KC only handles {"Field-Addition,
Making a field optional, changing type"}, KC will not detect the missing field
and the writers will try to write the record with schema {"A", "B", "C", "D"}.
Now if "D" was an optional columns the the writer will itself write "null" in
the parquet/avro file else the writer will fail.
So the write-default will never be used.
Let me know if I am missing something here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
