RussellSpitzer commented on code in PR #4301: URL: https://github.com/apache/iceberg/pull/4301#discussion_r858006034
########## format/spec.md: ########## @@ -194,6 +194,19 @@ Notes: For details on how to serialize a schema to JSON, see Appendix C. +#### Default values + +Default values can be tracked for struct fields (both nested structs and the top-level schema's struct). There can be two defaults with a field: +- `initial-default` is used to populate the field's value for all records that were written before the field was added to the schema +- `write-default` is used to populate the field's value for any records written after the field was added to the schema, if the writer does not supply the field's value + +The `initial-default` is set only when a field is added to an existing schema. The `write-default` is initially set to the same value as `initial-default` and can be changed through schema evolution. If either default is not set for an optional field, then the default value is null for compatibility with older spec versions. + +Together, the `initial-default` and `write-default` produce SQL default value behavior without rewriting data files. That is, default value changes may only affect future records and all known fields are written into data files. To produce this behavior, omitting a known field when writing a data file is not allowed. The write default for a field must be written if a field is not supplied to a write. If the write default for a required field is not set, the writer must fail. Review Comment: Based on the evolution rules below I think I understand ```An optional field may be skipped by a writer. When an optional field is skipped the write-default should be used in place of the missing value and this default may be null. A required field may only be skipped by a writer if a write-default exists for that field and this default must not be null.``` ########## format/spec.md: ########## @@ -194,6 +194,19 @@ Notes: For details on how to serialize a schema to JSON, see Appendix C. +#### Default values + +Default values can be tracked for struct fields (both nested structs and the top-level schema's struct). There can be two defaults with a field: +- `initial-default` is used to populate the field's value for all records that were written before the field was added to the schema +- `write-default` is used to populate the field's value for any records written after the field was added to the schema, if the writer does not supply the field's value + +The `initial-default` is set only when a field is added to an existing schema. The `write-default` is initially set to the same value as `initial-default` and can be changed through schema evolution. If either default is not set for an optional field, then the default value is null for compatibility with older spec versions. + +Together, the `initial-default` and `write-default` produce SQL default value behavior without rewriting data files. That is, default value changes may only affect future records and all known fields are written into data files. To produce this behavior, omitting a known field when writing a data file is not allowed. The write default for a field must be written if a field is not supplied to a write. If the write default for a required field is not set, the writer must fail. Review Comment: Based on the evolution rules below I think I understand ``` An optional field may be skipped by a writer. When an optional field is skipped the write-default should be used in place of the missing value and this default may be null. A required field may only be skipped by a writer if a write-default exists for that field and this default must not be null. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
