rdblue commented on a change in pull request #4301: URL: https://github.com/apache/iceberg/pull/4301#discussion_r830221563
##########
File path: format/spec.md
##########
@@ -951,24 +958,45 @@ Types are serialized according to this table:
|Type|JSON representation|Example|
|--- |--- |--- |
-|**`boolean`**|`JSON string: "boolean"`|`"boolean"`|
-|**`int`**|`JSON string: "int"`|`"int"`|
-|**`long`**|`JSON string: "long"`|`"long"`|
-|**`float`**|`JSON string: "float"`|`"float"`|
-|**`double`**|`JSON string: "double"`|`"double"`|
-|**`date`**|`JSON string: "date"`|`"date"`|
-|**`time`**|`JSON string: "time"`|`"time"`|
-|**`timestamp without zone`**|`JSON string: "timestamp"`|`"timestamp"`|
-|**`timestamp with zone`**|`JSON string: "timestamptz"`|`"timestamptz"`|
-|**`string`**|`JSON string: "string"`|`"string"`|
-|**`uuid`**|`JSON string: "uuid"`|`"uuid"`|
-|**`fixed(L)`**|`JSON string: "fixed[<L>]"`|`"fixed[16]"`|
-|**`binary`**|`JSON string: "binary"`|`"binary"`|
-|**`decimal(P, S)`**|`JSON string: "decimal(<P>,<S>)"`|`"decimal(9,2)"`,<br
/>`"decimal(9, 2)"`|
-|**`struct`**|`JSON object: {`<br /> `"type": "struct",`<br
/> `"fields": [ {`<br /> `"id": <field id
int>,`<br /> `"name": <name string>,`<br
/> `"required": <boolean>,`<br
/> `"type": <type JSON>,`<br
/> `"doc": <comment string>`<br
/> `}, ...`<br /> `] }`|`{`<br
/> `"type": "struct",`<br /> `"fields": [ {`<br
/> `"id": 1,`<br /> `"name":
"id",`<br /> `"required": true,`<br
/> `"type": "uuid"`<br /> `}, {`<br
/> `"id": 2,`<br /> `"name":
"data",`<br /> `"required": false,`<br
/> `"type": {`<br
/> `"type": "list",`<br
/> `...`<br />
`}`<br /> `} ]`<br />`}`|
-|**`list`**|`JSON object: {`<br /> `"type": "list",`<br
/> `"element-id": <id int>,`<br /> `"element-required":
<bool>`<br /> `"element": <type JSON>`<br />`}`|`{`<br
/> `"type": "list",`<br /> `"element-id": 3,`<br
/> `"element-required": true,`<br /> `"element":
"string"`<br />`}`|
-|**`map`**|`JSON object: {`<br /> `"type": "map",`<br
/> `"key-id": <key id int>,`<br /> `"key": <type
JSON>,`<br /> `"value-id": <val id int>,`<br
/> `"value-required": <bool>`<br /> `"value": <type
JSON>`<br />`}`|`{`<br /> `"type": "map",`<br
/> `"key-id": 4,`<br /> `"key": "string",`<br
/> `"value-id": 5,`<br /> `"value-required": false,`<br
/> `"value": "double"`<br />`}`|
-
+| **`boolean`** | `JSON string: "boolean"`
| `"boolean"`
|
+| **`int`** | `JSON string: "int"`
| `"int"`
|
+| **`long`** | `JSON string: "long"`
| `"long"`
|
+| **`float`** | `JSON string: "float"`
| `"float"`
|
+| **`double`** | `JSON string: "double"`
| `"double"`
|
+| **`date`** | `JSON string: "date"`
| `"date"`
|
+| **`time`** | `JSON string: "time"`
| `"time"`
|
+| **`timestamp without zone`** | `JSON string: "timestamp"`
| `"timestamp"`
|
+| **`timestamp with zone`** | `JSON string: "timestamptz"`
| `"timestamptz"`
|
+| **`string`** | `JSON string: "string"`
| `"string"`
|
+| **`uuid`** | `JSON string: "uuid"`
| `"uuid"`
|
+| **`fixed(L)`** | `JSON string: "fixed[<L>]"`
| `"fixed[16]"`
|
+| **`binary`** | `JSON string: "binary"`
| `"binary"`
|
+| **`decimal(P, S)`** | `JSON string: "decimal(<P>,<S>)"`
| `"decimal(9,2)"`,<br
/>`"decimal(9, 2)"`
|
+| **`struct`** | `JSON object: {`<br /> `"type":
"struct",`<br /> `"fields": [ {`<br /> `"id":
<field id int>,`<br /> `"name": <name string>,`<br
/> `"required": <boolean>,`<br
/> `"type": <type JSON>,`<br
/> `"doc": <comment string>,`<br
/> `"default": <JSON encoding of default value>`<br
/> `}, ...`<br /> `] }` | `{`<br
/> `"type": "struct",`<br /> `"fields": [ {`<br
/> `"id": 1,`<br /> `"name":
"id",`<br /> `"required": true,`<br
/> `"type": "uuid",`<br
/> `"default":
"0db3e2a8-9d1d-42b9-aa7b-74ebe558dceb"`<br /> `}, {`<br
/> `"id": 2,`<br /> `"name":
"data",`<br />
`"required": false,`<br /> `"type": {`<br
/> `"type": "list",`<br
/> `...`<br /> `}`<br
/> `} ]`<br />`}` |
+| **`list`** | `JSON object: {`<br /> `"type":
"list",`<br /> `"element-id": <id int>,`<br
/> `"element-required": <bool>`<br /> `"element": <type
JSON>`<br />`}`
|
`{`<br /> `"type": "list",`<br /> `"element-id": 3,`<br
/> `"element-required": true,`<br /> `"element":
"string"`<br />`}`
|
+| **`map`** | `JSON object: {`<br /> `"type":
"map",`<br /> `"key-id": <key id int>,`<br /> `"key":
<type JSON>,`<br /> `"value-id": <val id int>,`<br
/> `"value-required": <bool>`<br /> `"value": <type
JSON>`<br />`}`
|
`{`<br /> `"type": "map",`<br /> `"key-id": 4,`<br
/> `"key": "string",`<br /> `"value-id": 5,`<br
/> `"value-required": false,`<br /> `"value":
"double"`<br />`}`
|
+
+For default values, the serialization depends on the type of the corresponding
column or nested field. The mapping of types and their corresponding default
value JSON serialization is described in the following table:
+
+| Type | Json type | Example |
Note
|
+|--------------------|-------------------|----------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **`boolean`** | **`boolean`** | `true` |
|
+| **`int`** | **`json int`** | `1` |
|
+| **`long`** | **`json long`** | `1` |
|
+| **`float`** | **`json float`** | `1.1` |
|
+| **`double`** | **`json double`** | `1.1` |
|
+| **`decimal(P,S)`** | **`string`** | `"0x3162"` |
Stores the unscaled value, as the two's-complement big-endian binary using the
minimum number of bytes, converted to a hexadecimal string prefixed by `0x`
|
+| **`date`** | **`json int`** | `19054` |
Stores days from the 1970-01-01
|
+| **`time`** | **`json long`** | `36000000000` |
Stores microseconds from midnight
|
+| **`timestamp`** | **`json long`** | `1646277378000000` |
Stores microseconds from 1970-01-01 00:00:00.000000
|
+| **`timestamptz`** | **`json long`** | `1646277378000000` |
Stores microseconds from 1970-01-01 00:00:00.000000 UTC
|
+| **`string`** | **`string`** | `"foo"` |
|
+| **`uuid`** | **`string`** |
`"eb26bdb1-a1d8-4aa6-990e-da940875492c"` | Stores the lowercase uuid string
|
+| **`fixed(L)`** | **`string`** | `"0x3162"` |
Stored as a hexadecimal byte literal string, prefixex by `0x`
|
+| **`binary`** | **`string`** | `"0x3162"` |
Stored as a hexadecimal byte literal string, prefixex by `0x`
|
+| **`struct`** | **`object`** | `{"a": 1, "foo": "bar"}` |
Use a JSON map to represent struct data, the keys are the nested fields' names
in the struct schema, and the values are value literals of corresponding
fields' type
|
Review comment:
This requires a fairly complicated process to rewrite the default value
when the schema changes because it uses field names instead of field IDs. Have
you considered other representations, like a list of fields: `[ {"field-id": 5,
"value": "s"}, ... ]`?
Also, will these inherit default values when the struct evolves? Is that
explicit or implicit?
For example, if I have a default that is `{ "a": 1, "b": 12.34 }` and I add
an optional field `c` with the default value `"s"`, will the struct default be
`{ "a": 1, "b": 12.34, "c": null }` or `{ "a": 1, "b": 12.34, "c": "s" }`?
What happens if the default value is not updated at the same time and only
stores the original, `{ "a": 1, "b": 12.34 }`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
