zhangfengcdt commented on code in PR #7027:
URL: https://github.com/apache/datafusion-comet/pull/7027#discussion_r4634403597


##########
docs/source/user-guide/latest/iceberg.md:
##########
@@ -535,26 +535,26 @@ iceberg-java resolves per-write options and 
`spark.sql.iceberg.*` session overri
 
 A write is eligible only when ALL of the following hold:
 
-| Setting                                                                      
                                                               | Supported 
values                                                                          
                                                                                
                                                                                
                                                                                
                                                                                
                                                      |
-| 
-------------------------------------------------------------------------------------------------------------------------------------------
 | 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 |
-| resolved write format (`write-format` option overlaid on 
`write.format.default`)                                                         
   | `parquet`                                                                  
                                                                                
                                                                                
                                                                                
                                                                                
                                                                     |
-| `format-version`                                                             
                                                               | `1`, `2` or 
`3` (on `3`, writes that carry row lineage fall back; see below)                
                                                                                
                                                                                
                                                                                
                                                                                
                                                    |
-| `write.parquet.compression-codec` / `compression-level` / 
`row-group-size-bytes` / `page-size-bytes` / `page-row-limit` / 
`dict-size-bytes` | the sizes/limit must be positive Java ints 
(`Integer.parseInt` semantics — no trimming, no values past `Int.MaxValue` — 
matching iceberg-java, whose writer fails on anything else); 
`compression-level` must be a Java int within the native writer's per-codec 
range (zstd 1–22, gzip 0–9, brotli 0–11; ignored for snappy/lz4/none), which 
iceberg-java never validates, so other levels fall back; `brotli` needs 
Hadoop's `BrotliCodec` class on the classpath (see below) |
-| `write.parquet.row-group-check-min-record-count`                             
                                                               | unset or `100` 
(the default)                                                                   
                                                                                
                                                                                
                                                                                
                                                                                
                                                 |
-| `write.parquet.row-group-check-max-record-count`                             
                                                               | unset or 
`10000` (the default)                                                           
                                                                                
                                                                                
                                                                                
                                                                                
                                                       |
-| `write.parquet.page-version`                                                 
                                                               | unset or `v1`  
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                 |
-| `write.parquet.shred-variants`                                               
                                                               | any value 
(iceberg-java shreds only top-level `variant` columns, and a `variant` column 
falls back on its type)                                                         
                                                                                
                                                                                
                                                                                
                                                        |
-| `write.parquet.variant-inference-buffer-size`                                
                                                               | any value 
(only meaningful when shredding a `variant` column, which falls back)           
                                                                                
                                                                                
                                                                                
                                                                                
                                                      |
-| `write.parquet.bloom-filter-enabled.column.<col>`                            
                                                               | unset or 
`false`                                                                         
                                                                                
                                                                                
                                                                                
                                                                                
                                                       |
-| `write.metadata.metrics.*`                                                   
                                                               | any value 
(manifest metrics are re-derived on the JVM with Iceberg's own logic)           
                                                                                
                                                                                
                                                                                
                                                                                
                                                      |
-| `write.spark.fanout.enabled`                                                 
                                                               | any value (the 
native writer implements both clustered and fanout modes)                       
                                                                                
                                                                                
                                                                                
                                                                                
                                                 |
-| `write.target-file-size-bytes`                                               
                                                               | any value (the 
two writers can choose different roll points; see accepted divergences)         
                                                                                
                                                                                
                                                                                
                                                                                
                                                 |
-| data location URI scheme                                                     
                                                               | `file`, 
`memory`, `s3`, `s3a`, `gs`, matched case-sensitively (`S3://` falls back). 
`s3`, `s3a` and `gs` need a bucket in the authority (`s3://bucket/...`), so a 
hostless form such as `s3:/bucket/key` falls back. `gs` only when the `FileIO` 
opening the data location is a `GCSFileIO`; see below                           
                                                                                
                                                               |
-| resolved `table.locationProvider()`                                          
                                                               | Iceberg's 
built-in `DefaultLocationProvider`                                              
                                                                                
                                                                                
                                                                                
                                                                                
                                                      |
-| Hadoop S3A settings for an `s3` / `s3a` data location                        
                                                               | 
`fs.s3a.access.key`, `secret.key`, `session.token`, `endpoint`, 
`endpoint.region`, and `path.style.access`, plus the ignored, encryption and 
credential settings described below, including their 
`fs.s3a.bucket.<data-bucket>.*` forms; any other effective `fs.s3a.*` setting 
falls back                                                                      
                                                                                
                                |
-| Iceberg `FileIO` S3 settings for an `s3` / `s3a` data location               
                                                               | the S3 
endpoint, region, static/session credentials, path-style, SSE (`none`, `s3`, 
`kms`, or `custom`; not `dsse-kms`), assume-role, anonymous/config-chain 
settings parsed by the pinned iceberg-rust version, plus Comet's 
credential-provider class and built-in web-identity properties. When a custom 
provider is configured, its vendor-owned `s3.*` / `client.*` properties are 
also forwarded; unsupported Iceberg-defined S3 settings still fall back         
        |
-| partition spec                                                               
                                                               | any, except a 
`void` field whose source column was dropped beside a live field, or one 
sourced from an `unknown` column (see below)                                    
                                                                                
                                                                                
                                                                                
                                                         |
-| column types                                                                 
                                                               | any except 
`uuid` (Spark plans it as a string; no Arrow cast reaches `fixed(16)`), the v3 
types `variant`, `timestamp_ns`, `geometry` and `geography`, `unknown` inside a 
nested type, and a table whose columns are all `unknown`                        
                                                                                
                                                                                
                                                      |
+| Setting                                                                      
                                                               | Supported 
values                                                                          
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                       |
+| 
-------------------------------------------------------------------------------------------------------------------------------------------
 | 
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 |
+| resolved write format (`write-format` option overlaid on 
`write.format.default`)                                                         
   | `parquet`                                                                  
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      |
+| `format-version`                                                             
                                                               | `1`, `2` or 
`3` (on `3`, writes that carry row lineage fall back; see below)                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                     |
+| `write.parquet.compression-codec` / `compression-level` / 
`row-group-size-bytes` / `page-size-bytes` / `page-row-limit` / 
`dict-size-bytes` | the codec must be a name iceberg-java accepts 
(`uncompressed`, `snappy`, `gzip`, `lz4`, `zstd`, `brotli`, matched 
case-insensitively and without trimming; `none` is ORC's spelling and falls 
back, as do `lzo` and `lz4_raw`, which iceberg-java accepts but the native 
writer cannot write); the sizes/limit must be positive Java ints 
(`Integer.parseInt` semantics — no trimming, no values past `Int.MaxValue` — 
matching iceberg-java, whose writer fails on anything else); 
`compression-level` must be a Java int within the native writer's per-codec 
range (zstd 1–22, gzip 0–9, brotli 0–11; ignored by both writers for 
snappy/lz4/uncompressed) — iceberg-java never validates the level, so an 
out-of-range value falls back rather than becoming a native task failure; 
`brotli` needs Hadoop's `BrotliCodec` class on the classpath (see below) |
+| `write.parquet.row-group-check-min-record-count`                             
                                                               | unset or `100` 
(the default); a value iceberg-java cannot parse as a Java int, such as one 
with whitespace, falls back                                                     
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                      |
+| `write.parquet.row-group-check-max-record-count`                             
                                                               | unset or 
`10000` (the default); a value iceberg-java cannot parse as a Java int, such as 
one with whitespace, falls back                                                 
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                        |
+| `write.parquet.page-version`                                                 
                                                               | unset or `v1`, 
spelled exactly (iceberg-java rejects `V1`)                                     
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                  |

Review Comment:
   It looks like only v1.11 will reject "V1" value, but v1.10 actually accept 
it and other values than "v1" that  fallbacks to spark. Maybe reword it to be 
more accurate.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to