jonathansudhakar1 opened a new pull request, #41019: URL: https://github.com/apache/beam/pull/41019
Fixes #41018. The Parquet **Schema** table on the [CREATE EXTERNAL TABLE page](https://beam.apache.org/documentation/dsls/sql/extensions/create-external-table/#parquet) gave the wrong Avro type for several Beam SQL types. `ParquetTable` converts the table schema with `AvroUtils.toAvroSchema`, and the table now matches what that produces: | Beam SQL type | Before | After | |---|---|---| | TINYINT, SMALLINT, INTEGER | long | int | | BIGINT | long | long (unchanged) | | FLOAT | double | float | | DOUBLE | double | double (unchanged) | | DECIMAL | bytes (with logical type) | bytes (`decimal` logical type) | | DATE | long (with logical type) | int (`date` logical type) | | TIME | long (with logical type) | Not supported yet (#40994) | | TIMESTAMP | long (with logical type) | long (`timestamp-millis` logical type) | | BINARY, VARBINARY | *(missing)* | bytes | | MAP | *(missing)* | map (keys must be VARCHAR) | BOOLEAN, CHAR/VARCHAR, ARRAY and ROW were already correct. This is a docs-only change to one HTML table in `create-external-table.md`. ## Testing I used a temporary JUnit test in `sdks/java/extensions/sql`, which isn't part of this PR, on master at 74ab567d2e3 with the DirectRunner: ``` ./gradlew :sdks:java:extensions:sql:test --tests '...parquet.TmpParquetTypeDocsTest' -PenableCheckerFramework=false ``` 1. **Mapping.** For each type, the test runs `CREATE EXTERNAL TABLE t (c <type>) TYPE parquet LOCATION '...'`. It gets the table schema with `CalciteUtils.toSchema(env.parseQuery("SELECT * FROM t").getRowType())` and prints `AvroUtils.toAvroSchema(schema).getField("c").schema()`. Output: ``` TINYINT | beam=BYTE | avro=["null","int"] SMALLINT | beam=INT16 | avro=["null","int"] INTEGER | beam=INT32 | avro=["null","int"] BIGINT | beam=INT64 | avro=["null","long"] FLOAT | beam=FLOAT | avro=["null","float"] DOUBLE | beam=DOUBLE | avro=["null","double"] DECIMAL | beam=DECIMAL | avro=["null",{"type":"bytes","logicalType":"decimal","precision":2147483647,"scale":0}] BOOLEAN | beam=BOOLEAN | avro=["null","boolean"] DATE | beam=LOGICAL_TYPE<beam:logical_type:date:v1> | avro=["null",{"type":"int","logicalType":"date"}] TIME | beam=LOGICAL_TYPE<beam:logical_type:time:v1> | avro=EXCEPTION RuntimeException: Unhandled logical type beam:logical_type:time:v1 TIMESTAMP | beam=DATETIME | avro=["null",{"type":"long","logicalType":"timestamp-millis"}] CHAR | beam=STRING | avro=["null","string"] VARCHAR | beam=STRING | avro=["null","string"] BINARY | beam=BYTES | avro=["null","bytes"] VARBINARY | beam=BYTES | avro=["null","bytes"] ARRAY<INTEGER> | beam=ARRAY<INT32 NOT NULL> | avro=["null",{"type":"array","items":"int"}] MAP<VARCHAR, INTEGER> | beam=MAP<STRING NOT NULL, INT32 NOT NULL> | avro=["null",{"type":"map","values":"int"}] MAP<INTEGER, VARCHAR> | beam=MAP<INT32 NOT NULL, STRING NOT NULL> | avro=EXCEPTION IllegalArgumentException: Avro only supports maps with string keys ROW<a INTEGER, b VARCHAR> | beam=ROW<a INT32, b STRING> | avro=["null",{"type":"record","name":"c",...}] ``` (The `["null", ...]` unions are there because DDL columns are nullable by default.) 2. **Round trip.** The test writes one row with `INSERT INTO ... VALUES (...)` and reads it back with `SELECT *`. - Every value read back as written for TINYINT, SMALLINT, INTEGER, BIGINT, FLOAT, DOUBLE, DECIMAL, BOOLEAN, TIMESTAMP, CHAR, VARCHAR, BINARY, VARBINARY, `ARRAY<INTEGER>` and `MAP<VARCHAR, INTEGER>`. - DATE is written correctly, but reading it fails, which I filed as #41017. - ROW values are flattened on `INSERT`, which is the long-standing #18883. Neither DATE nor ROW changes the type mapping documented here, so this PR doesn't mention them. I also ran `pre-commit run --files <changed file>` (no applicable hooks) and `git diff --check` (clean). The page already uses raw HTML tables, and `unsafe = true` is set for Goldmark in `website/www/site/config.toml`, so the `<code>` and `<a>` tags in the cells render as HTML. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
