Daniel Vanko has posted comments on this change. ( http://gerrit.cloudera.org:8080/24570 )
Change subject: IMPALA-12675: Set PARQUET_ANNOTATE_STRINGS_UTF8 to true by default and turn on validation ...................................................................... Patch Set 16: (3 comments) Thanks for the comments, I cleared up the too verbose comments. http://gerrit.cloudera.org:8080/#/c/24570/15//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/24570/15//COMMIT_MSG@23 PS15, Line 23: CHAR/VARCHAR (always text) > What is the reason for not validating these? My guess there's no specific reason for this. >From my quick research on other engines: - DuckDB seems to be the strictest and only writes valid UTF-8 (error raised) - Hive replaces invalid bytes silently - Trino can write invalid UTF-8, if it's a pass-through from another file If we want to turn on validation for CHAR/VARCHAR columns, I think that should be a seperate patch, but Impala 5.0 would be good point to introduce this kind of change. http://gerrit.cloudera.org:8080/#/c/24570/14/be/src/exec/parquet/hdfs-parquet-table-writer.cc File be/src/exec/parquet/hdfs-parquet-table-writer.cc: http://gerrit.cloudera.org:8080/#/c/24570/14/be/src/exec/parquet/hdfs-parquet-table-writer.cc@940 PS14, Line 940: any partial : // state is committed. Runs once per non-null value here; ProcessValue() below may be : // retried on a new > Doesn't seem very useful info. Generally comments look a bit verbose, I ass Done http://gerrit.cloudera.org:8080/#/c/24570/15/tests/query_test/test_scanners.py File tests/query_test/test_scanners.py: http://gerrit.cloudera.org:8080/#/c/24570/15/tests/query_test/test_scanners.py@940 PS15, Line 940: t = get_parquet_meta > There is also get_parquet_metadata_from_hdfs_folder, why not use that? Done -- To view, visit http://gerrit.cloudera.org:8080/24570 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia95bcba978863ffb5d603fdf81bb2c75ba06f7da Gerrit-Change-Number: 24570 Gerrit-PatchSet: 16 Gerrit-Owner: Daniel Vanko <[email protected]> Gerrit-Reviewer: Arnab Karmakar <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Daniel Vanko <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Mihaly Szjatinya <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Tue, 15 Sep 2026 19:46:24 +0000 Gerrit-HasComments: Yes
