Hemanath created SPARK-58117:
--------------------------------
Summary: Spark rewrite_data_files procedure fails to compact
hash-partitioned Iceberg tables with WHERE clauses
Key: SPARK-58117
URL: https://issues.apache.org/jira/browse/SPARK-58117
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.0.1
Reporter: Hemanath
When attempting to run partition-by-partition compaction on Iceberg tables with
hash bucket partitioning using Spark's {{rewrite_data_files}} procedure, the
operation fails or produces no results. This prevents efficient compaction of
large hash-partitioned tables.
*Reproducible Steps*
Create an Iceberg table with hash bucket partitioning and Attempt
partition-by-partition compaction using partition field name:
pythonspark.sql(""" CALL system.rewrite_data_files( table =>
'namespace.table', where => 'TEST_PROGRAM_bucket = 12' )""")
*Getting the below error:*
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter
with name TEST_PROGRAM_bucket cannot be resolved. Did you mean one of the
following? [TEST_PROGRAM, TEST_PROGRAM_VERSION, CONTROL_HIGH, CONTROL_LOW,
PRODUCT]. SQLSTATE: 42703 java.lang.IllegalArgumentException: Cannot parse
predicates in where option: TEST_PROGRAM_bucket = 12
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
