[
https://issues.apache.org/jira/browse/HIVE-30023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hidataplus updated HIVE-30023:
------------------------------
Description:
Running {{ANALYZE TABLE <iceberg_table> COMPUTE STATISTICS FOR COLUMNS}} on an
Iceberg table whose *partition column contains a NULL value* fails with:
java.lang.NullPointerException: Cannot invoke "String.trim()" because
"identifier" is null
The NPE is thrown *server-side* in the metastore
({{{}ObjectStore.getMTable{}}}) while processing the {{alter_partitions}} RPC
that the column-stats path issues to write back partition parameters. It is
caused by {{ObjectStore.alterPartitionNoTxn}} trusting the incoming Thrift
{{Partition}} object's own {{dbName}} (which is null for the Iceberg null-value
partition) instead of the request-level database name.
h2. Reproduction steps
# Create an Iceberg table partitioned by a bigint column:
CREATE TABLE db.t (id INT) PARTITIONED BY (pt BIGINT) STORED BY ICEBERG;
# Insert a row with a NULL partition value:
INSERT INTO db.t VALUES (1, NULL);
# Run column-statistics analysis:
ANALYZE TABLE db.t COMPUTE STATISTICS FOR COLUMNS;
*Expected:* the statement succeeds (or the null partition is handled
gracefully).
*Actual:* {{{}NullPointerException: Cannot invoke "String.trim()" because
"identifier" is null{}}}.
Notes:
* Inserting the NULL value itself succeeds; only {{ANALYZE ... FOR COLUMNS}}
fails.
* {{ANALYZE TABLE ... COMPUTE STATISTICS}} (basic stats, no {{{}FOR
COLUMNS{}}}) does not hit this path.
* Tables without a NULL partition value are unaffected.
h2. Stack traces（not al)
Client side (the NPE is deserialized from the Thrift response):
org.apache.hadoop.hive.ql.metadata.Hive.alterPartitions(Hive.java:1218)
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_partitions_req_result$alter_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java)
Server side (metastore log), the actual throw site:
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2029)
h2. Root cause analysis (verified against rel/release-4.2.0)
# {{ANALYZE ... FOR COLUMNS}} computes column stats, then writes back
partition parameters via {{Hive.alterPartitions}}
({{{}ql/.../metadata/Hive.java{}}}, ~L1211-1218), issuing the
{{alter_partitions}} Thrift RPC.
# Server: {{HMSHandler.alter_partitions}} →
{{HiveAlterHandler.alterPartitions}} →
{{{}ObjectStore.alterPartitionNoTxn(catName, dbname, name, part_vals, newPart,
...){}}}.
# {{ObjectStore.alterPartitionNoTxn}} resolves the table using the {*}incoming
Partition object's own names{*}, not the request-level parameters
({{{}ObjectStore.java:4916{}}}):
MTable table = this.getMTable(newPart.getCatName(), newPart.getDbName(),
newPart.getTableName());
# {{ObjectStore.getMTable(catName, db, table, retrieveCD)}} normalizes {{db}}
*without a null guard* ({{{}ObjectStore.java:2028-2030{}}}):
catName =
normalizeIdentifier(Optional.ofNullable(catName).orElse(getDefaultCatalog(conf)));
// guarded
db = normalizeIdentifier(db); // <-- L2029: NPE when db == null
table = normalizeIdentifier(table);where {{normalizeIdentifier}} is
{{identifier.trim().toLowerCase()}}
({{{}metastore-common/.../StringUtils.java:93-95{}}}).
# For an Iceberg table, the partition whose partition-column value is NULL is
represented client-side by a Thrift {{Partition}} whose {{dbName}} is *not
populated* (the Iceberg storage handler synthesizes/maps this partition without
filling db/table names). When it reaches {{{}alterPartitionNoTxn{}}},
{{newPart.getDbName()}} is null → {{getMTable(..., null, ...)}} →
{{normalizeIdentifier(null)}} → NPE.
# Normal (non-null) partitions carry populated
{{{}dbName{}}}/{{{}tableName{}}}, so only the null-value partition triggers the
failure. The NPE is serialized into the Thrift response and re-thrown on the
client (hence the client frames above).
h2. Suggested fix
Preferred (semantically correct, server-side): in
{{{}ObjectStore.alterPartitionNoTxn{}}}, fall back to the request-level names
when the incoming {{{}Partition{}}}'s are unset:
// ObjectStore.java:4916, before
MTable table = this.getMTable(newPart.getCatName(), newPart.getDbName(),
newPart.getTableName());
// after
MTable table = this.getMTable(
newPart.isSetCatName() ? newPart.getCatName() : catName,
newPart.getDbName() != null ? newPart.getDbName() : dbname,
newPart.getTableName()!= null ? newPart.getTableName(): name);
Alternative / complementary:
* Add a defensive null guard for {{{}db{}}}/{{{}table{}}} in {{getMTable}}
mirroring the existing {{catName}} pattern (or fail fast with a clear
{{InvalidObjectException}} instead of an NPE).
* Ensure the Iceberg storage handler populates
{{{}dbName{}}}/{{{}tableName{}}}/{{{}catName{}}} on every {{Partition}} it
synthesizes, so clients never send partially-populated partition objects.
was:
Running {{ANALYZE TABLE <iceberg_table> COMPUTE STATISTICS FOR COLUMNS}} on an
Iceberg table whose *partition column contains a NULL value* fails with:
java.lang.NullPointerException: Cannot invoke "String.trim()" because
"identifier" is null
The NPE is thrown *server-side* in the metastore
({{{}ObjectStore.getMTable{}}}) while processing the {{alter_partitions}} RPC
that the column-stats path issues to write back partition parameters. It is
caused by {{ObjectStore.alterPartitionNoTxn}} trusting the incoming Thrift
{{Partition}} object's own {{dbName}} (which is null for the Iceberg null-value
partition) instead of the request-level database name.
h2. Reproduction steps
# Create an Iceberg table partitioned by a bigint column:
CREATE TABLE db.t (id INT) PARTITIONED BY (pt BIGINT) STORED BY ICEBERG;
# Insert a row with a NULL partition value:
INSERT INTO db.t VALUES (1, NULL);
# Run column-statistics analysis:
ANALYZE TABLE db.t COMPUTE STATISTICS FOR COLUMNS;
*Expected:* the statement succeeds (or the null partition is handled
gracefully).
*Actual:* {{{}NullPointerException: Cannot invoke "String.trim()" because
"identifier" is null{}}}.
Notes:
* Inserting the NULL value itself succeeds; only {{ANALYZE ... FOR COLUMNS}}
fails.
* {{ANALYZE TABLE ... COMPUTE STATISTICS}} (basic stats, no {{{}FOR
COLUMNS{}}}) does not hit this path.
* Tables without a NULL partition value are unaffected.
h2. Stack traces
Client side (the NPE is deserialized from the Thrift response):
org.apache.hadoop.hive.ql.metadata.Hive.alterPartitions(Hive.java:1218)
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_partitions_req_result$alter_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java)
Server side (metastore log), the actual throw site:
org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2029)
h2. Root cause analysis (verified against rel/release-4.2.0)
# {{ANALYZE ... FOR COLUMNS}} computes column stats, then writes back
partition parameters via {{Hive.alterPartitions}}
({{{}ql/.../metadata/Hive.java{}}}, ~L1211-1218), issuing the
{{alter_partitions}} Thrift RPC.
# Server: {{HMSHandler.alter_partitions}} →
{{HiveAlterHandler.alterPartitions}} →
{{{}ObjectStore.alterPartitionNoTxn(catName, dbname, name, part_vals, newPart,
...){}}}.
# {{ObjectStore.alterPartitionNoTxn}} resolves the table using the {*}incoming
Partition object's own names{*}, not the request-level parameters
({{{}ObjectStore.java:4916{}}}):
MTable table = this.getMTable(newPart.getCatName(), newPart.getDbName(),
newPart.getTableName());
# {{ObjectStore.getMTable(catName, db, table, retrieveCD)}} normalizes {{db}}
*without a null guard* ({{{}ObjectStore.java:2028-2030{}}}):
catName =
normalizeIdentifier(Optional.ofNullable(catName).orElse(getDefaultCatalog(conf)));
// guarded
db = normalizeIdentifier(db); // <-- L2029: NPE when db == null
table = normalizeIdentifier(table);where {{normalizeIdentifier}} is
{{identifier.trim().toLowerCase()}}
({{{}metastore-common/.../StringUtils.java:93-95{}}}).
# For an Iceberg table, the partition whose partition-column value is NULL is
represented client-side by a Thrift {{Partition}} whose {{dbName}} is *not
populated* (the Iceberg storage handler synthesizes/maps this partition without
filling db/table names). When it reaches {{{}alterPartitionNoTxn{}}},
{{newPart.getDbName()}} is null → {{getMTable(..., null, ...)}} →
{{normalizeIdentifier(null)}} → NPE.
# Normal (non-null) partitions carry populated
{{{}dbName{}}}/{{{}tableName{}}}, so only the null-value partition triggers the
failure. The NPE is serialized into the Thrift response and re-thrown on the
client (hence the client frames above).
h2. Suggested fix
Preferred (semantically correct, server-side): in
{{{}ObjectStore.alterPartitionNoTxn{}}}, fall back to the request-level names
when the incoming {{{}Partition{}}}'s are unset:
// ObjectStore.java:4916, before
MTable table = this.getMTable(newPart.getCatName(), newPart.getDbName(),
newPart.getTableName());
// after
MTable table = this.getMTable(
newPart.isSetCatName() ? newPart.getCatName() : catName,
newPart.getDbName() != null ? newPart.getDbName() : dbname,
newPart.getTableName()!= null ? newPart.getTableName(): name);
Alternative / complementary:
* Add a defensive null guard for {{{}db{}}}/{{{}table{}}} in {{getMTable}}
mirroring the existing {{catName}} pattern (or fail fast with a clear
{{InvalidObjectException}} instead of an NPE).
* Ensure the Iceberg storage handler populates
{{{}dbName{}}}/{{{}tableName{}}}/{{{}catName{}}} on every {{Partition}} it
synthesizes, so clients never send partially-populated partition objects.
h2. Workaround (until fixed)
* Avoid NULL values in the partition column (use a sentinel such as
{{{}-1{}}}/{{{}0{}}}), or rewrite/drop the null partition.
* Analyze only non-null partitions: {{ANALYZE TABLE db.t
PARTITION(pt=<non-null>) COMPUTE STATISTICS FOR COLUMNS;}}
* Or use basic stats only: {{ANALYZE TABLE db.t COMPUTE STATISTICS;}}
h2. Additional context
* The exact message {{Cannot invoke "String.trim()" because "identifier" is
null}} has no prior report in apache/hive (searched), so this appears to be a
new issue.
* Related but distinct: this is independent of the Iceberg multi-spec
partition-evolution {{Cannot project non-struct}} family of bugs; it is purely
a metastore null-name handling defect triggered by Iceberg null partition
values.
> NPE Cannot invoke "String.trim()" because "identifier" is null in
> ObjectStore.getMTable when running ANALYZE TABLE ... COMPUTE STATISTICS FOR
> COLUMNS on an Iceberg table with a NULL partition value
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30023
> URL: https://issues.apache.org/jira/browse/HIVE-30023
> Project: Hive
> Issue Type: Bug
> Environment: * Apache Hive *4.2.0* ({{{}rel/release-4.2.0{}}}); also
> expected on master
> * Bundled Iceberg *1.9.1* ({{{}iceberg/pom.xml{}}})
> * Table: *Iceberg* table, partitioned by a *bigint* partition column
> * The partition column contains a *NULL* value (Iceberg treats NULL as a
> first-class partition value)
> Reporter: hidataplus
> Priority: Major
>
> Running {{ANALYZE TABLE <iceberg_table> COMPUTE STATISTICS FOR COLUMNS}} on
> an Iceberg table whose *partition column contains a NULL value* fails with:
> java.lang.NullPointerException: Cannot invoke "String.trim()" because
> "identifier" is null
> The NPE is thrown *server-side* in the metastore
> ({{{}ObjectStore.getMTable{}}}) while processing the {{alter_partitions}} RPC
> that the column-stats path issues to write back partition parameters. It is
> caused by {{ObjectStore.alterPartitionNoTxn}} trusting the incoming Thrift
> {{Partition}} object's own {{dbName}} (which is null for the Iceberg
> null-value partition) instead of the request-level database name.
>
> h2. Reproduction steps
> # Create an Iceberg table partitioned by a bigint column:
> CREATE TABLE db.t (id INT) PARTITIONED BY (pt BIGINT) STORED BY ICEBERG;
>
> # Insert a row with a NULL partition value:
> INSERT INTO db.t VALUES (1, NULL);
>
> # Run column-statistics analysis:
> ANALYZE TABLE db.t COMPUTE STATISTICS FOR COLUMNS;
>
> *Expected:* the statement succeeds (or the null partition is handled
> gracefully).
> *Actual:* {{{}NullPointerException: Cannot invoke "String.trim()" because
> "identifier" is null{}}}.
> Notes:
> * Inserting the NULL value itself succeeds; only {{ANALYZE ... FOR COLUMNS}}
> fails.
> * {{ANALYZE TABLE ... COMPUTE STATISTICS}} (basic stats, no {{{}FOR
> COLUMNS{}}}) does not hit this path.
> * Tables without a NULL partition value are unaffected.
> h2. Stack traces（not al)
> Client side (the NPE is deserialized from the Thrift response):
> org.apache.hadoop.hive.ql.metadata.Hive.alterPartitions(Hive.java:1218)
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$alter_partitions_req_result$alter_partitions_req_resultStandardScheme.read(ThriftHiveMetastore.java)
>
> Server side (metastore log), the actual throw site:
> org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:2029)
>
> h2. Root cause analysis (verified against rel/release-4.2.0)
> # {{ANALYZE ... FOR COLUMNS}} computes column stats, then writes back
> partition parameters via {{Hive.alterPartitions}}
> ({{{}ql/.../metadata/Hive.java{}}}, ~L1211-1218), issuing the
> {{alter_partitions}} Thrift RPC.
> # Server: {{HMSHandler.alter_partitions}} →
> {{HiveAlterHandler.alterPartitions}} →
> {{{}ObjectStore.alterPartitionNoTxn(catName, dbname, name, part_vals,
> newPart, ...){}}}.
> # {{ObjectStore.alterPartitionNoTxn}} resolves the table using the
> {*}incoming Partition object's own names{*}, not the request-level parameters
> ({{{}ObjectStore.java:4916{}}}):
> MTable table = this.getMTable(newPart.getCatName(), newPart.getDbName(),
> newPart.getTableName());
> # {{ObjectStore.getMTable(catName, db, table, retrieveCD)}} normalizes
> {{db}} *without a null guard* ({{{}ObjectStore.java:2028-2030{}}}):
> catName =
> normalizeIdentifier(Optional.ofNullable(catName).orElse(getDefaultCatalog(conf)));
> // guarded
> db = normalizeIdentifier(db); // <-- L2029: NPE when db == null
> table = normalizeIdentifier(table);where {{normalizeIdentifier}} is
> {{identifier.trim().toLowerCase()}}
> ({{{}metastore-common/.../StringUtils.java:93-95{}}}).
> # For an Iceberg table, the partition whose partition-column value is NULL
> is represented client-side by a Thrift {{Partition}} whose {{dbName}} is *not
> populated* (the Iceberg storage handler synthesizes/maps this partition
> without filling db/table names). When it reaches {{{}alterPartitionNoTxn{}}},
> {{newPart.getDbName()}} is null → {{getMTable(..., null, ...)}} →
> {{normalizeIdentifier(null)}} → NPE.
> # Normal (non-null) partitions carry populated
> {{{}dbName{}}}/{{{}tableName{}}}, so only the null-value partition triggers
> the failure. The NPE is serialized into the Thrift response and re-thrown on
> the client (hence the client frames above).
> h2. Suggested fix
> Preferred (semantically correct, server-side): in
> {{{}ObjectStore.alterPartitionNoTxn{}}}, fall back to the request-level names
> when the incoming {{{}Partition{}}}'s are unset:
> // ObjectStore.java:4916, before
> MTable table = this.getMTable(newPart.getCatName(), newPart.getDbName(),
> newPart.getTableName());
> // after
> MTable table = this.getMTable(
> newPart.isSetCatName() ? newPart.getCatName() : catName,
> newPart.getDbName() != null ? newPart.getDbName() : dbname,
> newPart.getTableName()!= null ? newPart.getTableName(): name);
> Alternative / complementary:
> * Add a defensive null guard for {{{}db{}}}/{{{}table{}}} in {{getMTable}}
> mirroring the existing {{catName}} pattern (or fail fast with a clear
> {{InvalidObjectException}} instead of an NPE).
> * Ensure the Iceberg storage handler populates
> {{{}dbName{}}}/{{{}tableName{}}}/{{{}catName{}}} on every {{Partition}} it
> synthesizes, so clients never send partially-populated partition objects.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
