[
https://issues.apache.org/jira/browse/HUDI-5971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alberic Liu updated HUDI-5971:
------------------------------
Description:
reproduce steps:
* create the test table:
{code:java}
CREATE TABLE default.schema_evolution_test (
id bigint,
test_array array<int>,
test_array_struct ARRAY<STRUCT<name:STRING, age:INT>>)
USING hudi
TBLPROPERTIES (
type = 'cow',
primaryKey = 'id')
LOCATION 's3://trino-ci-test/schema_evolution_test'
{code}
* enable the full schema evolution
{code:java}
SET hoodie.schema.on.read.enable=true{code}
* alter the nested table column type, there is exception
{code:java}
SQL A: alter table default.schema_evolution_test alter column test_array_struct
type ARRAY<STRUCT<name:STRING, age:FLOAT>>;{code}
{code:java}
Exception: Failed in [alter table default.schema_evolution_test alter column
test_array_struct type ARRAY<STRUCT<name:STRING, age:FLOAT>>]
java.lang.IllegalArgumentException: only support update primitive type but find
nest column: test_array_struct
at
org.apache.hudi.internal.schema.action.TableChanges$ColumnUpdateChange.updateColumnType(TableChanges.java:89)
at
org.apache.spark.sql.hudi.command.AlterTableCommand.$anonfun$applyUpdateAction$1(AlterTableCommand.scala:149)
{code}
* alter the inner struct type, it can execute successfully
{code:java}
SQL B: alter table default.schema_evolution_test alter column
test_array_struct.element.age type FLOAT;{code}
* Describe the table schema；
{code:java}
desc default.schema_evolution_test;
id bigint
test_array array<int>
test_array_struct array<struct<name:string,age:float>> {code}
* Questions:
## is this a bug of alter table column type for Hudi?
## Both SQL A and SQL B is used to change the type from
array<struct<name:string,age:int>> to array<struct<name:string,age:float>>, why
SQL A do not supported?
## SQL B is worked as expected or not? and SQL A will be supported in next
release?
was:
reproduce steps:
* create the test table:
CREATE TABLE default.schema_evolution_test (
id bigint,
test_array array<int>,
test_array_struct ARRAY<STRUCT<name:STRING, age:INT>>)
USING hudi
TBLPROPERTIES (
type = 'cow',
primaryKey = 'id')
LOCATION 's3://trino-ci-test/schema_evolution_test'
* enable the full schema evolution
SET hoodie.schema.on.read.enable=true
* alter the nested table column type, there is exception
{color:#de350b}SQL A:{color} alter table default.schema_evolution_test alter
column test_array_struct type ARRAY<STRUCT<name:STRING, age:FLOAT>>;
Exception: Failed in [alter table default.schema_evolution_test alter column
test_array_struct type ARRAY<STRUCT<name:STRING, age:FLOAT>>]
java.lang.IllegalArgumentException: only support update primitive type but find
nest column: test_array_struct
at
org.apache.hudi.internal.schema.action.TableChanges$ColumnUpdateChange.updateColumnType(TableChanges.java:89)
at
org.apache.spark.sql.hudi.command.AlterTableCommand.$anonfun$applyUpdateAction$1(AlterTableCommand.scala:149)
* alter the inner struct type, it can execute successfully
{color:#de350b}SQL B:{color} alter table default.schema_evolution_test alter
column test_array_struct.element.age type FLOAT;
* Describe the table schema；
desc default.schema_evolution_test;
id bigint
test_array array<int>
test_array_struct array<struct<name:string,age:float>>
* Questions:
## is this a bug of alter table column type for Hudi?
## Both SQL A and SQL B is used to change the type from
array<struct<name:string,age:int>> to array<struct<name:string,age:float>>, why
SQL A do not supported?
## SQL B is worked as expected or not? and SQL A will be supported in next
release?
> About full schema evolution, alter column type do not support nest column but
> can alter inside struct type
> ----------------------------------------------------------------------------------------------------------
>
> Key: HUDI-5971
> URL: https://issues.apache.org/jira/browse/HUDI-5971
> Project: Apache Hudi
> Issue Type: Bug
> Components: spark-sql
> Affects Versions: 0.12.2
> Reporter: Alberic Liu
> Priority: Major
> Labels: schema-evolution
>
> reproduce steps:
> * create the test table:
>
> {code:java}
> CREATE TABLE default.schema_evolution_test (
> id bigint,
> test_array array<int>,
> test_array_struct ARRAY<STRUCT<name:STRING,
> age:INT>>)
> USING hudi
> TBLPROPERTIES (
> type = 'cow',
> primaryKey = 'id')
> LOCATION 's3://trino-ci-test/schema_evolution_test'
> {code}
>
> * enable the full schema evolution
> {code:java}
> SET hoodie.schema.on.read.enable=true{code}
> * alter the nested table column type, there is exception
> {code:java}
> SQL A: alter table default.schema_evolution_test alter column
> test_array_struct type ARRAY<STRUCT<name:STRING, age:FLOAT>>;{code}
> {code:java}
> Exception: Failed in [alter table default.schema_evolution_test alter column
> test_array_struct type ARRAY<STRUCT<name:STRING, age:FLOAT>>]
> java.lang.IllegalArgumentException: only support update primitive type but
> find nest column: test_array_struct
> at
> org.apache.hudi.internal.schema.action.TableChanges$ColumnUpdateChange.updateColumnType(TableChanges.java:89)
> at
> org.apache.spark.sql.hudi.command.AlterTableCommand.$anonfun$applyUpdateAction$1(AlterTableCommand.scala:149)
> {code}
> * alter the inner struct type, it can execute successfully
> {code:java}
> SQL B: alter table default.schema_evolution_test alter column
> test_array_struct.element.age type FLOAT;{code}
> * Describe the table schema；
> {code:java}
> desc default.schema_evolution_test;
> id bigint
>
> test_array array<int>
> test_array_struct array<struct<name:string,age:float>> {code}
> * Questions:
> ## is this a bug of alter table column type for Hudi?
> ## Both SQL A and SQL B is used to change the type from
> array<struct<name:string,age:int>> to array<struct<name:string,age:float>>,
> why SQL A do not supported?
> ## SQL B is worked as expected or not? and SQL A will be supported in next
> release?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
