zaminhassnain06 opened a new issue, #11431:
URL: https://github.com/apache/hudi/issues/11431
Hi,
I tried to update my hudi version from 0.6.0 to 0.11. I updated it gradually
version by version starting from 0.6 to 0.9 and then from 0.9 to 0.10 and
finally from 0.10 to 0.11. I am running it using EMR and querying the hudi
table on athena. The table version was updated correctly after each update in
hoodie.properties file on S3. However when I tried to Run Alter table command
to Alter data type of colum from int to big in on 0.11 it is giving me folowing
error
`pyspark.sql.utils.AnalysisException: ALTER TABLE CHANGE COLUMN is not
supported for changing column 'id' with type 'IntegerType' to 'id' with type
'LongType'`
Do we have to rebuild the tables to the newer version directly ???
Following is my hoodie.properties file content
hoodie.table.timeline.timezone=LOCAL
hoodie.table.keygenerator.class=org.apache.hudi.keygen.NonpartitionedKeyGenerator
hoodie.table.precombine.field=when_updated
hoodie.table.version=4
hoodie.database.name=
hoodie.datasource.write.hive_style_partitioning=true
hoodie.table.checksum=2716619607
hoodie.partition.metafile.use.base.format=false
hoodie.archivelog.folder=archived
hoodie.table.name=amz_hudi_vc_11_accounts
hoodie.populate.meta.fields=true
hoodie.table.type=COPY_ON_WRITE
hoodie.datasource.write.partitionpath.urlencode=false
hoodie.table.base.file.format=PARQUET
hoodie.datasource.write.drop.partition.columns=false
hoodie.table.metadata.partitions=files
hoodie.timeline.layout.version=1
hoodie.table.recordkey.fields=id
hoodie.table.partition.fields=
Following is the complete error
An error was encountered:
ALTER TABLE CHANGE COLUMN is not supported for changing column 'id' with
type 'IntegerType' to 'id' with type 'LongType'
Traceback (most recent call last):
File
"/mnt/yarn/usercache/livy/appcache/application_1718084941070_0004/container_1718084941070_0004_01_000001/pyspark.zip/pyspark/sql/session.py",
line 723, in sql
return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
File
"/mnt/yarn/usercache/livy/appcache/application_1718084941070_0004/container_1718084941070_0004_01_000001/py4j-0.10.9.3-src.zip/py4j/java_gateway.py",
line 1322, in __call__
answer, self.gateway_client, self.target_id, self.name)
File
"/mnt/yarn/usercache/livy/appcache/application_1718084941070_0004/container_1718084941070_0004_01_000001/pyspark.zip/pyspark/sql/utils.py",
line 117, in deco
raise converted from None
pyspark.sql.utils.AnalysisException: ALTER TABLE CHANGE COLUMN is not
supported for changing column 'id' with type 'IntegerType' to 'id' with type
'LongType'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
