lei-su-awx commented on issue #10309:
URL: https://github.com/apache/hudi/issues/10309#issuecomment-1918659000
I had a similar question, when table schema is double, and the incoming data
schema is long, then why the data can not upsert into table? I think double can
handle long. Below is my code(hudi 0.14.0, spark3.4.1):
```
from pyspark.sql.types import StructType, StructField, StringType,
ArrayType, MapType, DecimalType, LongType, BooleanType, DoubleType, IntegerType
from pyspark.sql import Row
schema1 = StructType(
[
StructField("id", IntegerType(), True),
StructField("value", DoubleType(), True)
]
)
schema2 = StructType(
[
StructField("id", IntegerType(), True),
StructField("value", LongType(), True)
]
)
data1 = [
Row(1, 10000000000.0),
Row(2, 10000000000.0),
Row(3,10000000000.0),
]
data2 = [
Row(1, 100),
Row(2, 200),
Row(3,10000000000),
]
hudi_configs = {
"hoodie.table.name": 'table',
"hoodie.datasource.write.precombine.field":"value",
"hoodie.datasource.write.recordkey.field":"id",
'hoodie.datasource.write.reconcile.schema': 'true',
'hoodie.schema.on.read.enable': 'true',
}
PATH = 'gs://data_lake_staging_hk/data/raw_lei/test/'
df = spark.createDataFrame(spark.sparkContext.parallelize(data1), schema1)
df.write.format("org.apache.hudi").options(**hudi_configs).mode("append").save(PATH)
spark.read.format("org.apache.hudi").load(PATH).printSchema()
spark.read.format("org.apache.hudi").load(PATH).show()
df = spark.createDataFrame(spark.sparkContext.parallelize(data2), schema2)
df.write.format("org.apache.hudi").options(**hudi_configs).mode("append").save(PATH)
spark.read.format("org.apache.hudi").load(PATH).printSchema()
spark.read.format("org.apache.hudi").load(PATH).show()
```
the stack trace is:
`IllegalArgumentException: cannot update origin type: double to a
incompatibility type: long`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
