陈磊 created HUDI-7609:
------------------------
Summary: Spark cannot write the hudi table containing array type
created by flink
Key: HUDI-7609
URL: https://issues.apache.org/jira/browse/HUDI-7609
Project: Apache Hudi
Issue Type: Bug
Reporter: 陈磊
When flink creates a Hudi table containing an array field, the elements of the
default array field cannot be nullable. When using Spark SQL to read data from
the Hive table to the Hudi table, a field verification exception will occur.
{code:java}
2024-03-27 12:47:51 INFO
org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to
dataType on unresolved object, tree: 'decentral_level1
2024-03-27 12:47:51 INFO at
org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.dataType(unresolved.scala:138)
2024-03-27 12:47:51 INFO at
org.apache.spark.sql.types.StructType$.$anonfun$fromAttributes$1(StructType.scala:549)
2024-03-27 12:47:51 INFO at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
2024-03-27 12:47:51 INFO at
scala.collection.immutable.List.foreach(List.scala:392)
2024-03-27 12:47:51 INFO at
scala.collection.TraversableLike.map(TraversableLike.scala:238)
2024-03-27 12:47:51 INFO at
scala.collection.TraversableLike.map$(TraversableLike.scala:231)
2024-03-27 12:47:51 INFO at scala.collection.immutable.List.map(List.scala:298)
2024-03-27 12:47:51 INFO at
org.apache.spark.sql.types.StructType$.fromAttributes(StructType.scala:549)
2024-03-27 12:47:51 INFO at
org.apache.spark.sql.catalyst.plans.QueryPlan.schema$lzycompute(QueryPlan.scala:281)
2024-03-27 12:47:51 INFO at
org.apache.spark.sql.catalyst.plans.QueryPlan.schema(QueryPlan.scala:281)
2024-03-27 12:47:51 INFO at
org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand$.alignQueryOutput(InsertIntoHoodieTableCommand.scala:153)
2024-03-27 12:47:51 INFO at
org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand$.run(InsertIntoHoodieTableCommand.scala:105)
2024-03-27 12:47:51 INFO at
org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand.run(InsertIntoHoodieTableCommand.scala:60)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
