[
https://issues.apache.org/jira/browse/HUDI-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-7609:
----------------------------
Story Points: 4
> Spark cannot write the hudi table containing array type created by flink
> ------------------------------------------------------------------------
>
> Key: HUDI-7609
> URL: https://issues.apache.org/jira/browse/HUDI-7609
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: 陈磊
> Assignee: Jonathan Vexler
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.16.0, 1.0.0
>
>
> When flink creates a Hudi table containing an array field, the elements of
> the default array field cannot be nullable. When using Spark SQL to read data
> from the Hive table to the Hudi table, a field verification exception will
> occur.
> {code:java}
> 2024-03-27 12:47:51 INFO
> org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to
> dataType on unresolved object, tree: 'decentral_level1
> 2024-03-27 12:47:51 INFO at
> org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.dataType(unresolved.scala:138)
> 2024-03-27 12:47:51 INFO at
> org.apache.spark.sql.types.StructType$.$anonfun$fromAttributes$1(StructType.scala:549)
> 2024-03-27 12:47:51 INFO at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> 2024-03-27 12:47:51 INFO at
> scala.collection.immutable.List.foreach(List.scala:392)
> 2024-03-27 12:47:51 INFO at
> scala.collection.TraversableLike.map(TraversableLike.scala:238)
> 2024-03-27 12:47:51 INFO at
> scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> 2024-03-27 12:47:51 INFO at
> scala.collection.immutable.List.map(List.scala:298)
> 2024-03-27 12:47:51 INFO at
> org.apache.spark.sql.types.StructType$.fromAttributes(StructType.scala:549)
> 2024-03-27 12:47:51 INFO at
> org.apache.spark.sql.catalyst.plans.QueryPlan.schema$lzycompute(QueryPlan.scala:281)
> 2024-03-27 12:47:51 INFO at
> org.apache.spark.sql.catalyst.plans.QueryPlan.schema(QueryPlan.scala:281)
> 2024-03-27 12:47:51 INFO at
> org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand$.alignQueryOutput(InsertIntoHoodieTableCommand.scala:153)
> 2024-03-27 12:47:51 INFO at
> org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand$.run(InsertIntoHoodieTableCommand.scala:105)
> 2024-03-27 12:47:51 INFO at
> org.apache.spark.sql.hudi.command.InsertIntoHoodieTableCommand.run(InsertIntoHoodieTableCommand.scala:60)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
