nsivabalan commented on issue #2508:
URL: https://github.com/apache/hudi/issues/2508#issuecomment-772655440
I did ran some local tests w/ array column and things are fine. given below
is the schema that I tested locally. Also, from the stack trace, don't think
encoders are the issue. but I too don't have much idea as of now. need more
time to triage.
@n3nash / @bvaradar : Can you guys spot any issues based on information
given buy author.
```
diagnostics: User class threw exception: java.lang.Exception: Error on
Table: order, Error Message: org.apache.spark.SparkException: Job aborted due
to stage failure: Task 7 in stage 28.0 failed 4 times, most recent failure:
Lost task 7.3 in stage 28.0 (TID 530,
ip-10-0-29-119.us-west-2.compute.internal, executor 5):
org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType
UPDATE for partition :7
at
org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:279)
at
org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$execute$ecf5068c$1(BaseSparkCommitActionExecutor.java:135)
```
```
public static final StructType STRUCT_TYPE = new StructType(new
StructField[] {
new StructField(HoodieRecord.COMMIT_TIME_METADATA_FIELD,
DataTypes.StringType, false, Metadata.empty()),
new StructField(HoodieRecord.COMMIT_SEQNO_METADATA_FIELD,
DataTypes.StringType, false, Metadata.empty()),
new StructField(HoodieRecord.RECORD_KEY_METADATA_FIELD,
DataTypes.StringType, false, Metadata.empty()),
new StructField(HoodieRecord.PARTITION_PATH_METADATA_FIELD,
DataTypes.StringType, false, Metadata.empty()),
new StructField(HoodieRecord.FILENAME_METADATA_FIELD,
DataTypes.StringType, false, Metadata.empty()),
new StructField("randomInt", DataTypes.IntegerType, false,
Metadata.empty()),
new StructField("randomLong", DataTypes.LongType, false,
Metadata.empty()),
new StructField("array_long",
DataTypes.createArrayType(DataTypes.LongType), false, Metadata.empty())
});public static final StructType STRUCT_TYPE = new StructType(new
StructField[] {
new StructField(HoodieRecord.COMMIT_TIME_METADATA_FIELD,
DataTypes.StringType, false, Metadata.empty()),
new StructField(HoodieRecord.COMMIT_SEQNO_METADATA_FIELD,
DataTypes.StringType, false, Metadata.empty()),
new StructField(HoodieRecord.RECORD_KEY_METADATA_FIELD,
DataTypes.StringType, false, Metadata.empty()),
new StructField(HoodieRecord.PARTITION_PATH_METADATA_FIELD,
DataTypes.StringType, false, Metadata.empty()),
new StructField(HoodieRecord.FILENAME_METADATA_FIELD,
DataTypes.StringType, false, Metadata.empty()),
new StructField("randomInt", DataTypes.IntegerType, false,
Metadata.empty()),
new StructField("randomLong", DataTypes.LongType, false,
Metadata.empty()),
new StructField("array_long",
DataTypes.createArrayType(DataTypes.LongType), false, Metadata.empty())
});
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
