[
https://issues.apache.org/jira/browse/HUDI-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890760#comment-17890760
]
sivabalan narayanan commented on HUDI-8178:
-------------------------------------------
1: Its expected. When we first create the partition stats partition in MDT, and
if empty table, there won't be any data files added. So, the hoodie partition
meta file may not be present. but after 1st commit, I validated that the meta
file is created.
2: This is an issue only for schema evolution cases. We can punt it to 1.1.0.
https://issues.apache.org/jira/browse/HUDI-8391
3: We should fix this.
4: This is specific to columns having "Date" data type. We can take this up
later in 1.1.0
https://issues.apache.org/jira/browse/HUDI-8392
> Enable partition stats by default
> ---------------------------------
>
> Key: HUDI-8178
> URL: https://issues.apache.org/jira/browse/HUDI-8178
> Project: Apache Hudi
> Issue Type: Task
> Reporter: Sagar Sumit
> Assignee: sivabalan narayanan
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.0.0
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> There are two issues that need to be fixed before we can enable partition
> stats by default:
> 1. There is a [class cast
> exception|https://dev.azure.com/apachehudi/hudi-oss-ci/_build/results?buildId=151&view=logs&j=b1544eb9-7ff1-5db9-0187-3e05abf459bc&t=e0ae894b-41c9-5f4b-7ed2-bdf5243b02e7]
> under certain curcumstance while collecting stats for log files. This
> happens even for colstats (run
> `[testReadPathsForOnlyLogFiles|https://github.com/apache/hudi/blob/33a02987b7fc385253dc0e0efbf112066b8cf190/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala#L885]`
> test with column stats enabled).
>
> {code:java}
> Caused by: java.lang.ClassCastException:
> org.apache.spark.sql.catalyst.expressions.UnsafeRow cannot be cast to
> org.apache.avro.generic.IndexedRecord
> at org.apache.avro.generic.GenericData.getField(GenericData.java:858)
> at org.apache.avro.generic.GenericData.compare(GenericData.java:1229)
> at org.apache.avro.generic.GenericData.compare(GenericData.java:1148)
> at
> org.apache.hudi.metadata.HoodieTableMetadataUtil.lambda$null$1(HoodieTableMetadataUtil.java:239)
> at java.util.ArrayList.forEach(ArrayList.java:1259)
> at
> org.apache.hudi.metadata.HoodieTableMetadataUtil.lambda$collectColumnRangeMetadata$2(HoodieTableMetadataUtil.java:222)
> at java.util.ArrayList.forEach(ArrayList.java:1259)
> at
> org.apache.hudi.metadata.HoodieTableMetadataUtil.collectColumnRangeMetadata(HoodieTableMetadataUtil.java:219)
> at
> org.apache.hudi.io.HoodieAppendHandle.processAppendResult(HoodieAppendHandle.java:424)
> at
> org.apache.hudi.io.HoodieAppendHandle.appendDataAndDeleteBlocks(HoodieAppendHandle.java:475)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
