hudi-bot opened a new issue, #17225:
URL: https://github.com/apache/hudi/issues/17225
Currently using `CustomKeyGenerator` with the partition-path config
\{hoodie.datasource.write.partitionpath.field=ts:timestamp} fails w/
{code:java}
Caused by: java.lang.RuntimeException: Failed to cast value `2022-05-11` to
`LongType` for partition column `ts_ms`
at
org.apache.spark.sql.execution.datasources.Spark3ParsePartitionUtil.$anonfun$parsePartition$2(Spark3ParsePartitionUtil.scala:72)
at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at
scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at
org.apache.spark.sql.execution.datasources.Spark3ParsePartitionUtil.$anonfun$parsePartition$1(Spark3ParsePartitionUtil.scala:65)
at scala.Option.map(Option.scala:230)
at
org.apache.spark.sql.execution.datasources.Spark3ParsePartitionUtil.parsePartition(Spark3ParsePartitionUtil.scala:63)
at
org.apache.hudi.SparkHoodieTableFileIndex.parsePartitionPath(SparkHoodieTableFileIndex.scala:274)
at
org.apache.hudi.SparkHoodieTableFileIndex.parsePartitionColumnValues(SparkHoodieTableFileIndex.scala:258)
at
org.apache.hudi.BaseHoodieTableFileIndex.lambda$getAllQueryPartitionPaths$3(BaseHoodieTableFileIndex.java:190)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
at
org.apache.hudi.BaseHoodieTableFileIndex.getAllQueryPartitionPaths(BaseHoodieTableFileIndex.java:193)
{code}
This occurs b/c SparkHoodieTableFileIndex produces incorrect partition
schema at XXX
where it properly handles only `TimestampBasedKeyGenerator`s but not the
other key-generators that might be changing the data-type of the
partition-value as compared to the source partition-column (in this case it has
`ts` as a long in the source table schema, but it produces partition-value as
string)
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4818
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9113
- Fix version(s):
- 1.1.0
---
## Comments
28/Feb/24 12:30;adityagoenka;This is still open -
One more GitHub issue reported -
[https://github.com/apache/hudi/issues/10678]
Reproducible code -
```
data = [("id1", "name1", time.time_ns()),
("id2", "name2", time.time_ns()+ 1)]
schema = StructType([
StructField("id", StringType(), True),
StructField("name", StringType(), True),
StructField("ts", LongType(), True)
])
df = spark.createDataFrame(data, schema=schema)
(df.write.format("hudi")
.option("hoodie.datasource.write.keygenerator.class",
"org.apache.hudi.keygen.CustomKeyGenerator")
.option("hoodie.datasource.write.partitionpath.field", "ts:timestamp")
.option("hoodie.datasource.write.recordkey.field", "id")
.option("hoodie.datasource.write.precombined.field", "name")
.option("hoodie.table.name", "hudi_cow2")
.option("hoodie.keygen.timebased.timestamp.type", "EPOCHMILLISECONDS")
.option("hoodie.keygen.timebased.output.dateformat", "yyyyMMdd-HH")
.mode("overwrite")
.save(PATH))
df_read_parquet = spark.read.parquet(PATH + "/5*")
df_read_parquet.show()
df_read_hudi = (spark.read.format("hudi")
.option("hoodie.schema.on.read.enable", "true")
.load(PATH))
df_read_hudi.show()
```;;;
---
17/Sep/24 02:31;guoyihua;I have a WIP fix (see the branch below) on making
partition pruning work for the timestamp-based partition field with the
timestamp or custom key generator. If this issue is related to the partition
pruning, I'll push through this fix.
[https://github.com/yihua/hudi/tree/HUDI-fix-timestamp-partition]
;;;
---
15/Oct/24 23:00;linliu;[~yihua], Do you need us to investigate this ticket,
and let you know the root cause? Since you already have the PR, should we land
it now?;;;
---
16/Oct/24 00:52;linliu;I reproduced the issue on my local, and I confirmed
this is related to the partition pruning. Would you please push your fix?
[~yihua]
;;;
---
21/Oct/24 16:10;linliu;[~yihua], I applied your fix, but it did not fix the
issue. So I have to debug then.;;;
---
21/Oct/24 18:13;linliu;The root cause of the failure is that
`.option("hoodie.keygen.timebased.output.dateformat", "yyyyMMdd-HH")`. the out
put format of the timestamp contains '\-', which cannot be cast into long type.
After removing the '-', it worked.;;;
---
22/Oct/24 01:38;yihua;I think we still need to fix this for timestamp or
custom key generator to work properly with the partition pruning.;;;
---
22/Oct/24 05:45;linliu;The description of this ticket does not mention about
partition pruning.
Please add property description or create a new ticket.;;;
---
26/Nov/24 19:30;yihua;Let's validate the behavior of the following scenarios
on both Hudi 0.14 and Hudi 1.0 to make sure there is no regression:
Key generators:
(1) Timestamp keygen, timestamp partition column (long type), SCALAR
timestamp type, yyyy-MM-dd as output format
(2) Custom keygen, simple (string type) and timestamp partition column (long
type), SCALAR timestamp type, yyyy-MM-dd as output format
Queries to test:
(1) full table scan
(2) filter for partition with string, e.g., `partition='yyyy-MM-dd'`
(3) filter for partition with timestamp, e.g., `partition between 123 and
456`, where "123" and "456" are the raw values from the partition column.;;;
---
27/Nov/24 20:07;jonvex;Here is the prototype for a fix
[https://github.com/jonvex/hudi/tree/fix_timestamp_keygen_vectorized]
in
[https://github.com/apache/spark/blob/32232e9ed33bb16b93ad58cfde8b82e0f07c0970/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala#L206]
it removes the partition only filters from the list of filters that that
should be run after reading the files. We need to not remove filters for
timestamp keygen cols;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
