hudi-bot opened a new issue, #17224:
URL: https://github.com/apache/hudi/issues/17224
Hi,
I have discovered an issue that doesn't play nicely with the custom key
generatosr, basically anything that is not TimestampBasedKeyGenerator or
TimestampBasedAvroKeyGenerator.
{{While trying to read a table that was created with these parameters(the
rest don't matter):}}
{code:java}
hoodie.datasource.write.recordkey.field=query_id,event_type
hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.CustomKeyGenerator
hoodie.datasource.write.partitionpath.field=create_time_epoch_seconds:timestamp
hoodie.deltastreamer.keygen.timebased.timestamp.type=UNIX_TIMESTAMP
hoodie.deltastreamer.keygen.timebased.output.dateformat=yyyy/MM/dd{code}
{color:#172b4d}I get and error that looks like:{color}
{code:java}
22/07/20 20:32:48 DEBUG Spark32HoodieParquetFileFormat: Appending
StructType(StructField(create_time_epoch_seconds,LongType,true)) [2022/07/13]
22/07/20 20:32:48 ERROR Executor: Exception in task 0.0 in stage 5.0 (TID 5)
java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String
cannot be cast to java.lang.Long
at scala.runtime.BoxesRunTime.unboxToLong(BoxesRunTime.java:107)
at
org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getLong(rows.scala:42)
at
org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow.getLong$(rows.scala:42)
at
org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getLong(rows.scala:195)
at
org.apache.spark.sql.execution.vectorized.ColumnVectorUtils.populate(ColumnVectorUtils.java:66)
at
org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.initBatch(VectorizedParquetRecordReader.java:245)
{code}
Apparently the issue is in _partitionSchemaFromProperties function in file:
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkHoodieTableFileIndex.scala
that checks for a class type it uses StructType of String for.
Once it is any non Timestamp based known class it basically uses whatever
type it is and then fails to retrieve the value for.
I have a proposal here which we probably need: Give a user a way to force a
string type if needed and add ability to add a prefixed column that contains a
processed partition value. It could be done as two separate features.
This problem is critical for me, so I have to change Hoodie source code on
my end temporary to make it work.
Here is how I roughly changed the referenced function:
{code:java}
/**
* Get the partition schema from the hoodie.properties.
*/
private lazy val _partitionSchemaFromProperties: StructType = {
val tableConfig = metaClient.getTableConfig
val partitionColumns = tableConfig.getPartitionFields
if (partitionColumns.isPresent) {
val partitionFields = partitionColumns.get().map(column =>
StructField("_hoodie_"+column, StringType))
StructType(partitionFields)
} else {
// If the partition columns have not stored in hoodie.properties(the
table that was
// created earlier), we trait it as a non-partitioned table.
logWarning("No partition columns available from hoodie.properties." +
" Partition pruning will not work")
new StructType()
}
} {code}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4430
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9113
- Affects version(s):
- 0.12.0
- Fix version(s):
- 1.1.0
---
## Comments
20/Jul/22 21:00;vburenin;{{Here is how I read it:}}
{code:java}
spark.read().format("org.apache.hudi")
.load(BASE_PATH + "/test_table").createOrReplaceTempView("some_table");
spark.sql("SELECT * FROM some_table").show();
spark.stop();
{code}
;;;
---
22/Jul/22 06:28;rmahindra;Looks like your input column is of type string,
did you try with the following config:
|{{hoodie.deltastreamer.keygen.timebased.timestamp.type="DATE_STRING"}}|
and
|{{hoodie.deltastreamer.keygen.timebased.input.dateformat="yyyy-MM-dd'T'HH:mm:ssZ,yyyy-MM-dd'T'HH:mm:ss.SSSZ,yyyyMMdd"}}|
depending on the actual format;;;
---
22/Jul/22 14:02;vburenin;[~rmahindra] The input column is a unix timestamp.
It is integer and appears like integer in parquet file. The processed output
looks exactly how I would expect. The problem is on the read side. Once I
change the code the way I mentioned, I can read the table, the
'create_time_epoch_seconds' appears as integer in the table as well as I also
get
_hoodie_create_time_epoch_seconds containing "2020/07/22" without quotes.
The problem with the original code is that it looks at the hardcoded class
names it is using and doesn't allow to use anything else to get a string type:
{code:java}
if
(classOf[TimestampBasedKeyGenerator].getName.equalsIgnoreCase(keyGeneratorClassName)
||
classOf[TimestampBasedAvroKeyGenerator].getName.equalsIgnoreCase(keyGeneratorClassName)){code}
;;;
---
17/Sep/22 19:12;vburenin;Here is an example PR how it could be fixed:
https://github.com/apache/hudi/pull/6172;;;
---
19/Sep/22 19:38;rmahindra;[~alexey.kudinkin] Will help with this.;;;
---
25/Oct/24 11:24;ljain;I have verified that this issue is not seen anymore
with the latest fixes in HUDI-7996 and HUDI-8036.
[~vburenin] Can you please try again with the latest master?;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
