[
https://issues.apache.org/jira/browse/HUDI-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259324#comment-17259324
]
Prashant Wason commented on HUDI-1509:
--------------------------------------
So calling getCombinedFieldsToWrite() is adding 275usec for writing each record
to ParquetWriter which takes approximately 200usec or so for the write call.
Everything else remaining same this nearly double the time to complete the
ingestion.
> Major performance degradation due to rewriting records with default values
> --------------------------------------------------------------------------
>
> Key: HUDI-1509
> URL: https://issues.apache.org/jira/browse/HUDI-1509
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Prashant Wason
> Priority: Blocker
>
> During the in-house testing for 0.5x to 0.6x release upgrade, I have detected
> a performance degradation for writes into HUDI. I have traced the issue due
> to the changes in the following commit
> [[HUDI-727]: Copy default values of fields if not present when rewriting
> incoming record with new
> schema|https://github.com/apache/hudi/commit/6d7ca2cf7e441ad19d32d7a25739e454f39ed253]
> I wrote a unit test to reduce the scope of testing as follows:
> # Take an existing parquet file from production dataset (size=690MB,
> #records=960K)
> # Read all the records from this parquet into a JavaRDD
> # Time the call HoodieWriteClient.bulkInsertPrepped().
> (bulkInsertParallelism=1)
> The above scenario is directly taken from our production pipelines where each
> executor will ingest about a million record creating a single parquet file in
> a COW dataset. This is bulk insert only dataset.
> The time to complete the bulk insert prepped *decreased from 680seconds to
> 380seconds* when I reverted the above commit.
> Schema details: This HUDI dataset uses a large schema with 51 fields in the
> record.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
