mzheng-plaid commented on issue #9977:
URL: https://github.com/apache/hudi/issues/9977#issuecomment-1811115357
@ad1happy2go I haven't been able to reproduce this with dummy data at ~50gb
scale
```
from pyspark.sql.types import StringType
from pyspark.sql import functions as F
import uuid
import math
# Size calculation
# Each UUID is 36 characters, and we can estimate 2 bytes per character
(UTF-8 encoding)
# Adding some overhead for Row object storage, let’s use 100 bytes as a
rough estimate per row
row_size_bytes = 100
# Target size for the DataFrame in bytes (50gb)
target_size_bytes = 50 * 1024 * 1024 * 1024
# Calculate the number of rows needed to generate the DataFrame
num_rows = math.ceil(target_size_bytes / row_size_bytes)
print(num_rows)
uuid_udf = F.udf(lambda: str(uuid.uuid4()), StringType())
df = spark.range(num_rows).withColumn("id", uuid_udf())
df = df.withColumn("partition", F.lit("a"))
df.show(5, truncate=False)
```
```
536870912
+------------------------------------+---------+
|id |partition|
+------------------------------------+---------+
|fec3f590-f3f5-41bb-8c8c-0542ebf9e911|a |
|383880f2-943c-40f0-abe1-617a026b6848|a |
|4252b669-8d57-4346-8e1f-58bf75a7e553|a |
|d0152ad5-83e7-4e94-a488-3f09b28335ce|a |
|205f1853-61ee-4f5c-9388-b39cca5a1a67|a |
+------------------------------------+---------+
only showing top 5 rows
```
I upserted all of these rows, then clustered with a dummy upsert. I'm not
sure how to reproduce this issue with dummy data - this seems to be only a
problem at a large scale
The biggest clues I have are:
- partition for dataset where this failed is ~150gb and 400 million rows
- the base file touched by the upsert is the only one affected. If I upsert
partition X but cluster partition Y, there is no data loss.
- exactly half the rows in the base file are missing after clustering (are
base files internally split in some way? some sort of grouping?)
- clustering replaced the base file that was upserted to
I'm not sure at what point I'll have more bandwidth to dig into this issue
more, but given the seriousness of silent data loss we're just abandoning using
clustering at this point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
