[
https://issues.apache.org/jira/browse/HUDI-2745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-2745:
----------------------------
Description:
Spark Shell command to do snapshot query:
{code:java}
val basePath = "/tmp/hoodie/hudi-test-topic"
val df = spark.read.format("hudi").load(basePath)
df.createOrReplaceTempView("hudi_test_table")
spark.sql("select count(*) from hudi_test_table").show() {code}
Two cases of count mismatch:
(1) Compaction scheduled, more deltacommits later on: the count does not match
input size. After compaction is executed. The count becomes correct.
(2) Clustering scheduled, more deltacommits later on: the count is correct,
equal to the input size. After clustering is executed, the count drops and
becomes incorrect.
> Record count does not match input after compaction is scheduled when running
> Hudi Kafka Connect sink
> ----------------------------------------------------------------------------------------------------
>
> Key: HUDI-2745
> URL: https://issues.apache.org/jira/browse/HUDI-2745
> Project: Apache Hudi
> Issue Type: Bug
> Components: Compaction
> Reporter: Ethan Guo
> Assignee: Ethan Guo
> Priority: Blocker
> Fix For: 0.10.0
>
>
> Spark Shell command to do snapshot query:
> {code:java}
> val basePath = "/tmp/hoodie/hudi-test-topic"
> val df = spark.read.format("hudi").load(basePath)
> df.createOrReplaceTempView("hudi_test_table")
> spark.sql("select count(*) from hudi_test_table").show() {code}
> Two cases of count mismatch:
> (1) Compaction scheduled, more deltacommits later on: the count does not
> match input size. After compaction is executed. The count becomes correct.
> (2) Clustering scheduled, more deltacommits later on: the count is correct,
> equal to the input size. After clustering is executed, the count drops and
> becomes incorrect.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
