TengHuo commented on code in PR #3774:
URL: https://github.com/apache/hudi/pull/3774#discussion_r950987984
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -533,4 +534,42 @@ public void close() throws Exception {
* @param instantTime The timestamp to use for the deltacommit.
*/
protected abstract void commit(List<HoodieRecord> records, String
partitionName, String instantTime);
+
+ /**
+ * Perform a compaction on the Metadata Table.
+ *
+ * Cases to be handled:
+ * 1. We cannot perform compaction if there are previous inflight
operations on the dataset. This is because
+ * a compacted metadata base file at time Tx should represent all the
actions on the dataset till time Tx.
+ *
+ * 2. In multi-writer scenario, a parallel operation with a greater
instantTime may have completed creating a
+ * deltacommit.
+ */
+ protected void compactIfNecessary(AbstractHoodieWriteClient writeClient,
String instantTime) {
+ String latestDeltacommitTime =
metadataMetaClient.getActiveTimeline().getDeltaCommitTimeline().filterCompletedInstants().lastInstant()
+ .get().getTimestamp();
+ List<HoodieInstant> pendingInstants =
dataMetaClient.reloadActiveTimeline().filterInflightsAndRequested()
+
.findInstantsBefore(latestDeltacommitTime).getInstants().collect(Collectors.toList());
+
+ if (!pendingInstants.isEmpty()) {
+ LOG.info(String.format("Cannot compact metadata table as there are %d
inflight instants before latest deltacommit %s: %s",
+ pendingInstants.size(), latestDeltacommitTime,
Arrays.toString(pendingInstants.toArray())));
+ return;
+ }
+
+ // Trigger compaction with suffixes based on the same instant time. This
ensures that any future
+ // delta commits synced over will not have an instant time lesser than the
last completed instant on the
+ // metadata table.
+ final String compactionInstantTime = latestDeltacommitTime + "001";
Review Comment:
Hi @danny0405
I encountered a `DateTimeParseException` in this method today. This is the
error stack when I run a test case
`TestMetadataTableWithSparkDataSource.testReadability` with metrics function
enabled.
```log
java.time.format.DateTimeParseException: Text '00000000000000001' could not
be parsed: Invalid value for YearOfEra (valid values 1 - 999999999/1000000000): 0
at
java.time.format.DateTimeFormatter.createError(DateTimeFormatter.java:1920)
at java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1855)
at java.time.LocalDateTime.parse(LocalDateTime.java:492)
at
org.apache.hudi.common.table.timeline.HoodieInstantTimeGenerator.parseDateFromInstantTime(HoodieInstantTimeGenerator.java:102)
at
org.apache.hudi.common.table.timeline.HoodieActiveTimeline.parseDateFromInstantTime(HoodieActiveTimeline.java:84)
at
org.apache.hudi.client.SparkRDDWriteClient.completeCompaction(SparkRDDWriteClient.java:322)
at
org.apache.hudi.client.SparkRDDWriteClient.completeTableService(SparkRDDWriteClient.java:461)
at
org.apache.hudi.client.SparkRDDWriteClient.compact(SparkRDDWriteClient.java:346)
...
```
The exception shows there is an illegal date text, so I checked code, found
out the illegal date coming from here, where `latestDeltacommitTime` is
"00000000000000", then `compactionInstantTime` is "00000000000000001", which is
an illegal date for `HoodieInstantTimeGenerator.parseDateFromInstantTime`.
As I understand, the value "00000000000000" in `latestDeltacommitTime` comes
from the metadata table timeline, which is an init timestamp in metadata table.
May I ask if it is a good solution that I disable Hudi metrics in
`metadataWriteConfig` by default? Do you have a better idea?
There is a similar issue I found before in PR:
https://github.com/apache/hudi/pull/6000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
