yihua commented on a change in pull request #5004:
URL: https://github.com/apache/hudi/pull/5004#discussion_r831461154
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieHFileConfig.java
##########
@@ -42,12 +42,12 @@
private final boolean dropBehindCacheCompaction;
private final Configuration hadoopConf;
private final BloomFilter bloomFilter;
- private final KeyValue.KVComparator hfileComparator;
+ private final CellComparator hfileComparator;
Review comment:
In this way, we don't need to ship our own comparator in
`org.apache.hadoop.hbase` and exclude that from shading.
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieHFileConfig.java
##########
@@ -42,12 +42,12 @@
private final boolean dropBehindCacheCompaction;
private final Configuration hadoopConf;
private final BloomFilter bloomFilter;
- private final KeyValue.KVComparator hfileComparator;
+ private final CellComparator hfileComparator;
Review comment:
Actually, I took another approach here. In HBase, there is already
backward compatibility to transform the old comparator. When we package the
Hudi bundles, we need to make sure that
`org.apache.hadoop.hbase.KeyValue$KeyComparator` (written to the metadata table
HFiles in Hudi 0.9.0) is not shaded, not to break the backward compatibility
logic in HBase. The new rule is added for this. More details can be found in
the PR description.
```
<relocation>
<pattern>org.apache.hadoop.hbase.</pattern>
<shadedPattern>org.apache.hudi.org.apache.hadoop.hbase.</shadedPattern>
<excludes>
<exclude>org.apache.hadoop.hbase.KeyValue$KeyComparator</exclude>
</excludes>
</relocation>
```
##########
File path: packaging/hudi-flink-bundle/pom.xml
##########
@@ -162,6 +171,22 @@
<pattern>org.apache.avro.</pattern>
<shadedPattern>${flink.bundle.shade.prefix}org.apache.avro.</shadedPattern>
</relocation>
+ <relocation>
+ <pattern>org.apache.commons.io.</pattern>
Review comment:
Actually, bootstrap index HFiles always use our own comparator class
under `org.apache.hudi` package so we're good here. The problem is with the
metadata table HFiles in Hudi 0.9.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
