leaves12138 commented on code in PR #8613:
URL: https://github.com/apache/paimon/pull/8613#discussion_r3577729804
##########
paimon-common/src/main/java/org/apache/paimon/globalindex/bitmap/BitmapIndexReader.java:
##########
@@ -346,26 +344,27 @@ private RoaringNavigableMap64 scanSerializedDictionary(
private int firstPossibleDictionaryBlock(
List<BitmapGlobalIndexFormat.DictionaryBlockMeta> blocks,
BitmapGlobalIndexFormat.SerializedKey key) {
- int index = findDictionaryBlockIndex(blocks, key);
+ int index = findSerializedDictionaryBlockIndex(blocks, key);
return Math.max(index, 0);
}
- private BitmapGlobalIndexFormat.BlockInfo findBitmapBlock(
- BitmapGlobalIndexFormat.SerializedKey key) {
+ private BitmapGlobalIndexFormat.BlockInfo findBitmapBlock(Object key) {
List<BitmapGlobalIndexFormat.DictionaryBlockMeta> blocks =
dictionaryBlocks.get();
if (blocks.isEmpty()) {
return null;
}
- int index = findDictionaryBlockIndex(blocks, key);
+ int index = findLogicalDictionaryBlockIndex(blocks, key);
Review Comment:
That makes sense. Since #8276 has not shipped in a release, keeping version
1 for this pre-release format change is fine. Thanks for clarifying.
##########
paimon-common/src/main/java/org/apache/paimon/globalindex/bitmap/BitmapGlobalIndexWriter.java:
##########
@@ -110,12 +121,30 @@ public List<ResultEntry> finish() {
return Collections.singletonList(new ResultEntry(fileName, rowCount,
meta));
}
- private void updateMinMax(Object key) {
- if (firstKey == null || comparator.compare(key, firstKey) < 0) {
- firstKey = key;
+ private void flushCurrentBitmap() {
+ if (currentBitmap.isEmpty()) {
+ return;
+ }
+ try {
+ streamingWriter()
+ .write(
+ BitmapGlobalIndexFormat.SerializedKey.fromObject(
+ keySerializer, lastKey),
+ currentBitmap);
+ currentBitmap.clear();
+ } catch (IOException e) {
+ throw new RuntimeException("Error in writing bitmap index files.",
e);
}
- if (lastKey == null || comparator.compare(key, lastKey) > 0) {
- lastKey = key;
+ }
+
+ private BitmapGlobalIndexFormat.StreamingWriter streamingWriter() throws
IOException {
+ if (streamingWriter == null) {
+ fileName =
fileWriter.newFileName(BitmapGlobalIndexerFactory.IDENTIFIER);
+ outputStream = fileWriter.newOutputStream(fileName);
Review Comment:
Addressed by `9cd6406`: `SortedSingleColumnIndexWriter` now closes its
active writer through try-with-resources, and Flink's `WriteIndexOperator`
closes the active writer from its lifecycle cleanup. I also reran the focused
core and Flink tests successfully. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
