>From Ritik Raj <[email protected]>:
Ritik Raj has uploaded this change for review. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21371?usp=email )
Change subject: MB-72604: flush .dic LAF pages in ascending page-id order
......................................................................
MB-72604: flush .dic LAF pages in ascending page-id order
On cloud-backed deployments the columnar .dic (Look-Aside File) is written
through an append-only cloud writer whose ensurePosition invariant requires
pages to arrive in strictly ascending offset (page-id) order. LAFWriter did
not honor that:
- writeFullPage() flushed a LAF frame as soon as it filled, i.e. in
completion order, not page-id order; and
- endWriting() flushed the remaining frames in HashMap iteration order.
During a large MERGE the bulkloader writes a leaf's column pages first and its
page-zero last, and multi-page column values add extra-page entries that
straddle LAF-page boundaries. So a higher LAF page could fill and flush while a
lower one was still incomplete. Harmless on the random-access local file, but
fatal on the append-only cloud copy: AbstractCloudIOManager.ensurePosition
throws "Misaligned positions" and the merge thread halts the JVM, which on the
affected collection turned into a crash/rollback/re-backfill loop.
Fix: hold a LAF frame back until every lower-id page has been flushed.
A nextPageToFlush cursor releases only the contiguous ascending prefix of
completed frames; endWriting flushes the remainder in ascending page-id order.
The local path is random-access and unaffected; only the append-only cloud
writer needs the ordering guarantee, and ensurePosition stays intact as a
corruption guard.
Adds runtime regression test cloud_storage/dic-merge-misalign: a COLUMN dataset
with incompressible columns inserted in batches, then COMPACT, forces a merged
component whose .dic spans multiple LAF pages. Without this change it halts the
JVM during COMPACT (in-JVM mock S3, append-only cloud writer); with it the merge
completes and the count is correct.
To surface the bug with little data, shrinks the cloud-storage test config so a
LAF page holds few entries and inserts flush into many small components:
buffercache.pagesize 32KB->2KB, memorycomponent.globalbudget 512MB->8MB, and
column.max.tuple.count=200 in cc-cloud-storage.conf.ftl. These apply to the
whole
CloudStorageTest suite.
Change-Id: Ib93e30635afe0b5f5a70a876a731a4e384133e20
---
M asterixdb/asterix-app/src/test/resources/cc-cloud-storage.conf.ftl
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.000.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.001.update.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.002.update.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.003.update.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.004.update.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.005.query.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.999.ddl.sqlpp
A
asterixdb/asterix-app/src/test/resources/runtimets/results/cloud_storage/dic-merge-misalign/mis.005.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_cloud_storage.xml
M
hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/compression/file/LAFWriter.java
11 files changed, 285 insertions(+), 16 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/71/21371/1
diff --git a/asterixdb/asterix-app/src/test/resources/cc-cloud-storage.conf.ftl
b/asterixdb/asterix-app/src/test/resources/cc-cloud-storage.conf.ftl
index 74c557a..1bd6c5d 100644
--- a/asterixdb/asterix-app/src/test/resources/cc-cloud-storage.conf.ftl
+++ b/asterixdb/asterix-app/src/test/resources/cc-cloud-storage.conf.ftl
@@ -40,7 +40,7 @@
app.class=org.apache.asterix.hyracks.bootstrap.NCApplication
jvm.args=-Xmx4096m
--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED
--add-opens=java.management/sun.management=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
-Dnode.Resolver="org.apache.asterix.external.util.IdentitiyResolverFactory"
storage.buffercache.size=128MB
-storage.memorycomponent.globalbudget=512MB
+storage.memorycomponent.globalbudget=8MB
storage.max.columns.in.zeroth.segment=800
storage.page.zero.writer=default
@@ -65,7 +65,8 @@
messaging.frame.size=4096
messaging.frame.count=512
cloud.deployment=true
-storage.buffercache.pagesize=32KB
+storage.buffercache.pagesize=2KB
+storage.column.max.tuple.count=200
storage.partitioning=static
cloud.storage.scheme=s3
cloud.storage.bucket=cloud-storage-container
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.000.ddl.sqlpp
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.000.ddl.sqlpp
new file mode 100644
index 0000000..03d570f
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.000.ddl.sqlpp
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+-- CBSE-23088-4 repro: out-of-order .dic LAF page flush vs append-only cloud
writer.
+-- COLUMN-format dataset with FAT nested-array columns (mimics
scenario_execution_detail
+-- ExecutionDetail / ScenarioResult). The fat columns make a single column
span many
+-- data pages so the component's .dic spans more than one LAF page during the
merge.
+
+DROP DATAVERSE test IF EXISTS;
+CREATE DATAVERSE test;
+
+USE test;
+
+CREATE TYPE OpenType AS {
+ id: bigint
+};
+
+CREATE DATASET ColumnDataset(OpenType)
+PRIMARY KEY id WITH {
+ "storage-format": {"format" : "column"}
+};
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.001.update.sqlpp
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.001.update.sqlpp
new file mode 100644
index 0000000..b8de7e6
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.001.update.sqlpp
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+-- Batch 1. HIGH-CARDINALITY incompressible columns: every field is a distinct
pseudo-random
+-- int per (gen,e) so dictionary/delta encoding cannot collapse the column ->
one column
+-- spans many data pages -> the component .dic spans many LAF pages during
MERGE.
+
+USE test;
+
+INSERT INTO ColumnDataset
+SELECT VALUE {
+ "id": gen,
+ "scenarioResult": (SELECT VALUE {
+ "k": (gen*2654435761 + e*40503) % 2147483647,
+ "v": (e*2654435761 + gen*40503) % 2147483647,
+ "w": (gen*e*2246822519 + e*7919) % 2147483647
+ } FROM range(1, 250) AS e),
+ "executionDetail": (SELECT VALUE {
+ "step": (gen*1000003 + e*97) % 2147483647,
+ "ts": (gen*2147483629 + e*65537) % 2147483647,
+ "note": (e*2654435761 + gen*99991) % 2147483647
+ } FROM range(1, 250) AS e)
+}
+FROM range(1, 6000) AS gen;
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.002.update.sqlpp
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.002.update.sqlpp
new file mode 100644
index 0000000..00ba2d8
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.002.update.sqlpp
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+-- Batch 2. HIGH-CARDINALITY incompressible columns: every field is a distinct
pseudo-random
+-- int per (gen,e) so dictionary/delta encoding cannot collapse the column ->
one column
+-- spans many data pages -> the component .dic spans many LAF pages during
MERGE.
+
+USE test;
+
+INSERT INTO ColumnDataset
+SELECT VALUE {
+ "id": gen,
+ "scenarioResult": (SELECT VALUE {
+ "k": (gen*2654435761 + e*40503) % 2147483647,
+ "v": (e*2654435761 + gen*40503) % 2147483647,
+ "w": (gen*e*2246822519 + e*7919) % 2147483647
+ } FROM range(1, 250) AS e),
+ "executionDetail": (SELECT VALUE {
+ "step": (gen*1000003 + e*97) % 2147483647,
+ "ts": (gen*2147483629 + e*65537) % 2147483647,
+ "note": (e*2654435761 + gen*99991) % 2147483647
+ } FROM range(1, 250) AS e)
+}
+FROM range(2500001, 2506000) AS gen;
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.003.update.sqlpp
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.003.update.sqlpp
new file mode 100644
index 0000000..b4b4fc9
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.003.update.sqlpp
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+-- Batch 3. HIGH-CARDINALITY incompressible columns: every field is a distinct
pseudo-random
+-- int per (gen,e) so dictionary/delta encoding cannot collapse the column ->
one column
+-- spans many data pages -> the component .dic spans many LAF pages during
MERGE.
+
+USE test;
+
+INSERT INTO ColumnDataset
+SELECT VALUE {
+ "id": gen,
+ "scenarioResult": (SELECT VALUE {
+ "k": (gen*2654435761 + e*40503) % 2147483647,
+ "v": (e*2654435761 + gen*40503) % 2147483647,
+ "w": (gen*e*2246822519 + e*7919) % 2147483647
+ } FROM range(1, 250) AS e),
+ "executionDetail": (SELECT VALUE {
+ "step": (gen*1000003 + e*97) % 2147483647,
+ "ts": (gen*2147483629 + e*65537) % 2147483647,
+ "note": (e*2654435761 + gen*99991) % 2147483647
+ } FROM range(1, 250) AS e)
+}
+FROM range(5000001, 5006000) AS gen;
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.004.update.sqlpp
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.004.update.sqlpp
new file mode 100644
index 0000000..970f36d
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.004.update.sqlpp
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+-- THE TRIGGER. COMPACT folds the flushed components into one large component
via the
+-- MERGE path (ColumnBTreeBulkloader.persist -> writeColumnsPages). The merged
component's
+-- .dic spans multiple LAF pages (fat columns -> many data pages); page-zero
is written
+-- LAST (low LAF id) so a higher LAF page flushes first -> out-of-order append
to the
+-- append-only cloud .dic writer -> AbstractCloudIOManager.ensurePosition
throws
+-- "Misaligned positions" -> MERGE thread halts the JVM.
+
+USE test;
+
+COMPACT DATASET ColumnDataset;
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.005.query.sqlpp
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.005.query.sqlpp
new file mode 100644
index 0000000..60fb41d
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.005.query.sqlpp
@@ -0,0 +1,25 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+-- Only reached if the merge did NOT crash. If the bug reproduced, the JVM
halted during
+-- step 004 and this step never runs (the whole test process dies = repro
confirmed).
+
+USE test;
+
+SELECT VALUE COUNT(*) FROM ColumnDataset;
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.999.ddl.sqlpp
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.999.ddl.sqlpp
new file mode 100644
index 0000000..20dc6fd
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/cloud_storage/dic-merge-misalign/mis.999.ddl.sqlpp
@@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+DROP DATAVERSE test IF EXISTS;
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/results/cloud_storage/dic-merge-misalign/mis.005.adm
b/asterixdb/asterix-app/src/test/resources/runtimets/results/cloud_storage/dic-merge-misalign/mis.005.adm
new file mode 100644
index 0000000..8f31360
--- /dev/null
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/results/cloud_storage/dic-merge-misalign/mis.005.adm
@@ -0,0 +1 @@
+18000
diff --git
a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_cloud_storage.xml
b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_cloud_storage.xml
index 683c348..b0aa911 100644
---
a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_cloud_storage.xml
+++
b/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_cloud_storage.xml
@@ -124,6 +124,13 @@
<output-dir compare="Clean-JSON">collection-exists</output-dir>
</compilation-unit>
</test-case>
+ <!-- MB-72604: large columnar MERGE must not flush .dic LAF pages to the
append-only
+ cloud writer out of order. A JVM halt during COMPACT = regression. -->
+ <test-case FilePath="cloud_storage">
+ <compilation-unit name="dic-merge-misalign">
+ <output-dir compare="Text">dic-merge-misalign</output-dir>
+ </compilation-unit>
+ </test-case>
</test-group>
&sqlpp_queries;
</test-suite>
diff --git
a/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/compression/file/LAFWriter.java
b/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/compression/file/LAFWriter.java
index 1ed462b..59a0bef 100644
---
a/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/compression/file/LAFWriter.java
+++
b/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/compression/file/LAFWriter.java
@@ -44,6 +44,13 @@
*/
@NotThreadSafe
class LAFWriter implements ICompressedPageWriter {
+ // Lowest LAF page id not yet flushed. A frame is released to the (dual
local+cloud) page
+ // writer only after every lower-id page has been flushed, so the
append-only cloud .dic
+ // writer always receives pages in ascending offset order (otherwise a
higher LAF page can
+ // flush while a lower one is still incomplete ->
AbstractCloudIOManager.ensurePosition
+ // "Misaligned positions" -> merge halts the JVM). The local file is
random-access and
+ // indifferent to flush order; only the cloud copy needs this guarantee.
See MB-72604.
+ private int nextPageToFlush;
private final CompressedFileManager compressedFileManager;
private final IBufferCache bufferCache;
private final IFIFOPageWriter pageWriter;
@@ -73,6 +80,7 @@
totalNumOfPages = 0;
maxPageId = -1;
currentPageId = -1;
+ nextPageToFlush = 0;
}
/* ************************************
@@ -140,8 +148,14 @@
*/
lastPage.setEOF();
}
- for (Entry<Integer, LAFFrame> entry : cachedFrames.entrySet()) {
- pageWriter.write(entry.getValue().cPage);
+ // Flush remaining frames in ASCENDING page-id order. The append-only
cloud .dic writer
+ // requires ascending offsets; HashMap iteration order does not
guarantee that and would
+ // trip AbstractCloudIOManager.ensurePosition ("Misaligned
positions"). See MB-72604.
+ for (int pageId = nextPageToFlush; pageId <= maxPageId; pageId++) {
+ final LAFFrame frame = cachedFrames.remove(pageId);
+ if (frame != null) {
+ flushFrame(pageId, frame);
+ }
}
//Signal the compressedFileManager to change its state
@@ -169,7 +183,7 @@
lastOffset += size;
totalNumOfPages++;
- writeFullPage();
+ flushReadyPages();
return pageOffset;
}
@@ -194,20 +208,35 @@
return frame;
}
- private void writeFullPage() throws HyracksDataException {
- if (currentFrame.isFull()) {
- //The LAF page is filled. We do not need to keep it.
- //Write it to the file and remove it from the cachedFrames map
- pageWriter.write(currentFrame.cPage);
- //Recycle the frame
- final LAFFrame frame = cachedFrames.remove(currentPageId);
- frame.setCachedPage(null);
- recycledFrames.add(frame);
- currentFrame = null;
- currentPageId = -1;
+ /**
+ * Flush the longest contiguous run of FULL LAF pages starting at {@link
#nextPageToFlush}.
+ * A higher LAF page that fills before a lower one is held back (it stays
in
+ * {@code cachedFrames}) until every lower-id page has been flushed, so
the append-only
+ * cloud .dic writer always receives pages in ascending offset order. The
local file is
+ * random-access and indifferent to order; only the cloud copy needs this
guarantee.
+ */
+ private void flushReadyPages() throws HyracksDataException {
+ LAFFrame frame;
+ while ((frame = cachedFrames.get(nextPageToFlush)) != null &&
frame.isFull()) {
+ final int pageId = nextPageToFlush;
+ cachedFrames.remove(pageId);
+ flushFrame(pageId, frame);
+ if (currentPageId == pageId) {
+ currentFrame = null;
+ currentPageId = -1;
+ }
}
}
+ /** Write one LAF page to the (dual) page writer and recycle its frame.
Advances
+ * {@link #nextPageToFlush}; callers MUST invoke this in ascending
page-id order. */
+ private void flushFrame(int pageId, LAFFrame frame) throws
HyracksDataException {
+ pageWriter.write(frame.cPage);
+ frame.setCachedPage(null);
+ recycledFrames.add(frame);
+ nextPageToFlush = pageId + 1;
+ }
+
private int getLAFEntryPageId(int compressedPageId) {
return compressedPageId * ENTRY_LENGTH / bufferCache.getPageSize();
}
--
To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21371?usp=email
To unsubscribe, or for help writing mail filters, visit
https://asterix-gerrit.ics.uci.edu/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: asterixdb
Gerrit-Branch: lumina
Gerrit-Change-Id: Ib93e30635afe0b5f5a70a876a731a4e384133e20
Gerrit-Change-Number: 21371
Gerrit-PatchSet: 1
Gerrit-Owner: Ritik Raj <[email protected]>
