[jira] [Commented] (ASTERIXDB-2552) Implement micro external sort operator for subplans
[ https://issues.apache.org/jira/browse/ASTERIXDB-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833162#comment-16833162 ] ASF subversion and git services commented on ASTERIXDB-2552: Commit 631846bfb6f6e8546ac6a544afbd071a6b724344 in asterixdb's branch refs/heads/master from Ali Alsuliman [ https://gitbox.apache.org/repos/asf?p=asterixdb.git;h=631846b ] [ASTERIXDB-2552][RT] Reuse objects on reseting buffer manager - user model changes: no - storage format changes: no - interface changes: yes Details: The "VariableFrameMemoryManager" releases the physical frames objects, the logical frames objects and the free slots objects on reset(). This patch is to retain those objects and only release them on close() (as also stated in the interface). - added close() to IFrameFreeSlotPolicy to make the interface similar to the other interfaces (IFrameBufferManager, IFramePool, ... etc) where close() is supposed to release while reset() is supposed to only reset but retain resources. - renamed "InMemorySortRuntimeFactory" to "MicroSortRuntimeFactory". Change-Id: I88bea27e5024c621412ef609475e2a7ba1913afa Reviewed-on: https://asterix-gerrit.ics.uci.edu/3372 Contrib: Jenkins Tested-by: Jenkins Integration-Tests: Jenkins Reviewed-by: Dmitry Lychagin Reviewed-by: Till Westmann > Implement micro external sort operator for subplans > --- > > Key: ASTERIXDB-2552 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-2552 > Project: Apache AsterixDB > Issue Type: Bug > Components: RT - Runtime >Affects Versions: 0.9.4 >Reporter: Ali Alsuliman >Assignee: Ali Alsuliman >Priority: Major > Fix For: 0.9.4.2 > > > The external sort operator deals gracefully with insufficient memory. > However, the sort operator that is used in subplans currently is in memory > sort and not an external sort and thus could run out of memory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Change in asterixdb[master]: [ASTERIXDB-2552][RT] Reuse objects on reseting buffer manager
Ali Alsuliman has submitted this change and it was merged. ( https://asterix-gerrit.ics.uci.edu/3372 ) Change subject: [ASTERIXDB-2552][RT] Reuse objects on reseting buffer manager .. [ASTERIXDB-2552][RT] Reuse objects on reseting buffer manager - user model changes: no - storage format changes: no - interface changes: yes Details: The "VariableFrameMemoryManager" releases the physical frames objects, the logical frames objects and the free slots objects on reset(). This patch is to retain those objects and only release them on close() (as also stated in the interface). - added close() to IFrameFreeSlotPolicy to make the interface similar to the other interfaces (IFrameBufferManager, IFramePool, ... etc) where close() is supposed to release while reset() is supposed to only reset but retain resources. - renamed "InMemorySortRuntimeFactory" to "MicroSortRuntimeFactory". Change-Id: I88bea27e5024c621412ef609475e2a7ba1913afa Reviewed-on: https://asterix-gerrit.ics.uci.edu/3372 Contrib: Jenkins Tested-by: Jenkins Integration-Tests: Jenkins Reviewed-by: Dmitry Lychagin Reviewed-by: Till Westmann --- M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/MicroStableSortPOperator.java R hyracks-fullstack/algebricks/algebricks-runtime/src/main/java/org/apache/hyracks/algebricks/runtime/operators/sort/MicroSortRuntimeFactory.java M hyracks-fullstack/algebricks/algebricks-tests/src/test/java/org/apache/hyracks/algebricks/tests/pushruntime/PushRuntimeTest.java M hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/buffermanager/FrameFreeSlotBiggestFirst.java M hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/buffermanager/FrameFreeSlotLastFit.java M hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/buffermanager/FrameFreeSlotSmallestFit.java M hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/buffermanager/IFrameBufferManager.java M hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/buffermanager/IFrameFreeSlotPolicy.java M hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/buffermanager/VariableDeletableTupleMemoryManager.java M hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/buffermanager/VariableFrameMemoryManager.java 10 files changed, 90 insertions(+), 43 deletions(-) Approvals: Jenkins: Verified; ; Verified Anon. E. Moose (1000171): Dmitry Lychagin: Looks good to me, but someone else must approve Till Westmann: Looks good to me, approved Objections: Jenkins: Violations found diff --git a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/MicroStableSortPOperator.java b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/MicroStableSortPOperator.java index c6a7d9d..413c1a4 100644 --- a/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/MicroStableSortPOperator.java +++ b/hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/MicroStableSortPOperator.java @@ -32,7 +32,7 @@ import org.apache.hyracks.algebricks.data.IBinaryComparatorFactoryProvider; import org.apache.hyracks.algebricks.data.INormalizedKeyComputerFactoryProvider; import org.apache.hyracks.algebricks.runtime.base.IPushRuntimeFactory; -import org.apache.hyracks.algebricks.runtime.operators.sort.InMemorySortRuntimeFactory; +import org.apache.hyracks.algebricks.runtime.operators.sort.MicroSortRuntimeFactory; import org.apache.hyracks.api.dataflow.value.IBinaryComparatorFactory; import org.apache.hyracks.api.dataflow.value.INormalizedKeyComputerFactory; import org.apache.hyracks.api.dataflow.value.RecordDescriptor; @@ -80,7 +80,7 @@ i++; } -IPushRuntimeFactory runtime = new InMemorySortRuntimeFactory(sortFields, nkcf, comps, null, maxNumberOfFrames); +IPushRuntimeFactory runtime = new MicroSortRuntimeFactory(sortFields, nkcf, comps, null, maxNumberOfFrames); builder.contributeMicroOperator(op, runtime, recDescriptor); ILogicalOperator src = op.getInputs().get(0).getValue(); builder.contributeGraphEdge(src, 0, op, 0); diff --git a/hyracks-fullstack/algebricks/algebricks-runtime/src/main/java/org/apache/hyracks/algebricks/runtime/operators/sort/InMemorySortRuntimeFactory.java b/hyracks-fullstack/algebricks/algebricks-runtime/src/main/java/org/apache/hyracks/algebricks/runtime/operators/sort/MicroSortRuntimeFactory.java similarity index 96%
Change in asterixdb[master]: [ASTERIXDB-2552][RT] Reuse objects on reseting buffer manager
Till Westmann has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/3372 ) Change subject: [ASTERIXDB-2552][RT] Reuse objects on reseting buffer manager .. Patch Set 2: Code-Review+2 -- To view, visit https://asterix-gerrit.ics.uci.edu/3372 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88bea27e5024c621412ef609475e2a7ba1913afa Gerrit-Change-Number: 3372 Gerrit-PatchSet: 2 Gerrit-Owner: Ali Alsuliman Gerrit-Reviewer: Ali Alsuliman Gerrit-Reviewer: Anon. E. Moose (1000171) Gerrit-Reviewer: Dmitry Lychagin Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-Comment-Date: Sat, 04 May 2019 21:18:18 + Gerrit-HasComments: No
Change in asterixdb[master]: [NO ISSUE][FUN] Simplify error message + cleanup
Anon. E. Moose (1000171) has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/3381 ) Change subject: [NO ISSUE][FUN] Simplify error message + cleanup .. Patch Set 1: Analytics Compatibility Tests Failed https://cbjenkins.page.link/RSz6P5DtfSkofAmu5 : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/3381 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bbcb63f266685befc80de957687249191345063 Gerrit-Change-Number: 3381 Gerrit-PatchSet: 1 Gerrit-Owner: Till Westmann Gerrit-Reviewer: Anon. E. Moose (1000171) Gerrit-Reviewer: Hussain Towaileb Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-Comment-Date: Sat, 04 May 2019 18:12:36 + Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIX-2559] Prevent missing parameter in external UDF
Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/3379 ) Change subject: [ASTERIX-2559] Prevent missing parameter in external UDF .. Patch Set 3: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/8493/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/3379 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1fe2098d1e1c5e179a0635d9d6cc3a1534278259 Gerrit-Change-Number: 3379 Gerrit-PatchSet: 3 Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Anon. E. Moose (1000171) Gerrit-Reviewer: Jenkins Gerrit-Comment-Date: Sat, 04 May 2019 05:54:08 + Gerrit-HasComments: No
Change in asterixdb[master]: [ASTERIX-2559] Prevent missing parameter in external UDF
Anon. E. Moose (1000171) has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/3379 ) Change subject: [ASTERIX-2559] Prevent missing parameter in external UDF .. Patch Set 3: Contrib-2 Analytics Compatibility Tests Failed https://cbjenkins.page.link/ccXvsFy4FsXHi9K66 : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/3379 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1fe2098d1e1c5e179a0635d9d6cc3a1534278259 Gerrit-Change-Number: 3379 Gerrit-PatchSet: 3 Gerrit-Owner: Xikui Wang Gerrit-Reviewer: Anon. E. Moose (1000171) Gerrit-Reviewer: Jenkins Gerrit-Comment-Date: Sat, 04 May 2019 06:07:32 + Gerrit-HasComments: No
