Change in asterixdb[master]: ASTERIXDB-1516: Unable to find free page in buffer cache aft...
Jenkins has posted comments on this change. Change subject: ASTERIXDB-1516: Unable to find free page in buffer cache after 3 cycles .. Patch Set 3: Integration Tests Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/240/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1038 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I46fa6bbda8c2f81e5e570dd6c07e4f4b794ef5bb Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Murtadha Hubail Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Yingyi Bu Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1516: Unable to find free page in buffer cache aft...
Jenkins has posted comments on this change. Change subject: ASTERIXDB-1516: Unable to find free page in buffer cache after 3 cycles .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/2112/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1038 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I46fa6bbda8c2f81e5e570dd6c07e4f4b794ef5bb Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Murtadha Hubail Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Yingyi Bu Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1516: Unable to find free page in buffer cache aft...
Hello Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/1038 to look at the new patch set (#3). Change subject: ASTERIXDB-1516: Unable to find free page in buffer cache after 3 cycles .. ASTERIXDB-1516: Unable to find free page in buffer cache after 3 cycles Update exhaustion logic to be two-tiered: - emit warning when cycle count exceeds warning threshold (3) - fail if cycle count reaches the failure threshold (1000) Change-Id: I46fa6bbda8c2f81e5e570dd6c07e4f4b794ef5bb --- M hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java 1 file changed, 392 insertions(+), 417 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/38/1038/3 -- To view, visit https://asterix-gerrit.ics.uci.edu/1038 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I46fa6bbda8c2f81e5e570dd6c07e4f4b794ef5bb Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Murtadha Hubail Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Yingyi Bu
Change in asterixdb[master]: ASTERIXDB-1516: Unable to find free page in buffer cache aft...
Jenkins has posted comments on this change. Change subject: ASTERIXDB-1516: Unable to find free page in buffer cache after 3 cycles .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/2111/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1038 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I46fa6bbda8c2f81e5e570dd6c07e4f4b794ef5bb Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Murtadha Hubail Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Yingyi Bu Gerrit-HasComments: No
Change in asterixdb[master]: Align Jetty Version, ResultExtractor.extract Accessible
Till Westmann has posted comments on this change. Change subject: Align Jetty Version, ResultExtractor.extract Accessible .. Patch Set 1: Code-Review+2 -- To view, visit https://asterix-gerrit.ics.uci.edu/1047 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iaa0a6cbd5dc6a4aaa153b0f0e2414ec6402256ce Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: Align Jetty Version, ResultExtractor.extract Accessible
Jenkins has posted comments on this change. Change subject: Align Jetty Version, ResultExtractor.extract Accessible .. Patch Set 1: Integration Tests Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/239/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1047 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: Iaa0a6cbd5dc6a4aaa153b0f0e2414ec6402256ce Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: handle missing format parameter
Jenkins has posted comments on this change. Change subject: handle missing format parameter .. Patch Set 2: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/238/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/1046 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I90b6da30e2e3a52d68cdf29b0026125dca4da3e4 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Till WestmannGerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1479: Change storage valid int and add explicit ve...
Ian Maxon has submitted this change and it was merged. Change subject: ASTERIXDB-1479: Change storage valid int and add explicit version .. ASTERIXDB-1479: Change storage valid int and add explicit version This change will stop old versions from possibly corrupting data written by newer versions of Hyracks. It also adds an explicit version to the tree metadata that must match the version at runtime. Change-Id: I888ff0eacf5b3cb6ad7ec002c74f113c6ffcd496 Reviewed-on: https://asterix-gerrit.ics.uci.edu/919 Sonar-Qube: JenkinsTested-by: Jenkins Integration-Tests: Jenkins Reviewed-by: Michael Blow --- M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataBootstrap.java M asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceFactory.java M asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexMetaDataFrame.java M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/dataflow/IndexDataflowHelper.java M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/frames/LIFOMetaDataFrame.java M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/freepage/LinkedMetaDataPageManager.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/impls/AbstractLSMIndexFileManager.java M hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/file/ILocalResourceFactory.java M hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/file/LocalResource.java M hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/file/TransientLocalResourceFactory.java 11 files changed, 89 insertions(+), 31 deletions(-) Approvals: Michael Blow: Looks good to me, approved Jenkins: Verified; No violations found; Verified diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataBootstrap.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataBootstrap.java index c180be4..757694d 100644 --- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataBootstrap.java +++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataBootstrap.java @@ -77,6 +77,7 @@ import org.apache.hyracks.api.dataflow.value.ITypeTraits; import org.apache.hyracks.api.io.FileReference; import org.apache.hyracks.api.io.IIOManager; +import org.apache.hyracks.storage.am.common.frames.LIFOMetaDataFrame; import org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree; import org.apache.hyracks.storage.am.lsm.btree.util.LSMBTreeUtils; import org.apache.hyracks.storage.am.lsm.common.api.ILSMMergePolicyFactory; @@ -373,7 +374,7 @@ localResourceMetadata, LocalResource.LSMBTreeResource); ILocalResourceFactory localResourceFactory = localResourceFactoryProvider.getLocalResourceFactory(); localResourceRepository.insert(localResourceFactory.createLocalResource(resourceID, resourceName, -metadataPartition.getPartitionId(), absolutePath)); +metadataPartition.getPartitionId(), LIFOMetaDataFrame.VERSION, absolutePath)); dataLifecycleManager.register(absolutePath, lsmBtree); } else { final LocalResource resource = localResourceRepository.getResourceByPath(absolutePath); diff --git a/asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceFactory.java b/asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceFactory.java index 15224e2..33e5bf2 100644 --- a/asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceFactory.java +++ b/asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceFactory.java @@ -33,7 +33,9 @@ } @Override -public LocalResource createLocalResource(long resourceId, String resourceName, int partition, String resourcePath) { -return new LocalResource(resourceId, resourceName, partition, resourcePath, resourceType, localResourceMetadata); +public LocalResource createLocalResource(long
Change in asterixdb[master]: handle missing format parameter
Jenkins has posted comments on this change. Change subject: handle missing format parameter .. Patch Set 2: Integration Tests Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/238/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1046 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I90b6da30e2e3a52d68cdf29b0026125dca4da3e4 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Till WestmannGerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: Filter Sample Cluster Scripts To Enable Extension
Jenkins has posted comments on this change. Change subject: Filter Sample Cluster Scripts To Enable Extension .. Patch Set 2: Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/237/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/1044 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6f6f27ddcc2311630e8e234ce2770cbb2c66d253 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: handle missing format parameter
Jenkins has posted comments on this change. Change subject: handle missing format parameter .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/2108/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1046 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I90b6da30e2e3a52d68cdf29b0026125dca4da3e4 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Till WestmannGerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: handle missing format parameter
Till Westmann has uploaded a new change for review. https://asterix-gerrit.ics.uci.edu/1046 Change subject: handle missing format parameter .. handle missing format parameter Change-Id: I90b6da30e2e3a52d68cdf29b0026125dca4da3e4 --- M asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/servlet/QueryServiceServlet.java 1 file changed, 8 insertions(+), 1 deletion(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/46/1046/1 diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/servlet/QueryServiceServlet.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/servlet/QueryServiceServlet.java index f208694..7f54bb8 100644 --- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/servlet/QueryServiceServlet.java +++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/servlet/QueryServiceServlet.java @@ -215,6 +215,9 @@ } private static String getParameterValue(String content, String attribute) { +if (content == null || attribute == null) { +return null; +} int sc = content.indexOf(';'); if (sc < 0) { return null; @@ -227,6 +230,10 @@ return content.substring(eq + 1).trim().toLowerCase(); } return null; +} + +private static String toLower(String s) { +return s != null ? s.toLowerCase() : s; } private static SessionConfig.OutputFormat getFormat(String format) { @@ -262,7 +269,7 @@ return app; }; -String formatstr = request.getParameter(Parameter.FORMAT.str()).toLowerCase(); +String formatstr = toLower(request.getParameter(Parameter.FORMAT.str())); SessionConfig.OutputFormat format = getFormat(formatstr); SessionConfig sessionConfig = new SessionConfig(resultWriter, format, resultPrefix, resultPostfix); sessionConfig.set(SessionConfig.FORMAT_WRAPPER_ARRAY, true); -- To view, visit https://asterix-gerrit.ics.uci.edu/1046 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I90b6da30e2e3a52d68cdf29b0026125dca4da3e4 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Till Westmann
Change in asterixdb[master]: Filter Sample Cluster Scripts To Enable Extension
Till Westmann has posted comments on this change. Change subject: Filter Sample Cluster Scripts To Enable Extension .. Patch Set 2: Code-Review+2 -- To view, visit https://asterix-gerrit.ics.uci.edu/1044 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6f6f27ddcc2311630e8e234ce2770cbb2c66d253 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: Filter Sample Cluster Scripts To Enable Extension
Jenkins has posted comments on this change. Change subject: Filter Sample Cluster Scripts To Enable Extension .. Patch Set 2: Integration Tests Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/237/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1044 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6f6f27ddcc2311630e8e234ce2770cbb2c66d253 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: Filter Sample Cluster Scripts To Enable Extension
Jenkins has posted comments on this change. Change subject: Filter Sample Cluster Scripts To Enable Extension .. Patch Set 2: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/2107/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1044 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6f6f27ddcc2311630e8e234ce2770cbb2c66d253 Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
[jira] [Commented] (ASTERIXDB-1560) Misleading SQL++ error message when variables are undefined
[ https://issues.apache.org/jira/browse/ASTERIXDB-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406529#comment-15406529 ] Yingyi Bu commented on ASTERIXDB-1560: -- >>That actually raises an orthogonal issue - perhaps we should (someday) add >>ANY as a supported typename and then you wouldn't need two separate >>mechanisms. +1 :-) > Misleading SQL++ error message when variables are undefined > > > Key: ASTERIXDB-1560 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1560 > Project: Apache AsterixDB > Issue Type: Bug > Components: AsterixDB, Error Reporting >Reporter: Michael J. Carey >Assignee: Yingyi Bu >Priority: Minor > > This query: > SELECT u.name AS uname, m.message AS message > FROM GleambookUsers u > UNNEST GleambookMessages m > WHERE m.authorId = user.id; > Yields this error message: > Cannot resolve ambiguous alias (variable) reference for identifier user > [AlgebricksException] > The reference is undefined, not ambiguous, in this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ASTERIXDB-1560) Misleading SQL++ error message when variables are undefined
[ https://issues.apache.org/jira/browse/ASTERIXDB-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406520#comment-15406520 ] Michael J. Carey commented on ASTERIXDB-1560: - Thx! Meanwhile, regarding the last/long comment - COOL! That actually raises an orthogonal issue - perhaps we should (someday) add ANY as a supported typename and then you wouldn't need two separate mechanisms. (We'd have a type-system-supported way of having fields with known names but variable types.) > Misleading SQL++ error message when variables are undefined > > > Key: ASTERIXDB-1560 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1560 > Project: Apache AsterixDB > Issue Type: Bug > Components: AsterixDB, Error Reporting >Reporter: Michael J. Carey >Assignee: Yingyi Bu >Priority: Minor > > This query: > SELECT u.name AS uname, m.message AS message > FROM GleambookUsers u > UNNEST GleambookMessages m > WHERE m.authorId = user.id; > Yields this error message: > Cannot resolve ambiguous alias (variable) reference for identifier user > [AlgebricksException] > The reference is undefined, not ambiguous, in this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ASTERIXDB-1556) Prefix-based multi-way Fuzzy-join generates an exception.
[ https://issues.apache.org/jira/browse/ASTERIXDB-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406474#comment-15406474 ] Michael J. Carey commented on ASTERIXDB-1556: - Just one thing to add/clarify/double-check. Where one of the points above says HASH TABLE SIZE = HASH TABLE SPACE I'm not quite sure what is meant. We should be sure to account for all the hash table's space use, in frames. (Which given how it's built under the covers should not be a hard problem at all.) I just want to be clear that "HASH TABLE SIZE" should mean its total footprint, not to be confused with the number of initial entries (array elements) in its directory/table structure. > Prefix-based multi-way Fuzzy-join generates an exception. > - > > Key: ASTERIXDB-1556 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1556 > Project: Apache AsterixDB > Issue Type: Bug >Reporter: Taewoo Kim >Assignee: Taewoo Kim > Attachments: 2wayjoin.pdf, 2wayjoin.rtf, 2wayjoinplan.rtf, > 3wayjoin.pdf, 3wayjoin.rtf, 3wayjoinplan.rtf > > > When we enable prefix-based fuzzy-join and apply the multi-way fuzzy-join ( > > 2), the system generates an out-of-memory exception. > Since a fuzzy-join is created using 30-40 lines of AQL codes and this AQL is > translated into massive number of operators (more than 200 operators in the > plan for a 3-way fuzzy join), it could generate out-of-memory exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ASTERIXDB-1556) Prefix-based multi-way Fuzzy-join generates an exception.
[ https://issues.apache.org/jira/browse/ASTERIXDB-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406282#comment-15406282 ] Jianfeng Jia commented on ASTERIXDB-1556: - I'd like to be in the discussion if needed :-) > Prefix-based multi-way Fuzzy-join generates an exception. > - > > Key: ASTERIXDB-1556 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1556 > Project: Apache AsterixDB > Issue Type: Bug >Reporter: Taewoo Kim >Assignee: Taewoo Kim > Attachments: 2wayjoin.pdf, 2wayjoin.rtf, 2wayjoinplan.rtf, > 3wayjoin.pdf, 3wayjoin.rtf, 3wayjoinplan.rtf > > > When we enable prefix-based fuzzy-join and apply the multi-way fuzzy-join ( > > 2), the system generates an out-of-memory exception. > Since a fuzzy-join is created using 30-40 lines of AQL codes and this AQL is > translated into massive number of operators (more than 200 operators in the > plan for a 3-way fuzzy join), it could generate out-of-memory exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Change in asterixdb[master]: ASTERIXDB-1516: Unable to find free page in buffer cache aft...
Murtadha Hubail has posted comments on this change. Change subject: ASTERIXDB-1516: Unable to find free page in buffer cache after 3 cycles .. Patch Set 1: (4 comments) Just minor comments and suggestions. https://asterix-gerrit.ics.uci.edu/#/c/1038/1/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java File hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/BufferCache.java: Line 370: while (cPage != null) { This loop is repeated three times inside this method. You may refactor it as private CachedPage findWinnerInBucket(long dpid, CacheBucket bucket, CachedPage victim) Line 1178: if (victim != null) { I think if you do similar check in here as findPageInner, the code will be easier to follow. i.e. if (victim == null) { return null; } Line 1259: finally{ formatting Line 1298: do { do we need this do/while loop? -- To view, visit https://asterix-gerrit.ics.uci.edu/1038 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I46fa6bbda8c2f81e5e570dd6c07e4f4b794ef5bb Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Murtadha Hubail Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: Yingyi Bu Gerrit-HasComments: Yes
Change in asterixdb[master]: Filter Sample Cluster Scripts To Enable Extension
Till Westmann has posted comments on this change. Change subject: Filter Sample Cluster Scripts To Enable Extension .. Patch Set 1: Code-Review+2 I think that I understand better now. -- To view, visit https://asterix-gerrit.ics.uci.edu/1044 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6f6f27ddcc2311630e8e234ce2770cbb2c66d253 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
[jira] [Comment Edited] (ASTERIXDB-1556) Prefix-based multi-way Fuzzy-join generates an exception.
[ https://issues.apache.org/jira/browse/ASTERIXDB-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406254#comment-15406254 ] Taewoo Kim edited comment on ASTERIXDB-1556 at 8/3/16 5:21 PM: --- (2) spilling the data frame is already implemented. We just don't have any within budget protection mechanism regarding the hash table size allocation and its overflow. We need to add two logics: #1. allocating a hash table frame from the global budget #2. spilling whenever data or hash table can't allocate another frame. Regarding the time, I need to understand the current structure of hash table and data spilling logic so it may take a while. But, without applying this protection mechanism, we can't say that reducing the table size of external group-by solves the out of memory issue regarding the fuzzy join 100%. So, I will do my best to reduce the time. was (Author: wangsaeu): (2) spilling the data frame is already implemented. We just don't have any within budget protection mechanism regarding the hash table size allocation and its overflow. We need to add two logics: #1. allocating a hash table frame from the global budget #2. spilling whenever data or hash table can't allocate another frame. Regarding the time, I need to understand the current structure of hash table and data spilling logic so it may take a while. But, without apply this protection mechanism, we can't say that reducing the table size of external group-by solves the out of memory issue regarding the fuzzy join 100%. So, I will do my best to reduce the time. > Prefix-based multi-way Fuzzy-join generates an exception. > - > > Key: ASTERIXDB-1556 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1556 > Project: Apache AsterixDB > Issue Type: Bug >Reporter: Taewoo Kim >Assignee: Taewoo Kim > Attachments: 2wayjoin.pdf, 2wayjoin.rtf, 2wayjoinplan.rtf, > 3wayjoin.pdf, 3wayjoin.rtf, 3wayjoinplan.rtf > > > When we enable prefix-based fuzzy-join and apply the multi-way fuzzy-join ( > > 2), the system generates an out-of-memory exception. > Since a fuzzy-join is created using 30-40 lines of AQL codes and this AQL is > translated into massive number of operators (more than 200 operators in the > plan for a 3-way fuzzy join), it could generate out-of-memory exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ASTERIXDB-1556) Prefix-based multi-way Fuzzy-join generates an exception.
[ https://issues.apache.org/jira/browse/ASTERIXDB-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406254#comment-15406254 ] Taewoo Kim commented on ASTERIXDB-1556: --- (2) spilling the data frame is already implemented. We just don't have any within budget protection mechanism regarding the hash table size allocation and its overflow. We need to add two logics: #1. allocating a hash table frame from the global budget #2. spilling whenever data or hash table can't allocate another frame. Regarding the time, I need to understand the current structure of hash table and data spilling logic so it may take a while. But, without apply this protection mechanism, we can't say that reducing the table size of external group-by solves the out of memory issue regarding the fuzzy join 100%. So, I will do my best to reduce the time. > Prefix-based multi-way Fuzzy-join generates an exception. > - > > Key: ASTERIXDB-1556 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1556 > Project: Apache AsterixDB > Issue Type: Bug >Reporter: Taewoo Kim >Assignee: Taewoo Kim > Attachments: 2wayjoin.pdf, 2wayjoin.rtf, 2wayjoinplan.rtf, > 3wayjoin.pdf, 3wayjoin.rtf, 3wayjoinplan.rtf > > > When we enable prefix-based fuzzy-join and apply the multi-way fuzzy-join ( > > 2), the system generates an out-of-memory exception. > Since a fuzzy-join is created using 30-40 lines of AQL codes and this AQL is > translated into massive number of operators (more than 200 operators in the > plan for a 3-way fuzzy join), it could generate out-of-memory exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ASTERIXDB-1556) Prefix-based multi-way Fuzzy-join generates an exception.
[ https://issues.apache.org/jira/browse/ASTERIXDB-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406181#comment-15406181 ] Chen Li commented on ASTERIXDB-1556: [~wangsaeu]: The discussion results make sense to me. A few questions: (1) Yourkit reported 80MB memory for each operator, which is very consistent with your calculation "INT_SIZE * 2 * table size (10,485,767)." What does this 80MB really include then? Hash Table + Data? (2) For the case we don't have enough memory, what do we do to "spill tuples to the disk"? Can you elaborate? (3) Seem we have two limit parameters, namely "MAX LIMIT OF FRAME" for data and "MAX LIMIT OF HASH TABLE FRAME" for the hash table. Can we simply it by having a single limit? We can talk again after you become available in a few days. > Prefix-based multi-way Fuzzy-join generates an exception. > - > > Key: ASTERIXDB-1556 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1556 > Project: Apache AsterixDB > Issue Type: Bug >Reporter: Taewoo Kim >Assignee: Taewoo Kim > Attachments: 2wayjoin.pdf, 2wayjoin.rtf, 2wayjoinplan.rtf, > 3wayjoin.pdf, 3wayjoin.rtf, 3wayjoinplan.rtf > > > When we enable prefix-based fuzzy-join and apply the multi-way fuzzy-join ( > > 2), the system generates an out-of-memory exception. > Since a fuzzy-join is created using 30-40 lines of AQL codes and this AQL is > translated into massive number of operators (more than 200 operators in the > plan for a 3-way fuzzy join), it could generate out-of-memory exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Change in asterixdb[master]: Align / Advance Jetty Version
Till Westmann has posted comments on this change. Change subject: Align / Advance Jetty Version .. Patch Set 1: Ok, maybe we can file an issue for that for now? However, I'm not sure understand the test failures. It seems that there should be no conflict on signer information if we use the same dependency - where is the origin of the signer information? -- To view, visit https://asterix-gerrit.ics.uci.edu/1045 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I2eba5eafc7b82ced978bed6c5b71e46b7a662f72 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: Filter Sample Cluster Scripts To Enable Extension
Till Westmann has posted comments on this change. Change subject: Filter Sample Cluster Scripts To Enable Extension .. Patch Set 1: (2 comments) Just a few questions to help me understand. https://asterix-gerrit.ics.uci.edu/#/c/1044/1/asterixdb/asterix-server/pom.xml File asterixdb/asterix-server/pom.xml: Line 152: ${project.basedir}/src/main/assembly/filter.properties How can this be overridden if I want to extend the AsterixDB? https://asterix-gerrit.ics.uci.edu/#/c/1044/1/asterixdb/pom.xml File asterixdb/pom.xml: Line 589: false This disables the use of non-Hyracks snapshot releases? -- To view, visit https://asterix-gerrit.ics.uci.edu/1044 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6f6f27ddcc2311630e8e234ce2770cbb2c66d253 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: Yes
Change in asterixdb[master]: Align / Advance Jetty Version
Michael Blow has posted comments on this change. Change subject: Align / Advance Jetty Version .. Patch Set 1: Not without some work to consume Hyracks dependencies from a BOM, but I think this actually might be a worthwhile improvement. -- To view, visit https://asterix-gerrit.ics.uci.edu/1045 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I2eba5eafc7b82ced978bed6c5b71e46b7a662f72 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: Align / Advance Jetty Version
Till Westmann has posted comments on this change. Change subject: Align / Advance Jetty Version .. Patch Set 1: As asterix-app depends on hyracks-control-cc, would it be ok to remove the dependency from asterix-app? Or is that undesirable as asterix-app wouldn't declare its direct dependencies correctly? -- To view, visit https://asterix-gerrit.ics.uci.edu/1045 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I2eba5eafc7b82ced978bed6c5b71e46b7a662f72 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Jenkins Gerrit-Reviewer: Till Westmann Gerrit-HasComments: No
Change in asterixdb[master]: Filter Sample Cluster Scripts To Enable Extension
Jenkins has posted comments on this change. Change subject: Filter Sample Cluster Scripts To Enable Extension .. Patch Set 1: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/236/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/1044 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6f6f27ddcc2311630e8e234ce2770cbb2c66d253 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: Align / Advance Jetty Version
Jenkins has posted comments on this change. Change subject: Align / Advance Jetty Version .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/2106/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1045 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I2eba5eafc7b82ced978bed6c5b71e46b7a662f72 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: Align / Advance Jetty Version
Michael Blow has uploaded a new change for review. https://asterix-gerrit.ics.uci.edu/1045 Change subject: Align / Advance Jetty Version .. Align / Advance Jetty Version Advances Jetty version from: - 8.0.0.RC0 -> 8.1.19.v20160209 (hyracks-control-cc) - 8.0.0.M2 -> 8.1.19.v20160209 (asterix-app) Change-Id: I2eba5eafc7b82ced978bed6c5b71e46b7a662f72 --- M asterixdb/asterix-app/pom.xml M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/pom.xml 2 files changed, 4 insertions(+), 4 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/45/1045/1 diff --git a/asterixdb/asterix-app/pom.xml b/asterixdb/asterix-app/pom.xml index 70c777d..149a36a 100644 --- a/asterixdb/asterix-app/pom.xml +++ b/asterixdb/asterix-app/pom.xml @@ -184,14 +184,14 @@ org.eclipse.jetty jetty-server - 8.0.0.M2 + 8.1.19.v20160209 jar compile org.eclipse.jetty jetty-servlet - 8.0.0.M2 + 8.1.19.v20160209 jar compile diff --git a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/pom.xml b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/pom.xml index 2b59165..5186ae0 100644 --- a/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/pom.xml +++ b/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/pom.xml @@ -41,14 +41,14 @@ org.eclipse.jetty jetty-server - 8.0.0.RC0 + 8.1.19.v20160209 jar compile org.eclipse.jetty jetty-webapp - 8.0.0.RC0 + 8.1.19.v20160209 jar compile -- To view, visit https://asterix-gerrit.ics.uci.edu/1045 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I2eba5eafc7b82ced978bed6c5b71e46b7a662f72 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael Blow
Change in asterixdb[master]: Filter Sample Cluster Scripts To Enable Extension
Jenkins has posted comments on this change. Change subject: Filter Sample Cluster Scripts To Enable Extension .. Patch Set 1: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/2105/ -- To view, visit https://asterix-gerrit.ics.uci.edu/1044 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I6f6f27ddcc2311630e8e234ce2770cbb2c66d253 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Jenkins Gerrit-HasComments: No
Change in asterixdb[master]: Filter Sample Cluster Scripts To Enable Extension
Michael Blow has uploaded a new change for review. https://asterix-gerrit.ics.uci.edu/1044 Change subject: Filter Sample Cluster Scripts To Enable Extension .. Filter Sample Cluster Scripts To Enable Extension Change-Id: I6f6f27ddcc2311630e8e234ce2770cbb2c66d253 --- M asterixdb/asterix-server/pom.xml M asterixdb/asterix-server/src/main/assembly/binary-assembly.xml A asterixdb/asterix-server/src/main/assembly/filter.properties M asterixdb/asterix-server/src/main/resources/samples/local/bin/start-sample-cluster.sh M asterixdb/asterix-server/src/main/resources/samples/local/bin/stop-sample-cluster.sh M asterixdb/asterix-server/src/main/resources/samples/local/conf/cc.conf M asterixdb/pom.xml 7 files changed, 29 insertions(+), 9 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/44/1044/1 diff --git a/asterixdb/asterix-server/pom.xml b/asterixdb/asterix-server/pom.xml index b083bbc..78e990f 100644 --- a/asterixdb/asterix-server/pom.xml +++ b/asterixdb/asterix-server/pom.xml @@ -148,8 +148,11 @@ + + ${project.basedir}/src/main/assembly/filter.properties + -src/main/assembly/binary-assembly.xml + ${project.basedir}/src/main/assembly/binary-assembly.xml package diff --git a/asterixdb/asterix-server/src/main/assembly/binary-assembly.xml b/asterixdb/asterix-server/src/main/assembly/binary-assembly.xml index 379a6f2..4afde5f 100644 --- a/asterixdb/asterix-server/src/main/assembly/binary-assembly.xml +++ b/asterixdb/asterix-server/src/main/assembly/binary-assembly.xml @@ -43,9 +43,21 @@ src/main/resources . -LICENSE* samples/** + +**/*.sh + + true + + + src/main/resources + . + +samples/**/*.sh + + true + 0755 target/appassembler/repo diff --git a/asterixdb/asterix-server/src/main/assembly/filter.properties b/asterixdb/asterix-server/src/main/assembly/filter.properties new file mode 100644 index 000..301aeaf --- /dev/null +++ b/asterixdb/asterix-server/src/main/assembly/filter.properties @@ -0,0 +1,5 @@ +NC_SERVICE_COMMAND=asterixncservice +CC_COMMAND=asterixcc +NC_COMMAND=asterixnc +LISTEN_PORT=19002 +SHUTDOWN_SERVLET_PATH=admin/shutdown diff --git a/asterixdb/asterix-server/src/main/resources/samples/local/bin/start-sample-cluster.sh b/asterixdb/asterix-server/src/main/resources/samples/local/bin/start-sample-cluster.sh index eb26384..39d5410 100755 --- a/asterixdb/asterix-server/src/main/resources/samples/local/bin/start-sample-cluster.sh +++ b/asterixdb/asterix-server/src/main/resources/samples/local/bin/start-sample-cluster.sh @@ -54,13 +54,13 @@ echo "--" ) | tee -a $LOGSDIR/blue-service.log | tee -a $LOGSDIR/red-service.log >> $LOGSDIR/cc.log -$CBASDIR/bin/asterixncservice -logdir - -config-file $CLUSTERDIR/conf/blue.conf >> $LOGSDIR/blue-service.log 2>&1 & -$CBASDIR/bin/asterixncservice -logdir - >> $LOGSDIR/red-service.log 2>&1 & -$CBASDIR/bin/asterixcc -config-file $CLUSTERDIR/conf/cc.conf >> $LOGSDIR/cc.log 2>&1 & +$CBASDIR/bin/${NC_SERVICE_COMMAND} -logdir - -config-file $CLUSTERDIR/conf/blue.conf >> $LOGSDIR/blue-service.log 2>&1 & +$CBASDIR/bin/${NC_SERVICE_COMMAND} -logdir - >> $LOGSDIR/red-service.log 2>&1 & +$CBASDIR/bin/${CC_COMMAND} -config-file $CLUSTERDIR/conf/cc.conf >> $LOGSDIR/cc.log 2>&1 & echo -n "Waiting for cluster to start." while [ 1 ]; do - curl -s -o /dev/null http://localhost:19002 && break + curl -s -o /dev/null http://localhost:${LISTEN_PORT} && break echo -n "." sleep 1s done diff --git a/asterixdb/asterix-server/src/main/resources/samples/local/bin/stop-sample-cluster.sh b/asterixdb/asterix-server/src/main/resources/samples/local/bin/stop-sample-cluster.sh index 4a64553..9836cb3 100755 --- a/asterixdb/asterix-server/src/main/resources/samples/local/bin/stop-sample-cluster.sh +++ b/asterixdb/asterix-server/src/main/resources/samples/local/bin/stop-sample-cluster.sh @@ -32,7 +32,7 @@ exit 2 } -curl -X POST http://localhost:19002/admin/shutdown +curl -X POST http://localhost:${LISTEN_PORT}/${SHUTDOWN_SERVLET_PATH} $JAVA_HOME/bin/jps | awk '/NCService/ { print $1 }' | xargs kill echo echo -n "Waiting for CCDriver to terminate." diff --git a/asterixdb/asterix-server/src/main/resources/samples/local/conf/cc.conf b/asterixdb/asterix-server/src/main/resources/samples/local/conf/cc.conf index 99a97c1c..5ef1cbf 100644 --- a/asterixdb/asterix-server/src/main/resources/samples/local/conf/cc.conf +++ b/asterixdb/asterix-server/src/main/resources/samples/local/conf/cc.conf @@ -12,7 +12,7 @@ [nc] storagedir=storage address=127.0.0.1 -command=asterixnc +command=${NC_COMMAND} [cc] cluster.address =