Change in asterixdb[master]: ASTERIXDB-1475: Fix Integration Tests
Michael Blow has uploaded a new change for review. https://asterix-gerrit.ics.uci.edu/922 Change subject: ASTERIXDB-1475: Fix Integration Tests .. ASTERIXDB-1475: Fix Integration Tests Ensure test feed classes are available on classpath for dataset-with-meta-record test. Change-Id: I1881becff30004a52711daaa475b177619010e47 --- M asterixdb/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/dataset-with-meta-record/create_and_start.sh 1 file changed, 4 insertions(+), 0 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/22/922/1 diff --git a/asterixdb/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/dataset-with-meta-record/create_and_start.sh b/asterixdb/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/dataset-with-meta-record/create_and_start.sh index 945f01d..b3e6d26 100755 --- a/asterixdb/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/dataset-with-meta-record/create_and_start.sh +++ b/asterixdb/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/dataset-with-meta-record/create_and_start.sh @@ -1 +1,5 @@ $MANAGIX_HOME/bin/managix create -n nc1 -c $MANAGIX_HOME/clusters/local/local.xml; +$MANAGIX_HOME/bin/managix stop -n nc1; +cp $MANAGIX_HOME/../../../asterix-external-data/target/asterix-external-data-*-tests.jar \ +$MANAGIX_HOME/clusters/local/working_dir/asterix/repo/ +$MANAGIX_HOME/bin/managix start -n nc1; -- To view, visit https://asterix-gerrit.ics.uci.edu/922 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I1881becff30004a52711daaa475b177619010e47 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael Blow
Change in asterixdb[master]: Files necessary for initial perf regression tests
Ian Maxon has submitted this change and it was merged. Change subject: Files necessary for initial perf regression tests .. Files necessary for initial perf regression tests These are files that are run as part of the asterix-perf job on our Jenkins CI server. Basically it is a set of queries that are run using the LSM experiments framework. Change-Id: I21e2d44ca46a4a28478d9ef256c4fa8d53f03dc9 Reviewed-on: https://asterix-gerrit.ics.uci.edu/884 Tested-by: JenkinsReviewed-by: Young-Seok Kim --- A asterixdb/asterix-experiments/src/main/java/org/apache/asterix/experiment/action/derived/RunSQLPPFileAction.java A asterixdb/asterix-experiments/src/main/java/org/apache/asterix/experiment/builder/AbstractPerfLoadBuilder.java A asterixdb/asterix-experiments/src/main/java/org/apache/asterix/experiment/builder/Experiment1EBuilder.java A asterixdb/asterix-experiments/src/main/java/org/apache/asterix/experiment/builder/PerfTestAggBuilder.java M asterixdb/asterix-experiments/src/main/java/org/apache/asterix/experiment/client/LSMExperimentConstants.java M asterixdb/asterix-experiments/src/main/java/org/apache/asterix/experiment/client/LSMExperimentSetRunner.java A asterixdb/asterix-experiments/src/main/java/org/apache/asterix/experiment/client/LSMPerfConstants.java A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/agg-3-months.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/agg-6-months.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/agg-day.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/agg-hour.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/agg-minute.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/agg-month.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/agg-week.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/countjoin-3-months.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/countjoin-6-months.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/countjoin-day.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/countjoin-hour.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/countjoin-minute.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/countjoin-month.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/countjoin-week.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/countjoin_index-3-months.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/countjoin_index-6-months.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/countjoin_index-day.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/countjoin_index-hour.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/countjoin_index-minute.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/countjoin_index-month.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/countjoin_index-week.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/join-3-months.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/join-6-months.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/join-day.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/join-hour.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/join-minute.sqlpp A asterixdb/asterix-experiments/src/main/resources/ingestion-experiment-binary-and-configs/aql/agg_bench/join-month.sqlpp A
Change in asterixdb[master]: ASTERIXDB-1479: Change storage valid int and add explicit ve...
Jenkins has posted comments on this change. Change subject: ASTERIXDB-1479: Change storage valid int and add explicit version .. Patch Set 8: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/1650/ -- To view, visit https://asterix-gerrit.ics.uci.edu/919 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I888ff0eacf5b3cb6ad7ec002c74f113c6ffcd496 Gerrit-PatchSet: 8 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Ian MaxonGerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
[jira] [Created] (ASTERIXDB-1479) Explicit Storage Versioning
Ian Maxon created ASTERIXDB-1479: Summary: Explicit Storage Versioning Key: ASTERIXDB-1479 URL: https://issues.apache.org/jira/browse/ASTERIXDB-1479 Project: Apache AsterixDB Issue Type: Improvement Components: Hyracks, Storage Reporter: Ian Maxon Assignee: Ian Maxon Right now storage versioning is largely implied, there's no exact indication of when an index structure has changed from one revision to the next. We should have a marker in the metadata page that explicitly is changed along with any sort of backwards-incompatible storage change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Change in asterixdb[master]: ASTERIXDB-1474: Cannot query dataset with big obj
Jenkins has posted comments on this change. Change subject: ASTERIXDB-1474: Cannot query dataset with big obj .. Patch Set 3: Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/1646/ -- To view, visit https://asterix-gerrit.ics.uci.edu/920 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I2f47c551a053e193519a4337c338934f3e171afd Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1474: Cannot query dataset with big obj
Hello Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/920 to look at the new patch set (#3). Change subject: ASTERIXDB-1474: Cannot query dataset with big obj .. ASTERIXDB-1474: Cannot query dataset with big obj Fixes ASTERIXDB-1474: Cannot query a dataset with big object after restart - large page information managed by buffer cache, outside of page - size header written out a prefix to pages Change-Id: I2f47c551a053e193519a4337c338934f3e171afd --- M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/AsterixStorageProperties.java A asterixdb/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/big_object_20M/big_object_20M.1.script.aql C asterixdb/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/big_object_20M/big_object_20M.2.ddl.aql C asterixdb/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/big_object_20M/big_object_20M.3.update.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/big_object_20M/big_object_20M.4.script.aql R asterixdb/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/big_object_20M/big_object_20M.5.query.aql A asterixdb/asterix-installer/src/test/resources/transactionts/queries/query_after_restart/big_object_20M/big_object_20M.6.script.aql A asterixdb/asterix-installer/src/test/resources/transactionts/results/query_after_restart/big_object_20M/big_object_20M.1.adm.template A asterixdb/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/big_object_20M/create_and_start.sh A asterixdb/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/big_object_20M/stop_and_delete.sh A asterixdb/asterix-installer/src/test/resources/transactionts/scripts/query_after_restart/big_object_20M/stop_and_start.sh M asterixdb/asterix-installer/src/test/resources/transactionts/testsuite.xml M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/IIOManager.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/io/IOManager.java M hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/api/IBTreeFrame.java M hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/api/IBTreeLeafFrame.java M hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeFieldPrefixNSMLeafFrame.java M hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeFieldPrefixNSMLeafFrameFactory.java D hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeLargeFrameHelper.java M hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMInteriorFrame.java M hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMInteriorFrameFactory.java M hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMLeafFrame.java M hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/frames/BTreeNSMLeafFrameFactory.java M hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/BTree.java M hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/BTreeCountingSearchCursor.java M hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/BTreeOpContext.java M hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/impls/BTreeRangeSearchCursor.java M hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/util/BTreeUtils.java M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexFrame.java M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/api/ITreeIndexFrameFactory.java M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/frames/TreeIndexNSMFrame.java M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/impls/AbstractTreeIndex.java M hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/impls/TreeIndexDiskOrderScanCursor.java M
Change in asterixdb[master]: ASTERIXDB-1474: Cannot query dataset with big obj
Michael Blow has posted comments on this change. Change subject: ASTERIXDB-1474: Cannot query dataset with big obj .. Patch Set 2: (6 comments) https://asterix-gerrit.ics.uci.edu/#/c/920/2/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/IIOManager.java File hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/IIOManager.java: Line 46: public long syncWrite(IFileHandle fHandle, long offset, ByteBuffer[] dataArray) throws HyracksDataException; > looking at this method definition, no one would guess that the dataArray is IIOManager is generic and can be used for lots of things, I don't think we would want to corrupt this interface with BufferCache's specific use pattern Line 50: public long syncRead(IFileHandle fHandle, long offset, ByteBuffer[] dataArray) throws HyracksDataException; > Remove unused? or comment future use? Done https://asterix-gerrit.ics.uci.edu/#/c/920/2/hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/io/IOManager.java File hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/io/IOManager.java: Line 189: @Override > remove method? Done https://asterix-gerrit.ics.uci.edu/#/c/920/2/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/util/TreeIndexBufferCacheWarmup.java File hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/util/TreeIndexBufferCacheWarmup.java: Line 47: > Am I seeing tabs?? Yes, I think we should remove tabs from files in a separate change (they're not being introduced here) https://asterix-gerrit.ics.uci.edu/#/c/920/2/hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/util/TreeIndexStatsGatherer.java File hyracks-fullstack/hyracks/hyracks-storage-am-common/src/main/java/org/apache/hyracks/storage/am/common/util/TreeIndexStatsGatherer.java: Line 46:ITreeIndexFrame interiorFrame, ITreeIndexMetaDataFrame metaFrame) > Huh! how come Sonar doesn't detect those? Because they are not new. https://asterix-gerrit.ics.uci.edu/#/c/920/2/hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/ClockPageReplacementStrategy.java File hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/buffercache/ClockPageReplacementStrategy.java: Line 228: // TODO(mblow): is dropping pages on the floor enough to adhere to memory budget? > I believe it is enough. would be nice to find a better source to confirm Done -- To view, visit https://asterix-gerrit.ics.uci.edu/920 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I2f47c551a053e193519a4337c338934f3e171afd Gerrit-PatchSet: 2 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Michael BlowGerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Till Westmann Gerrit-Reviewer: abdullah alamoudi Gerrit-HasComments: Yes
Change in asterixdb[master]: some code reduction/cleanup
Hello Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/916 to look at the new patch set (#4). Change subject: some code reduction/cleanup .. some code reduction/cleanup Change-Id: I179a16fc183bd50d5d58cc12321234df1615abfd --- M asterixdb/asterix-om/src/main/java/org/apache/asterix/dataflow/data/nontagged/serde/ARecordSerializerDeserializer.java M asterixdb/asterix-om/src/main/java/org/apache/asterix/om/pointables/ARecordVisitablePointable.java M asterixdb/asterix-om/src/main/java/org/apache/asterix/om/typecomputer/impl/RecordRemoveFieldsTypeComputer.java 3 files changed, 88 insertions(+), 117 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/16/916/4 -- To view, visit https://asterix-gerrit.ics.uci.edu/916 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I179a16fc183bd50d5d58cc12321234df1615abfd Gerrit-PatchSet: 4 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Till WestmannGerrit-Reviewer: Jenkins
[jira] [Commented] (ASTERIXDB-1264) Feed didn't release lock if the ingesting hit some exceptions
[ https://issues.apache.org/jira/browse/ASTERIXDB-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327921#comment-15327921 ] Jianfeng Jia commented on ASTERIXDB-1264: - I tried a bunch of actions, but the problems just didn't show up again. Will try to report here once it appears again. > Feed didn't release lock if the ingesting hit some exceptions > - > > Key: ASTERIXDB-1264 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1264 > Project: Apache AsterixDB > Issue Type: Bug > Components: Feeds >Reporter: Jianfeng Jia >Assignee: Abdullah Alamoudi > Attachments: cc.log, nc.log, nc.stack > > > This is a discussed issue in the mailing list. I copy it here to make it more > tractable and shareable. > I hit an wield issue that is reproducible, but only if the data has > duplications and also is large enough. Let me explained it step by step: > 1. The dataset is very simple that only has two fields. > DDL AQL: > {code} > drop dataverse test if exists; > create dataverse test; > use dataverse test; > create type t_test as closed{ > fa: int64, > fb : int64 > } > create dataset ds_test(t_test) primary key fa; > create feed fd_test using socket_adapter > ( > ("sockets"="nc1:10001"), > ("address-type"="nc"), > ("type-name"="t_test"), > ("format"="adm"), > ("duration"="1200") > ); > set wait-for-completion-feed "false"; > connect feed fd_test to dataset ds_test using policy AdvancedFT_Discard; > {code} > —— > That AdvancedFT_Discard policy will ignore the exception from the insertion > and keep ingesting. > 2. Ingesting the data by a very simple socked adapter which reads the record > one by one from an adm file. The src is > here:https://github.com/JavierJia/twitter-tracker/blob/master/src/main/java/edu/uci/ics/twitter/asterix/feed/FileFeedSocketAdapterClient.java > The data and the app package is provided here: > https://drive.google.com/folderview?id=0B423M7wGZj9dYVQ1TkpBNzcwSlE=sharing > To feed the data you can run: > ./bin/feedFile -u 172.17.0.2 -p 10001 -c 500 ~/data/twitter/test.adm > -u for sever url > -p for server port > -c for count of line you want to ingest > 3. After ingestion, all the requests about the ds_test was hanging. There is > no exception and no responds for hours. However it can respond any other > queries that on other datasets, like Metadata. > That data contains some duplicated records which should trigger the insert > exception. If I change the count from 500 to lower, let’s say 300, it > has no problems, although it contains duplications as well. > Answer from [~amoudi] : > I know exactly what is going on here. The problem is you pointed out is > caused by the duplicate keys. If I remember correctly, the main issue is > that locks that are placed on the primary keys are not released. -- This message was sent by Atlassian JIRA (v6.3.4#6332)