Change in asterixdb[master]: TweetParser Extension

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: TweetParser Extension
..


Patch Set 4:

@Till

Another submit addresses the Line # fix.

Just a side note on the coding style part. I got quite a few comments from 
SonarQube that says Missing curly brace for  if clause. Most of them are just 
one line statements. I am wondering is there any particular reason for that or 
it's just a fail-proof for future changes. I don't have strong opinion on this. 
Just out of curiosity due to my lack of project experience. :)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1002
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: TweetParser Extension

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: TweetParser Extension
..


Patch Set 4:

@Abdullah,

I checked the TwitterObjectFactory.getRawJSON method. When the JsonStoreEnabled 
is turned on. The TwitterObjectFactory will maintain an internal map which maps 
the existed Status object to JSON string, thus I think it will not trigger two 
requests to Twitter.

On the performance side, my actual concern is that I created one JSONObject per 
record, which may cause memory overhead. The reason to use JSONObject is in 
order to iterate over all fields. I couldn't find any other workaround on that 
part. Any suggestions?

For the test case part, the test cases I understood are the queries under  
asterixdb/asterixdb/asterix-app/src/test/resources/runtimets . I think you are 
referring to test cases elsewhere using mock objects. Any existed test case 
that I can refer to? I will put my test queries under the queries section.

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1002
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Add init parameter option for external functions.

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1006

to look at the new patch set (#6).

Change subject: Add init parameter option for external functions.
..

Add init parameter option for external functions.

1. Add one attribute to the external function descriptor.xml
STRING_VALUE
2. Add one test case 'Keyword detector' as a toy example for this
feature.

This init parameter option allows user to reuse the same external
function with different initialization parameters or configuration
files, such as keyword list, module file, etc.

Change-Id: I437880a700f61440ffab9f44035798e538fb9ff5
---
A asterixdb/asterix-app/data/external_function/KeywordsDetector_List1.txt
A asterixdb/asterix-app/data/external_function/KeywordsDetector_List2.txt
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/aql/translator/QueryTranslator.java
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/meta06/meta06.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_datatype/metadata_datatype.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/keyword_detector/keyword_detector.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/keyword_detector/keyword_detector.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/keyword_detector/keyword_detector.3.lib.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/keyword_detector/keyword_detector.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/keyword_detector/keyword_detector.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/external-library/keyword_detector/keyword_detector.1.adm
M 
asterixdb/asterix-app/src/test/resources/runtimets/results/user-defined-functions/single-line-definition/single-line-definition.1.adm
M 
asterixdb/asterix-app/src/test/resources/runtimets/results/user-defined-functions/udf28/udf28.1.adm
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IFunctionHelper.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/JavaFunctionHelper.java
M asterixdb/asterix-external-data/src/main/resources/schema/library.xsd
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library/KeywordsDetecterFactory.java
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library/KeywordsDetecterFunction.java
M asterixdb/asterix-external-data/src/test/resources/library_descriptor.xml
M 
asterixdb/asterix-installer/src/test/resources/integrationts/library/results/library-metadata/functionDataset/functionDataset.1.adm
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataTransactionContext.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataRecordTypes.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Function.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/FunctionTupleTranslator.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/functions/AsterixExternalScalarFunctionInfo.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/functions/ExternalFunctionCompilerUtil.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/AsterixExternalFunctionInfo.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/IExternalFunctionInfo.java
M 
asterixdb/asterix-yarn/src/test/resources/library/results/library-metadata/functionDataset/functionDataset.1.adm
30 files changed, 342 insertions(+), 32 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/06/1006/6
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1006
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I437880a700f61440ffab9f44035798e538fb9ff5
Gerrit-PatchSet: 6
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Till Westmann 

Change in asterixdb[master]: TweetParser Extension

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1002

to look at the new patch set (#6).

Change subject: TweetParser Extension
..

TweetParser Extension

This patch includes following changes:

1. ExtendedTweetParser to parse more than fix attributes.

2. Changed the twitter feeds message unit from Status to String.

Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/api/common/AsterixHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.2.update.aql
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/java/JObjectAccessors.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/TweetParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/TwitterUtil.java
M 
asterixdb/asterix-maven-plugins/lexer-generator-maven-plugin/src/main/resources/Lexer.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AOrderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AUnorderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/TypeTagUtil.java
17 files changed, 465 insertions(+), 152 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/02/1002/6
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1002
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
Gerrit-PatchSet: 6
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: TweetParser Extension

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: TweetParser Extension
..


Patch Set 4:

@Till

Thanks for your explanation. That makes totally sense to me. Will apply the 
formatter and add test cases at the next submit. Thanks! :)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1002
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Add init parameter option for external functions.

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Add init parameter option for external functions.
..


Patch Set 6:

@Till & Michael,

There are many comments from SonaQube that regarding to the existed coding 
style problems. Should I try to resolve that in my patch? How about the 
parameter number one, the one in AsterixExternalScalarFunctionInfo.java. 

btw, in case I am using tools in a wrong and wasting resources on the server 
due to my lack of experience. I am following the 'Introduction to the Build 
Lifecycle
' on Maven and trying to minimize the submit to Jenkins (I assume that's the 
goal?). I executed 'mvn test' on my local machine for unit tests and then ran 
'mvn verify' based on the procedure. 'mvn test' costs ~40mins, which is 
acceptable. 'mvn verify' takes absurdly long time, and get stuck on certain 
cases (just keep waiting on something). So I guess the right procedure will be 
'mvn test' locally and then 'gerrit submit' to check integration test result on 
jenkins? Thanks ahead. :)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1006
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I437880a700f61440ffab9f44035798e538fb9ff5
Gerrit-PatchSet: 6
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-HasComments: No

Change in asterixdb[master]: Add init parameter option for external functions.

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1006

to look at the new patch set (#4).

Change subject: Add init parameter option for external functions.
..

Add init parameter option for external functions.

1. Add one attribute to the external function descriptor.xml
STRING_VALUE
2. Add one test case 'Keyword detector' as a toy example for this
feature.

This init parameter option allows user to reuse the same external
function with different initialization parameters or configuration
files, such as keyword list, module file, etc.

Change-Id: I437880a700f61440ffab9f44035798e538fb9ff5
---
A asterixdb/asterix-app/data/external_function/KeywordsDetector_List1.txt
A asterixdb/asterix-app/data/external_function/KeywordsDetector_List2.txt
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/aql/translator/QueryTranslator.java
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/meta06/meta06.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_datatype/metadata_datatype.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/keyword_detector/keyword_detector.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/keyword_detector/keyword_detector.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/keyword_detector/keyword_detector.3.lib.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/keyword_detector/keyword_detector.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/keyword_detector/keyword_detector.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/external-library/keyword_detector/keyword_detector.1.adm
M 
asterixdb/asterix-app/src/test/resources/runtimets/results/user-defined-functions/single-line-definition/single-line-definition.1.adm
M 
asterixdb/asterix-app/src/test/resources/runtimets/results/user-defined-functions/udf28/udf28.1.adm
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IFunctionHelper.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/JavaFunctionHelper.java
M asterixdb/asterix-external-data/src/main/resources/schema/library.xsd
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library/KeywordsDetecterFactory.java
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library/KeywordsDetecterFunction.java
M asterixdb/asterix-external-data/src/test/resources/library_descriptor.xml
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataTransactionContext.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataRecordTypes.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Function.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/FunctionTupleTranslator.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/functions/AsterixExternalScalarFunctionInfo.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/functions/ExternalFunctionCompilerUtil.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/AsterixExternalFunctionInfo.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/IExternalFunctionInfo.java
M 
asterixdb/asterix-yarn/src/test/resources/library/results/library-metadata/functionDataset/functionDataset.1.adm
29 files changed, 330 insertions(+), 24 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/06/1006/4
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1006
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I437880a700f61440ffab9f44035798e538fb9ff5
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Till Westmann 

Change in asterixdb[master]: TweetParser Extension

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1002

to look at the new patch set (#5).

Change subject: TweetParser Extension
..

TweetParser Extension

This patch includes following changes:

1. ExtendedTweetParser to parse more than fix attributes.

2. Changed the twitter feeds message unit from Status to String.

Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/api/common/AsterixHyracksIntegrationUtil.java
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.2.update.aql
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/java/JObjectAccessors.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/TweetParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/TwitterUtil.java
M 
asterixdb/asterix-maven-plugins/lexer-generator-maven-plugin/src/main/resources/Lexer.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AOrderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AUnorderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/TypeTagUtil.java
17 files changed, 465 insertions(+), 152 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/02/1002/5
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1002
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: TweetParser Extension

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1002

to look at the new patch set (#8).

Change subject: TweetParser Extension
..

TweetParser Extension

This patch includes following changes:

1. ExtendedTweetParser to parse more than fix attributes.

2. Changed the twitter feeds message unit from Status to String.

Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
---
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.2.update.aql
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/java/JObjectAccessors.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/TweetParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/TwitterUtil.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AOrderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AUnorderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/TypeTagUtil.java
15 files changed, 516 insertions(+), 246 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/02/1002/8
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1002
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
Gerrit-PatchSet: 8
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: Add upsert option for feed

[Xikui Wang (Code Review)]

Xikui Wang has uploaded a new change for review.

  https://asterix-gerrit.ics.uci.edu/1068

Change subject: Add upsert option for feed
..

Add upsert option for feed

For ASTERIXDB-1567. Provide "upsert-feed"="true" for
feed configuration, which changes the default record
insert to upsert.

Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
4 files changed, 175 insertions(+), 186 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/68/1068/1

diff --git 
a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
 
b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
index 5081587..8521c01 100644
--- 
a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
+++ 
b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
@@ -81,6 +81,7 @@
 import org.apache.asterix.metadata.declared.ResultSetDataSink;
 import org.apache.asterix.metadata.declared.ResultSetSinkId;
 import org.apache.asterix.metadata.entities.Dataset;
+import org.apache.asterix.metadata.entities.Feed;
 import org.apache.asterix.metadata.entities.Function;
 import org.apache.asterix.metadata.entities.InternalDatasetDetails;
 import org.apache.asterix.metadata.feeds.FeedMetadataUtil;
@@ -181,13 +182,11 @@
 
FormatUtils.getDefaultFormat().registerRuntimeFunctions(FunctionCollection.getFunctionDescriptorFactories());
 }
 
-@Override
-public int getVarCounter() {
+@Override public int getVarCounter() {
 return context.getVarCounter();
 }
 
-@Override
-public ILogicalPlan translateLoad(ICompiledDmlStatement stmt) throws 
AlgebricksException {
+@Override public ILogicalPlan translateLoad(ICompiledDmlStatement stmt) 
throws AlgebricksException {
 CompiledLoadFromFileStatement clffs = (CompiledLoadFromFileStatement) 
stmt;
 Dataset dataset = 
metadataProvider.findDataset(clffs.getDataverseName(), clffs.getDatasetName());
 if (dataset == null) {
@@ -196,10 +195,10 @@
 "Unable to load dataset " + clffs.getDatasetName() + " 
since it does not exist");
 }
 IAType itemType = 
metadataProvider.findType(dataset.getItemTypeDataverseName(), 
dataset.getItemTypeName());
-IAType metaItemType =
-
metadataProvider.findType(dataset.getMetaItemTypeDataverseName(), 
dataset.getMetaItemTypeName());
-DatasetDataSource targetDatasource =
-validateDatasetInfo(metadataProvider, stmt.getDataverseName(), 
stmt.getDatasetName());
+IAType metaItemType = metadataProvider
+.findType(dataset.getMetaItemTypeDataverseName(), 
dataset.getMetaItemTypeName());
+DatasetDataSource targetDatasource = 
validateDatasetInfo(metadataProvider, stmt.getDataverseName(),
+stmt.getDatasetName());
 List partitionKeys = 
DatasetUtils.getPartitioningKeys(targetDatasource.getDataset());
 if (dataset.hasMetaPart()) {
 throw new AlgebricksException(
@@ -235,8 +234,8 @@
 List varRefsForLoading = new 
ArrayList<>();
 LogicalVariable payloadVar = payloadVars.get(0);
 for (List keyFieldName : partitionKeys) {
-PlanTranslationUtil.prepareVarAndExpression(keyFieldName, 
payloadVar, pkVars, pkExprs, varRefsForLoading,
-context);
+PlanTranslationUtil
+.prepareVarAndExpression(keyFieldName, payloadVar, pkVars, 
pkExprs, varRefsForLoading, context);
 }
 
 AssignOperator assign = new AssignOperator(pkVars, pkExprs);
@@ -262,8 +261,8 @@
 additionalFilteringExpressions = new ArrayList<>();
 
PlanTranslationUtil.prepareVarAndExpression(additionalFilteringField, 
payloadVar, additionalFilteringVars,
 additionalFilteringAssignExpressions, 
additionalFilteringExpressions, context);
-additionalFilteringAssign =
-new AssignOperator(additionalFilteringVars, 
additionalFilteringAssignExpressions);
+additionalFilteringAssign = new 
AssignOperator(additionalFilteringVars,
+additionalFilteringAssignExpressions);
 }
 
 InsertDeleteUpsertOperator insertOp = new 
InsertDeleteUpsertOperator(targetDatasource, payloadRef,
@@ -282,12 +281,10 

Change in asterixdb[master]: TweetParser Extension

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1002

to look at the new patch set (#9).

Change subject: TweetParser Extension
..

TweetParser Extension

This patch includes following changes:

1. ExtendedTweetParser to parse more than fix attributes.

2. Changed the twitter feeds message unit from Status to String.

Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
---
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.2.update.aql
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/java/JObjectAccessors.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/TweetParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/TwitterUtil.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AOrderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AUnorderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/TypeTagUtil.java
15 files changed, 526 insertions(+), 248 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/02/1002/9
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1002
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
Gerrit-PatchSet: 9
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: TweetParser Extension

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1002

to look at the new patch set (#10).

Change subject: TweetParser Extension
..

TweetParser Extension

This patch includes following changes:

1. ExtendedTweetParser to parse more than fix attributes.

2. Changed the twitter feeds message unit from Status to String.

Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
---
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.2.update.aql
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/java/JObjectAccessors.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/TweetParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/TwitterUtil.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AOrderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AUnorderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/TypeTagUtil.java
15 files changed, 526 insertions(+), 248 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/02/1002/10
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1002
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
Gerrit-PatchSet: 10
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: Add upsert option for feed

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1068

to look at the new patch set (#8).

Change subject: Add upsert option for feed
..

Add upsert option for feed

For ASTERIXDB-1567. Provide "upsert-feed"="true" for
feed configuration, which changes the default record
insert to upsert. Added one test case for upsert feed.

Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
A asterixdb/asterix-app/data/tinysocial/twu_update.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.10.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.12.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.3.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.4.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.5.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.6.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.7.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.8.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.9.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/upsert-feed/upsert-feed.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
18 files changed, 350 insertions(+), 13 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/68/1068/8
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1068
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
Gerrit-PatchSet: 8
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: Add upsert option for feed

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1068

to look at the new patch set (#7).

Change subject: Add upsert option for feed
..

Add upsert option for feed

For ASTERIXDB-1567. Provide "upsert-feed"="true" for
feed configuration, which changes the default record
insert to upsert.

Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
A asterixdb/asterix-app/data/tinysocial/twu_update.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.10.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.12.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.3.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.4.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.5.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.6.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.7.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.8.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.9.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/upsert-feed/upsert-feed.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
18 files changed, 350 insertions(+), 13 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/68/1068/7
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1068
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
Gerrit-PatchSet: 7
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: Add upsert option for feed

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Add upsert option for feed
..


Patch Set 7:

@Till

Thanks for your comments. It's good to have someone helping me improve my 
coding style and quality. I added one test case for this option. It includes 
insert & upsert case. Feel free to leave any comments. :)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1068
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
Gerrit-PatchSet: 7
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Fix for ASTERIXDB-1471

[Xikui Wang (Code Review)]

Xikui Wang has uploaded a new change for review.

  https://asterix-gerrit.ics.uci.edu/1012

Change subject: Fix for ASTERIXDB-1471
..

Fix for ASTERIXDB-1471

The line number is increased twice in setBuffer and updateLineColumn
which causes the line number is 2x larger.

Change-Id: Ibd91bc13c3adbbbe1dbfa0dccbdf6b378b280eae
---
M 
asterixdb/asterix-maven-plugins/lexer-generator-maven-plugin/src/main/resources/Lexer.java
1 file changed, 0 insertions(+), 1 deletion(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/12/1012/1

diff --git 
a/asterixdb/asterix-maven-plugins/lexer-generator-maven-plugin/src/main/resources/Lexer.java
 
b/asterixdb/asterix-maven-plugins/lexer-generator-maven-plugin/src/main/resources/Lexer.java
index 60e6e89..495a95e 100644
--- 
a/asterixdb/asterix-maven-plugins/lexer-generator-maven-plugin/src/main/resources/Lexer.java
+++ 
b/asterixdb/asterix-maven-plugins/lexer-generator-maven-plugin/src/main/resources/Lexer.java
@@ -85,7 +85,6 @@
 this.buffer = buffer;
 tokenBegin = bufpos = 0;
 containsEscapes = false;
-line++;
 tokenBegin = -1;
 }
 

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1012
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibd91bc13c3adbbbe1dbfa0dccbdf6b378b280eae
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 

Change in asterixdb[master]: Add init parameter option for external functions.

[Xikui Wang (Code Review)]

Xikui Wang has uploaded a new change for review.

  https://asterix-gerrit.ics.uci.edu/1006

Change subject: Add init parameter option for external functions.
..

Add init parameter option for external functions.

Add one attribute to the external function descriptor.xml
STRING_VALUE

Change-Id: I437880a700f61440ffab9f44035798e538fb9ff5
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/aql/translator/QueryTranslator.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/JavaFunctionHelper.java
M asterixdb/asterix-external-data/src/main/resources/schema/library.xsd
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataTransactionContext.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataRecordTypes.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Function.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/FunctionTupleTranslator.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/functions/AsterixExternalScalarFunctionInfo.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/functions/ExternalFunctionCompilerUtil.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/AsterixExternalFunctionInfo.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/IExternalFunctionInfo.java
12 files changed, 46 insertions(+), 12 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/06/1006/1

diff --git 
a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java
 
b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java
index 92ef062..536b664 100755
--- 
a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java
+++ 
b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java
@@ -247,7 +247,7 @@
 }
 Function f = new Function(dataverse, libraryName + "#" + 
function.getName().trim(), args.size(),
 args, function.getReturnType().trim(), 
function.getDefinition().trim(),
-library.getLanguage().trim(), 
function.getFunctionType().trim());
+library.getLanguage().trim(), 
function.getFunctionType().trim(), function.getInitParameters());
 MetadataManager.INSTANCE.addFunction(mdTxnCtx, f);
 if (LOGGER.isLoggable(Level.INFO)) {
 LOGGER.info("Installed function: " + libraryName + "#" 
+ function.getName().trim());
diff --git 
a/asterixdb/asterix-app/src/main/java/org/apache/asterix/aql/translator/QueryTranslator.java
 
b/asterixdb/asterix-app/src/main/java/org/apache/asterix/aql/translator/QueryTranslator.java
index 295b308..e1c5cde 100644
--- 
a/asterixdb/asterix-app/src/main/java/org/apache/asterix/aql/translator/QueryTranslator.java
+++ 
b/asterixdb/asterix-app/src/main/java/org/apache/asterix/aql/translator/QueryTranslator.java
@@ -1796,9 +1796,10 @@
 if (dv == null) {
 throw new AlgebricksException("There is no dataverse with this 
name " + dataverse + ".");
 }
+// for cfs, assign null to init_para for now.
 Function function = new Function(dataverse, functionName, 
cfs.getaAterixFunction().getArity(),
 cfs.getParamList(), Function.RETURNTYPE_VOID, 
cfs.getFunctionBody(), Function.LANGUAGE_AQL,
-FunctionKind.SCALAR.toString());
+FunctionKind.SCALAR.toString(), null);
 MetadataManager.INSTANCE.addFunction(mdTxnCtx, function);
 
 MetadataManager.INSTANCE.commitTransaction(mdTxnCtx);
diff --git 
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/JavaFunctionHelper.java
 
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/JavaFunctionHelper.java
index 50dc28e..b46 100644
--- 
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/JavaFunctionHelper.java
+++ 
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/JavaFunctionHelper.java
@@ -55,6 +55,7 @@
 private final JObjectPointableVisitor pointableVisitor;
 private final PointableAllocator pointableAllocator;
 private final Map poolTypeInfo;
+private final String initParameters;
 
 private boolean isValidResult = false;
 
@@ -71,6 +72,7 @@
 }
 this.resultHolder = objectPool.allocate(finfo.getReturnType());
 this.poolTypeInfo = new 

Change in asterixdb[master]: Add init parameter option for external functions.

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1006

to look at the new patch set (#2).

Change subject: Add init parameter option for external functions.
..

Add init parameter option for external functions.

Add one attribute to the external function descriptor.xml
STRING_VALUE

Change-Id: I437880a700f61440ffab9f44035798e538fb9ff5
---
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/ExternalLibraryUtils.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/aql/translator/QueryTranslator.java
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/meta06/meta06.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_datatype/metadata_datatype.1.adm
M 
asterixdb/asterix-app/src/test/resources/runtimets/results/user-defined-functions/single-line-definition/single-line-definition.1.adm
M 
asterixdb/asterix-app/src/test/resources/runtimets/results/user-defined-functions/udf28/udf28.1.adm
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/JavaFunctionHelper.java
M asterixdb/asterix-external-data/src/main/resources/schema/library.xsd
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataTransactionContext.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataRecordTypes.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entities/Function.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/entitytupletranslators/FunctionTupleTranslator.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/functions/AsterixExternalScalarFunctionInfo.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/functions/ExternalFunctionCompilerUtil.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/AsterixExternalFunctionInfo.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/IExternalFunctionInfo.java
16 files changed, 50 insertions(+), 16 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/06/1006/2
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1006
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I437880a700f61440ffab9f44035798e538fb9ff5
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: This patch includes following changes:

[Xikui Wang (Code Review)]

Xikui Wang has uploaded a new change for review.

  https://asterix-gerrit.ics.uci.edu/1002

Change subject: This patch includes following changes:
..

This patch includes following changes:

1. ExtendedTweetParser to parse more than fix attributes.

2. Changed the twitter feeds message unit from Status to String.

3. Fixed bug ASTERIXDB-1471.

4. Fixed bug ASTERIXDB-1352.

Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
---
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/java/JObjectAccessors.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/TweetParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/TwitterUtil.java
M 
asterixdb/asterix-maven-plugins/lexer-generator-maven-plugin/src/main/resources/Lexer.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AOrderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AUnorderedListType.java
12 files changed, 408 insertions(+), 149 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/02/1002/1

diff --git 
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java
 
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java
index e31325a..80e716d 100644
--- 
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java
+++ 
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java
@@ -28,13 +28,9 @@
 import org.apache.asterix.external.util.FeedLogManager;
 import org.apache.hyracks.api.exceptions.HyracksDataException;
 
-import twitter4j.Query;
-import twitter4j.QueryResult;
-import twitter4j.Status;
-import twitter4j.Twitter;
-import twitter4j.TwitterException;
+import twitter4j.*;
 
-public class TwitterPullRecordReader implements IRecordReader {
+public class TwitterPullRecordReader implements IRecordReader {
 
 private Query query;
 private Twitter twitter;
@@ -42,14 +38,14 @@
 private QueryResult result;
 private int nextTweetIndex = 0;
 private long lastTweetIdReceived = 0;
-private GenericRecord record;
+private GenericRecord record;
 
 public TwitterPullRecordReader(Twitter twitter, String keywords, int 
requestInterval) {
 this.twitter = twitter;
 this.requestInterval = requestInterval;
 this.query = new Query(keywords);
 this.query.setCount(100);
-this.record = new GenericRecord();
+this.record = new GenericRecord<>();
 }
 
 @Override
@@ -62,7 +58,7 @@
 }
 
 @Override
-public IRawRecord next() throws IOException, InterruptedException {
+public IRawRecord next() throws IOException, InterruptedException {
 if (result == null || nextTweetIndex >= result.getTweets().size()) {
 Thread.sleep(1000 * requestInterval);
 query.setSinceId(lastTweetIdReceived);
@@ -79,7 +75,8 @@
 if (lastTweetIdReceived < tweet.getId()) {
 lastTweetIdReceived = tweet.getId();
 }
-record.set(tweet);
+String jsonTweet = TwitterObjectFactory.getRawJSON(tweet); // 
transform tweet obj to json
+record.set(jsonTweet);
 return record;
 } else {
 return null;
diff --git 
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
 
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
index f04cdb9..02c3963 100644
--- 
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
+++ 
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
@@ -27,30 +27,25 @@
 import 

Change in asterixdb[master]: TweetParser Extension

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1002

to look at the new patch set (#3).

Change subject: TweetParser Extension
..

TweetParser Extension

This patch includes following changes:

1. ExtendedTweetParser to parse more than fix attributes.

2. Changed the twitter feeds message unit from Status to String.

3. Fixed bug ASTERIXDB-1471.

4. Fixed bug ASTERIXDB-1352.

Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
---
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/java/JObjectAccessors.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/TweetParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/TwitterUtil.java
M 
asterixdb/asterix-maven-plugins/lexer-generator-maven-plugin/src/main/resources/Lexer.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AOrderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AUnorderedListType.java
12 files changed, 407 insertions(+), 149 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/02/1002/3
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1002
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: TweetParser Extension

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1002

to look at the new patch set (#4).

Change subject: TweetParser Extension
..

TweetParser Extension

This patch includes following changes:

1. ExtendedTweetParser to parse more than fix attributes.

2. Changed the twitter feeds message unit from Status to String.

3. Fixed bug ASTERIXDB-1471.

4. Fixed bug ASTERIXDB-1352.

Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
---
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/java/JObjectAccessors.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/TweetParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/TwitterUtil.java
M 
asterixdb/asterix-maven-plugins/lexer-generator-maven-plugin/src/main/resources/Lexer.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AOrderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AUnorderedListType.java
12 files changed, 407 insertions(+), 150 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/02/1002/4
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1002
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: TweetParser Extension

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1002

to look at the new patch set (#11).

Change subject: TweetParser Extension
..

TweetParser Extension

This patch includes following changes:

1. ExtendedTweetParser to parse more than fix attributes.

2. Changed the twitter feeds message unit from Status to String.

Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
---
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.2.update.aql
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/java/JObjectAccessors.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/TweetParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/TwitterUtil.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AOrderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AUnorderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/TypeTagUtil.java
15 files changed, 482 insertions(+), 156 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/02/1002/11
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1002
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
Gerrit-PatchSet: 11
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: TweetParser Extension

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1002

to look at the new patch set (#12).

Change subject: TweetParser Extension
..

TweetParser Extension

This patch includes following changes:

1. ExtendedTweetParser to parse more than fix attributes.

2. Changed the twitter feeds message unit from Status to String.

Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
---
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.2.update.aql
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/java/JObjectAccessors.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/TweetParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/TwitterUtil.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AOrderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AUnorderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/TypeTagUtil.java
15 files changed, 483 insertions(+), 156 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/02/1002/12
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1002
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
Gerrit-PatchSet: 12
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: TweetParser Extension

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: TweetParser Extension
..


Patch Set 12:

Fixed it. Sorry about that.

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1002
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
Gerrit-PatchSet: 12
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Feed Connection Refactoring

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1259

to look at the new patch set (#11).

Change subject: Feed Connection Refactoring
..

Feed Connection Refactoring

1. The feed subscription network using FeedJoint is removed.
2. FeedConnection metadata dataset is added (pkeys: dataverseName,
feedName, datasetName).
3. Replaced the old intake job + collect job combination with one single
job using SplitOperator.
4. Now one feed can connect to multiple datasets.
5. The disconnect feed job is replaced by ActiveManagerMessage.
6. The new feed life cycle is:
  - Create feed
  - Connect feed to dataset0, dataset1, dataset2, etc.
  - Start feed
  - Stop feed
  - Disconnect feed
7. New feedEventListner framework by Abdullah

Change-Id: Ic36267eb9a10df21734ce1cc1f38583e23c9e8f0
---
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveEvent.java
D 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveJob.java
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveJobNotificationHandler.java
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveLifecycleListener.java
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActivityState.java
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/IActiveEntityEventsListener.java
A 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/IActiveEventSubscriber.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceRandomPartitioningFeedComputationRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/UnnestToDataScanRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
D 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedJoint.java
D 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedWorkCollection.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
A 
asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/FeedOperations.java
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_dataset/metadata_dataset.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_datatype/metadata_datatype.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_index/metadata_index.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_selfjoin/metadata_selfjoin.1.adm
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/big-object/big_object_feed_20M/big_object_feed_20M.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/typed_adapter/typed_adapter.4.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.4.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-in-value/change-feed-with-meta-pk-in-meta-open-index-in-value.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-with-missing/change-feed-with-meta-pk-in-meta-open-index-with-missing.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta/change-feed-with-meta-pk-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-with-mixed-index/change-feed-with-meta-with-mixed-index.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed/change-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.0.ddl.aql
C 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.1.update.aql
R 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.2.update.aql
C 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.3.query.aql
C 

Change in asterixdb[master]: Feed Connection Refactoring

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1259

to look at the new patch set (#10).

Change subject: Feed Connection Refactoring
..

Feed Connection Refactoring

1. The feed subscription network using FeedJoint is removed.
2. FeedConnection metadata dataset is added (pkeys: dataverseName,
feedName, datasetName).
3. Replaced the old intake job + collect job combination with one single
job using SplitOperator.
4. Now one feed can connect to multiple datasets.
5. The disconnect feed job is replaced by ActiveManagerMessage.
6. The new feed life cycle is:
  - Create feed
  - Connect feed to dataset0, dataset1, dataset2, etc.
  - Start feed
  - Stop feed
  - Disconnect feed
7. New feedEventListner framework by Abdullah

Change-Id: Ic36267eb9a10df21734ce1cc1f38583e23c9e8f0
---
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveEvent.java
D 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveJob.java
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveJobNotificationHandler.java
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveLifecycleListener.java
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActivityState.java
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/IActiveEntityEventsListener.java
A 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/IActiveEventSubscriber.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceRandomPartitioningFeedComputationRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/UnnestToDataScanRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
D 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedJoint.java
D 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedWorkCollection.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
A 
asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/FeedOperations.java
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_dataset/metadata_dataset.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_datatype/metadata_datatype.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_index/metadata_index.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_selfjoin/metadata_selfjoin.1.adm
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/big-object/big_object_feed_20M/big_object_feed_20M.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/typed_adapter/typed_adapter.4.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.4.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-in-value/change-feed-with-meta-pk-in-meta-open-index-in-value.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-with-missing/change-feed-with-meta-pk-in-meta-open-index-with-missing.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta/change-feed-with-meta-pk-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-with-mixed-index/change-feed-with-meta-with-mixed-index.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed/change-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.0.ddl.aql
C 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.1.update.aql
R 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.2.update.aql
C 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.3.query.aql
C 

Change in asterixdb[master]: WIP-Feed Connection Refactoring

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1259

to look at the new patch set (#8).

Change subject: WIP-Feed Connection Refactoring
..

WIP-Feed Connection Refactoring

1. The feed subscription network using FeedJoint is removed.
2. FeedConnection metadata dataset is added (pkeys: dataverseName,
feedName, datasetName).
3. Replaced the old intake job + collect job combination with one single
job using SplitOperator.
4. Now one feed can connect to multiple datasets.
5. The disconnect feed job is replaced by ActiveManagerMessage.
6. Twitter adaptor will run on all possible nodes now.
7. The new feed life cycle is:
  - Create feed
  - Connect feed to dataset0, dataset1, dataset2, etc.
  - Start feed
  - Stop feed
  - Disconnect feed
8. Fixed one sporadic error described in ASTERIXDB-1709.

Change-Id: Ic36267eb9a10df21734ce1cc1f38583e23c9e8f0
---
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveEvent.java
D 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveJob.java
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveJobNotificationHandler.java
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveLifecycleListener.java
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActivityState.java
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/IActiveEntityEventsListener.java
A 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/IActiveEventSubscriber.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceRandomPartitioningFeedComputationRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/UnnestToDataScanRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
D 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedJoint.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedOperations.java
D 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedWorkCollection.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_dataset/metadata_dataset.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_datatype/metadata_datatype.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_index/metadata_index.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_selfjoin/metadata_selfjoin.1.adm
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/big-object/big_object_feed_20M/big_object_feed_20M.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/typed_adapter/typed_adapter.4.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.4.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-in-value/change-feed-with-meta-pk-in-meta-open-index-in-value.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-with-missing/change-feed-with-meta-pk-in-meta-open-index-with-missing.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta/change-feed-with-meta-pk-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-with-mixed-index/change-feed-with-meta-with-mixed-index.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed/change-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.0.ddl.aql
C 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.1.update.aql
R 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.2.update.aql
C 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.3.query.aql

Change in asterixdb[master]: Modularize feed adaptors

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1430

to look at the new patch set (#6).

Change subject: Modularize feed adaptors
..

Modularize feed adaptors

1. Load adaptors from class-path.
2. Move rss adaptors/parsers to the new module, as well as related test
cases. All future parsers/adaptors that depend on various libraries should
be placed in this new module.

Change-Id: Ic4f95255f5493a813ee1f875b63a62e74bc47602
---
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
M asterixdb/asterix-external-data/pom.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/feed/test/DatasourceFactoryProviderTest.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/RecordWithPKTestReaderFactory.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/kv/KVTestReaderFactory.java
D 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/ParserFactoryProviderLoadParserTest.java
A asterixdb/asterix-villain/adaptor-rss/pom.xml
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/parser/RSSParser.java
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/parser/RSSParserFactory.java
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/reader/RSSRecordReader.java
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/reader/RSSRecordReaderFactory.java
A 
asterixdb/asterix-villain/adaptor-rss/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-villain/adaptor-rss/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A asterixdb/asterix-villain/adaptor-twitter/pom.xml
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/parser/TweetParser.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/parser/TweetParserFactory.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/reader/TwitterPullRecordReader.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/reader/TwitterPushRecordReader.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/reader/TwitterRecordReaderFactory.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/util/TwitterUtil.java
A 
asterixdb/asterix-villain/adaptor-twitter/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-villain/adaptor-twitter/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A asterixdb/asterix-villain/pom.xml
A asterixdb/asterix-villain/villain-test-framework/pom.xml
A 
asterixdb/asterix-villain/villain-test-framework/src/test/java/org/apache/asterix/villain/test/VillainExecutionTest.java
A 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/only.xml
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.1.ddl.aql
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.2.update.aql
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/twitter-feed/twitter-feed.1.ddl.aql
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/twitter-feed/twitter-feed.2.update.aql
A 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/testsuite.xml
M 

Change in asterixdb[master]: Modularize feed adaptors

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1430

to look at the new patch set (#7).

Change subject: Modularize feed adaptors
..

Modularize feed adaptors

1. Load adaptors from class-path.
2. Move rss adaptors/parsers to the new module, as well as related test
cases. All future parsers/adaptors that depend on various libraries should
be placed in this new module.

Change-Id: Ic4f95255f5493a813ee1f875b63a62e74bc47602
---
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
M asterixdb/asterix-external-data/pom.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/feed/test/DatasourceFactoryProviderTest.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/RecordWithPKTestReaderFactory.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/kv/KVTestReaderFactory.java
D 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/ParserFactoryProviderLoadParserTest.java
A asterixdb/asterix-villain/adaptor-rss/pom.xml
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/parser/RSSParser.java
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/parser/RSSParserFactory.java
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/reader/RSSRecordReader.java
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/reader/RSSRecordReaderFactory.java
A 
asterixdb/asterix-villain/adaptor-rss/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-villain/adaptor-rss/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A asterixdb/asterix-villain/adaptor-twitter/pom.xml
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/parser/TweetParser.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/parser/TweetParserFactory.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/reader/TwitterPullRecordReader.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/reader/TwitterPushRecordReader.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/reader/TwitterRecordReaderFactory.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/util/TwitterUtil.java
A 
asterixdb/asterix-villain/adaptor-twitter/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-villain/adaptor-twitter/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A asterixdb/asterix-villain/pom.xml
A asterixdb/asterix-villain/villain-test-framework/pom.xml
A 
asterixdb/asterix-villain/villain-test-framework/src/test/java/org/apache/asterix/villain/test/VillainExecutionTest.java
A 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/only.xml
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.1.ddl.aql
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.2.update.aql
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/twitter-feed/twitter-feed.1.ddl.aql
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/twitter-feed/twitter-feed.2.update.aql
A 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/testsuite.xml
M 

Change in asterixdb[master]: Modularize feed adaptors

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1430

to look at the new patch set (#9).

Change subject: Modularize feed adaptors
..

Modularize feed adaptors

1. Load adaptors from class-path.
2. Move rss adaptors/parsers to the new module, as well as related test
cases. All future parsers/adaptors that depend on various libraries should
be placed in this new module.

Change-Id: Ic4f95255f5493a813ee1f875b63a62e74bc47602
---
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
M asterixdb/asterix-external-data/pom.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/feed/test/DatasourceFactoryProviderTest.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/RecordWithPKTestReaderFactory.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/kv/KVTestReaderFactory.java
D 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/ParserFactoryProviderLoadParserTest.java
A asterixdb/asterix-villain/adaptor-rss/pom.xml
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/parser/RSSParser.java
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/parser/RSSParserFactory.java
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/reader/RSSRecordReader.java
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/reader/RSSRecordReaderFactory.java
A 
asterixdb/asterix-villain/adaptor-rss/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-villain/adaptor-rss/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A asterixdb/asterix-villain/adaptor-twitter/pom.xml
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/parser/TweetParser.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/parser/TweetParserFactory.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/reader/TwitterPullRecordReader.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/reader/TwitterPushRecordReader.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/reader/TwitterRecordReaderFactory.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/util/TwitterUtil.java
A 
asterixdb/asterix-villain/adaptor-twitter/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-villain/adaptor-twitter/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A asterixdb/asterix-villain/pom.xml
A asterixdb/asterix-villain/villain-test-framework/pom.xml
A 
asterixdb/asterix-villain/villain-test-framework/src/test/java/org/apache/asterix/villain/test/VillainExecutionTest.java
A 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/only.xml
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.1.ddl.aql
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.2.update.aql
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/twitter-feed/twitter-feed.1.ddl.aql
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/twitter-feed/twitter-feed.2.update.aql
A 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/testsuite.xml
M 

Change in asterixdb[master]: Modularize feed adaptors

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Modularize feed adaptors
..


Patch Set 9:

@Mike B, The further modularization comment turns out to be very useful! Quite 
some knowledge about Maven is gained from this process.. The quirky part about 
this patch is, I can't really have adaptor related dependencies resides at the 
villain module, because that will cause unused lib issue for the villain test 
framework part. So I duplicated some of them in each of the twitter-adaptor 
module and rss-adaptor module.

I guess the right answer would be use the AQLTestExcution load scripts from 
villain package and add villain-module as its dependency. Currently, These 
executionTests are included in asterix-app which will cause recursive 
dependency issue... I guess there is no better solution for that now?

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1430
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4f95255f5493a813ee1f875b63a62e74bc47602
Gerrit-PatchSet: 9
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-HasComments: No

Change in asterixdb[master]: Modularize feed adaptors

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1430

to look at the new patch set (#5).

Change subject: Modularize feed adaptors
..

Modularize feed adaptors

1. Load adaptors from class-path.
2. Move rss adaptors/parsers to the new module, as well as related test
cases. All future parsers/adaptors that depend on various libraries should
be placed in this new module.

Change-Id: Ic4f95255f5493a813ee1f875b63a62e74bc47602
---
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
M asterixdb/asterix-external-data/pom.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/feed/test/DatasourceFactoryProviderTest.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/RecordWithPKTestReaderFactory.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/kv/KVTestReaderFactory.java
D 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/ParserFactoryProviderLoadParserTest.java
A asterixdb/asterix-villain/adaptor-rss/pom.xml
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/parser/RSSParser.java
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/parser/RSSParserFactory.java
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/reader/RSSRecordReader.java
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/reader/RSSRecordReaderFactory.java
A 
asterixdb/asterix-villain/adaptor-rss/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-villain/adaptor-rss/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A asterixdb/asterix-villain/adaptor-twitter/pom.xml
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/parser/TweetParser.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/parser/TweetParserFactory.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/reader/TwitterPullRecordReader.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/reader/TwitterPushRecordReader.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/reader/TwitterRecordReaderFactory.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/util/TwitterUtil.java
A 
asterixdb/asterix-villain/adaptor-twitter/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-villain/adaptor-twitter/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A asterixdb/asterix-villain/pom.xml
A asterixdb/asterix-villain/villain-test-framework/pom.xml
A 
asterixdb/asterix-villain/villain-test-framework/src/test/java/org/apache/asterix/villain/test/VillainExecutionTest.java
A 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/only.xml
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.1.ddl.aql
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.2.update.aql
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/twitter-feed/twitter-feed.1.ddl.aql
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/twitter-feed/twitter-feed.2.update.aql
A 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/testsuite.xml
M 

Change in asterixdb[master]: Modularize feed adaptors

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1430

to look at the new patch set (#8).

Change subject: Modularize feed adaptors
..

Modularize feed adaptors

1. Load adaptors from class-path.
2. Move rss adaptors/parsers to the new module, as well as related test
cases. All future parsers/adaptors that depend on various libraries should
be placed in this new module.

Change-Id: Ic4f95255f5493a813ee1f875b63a62e74bc47602
---
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
M asterixdb/asterix-external-data/pom.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/feed/test/DatasourceFactoryProviderTest.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/RecordWithPKTestReaderFactory.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/kv/KVTestReaderFactory.java
D 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/ParserFactoryProviderLoadParserTest.java
A asterixdb/asterix-villain/adaptor-rss/pom.xml
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/parser/RSSParser.java
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/parser/RSSParserFactory.java
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/reader/RSSRecordReader.java
R 
asterixdb/asterix-villain/adaptor-rss/src/main/java/org/apache/asterix/villain/external/reader/RSSRecordReaderFactory.java
A 
asterixdb/asterix-villain/adaptor-rss/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-villain/adaptor-rss/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A asterixdb/asterix-villain/adaptor-twitter/pom.xml
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/parser/TweetParser.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/parser/TweetParserFactory.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/reader/TwitterPullRecordReader.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/reader/TwitterPushRecordReader.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/reader/TwitterRecordReaderFactory.java
R 
asterixdb/asterix-villain/adaptor-twitter/src/main/java/org/apache/asterix/villain/external/util/TwitterUtil.java
A 
asterixdb/asterix-villain/adaptor-twitter/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-villain/adaptor-twitter/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A asterixdb/asterix-villain/pom.xml
A asterixdb/asterix-villain/villain-test-framework/pom.xml
A 
asterixdb/asterix-villain/villain-test-framework/src/test/java/org/apache/asterix/villain/test/VillainExecutionTest.java
A 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/only.xml
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.1.ddl.aql
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.2.update.aql
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/twitter-feed/twitter-feed.1.ddl.aql
R 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/queries/feeds/twitter-feed/twitter-feed.2.update.aql
A 
asterixdb/asterix-villain/villain-test-framework/src/test/resources/runtimets/testsuite.xml
M 

Change in asterixdb[master]: Feed Connection Refactoring

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1259

to look at the new patch set (#13).

Change subject: Feed Connection Refactoring
..

Feed Connection Refactoring

1. The feed subscription network using FeedJoint is removed.
2. FeedConnection metadata dataset is added (pkeys: dataverseName,
feedName, datasetName).
3. Replaced the old intake job + collect job combination with one single
job using SplitOperator.
4. Now one feed can connect to multiple datasets.
5. The disconnect feed job is replaced by ActiveManagerMessage.
6. The new feed life cycle is:
  - Create feed
  - Connect feed to dataset0, dataset1, dataset2, etc.
  - Start feed
  - Stop feed
  - Disconnect feed
7. New feedEventListner framework by Abdullah

Change-Id: Ic36267eb9a10df21734ce1cc1f38583e23c9e8f0
---
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveEvent.java
D 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveJob.java
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveJobNotificationHandler.java
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActiveLifecycleListener.java
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/ActivityState.java
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/IActiveEntityEventsListener.java
A 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/IActiveEventSubscriber.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceRandomPartitioningFeedComputationRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/UnnestToDataScanRule.java
D 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedJoint.java
D 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedWorkCollection.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
A 
asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/FeedOperations.java
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_dataset/metadata_dataset.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_datatype/metadata_datatype.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_index/metadata_index.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_selfjoin/metadata_selfjoin.1.adm
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/big-object/big_object_feed_20M/big_object_feed_20M.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/typed_adapter/typed_adapter.4.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-csv/change-feed-with-meta-csv.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.4.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-in-value/change-feed-with-meta-pk-in-meta-open-index-in-value.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-with-missing/change-feed-with-meta-pk-in-meta-open-index-with-missing.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta/change-feed-with-meta-pk-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-with-mixed-index/change-feed-with-meta-with-mixed-index.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed/change-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.0.ddl.aql
C 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.1.update.aql
R 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.2.update.aql
C 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.3.query.aql
C 

Change in asterixdb[master]: Enable Feed Changes to work with BAD project

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Enable Feed Changes to work with BAD project
..


Patch Set 1:

This change looks good to me. One general comment about this change. It seems 
the repeatable flag is not used on the AsterixDB (probably only for now). I 
guess that serves the BAD side only?

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1524
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ib62184b67aff564475ef9b58790ff96409195b77
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Steven Jacobs 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Add init parameter option for external functions.

[Xikui Wang (Code Review)]

Xikui Wang has abandoned this change.

Change subject: Add init parameter option for external functions.
..


Abandoned

This patch is too old and some of the modifications are not applicable due to 
file deletion. Will submit another one if needed.

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1006
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: abandon
Gerrit-Change-Id: I437880a700f61440ffab9f44035798e538fb9ff5
Gerrit-PatchSet: 6
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 

Change in asterixdb-bad[master]: Added Procedures to BAD

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Added Procedures to BAD
..


Patch Set 5:

(6 comments)

I added several comments. Basically that's focusing on the EventListener and 
related issues. You can also discuss that with me on Skype if you find the UI 
here is not very convenient.

https://asterix-gerrit.ics.uci.edu/#/c/1473/5/asterix-bad/src/main/java/org/apache/asterix/bad/lang/statement/ChannelDropStatement.java
File 
asterix-bad/src/main/java/org/apache/asterix/bad/lang/statement/ChannelDropStatement.java:

Line 122: 
ActiveLifecycleListener.INSTANCE.notifyJobFinish(hyracksJobId);
I don't think you need to notifyJobFinish here explicitly. It seems the 
jobManager handles that. Check JobManager.java


https://asterix-gerrit.ics.uci.edu/#/c/1473/5/asterix-bad/src/main/java/org/apache/asterix/bad/lang/statement/CreateProcedureStatement.java
File 
asterix-bad/src/main/java/org/apache/asterix/bad/lang/statement/CreateProcedureStatement.java:

Line 142: duration = ((StringLiteral) ((LiteralExpr) 
period.getExprList().get(0)).getValue()).getValue();
Is there a better/simpler way to get this parameter? Not sure though... Limited 
knowledge here. Just curious. :)


Line 184: ActiveJobNotificationHandler.INSTANCE.monitorJob(jobId, 
distributedJobInfo);
Do we need to monitor the distributed job here?


https://asterix-gerrit.ics.uci.edu/#/c/1473/5/asterix-bad/src/main/java/org/apache/asterix/bad/lang/statement/ProcedureDropStatement.java
File 
asterix-bad/src/main/java/org/apache/asterix/bad/lang/statement/ProcedureDropStatement.java:

Line 118: 
ActiveLifecycleListener.INSTANCE.notifyJobFinish(hyracksJobId);
Same here. Probably no need to notifyJobFinish explicitly.


https://asterix-gerrit.ics.uci.edu/#/c/1473/5/asterix-bad/src/main/java/org/apache/asterix/bad/metadata/PrecompiledJobEventListener.java
File 
asterix-bad/src/main/java/org/apache/asterix/bad/metadata/PrecompiledJobEventListener.java:

Line 147: notifyEventSubscribers(event);
IMO, managing the ActiveLifecycleEvent is not necessary here. I don't think you 
really care the about the distributedJob's status. The one that you really care 
is status of the one who manages the distributed job(manager?). I guess that 
the reason why you put the active variable for that. If I understand correctly, 
the EventSubscribers here didn't do much. The only use that I noticed is when 
createChannels, there is an assert to make sure the distributed job is up(but 
only for the first cycle). But everything after that, I think the subscriber is 
not doing anything. Correct me if I am wrong. :)


https://asterix-gerrit.ics.uci.edu/#/c/1473/5/asterix-bad/src/test/resources/runtimets/queries/procedure/insert_procedure/insert_procedure.1.ddl.aql
File 
asterix-bad/src/test/resources/runtimets/queries/procedure/insert_procedure/insert_procedure.1.ddl.aql:

Maybe rename this test case to create_procedure?


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1473
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I03550a74e2c90179e72345103b3d2c4f98148631
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb-bad
Gerrit-Branch: master
Gerrit-Owner: Steven Jacobs 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: Yes

Change in asterixdb-bad[master]: Update gitignore file for BAD project

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Update gitignore file for BAD project
..


Patch Set 1:

@Steven,
 

As a test submit, I updated the .gitignore for bad project.

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1454
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I2feb1594e78c6c893da3b888f4c678d2c00934c7
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb-bad
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Steven Jacobs 
Gerrit-Reviewer: Xikui Wang 
Gerrit-HasComments: No

Change in asterixdb-bad[master]: Update gitignore file for BAD project

[Xikui Wang (Code Review)]

Xikui Wang has uploaded a new change for review.

  https://asterix-gerrit.ics.uci.edu/1454

Change subject: Update gitignore file for BAD project
..

Update gitignore file for BAD project

Add several file filters for project files generated by intelliJ

Change-Id: I2feb1594e78c6c893da3b888f4c678d2c00934c7
---
M .gitignore
1 file changed, 2 insertions(+), 0 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb-bad 
refs/changes/54/1454/1

diff --git a/.gitignore b/.gitignore
index 42fc47d..00a7ae5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,3 +7,5 @@
 .DS_Store
 *.swp
 build
+*.iml
+.idea/*

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1454
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2feb1594e78c6c893da3b888f4c678d2c00934c7
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb-bad
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 

Change in asterixdb[master]: Modularize feed adaptors

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1430

to look at the new patch set (#3).

Change subject: Modularize feed adaptors
..

Modularize feed adaptors

1. Load adaptors from class-path.
2. Move rss adaptors/parsers to the new module, as well as related test
cases. All future parsers/adaptors that depend on various libraries should
be placed in this new module.

Change-Id: Ic4f95255f5493a813ee1f875b63a62e74bc47602
---
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
M asterixdb/asterix-external-data/pom.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/feed/test/DatasourceFactoryProviderTest.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/RecordWithPKTestReaderFactory.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/kv/KVTestReaderFactory.java
D 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/ParserFactoryProviderLoadParserTest.java
A asterixdb/asterix-villain/pom.xml
R 
asterixdb/asterix-villain/src/main/java/org/apache/asterix/villain/external/parser/rss/RSSParser.java
R 
asterixdb/asterix-villain/src/main/java/org/apache/asterix/villain/external/parser/rss/RSSParserFactory.java
R 
asterixdb/asterix-villain/src/main/java/org/apache/asterix/villain/external/parser/twitter/TweetParser.java
R 
asterixdb/asterix-villain/src/main/java/org/apache/asterix/villain/external/parser/twitter/TweetParserFactory.java
R 
asterixdb/asterix-villain/src/main/java/org/apache/asterix/villain/external/reader/rss/RSSRecordReader.java
R 
asterixdb/asterix-villain/src/main/java/org/apache/asterix/villain/external/reader/rss/RSSRecordReaderFactory.java
R 
asterixdb/asterix-villain/src/main/java/org/apache/asterix/villain/external/reader/twitter/TwitterPullRecordReader.java
R 
asterixdb/asterix-villain/src/main/java/org/apache/asterix/villain/external/reader/twitter/TwitterPushRecordReader.java
R 
asterixdb/asterix-villain/src/main/java/org/apache/asterix/villain/external/reader/twitter/TwitterRecordReaderFactory.java
R 
asterixdb/asterix-villain/src/main/java/org/apache/asterix/villain/external/util/TwitterUtil.java
A 
asterixdb/asterix-villain/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
A 
asterixdb/asterix-villain/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A 
asterixdb/asterix-villain/src/test/java/org/apache/asterix/villain/test/VillainExecutionTest.java
A asterixdb/asterix-villain/src/test/resources/runtimets/only.xml
R 
asterixdb/asterix-villain/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.1.ddl.aql
R 
asterixdb/asterix-villain/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.2.update.aql
R 
asterixdb/asterix-villain/src/test/resources/runtimets/queries/feeds/twitter-feed/twitter-feed.1.ddl.aql
R 
asterixdb/asterix-villain/src/test/resources/runtimets/queries/feeds/twitter-feed/twitter-feed.2.update.aql
A asterixdb/asterix-villain/src/test/resources/runtimets/testsuite.xml
M asterixdb/pom.xml
36 files changed, 519 insertions(+), 193 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/30/1430/3
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1430
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic4f95255f5493a813ee1f875b63a62e74bc47602
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 

Change in asterixdb[master]: TweetParser Extension

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: TweetParser Extension
..


Patch Set 10: Code-Review+1

I went through all the changes. It looks fine to me now.

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1002
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
Gerrit-PatchSet: 10
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Add upsert option for feed

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Add upsert option for feed
..


Patch Set 8: Code-Review+1

Looks good to me too. 

The integration sometimes failed at timeout. Not sure the reason.

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1068
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
Gerrit-PatchSet: 8
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: WIP - Feed Adaptor and RecordReader Clean

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1535

to look at the new patch set (#2).

Change subject: WIP - Feed Adaptor and RecordReader Clean
..

WIP - Feed Adaptor and RecordReader Clean

1. Cleaned DatasourceFactoryProvider with service interface
2. Cleaned StreamRecordReaderProvider with serivce interface
3. Combined LineRecordReader with QuotedLineRecordReader to fit the
StreamRecordReader interface.
4. Delayed the Inputstream binding at the configuration phase.
5. Added one simple test case for StreamRecordReaderProvider.

Change-Id: I699657ddd8408fd00bcbd7df57b6610ef3692a1a
---
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
M asterixdb/asterix-external-data/pom.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/rss/RSSRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/EmptyLineSeparatedRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/LineRecordReader.java
D 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/QuotedLineRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/SemiStructuredRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java
A 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.input.record.reader.stream.StreamRecordReader
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/classad/test/ClassAdToADMTest.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/RecordWithPKTestReaderFactory.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/kv/KVTestReaderFactory.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/ByteBufUTF8DecodeTest.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/RecordWithMetaTest.java
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/StreamRecordReaderProviderTest.java
23 files changed, 477 insertions(+), 250 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/35/1535/2
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1535
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I699657ddd8408fd00bcbd7df57b6610ef3692a1a
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: Update the Feed Tutorial

[Xikui Wang (Code Review)]

Xikui Wang has uploaded a new change for review.

  https://asterix-gerrit.ics.uci.edu/1533

Change subject: Update the Feed Tutorial
..

Update the Feed Tutorial

1. Updated the Twitter and RSS part to work with the current new feed
connection model.
2. Added one section about the Socket feed.
3. Commented the policy section.

Change-Id: Ia18d0b2e3f483332058d9739350643e7f8773433
---
M asterixdb/asterix-doc/src/site/markdown/feeds/tutorial.md
1 file changed, 139 insertions(+), 38 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/33/1533/1

diff --git a/asterixdb/asterix-doc/src/site/markdown/feeds/tutorial.md 
b/asterixdb/asterix-doc/src/site/markdown/feeds/tutorial.md
index 948c7aa..9e53a68 100644
--- a/asterixdb/asterix-doc/src/site/markdown/feeds/tutorial.md
+++ b/asterixdb/asterix-doc/src/site/markdown/feeds/tutorial.md
@@ -23,12 +23,15 @@
 
 * [Introduction](#Introduction)
 * [Feed Adaptors](#FeedAdaptors)
-* [Feed Policies](#FeedPolicies)
+
 
 ## Introduction  ##
 
 In this document, we describe the support for data ingestion in
-AsterixDB.  Data feeds are a new mechanism for having continuous data arrive 
into a BDMS from external sources and incrementally populate a persisted 
dataset and associated indexes. We add a new BDMS architectural component, 
called a data feed, that makes a Big Data system the caretaker for 
functionality that
+AsterixDB. Data feeds are a new mechanism for having continuous
+data arrive into a BDMS from external sources and incrementally
+populate a persisted dataset and associated indexes. We add a new BDMS
+architectural component, called a data feed, that makes a Big Data system the 
caretaker for functionality that
 used to live outside, and we show how it improves users' lives and system 
performance.
 
 ## Feed Adaptors  ##
@@ -52,25 +55,28 @@
 to ingest data that is directed at a prescribed socket.
 
 
-In this tutorial, we shall describe building two example data ingestion 
pipelines that cover the popular scenario of ingesting data from (a) Twitter 
and (b) RSS Feed source.
+In this tutorial, we shall describe building two example data ingestion 
pipelines
+that cover the popular scenarios of ingesting data from (a) Twitter (b) RSS  
(c) Socket Feed source.
 
 Ingesting Twitter Stream
 We shall use the built-in push-based Twitter adaptor.
-As a pre-requisite, we must define a Tweet using the AsterixDB Data Model 
(ADM) and the AsterixDB Query Language (AQL). Given below are the type 
definition in AQL that create a Tweet datatype which is representative of a 
real tweet as obtained from Twitter.
+As a pre-requisite, we must define a Tweet using the AsterixDB Data Model (ADM)
+and the AsterixDB Query Language (AQL). Given below are the type definition in 
AQL
+that create a Tweet datatype which is representative of a real tweet as 
obtained from Twitter.
 
 create dataverse feeds;
 use dataverse feeds;
 
 create type TwitterUser as closed {
-screen_name: string,
-lang: string,
-friends_count: int32,
-statuses_count: int32
+screen_name: string,
+lang: string,
+friends_count: int32,
+statuses_count: int32
 };
 
 create type Tweet as open {
-  id: int64,
-  user: TwitterUser
+id: int64,
+user: TwitterUser
 }
 
 create dataset Tweets (Tweet)
@@ -81,7 +87,9 @@
 
 #Using the "push_twitter" feed adapter#
 The "push_twitter" adaptor requires setting up an application account with 
Twitter. To retrieve
-tweets, Twitter requires registering an application with Twitter. Registration 
involves providing a name and a brief description for the application. Each 
application has an associated OAuth authentication credential that includes 
OAuth keys and tokens. Accessing the
+tweets, Twitter requires registering an application with Twitter. Registration 
involves providing
+a name and a brief description for the application. Each application has an 
associated OAuth
+authentication credential that includes OAuth keys and tokens. Accessing the
 Twitter API requires providing the following.
 1. Consumer Key (API Key)
 2. Consumer Secret (API Secret)
@@ -89,7 +97,9 @@
 4. Access Token Secret
 
 The "push_twitter" adaptor takes as configuration the above mentioned
-parameters. End users are required to obtain the above authentication 
credentials prior to using the "push_twitter" adaptor. For further information 
on obtaining OAuth keys and tokens and registering an application with Twitter, 
please visit http://apps.twitter.com
+parameters. End users are required to obtain the above authentication 
credentials prior to
+using the "push_twitter" adaptor. For further information on obtaining OAuth 
keys and tokens and
+registering an application with Twitter, 

Change in asterixdb[master]: Feed Adaptor and RecordReader Clean

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1535

to look at the new patch set (#3).

Change subject: Feed Adaptor and RecordReader Clean
..

Feed Adaptor and RecordReader Clean

1. Cleaned DatasourceFactoryProvider with service interface
2. Cleaned StreamRecordReaderProvider with serivce interface
3. Combined LineRecordReader with QuotedLineRecordReader to fit the
StreamRecordReader interface.
4. Delayed the Inputstream binding to the configuration phase.
5. Added one simple test case for StreamRecordReaderProvider.

Change-Id: I699657ddd8408fd00bcbd7df57b6610ef3692a1a
---
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
M asterixdb/asterix-external-data/pom.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/rss/RSSRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/EmptyLineSeparatedRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/LineRecordReader.java
D 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/QuotedLineRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/SemiStructuredRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java
A 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.input.record.reader.stream.StreamRecordReader
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/classad/test/ClassAdToADMTest.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/RecordWithPKTestReaderFactory.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/kv/KVTestReaderFactory.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/ByteBufUTF8DecodeTest.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/RecordWithMetaTest.java
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/StreamRecordReaderProviderTest.java
23 files changed, 479 insertions(+), 250 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/35/1535/3
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1535
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I699657ddd8408fd00bcbd7df57b6610ef3692a1a
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: Feed Adaptor and RecordReader Clean

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Feed Adaptor and RecordReader Clean
..


Patch Set 3:

(3 comments)

Added one Q to StreamRecordReaderFactory

https://asterix-gerrit.ics.uci.edu/#/c/1535/3/asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
File asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties:

PS3, Line 148: name
> Could we add the adaptor name to the message?
Will address these two in next submit.


https://asterix-gerrit.ics.uci.edu/#/c/1535/3/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/LineRecordReader.java
File 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/LineRecordReader.java:

Line 107: {
> Why do we have 2 nested blocks here?
blindly copied from previous code. Will address this in next update.


https://asterix-gerrit.ics.uci.edu/#/c/1535/3/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReaderFactory.java
File 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReaderFactory.java:

PS3, Line 97: recordReader
> Just return the RecordReader here?
Quick Q: Here this factory will return one of the implementation of 
StreamRecordReader. The class hierarchy would be like this: 
RecordReader -> TwitterRecordReader, StreamRecordReader, etc. ->   
StreamRecordReaderImplementations(Semistructure,LineRecord, etc.) 
The constructor for StreamRecordReaders is slightly different as for the input 
stream. That's not necessary for the other record readers. In this case, 
probably StreamRecordReader is a better option?


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1535
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I699657ddd8408fd00bcbd7df57b6610ef3692a1a
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Modularize feed adaptors

[Xikui Wang (Code Review)]

Xikui Wang has abandoned this change.

Change subject: Modularize feed adaptors
..


Abandoned

useful parts are picked up in another patch. The separate package for 
twitter/rss feeds is delayed for future better ideas.

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1430
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: abandon
Gerrit-Change-Id: Ic4f95255f5493a813ee1f875b63a62e74bc47602
Gerrit-PatchSet: 9
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 

Change in asterixdb-bad[master]: Updated BAD to work with recent Asterix changes

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Updated BAD to work with recent Asterix changes
..


Patch Set 1:

(1 comment)

added one comment.

https://asterix-gerrit.ics.uci.edu/#/c/1539/1/asterix-bad/src/main/java/org/apache/asterix/bad/lang/statement/ExecuteProcedureStatement.java
File 
asterix-bad/src/main/java/org/apache/asterix/bad/lang/statement/ExecuteProcedureStatement.java:

Line 121: ResultUtil.printResults(resultReader, 
((QueryTranslator) statementExecutor).getSessionConfig(),
Is this not-open intentionally? Do we need to open it explicitly?


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1539
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I20f2e86ff47f92477ffc1d20514127af41340346
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb-bad
Gerrit-Branch: master
Gerrit-Owner: Steven Jacobs 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Xikui Wang 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Feed Adaptor and RecordReader Clean

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1535

to look at the new patch set (#4).

Change subject: Feed Adaptor and RecordReader Clean
..

Feed Adaptor and RecordReader Clean

1. Cleaned DatasourceFactoryProvider with service interface
2. Cleaned StreamRecordReaderProvider with serivce interface
3. Combined LineRecordReader with QuotedLineRecordReader to fit the
StreamRecordReader interface.
4. Delayed the Inputstream binding to the configuration phase.
5. Added one simple test case for StreamRecordReaderProvider.

Change-Id: I699657ddd8408fd00bcbd7df57b6610ef3692a1a
---
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
M asterixdb/asterix-external-data/pom.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/rss/RSSRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/EmptyLineSeparatedRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/LineRecordReader.java
D 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/QuotedLineRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/SemiStructuredRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java
A 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.input.record.reader.stream.StreamRecordReader
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/classad/test/ClassAdToADMTest.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/RecordWithPKTestReaderFactory.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/kv/KVTestReaderFactory.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/ByteBufUTF8DecodeTest.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/RecordWithMetaTest.java
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/StreamRecordReaderProviderTest.java
23 files changed, 474 insertions(+), 250 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/35/1535/4
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1535
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I699657ddd8408fd00bcbd7df57b6610ef3692a1a
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: Update the Feed Tutorial

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Update the Feed Tutorial
..


Patch Set 1:

(1 comment)

I think the IT failure is a false alarm... Will have it test again in next 
revision.

https://asterix-gerrit.ics.uci.edu/#/c/1533/1/asterixdb/asterix-doc/src/site/markdown/feeds/tutorial.md
File asterixdb/asterix-doc/src/site/markdown/feeds/tutorial.md:

Line 34: architectural component, called a data feed, that makes a Big Data 
system the caretaker for functionality that
Do we need to limit 120 characters in md file? If so, I will fix this in next 
version.


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1533
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia18d0b2e3f483332058d9739350643e7f8773433
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Carey 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Feed Exception Revise

[Xikui Wang (Code Review)]

Xikui Wang has uploaded a new change for review.

  https://asterix-gerrit.ics.uci.edu/1534

Change subject: Feed Exception Revise
..

Feed Exception Revise

1. Add compliation exception for missing Twitter4J library. Try to
expose this exception when create feed.
2. Removed 'invalid feed parameter exception'. Expose detailed exception
information to front end.

Change-Id: Iee65b2a5365067c9ba2aa1615ac23f0ace41b51e
---
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
M 
asterixdb/asterix-lang-aql/src/main/java/org/apache/asterix/lang/aql/statement/SubscribeFeedStatement.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataException.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
6 files changed, 21 insertions(+), 12 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/34/1534/1

diff --git 
a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
 
b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
index 83ff3a2..b03ed0f 100644
--- 
a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
+++ 
b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
@@ -87,9 +87,9 @@
 public static final int 
OPERATORS_FEED_INTAKE_OPERATOR_DESCRIPTOR_CLASSLOADER_NOT_CONFIGURED = 3004;
 public static final int PARSER_DELIMITED_NONOPTIONAL_NULL = 3005;
 public static final int PARSER_DELIMITED_ILLEGAL_FIELD = 3006;
-public static final int 
FEED_MANAGEMENT_ACTIVE_LIFE_CYCLE_EVENT_SUBSCRIBER_ACTIVE_JOB_FAILURE = 3007;
+public static final int ADAPTOR_TWITTER_TWITTER4J_LIB_NOT_FOUND = 3007;
 public static final int 
OPERATORS_FEED_INTAKE_OPERATOR_NODE_PUSHABLE_FAIL_AT_INGESTION = 3008;
-public static final int 
OPERATORS_FEED_MSG_OPERATOR_NODE_PUSHABLE_INVALID_SUBSCRIBABLE_RUNTIME = 3009;
+public static final int FEED_CREATE_FEED_DATATYPE_ERROR = 3009;
 public static final int PARSER_HIVE_NON_PRIMITIVE_LIST_NOT_SUPPORT = 3010;
 public static final int PARSER_HIVE_FIELD_TYPE = 3011;
 public static final int PARSER_HIVE_GET_COLUMNS = 3012;
diff --git 
a/asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties 
b/asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
index 3e96972..9373a0f 100644
--- a/asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
+++ b/asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
@@ -72,9 +72,9 @@
 3004 = Unable to create adapter as class loader not configured for library 
%1$s in dataverse %2$s
 3005 = At record: %1$s - Field %2$s is not privatean optional type so it 
cannot accept null value.
 3006 = Illegal field %1$s in closed type %2$s
-3007 = Failure in active job.
+3007 = Twitter4J library not found!
 3008 = Unable to ingest data
-3009 = Invalid subscribable runtime type %1$s
+3009 = Exception in get record type  %1$s for feed
 3010 = Doesn't support Hive data with list of non-primitive types
 3011 = Can't get hive type for field of type %1$s
 3012 = Failed to get columns of record
diff --git 
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
 
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
index 1428863..88d6898 100644
--- 
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
+++ 
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
@@ -38,6 +38,7 @@
 import org.apache.asterix.external.util.ExternalDataUtils;
 import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException;
 import org.apache.hyracks.api.exceptions.HyracksDataException;
+import twitter4j.conf.ConfigurationBuilder;
 
 public class DatasourceFactoryProvider {
 
@@ -108,6 +109,11 @@
 case ExternalDataConstants.READER_PUSH_TWITTER:
 case ExternalDataConstants.READER_PULL_TWITTER:
 case ExternalDataConstants.READER_USER_STREAM_TWITTER:
+try {
+ConfigurationBuilder cb = new ConfigurationBuilder();
+} catch (NoClassDefFoundError e) {
+throw new 
RuntimeDataException(ErrorCode.ADAPTOR_TWITTER_TWITTER4J_LIB_NOT_FOUND);
+}
 return new TwitterRecordReaderFactory();
 case ExternalDataConstants.ALIAS_SOCKET_ADAPTER:
 case 

Change in asterixdb[master]: WIP - Feed Adaptor and RecordReader Clean

[Xikui Wang (Code Review)]

Xikui Wang has uploaded a new change for review.

  https://asterix-gerrit.ics.uci.edu/1535

Change subject: WIP - Feed Adaptor and RecordReader Clean
..

WIP - Feed Adaptor and RecordReader Clean

1. Cleaned DatasourceFactoryProvider with service interface
2. Cleaned StreamRecordReaderProvider with serivce interface
3. Combined LineRecordReader with QuotedLineRecordReader to fit the
StreamRecordReader interface.
4. Delayed the Inputstream binding at the configuration phase.
5. Added one simple test case for StreamRecordReaderProvider.

Change-Id: I699657ddd8408fd00bcbd7df57b6610ef3692a1a
---
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
M asterixdb/asterix-external-data/pom.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/rss/RSSRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/EmptyLineSeparatedRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/LineRecordReader.java
D 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/QuotedLineRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/SemiStructuredRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java
A 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.input.record.reader.stream.StreamRecordReader
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/classad/test/ClassAdToADMTest.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/RecordWithPKTestReaderFactory.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/kv/KVTestReaderFactory.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/ByteBufUTF8DecodeTest.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/RecordWithMetaTest.java
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/StreamRecordReaderProviderTest.java
23 files changed, 473 insertions(+), 249 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/35/1535/1

diff --git 
a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
 
b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
index 83ff3a2..5365bd8 100644
--- 
a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
+++ 
b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
@@ -158,6 +158,8 @@
 public static final int UTIL_FILE_SYSTEM_WATCHER_NO_FILES_FOUND = 3076;
 public static final int UTIL_LOCAL_FILE_SYSTEM_UTILS_PATH_NOT_FOUND = 3077;
 public static final int UTIL_HDFS_UTILS_CANNOT_OBTAIN_HDFS_SCHEDULER = 
3078;
+public static final int READER_FACTORY_RECORD_READER_NOT_FOUND = 3079;
+public static final int 
PROVIDER_DATASOURCE_FACTORY_DUPLICATE_FORMAT_MAPPING = 3080;
 
 // Loads the map that maps error codes to error message templates.
 private static Map errorMessageMap = null;
diff --git 
a/asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties 
b/asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
index 3e96972..81dd289 100644
--- a/asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
+++ b/asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
@@ -143,3 +143,4 @@
 3076 = %1$s: no files found
 3077 = %1$s: path not found
 3078 = Cannot obtain hdfs scheduler
+3080 = Duplicate 

Change in asterixdb[master]: Feed Exception Revise

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1534

to look at the new patch set (#2).

Change subject: Feed Exception Revise
..

Feed Exception Revise

1. Add compliation exception for missing Twitter4J library. Try to
expose this exception when create feed.
2. Removed 'invalid feed parameter exception'. Expose detailed exception
information to front end.

Change-Id: Iee65b2a5365067c9ba2aa1615ac23f0ace41b51e
---
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
M 
asterixdb/asterix-lang-aql/src/main/java/org/apache/asterix/lang/aql/statement/SubscribeFeedStatement.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/MetadataException.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
6 files changed, 21 insertions(+), 12 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/34/1534/2
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1534
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iee65b2a5365067c9ba2aa1615ac23f0ace41b51e
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: Add upsert option for feed

[Xikui Wang (Code Review)]

Hello Michael Blow, Till Westmann, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1068

to look at the new patch set (#9).

Change subject: Add upsert option for feed
..

Add upsert option for feed

For ASTERIXDB-1567. Provide "upsert-feed"="true" for
feed configuration, which changes the default record
insert to upsert. Added one test case for upsert feed.

Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
A asterixdb/asterix-app/data/tinysocial/twu_update.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.10.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.11.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.3.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.4.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.5.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.6.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.7.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.8.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.9.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/upsert-feed/upsert-feed.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
18 files changed, 350 insertions(+), 12 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/68/1068/9
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1068
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
Gerrit-PatchSet: 9
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: Add upsert option for feed

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1068

to look at the new patch set (#13).

Change subject: Add upsert option for feed
..

Add upsert option for feed

For ASTERIXDB-1567. Provide "upsert-feed"="true" for
feed configuration, which changes the default record
insert to upsert. Added one test case for upsert feed.

Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
A asterixdb/asterix-app/data/tinysocial/twu_update.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.3.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.4.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.5.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.6.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.7.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.8.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/upsert-feed/upsert-feed.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
15 files changed, 279 insertions(+), 12 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/68/1068/13
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1068
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
Gerrit-PatchSet: 13
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: Add upsert option for feed

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1068

to look at the new patch set (#12).

Change subject: Add upsert option for feed
..

Add upsert option for feed

For ASTERIXDB-1567. Provide "upsert-feed"="true" for
feed configuration, which changes the default record
insert to upsert. Added one test case for upsert feed.

Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
A asterixdb/asterix-app/data/tinysocial/twu_update.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.3.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.4.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.5.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.6.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.7.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.8.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/upsert-feed/upsert-feed.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
15 files changed, 279 insertions(+), 12 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/68/1068/12
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1068
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
Gerrit-PatchSet: 12
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: Feed tutorial revise.

[Xikui Wang (Code Review)]

Xikui Wang has uploaded a new change for review.

  https://asterix-gerrit.ics.uci.edu/1152

Change subject: Feed tutorial revise.
..

Feed tutorial revise.

1. Fix 1352.
2. Revise the twitter feed part to work with the new tweet parser.

Change-Id: I110e2225a3036109bf58a9ef529407de955335ab
---
M asterixdb/asterix-doc/src/site/markdown/feeds/tutorial.md
1 file changed, 13 insertions(+), 15 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/52/1152/1

diff --git a/asterixdb/asterix-doc/src/site/markdown/feeds/tutorial.md 
b/asterixdb/asterix-doc/src/site/markdown/feeds/tutorial.md
index 39e04b1..54bfaa5 100644
--- a/asterixdb/asterix-doc/src/site/markdown/feeds/tutorial.md
+++ b/asterixdb/asterix-doc/src/site/markdown/feeds/tutorial.md
@@ -61,22 +61,18 @@
 create dataverse feeds;
 use dataverse feeds;
 
-create type TwitterUser if not exists as open{
-screen_name: string,
-language: string,
-friends_count: int32,
-status_count: int32,
-name: string,
-followers_count: int32
+create type TwitterUser as closed{
+screen_name: string,
+lang: string,
+friends_count: int32,
+statuses_count: int32
 };
-create type Tweet if not exists as open{
-id: string,
-user: TwitterUser,
-latitude:double,
-longitude:double,
-created_at:string,
-message_text:string
-};
+
+create type Tweet as open
+{
+  id: int64,
+  user: TwitterUser
+}
 
 create dataset Tweets (Tweet)
 primary key id;
@@ -103,6 +99,7 @@
 
 create feed TwitterFeed if not exists using "push_twitter"
 (("type-name"="Tweet"),
+ ("format"="twitter-status"),
  ("consumer.key"=""),
  ("consumer.secret"="**"),
  ("access.token"="**"),
@@ -185,6 +182,7 @@
 create feed my_feed using
 rss_feed (
("type-name"="Rss"),
+   ("format"="rss"),
("url"="http://rss.cnn.com/rss/edition.rss;)
 );
 

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1152
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I110e2225a3036109bf58a9ef529407de955335ab
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 

Change in asterixdb[master]: Add upsert option for feed

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1068

to look at the new patch set (#11).

Change subject: Add upsert option for feed
..

Add upsert option for feed

For ASTERIXDB-1567. Provide "upsert-feed"="true" for
feed configuration, which changes the default record
insert to upsert. Added one test case for upsert feed.

Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
A asterixdb/asterix-app/data/tinysocial/twu_update.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.3.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.4.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.5.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.6.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.7.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.8.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/upsert-feed/upsert-feed.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
15 files changed, 275 insertions(+), 12 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/68/1068/11
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1068
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
Gerrit-PatchSet: 11
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: Feed tutorial revise.

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1152

to look at the new patch set (#2).

Change subject: Feed tutorial revise.
..

Feed tutorial revise.

1. Fix 1352.
2. Revise the twitter feed part to work with the new tweet parser.

Change-Id: I110e2225a3036109bf58a9ef529407de955335ab
---
M asterixdb/asterix-doc/src/site/markdown/feeds/tutorial.md
1 file changed, 12 insertions(+), 15 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/52/1152/2
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1152
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I110e2225a3036109bf58a9ef529407de955335ab
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: Add upsert option for feed

[Xikui Wang (Code Review)]

Hello abdullah alamoudi, Michael Blow, Till Westmann, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1068

to look at the new patch set (#14).

Change subject: Add upsert option for feed
..

Add upsert option for feed

For ASTERIXDB-1567. Provide "upsert-feed"="true" for
feed configuration, which changes the default record
insert to upsert. Added one test case for upsert feed.

Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
A asterixdb/asterix-app/data/tinysocial/twu_update.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.3.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.4.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.5.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.6.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.7.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.8.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/upsert-feed/upsert-feed.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
15 files changed, 278 insertions(+), 12 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/68/1068/14
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1068
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
Gerrit-PatchSet: 14
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: Add upsert option for feed

[Xikui Wang (Code Review)]

Hello Till Westmann, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1068

to look at the new patch set (#10).

Change subject: Add upsert option for feed
..

Add upsert option for feed

For ASTERIXDB-1567. Provide "upsert-feed"="true" for
feed configuration, which changes the default record
insert to upsert. Added one test case for upsert feed.

Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/LangExpressionToPlanTranslator.java
A asterixdb/asterix-app/data/tinysocial/twu_update.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.3.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.4.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.5.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.6.server.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.7.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/upsert-feed/upsert-feed.8.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/upsert-feed/upsert-feed.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
15 files changed, 275 insertions(+), 12 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/68/1068/10
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1068
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic5133e7c6941fea4110cc9983f99502f364dc810
Gerrit-PatchSet: 10
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: TweetParser Extension

[Xikui Wang (Code Review)]

Hello abdullah alamoudi, Yingyi Bu, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1002

to look at the new patch set (#14).

Change subject: TweetParser Extension
..

TweetParser Extension

This patch includes following changes:

1. ExtendedTweetParser to parse more than fix attributes.

2. Changed the twitter feeds message unit from Status to String.

Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
---
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.2.update.aql
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPullRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/library/java/JObjectAccessors.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/TweetParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/Datatypes.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/TwitterUtil.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AOrderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/AUnorderedListType.java
M 
asterixdb/asterix-om/src/main/java/org/apache/asterix/om/types/TypeTagUtil.java
15 files changed, 482 insertions(+), 154 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/02/1002/14
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1002
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7021e7b779de05b9ec999a8d5f8464fb0ab413c0
Gerrit-PatchSet: 14
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: Yingyi Bu 
Gerrit-Reviewer: abdullah alamoudi 

Change in asterixdb[master]: Feed Connection Refactoring

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1259

to look at the new patch set (#3).

Change subject: Feed Connection Refactoring
..

Feed Connection Refactoring

1. The feed subscription network using FeedJoint is removed.
2. FeedConnection metadata dataset is added (pkeys: dataverseName,
feedName, datasetName).
3. Replaced the old intake job + collect job combination with one single
job using SplitOperator.
4. Now one feed can connect to multiple datasets.
5. The disconnect feed job is replaced by ActiveManagerMessage.
6. Twitter adaptor will run on all possible nodes now.
7. The new feed life cycle is:
  - Create feed
  - Connect feed to dataset0, dataset1, dataset2, etc.
  - Start feed
  - Stop feed
  - Disconnect feed
8. Fixed one sporadic error described in ASTERIXDB-1709.

Change-Id: Ic36267eb9a10df21734ce1cc1f38583e23c9e8f0
---
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/message/ActiveManagerMessage.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceRandomPartitioningFeedComputationRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/UnnestToDataScanRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
D 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedJoint.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedOperations.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_dataset/metadata_dataset.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_datatype/metadata_datatype.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_index/metadata_index.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_selfjoin/metadata_selfjoin.1.adm
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/big-object/big_object_feed_20M/big_object_feed_20M.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/typed_adapter/typed_adapter.4.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.4.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-in-value/change-feed-with-meta-pk-in-meta-open-index-in-value.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-with-missing/change-feed-with-meta-pk-in-meta-open-index-with-missing.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta/change-feed-with-meta-pk-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-with-mixed-index/change-feed-with-meta-with-mixed-index.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed/change-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.4.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-push-socket/feed-push-socket.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-push-socket/feed-push-socket.3.server.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-push-socket/feed-push-socket.5.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-with-external-function/feed-with-external-function.1.ddl.aql
M 

Change in asterixdb[master]: Feed Connection Refactoring

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1259

to look at the new patch set (#4).

Change subject: Feed Connection Refactoring
..

Feed Connection Refactoring

1. The feed subscription network using FeedJoint is removed.
2. FeedConnection metadata dataset is added (pkeys: dataverseName,
feedName, datasetName).
3. Replaced the old intake job + collect job combination with one single
job using SplitOperator.
4. Now one feed can connect to multiple datasets.
5. The disconnect feed job is replaced by ActiveManagerMessage.
6. Twitter adaptor will run on all possible nodes now.
7. The new feed life cycle is:
  - Create feed
  - Connect feed to dataset0, dataset1, dataset2, etc.
  - Start feed
  - Stop feed
  - Disconnect feed
8. Fixed one sporadic error described in ASTERIXDB-1709.

Change-Id: Ic36267eb9a10df21734ce1cc1f38583e23c9e8f0
---
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/message/ActiveManagerMessage.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceRandomPartitioningFeedComputationRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/UnnestToDataScanRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
D 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedJoint.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedOperations.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_dataset/metadata_dataset.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_datatype/metadata_datatype.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_index/metadata_index.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_selfjoin/metadata_selfjoin.1.adm
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/big-object/big_object_feed_20M/big_object_feed_20M.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/typed_adapter/typed_adapter.4.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.4.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-in-value/change-feed-with-meta-pk-in-meta-open-index-in-value.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-with-missing/change-feed-with-meta-pk-in-meta-open-index-with-missing.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta/change-feed-with-meta-pk-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-with-mixed-index/change-feed-with-meta-with-mixed-index.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed/change-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.4.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-push-socket/feed-push-socket.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-push-socket/feed-push-socket.3.server.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-push-socket/feed-push-socket.5.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-with-external-function/feed-with-external-function.1.ddl.aql
M 

Change in asterixdb[master]: Feed Connection Refactoring

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1259

to look at the new patch set (#5).

Change subject: Feed Connection Refactoring
..

Feed Connection Refactoring

1. The feed subscription network using FeedJoint is removed.
2. FeedConnection metadata dataset is added (pkeys: dataverseName,
feedName, datasetName).
3. Replaced the old intake job + collect job combination with one single
job using SplitOperator.
4. Now one feed can connect to multiple datasets.
5. The disconnect feed job is replaced by ActiveManagerMessage.
6. Twitter adaptor will run on all possible nodes now.
7. The new feed life cycle is:
  - Create feed
  - Connect feed to dataset0, dataset1, dataset2, etc.
  - Start feed
  - Stop feed
  - Disconnect feed
8. Fixed one sporadic error described in ASTERIXDB-1709.

Change-Id: Ic36267eb9a10df21734ce1cc1f38583e23c9e8f0
---
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/message/ActiveManagerMessage.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceRandomPartitioningFeedComputationRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/UnnestToDataScanRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
D 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedJoint.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedOperations.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_dataset/metadata_dataset.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_datatype/metadata_datatype.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_index/metadata_index.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_selfjoin/metadata_selfjoin.1.adm
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/big-object/big_object_feed_20M/big_object_feed_20M.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/typed_adapter/typed_adapter.4.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.4.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-in-value/change-feed-with-meta-pk-in-meta-open-index-in-value.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-with-missing/change-feed-with-meta-pk-in-meta-open-index-with-missing.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta/change-feed-with-meta-pk-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-with-mixed-index/change-feed-with-meta-with-mixed-index.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed/change-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.4.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-push-socket/feed-push-socket.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-push-socket/feed-push-socket.3.server.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-push-socket/feed-push-socket.5.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-with-external-function/feed-with-external-function.1.ddl.aql
M 

Change in asterixdb[master]: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser
..


Patch Set 4:

Hi Wail,

Thanks for your comments! My replies are as follow:

Line 220: Here the incoming data is checked first. If the incoming data doesn't 
have this attribute, and it's not optional, then it throws exception. For 
optional attribute, if incoming attribute is null, this check will be false as 
it's not closed field.

Line 213 - 230: Actually it's necessary. I found out if we assigned extra 
attribute to closed datatype, there will be a problem in inserting. So for the 
closed datatype, I am using cherry-pick like method, and for open datatype, I 
just pull whatever in that record.

Line 236, Yes. There is. The weird part of Tweets coming in is, there are 
certain attribute only has name but with content. The length == 0 case is when 
obj is a JSONArray.

Line 258, Fixed.

Thanks for your help!

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1339
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Wail Alkowaileet 
Gerrit-Reviewer: Xikui Wang 
Gerrit-HasComments: No

Change in asterixdb[master]: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1339

to look at the new patch set (#5).

Change subject: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser
..

Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

1. For ASTERIXDB-1609, add UNION type check in writeField, and add one
more case for orderedList.
2. For OrderedList bug, change UnorderedListBuilder to
OrderedListBuilder.

Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd
---
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
1 file changed, 68 insertions(+), 28 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/39/1339/5
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1339
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Wail Alkowaileet 
Gerrit-Reviewer: Xikui Wang 

Change in asterixdb[master]: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1339

to look at the new patch set (#4).

Change subject: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser
..

Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

1. For ASTERIXDB-1609, add UNION type check in writeField, and add one
more case for orderedList.
2. For OrderedList bug, change UnorderedListBuilder to
OrderedListBuilder.

Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd
---
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
1 file changed, 68 insertions(+), 28 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/39/1339/4
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1339
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Wail Alkowaileet 
Gerrit-Reviewer: Xikui Wang 

Change in asterixdb[master]: Introduce XML Reader & Parser

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Introduce XML Reader & Parser
..


Patch Set 6:

(1 comment)

https://asterix-gerrit.ics.uci.edu/#/c/1269/6/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/XMLFileParser.java
File 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/XMLFileParser.java:

Line 48: JSONObject xmlObj = XML.toJSONObject(record.toString());
> "Some information may be lost" doesn't sound ideal :) Is this aiming to be 
I guess this means it can only maintain those information that can be stored in 
JSON, as they are two different format. Thus it says 'Comments, prologs, DTDs, 
and <[ [ ]]> are ignored'. 

This was designed for CAPS message which each of the document is a small XML 
document with simple elements. Later Taewoo come to me with a application 
scenario that he needs to load xml documents into AsterixDB. So I extended this 
a little bit to fit his need.

After meeting with Taewoo today, It turns out he has a slightly different 
application scenario that he needs to map a nested element in XML document 
(e.g.  in  ...  ...  ... 
 into a record. One possible change is to allow user specify the 
level they want to do mapping, but I am not sure that's general enough.

Generally, IMO there are needs for processing XML in AsterixDB, and probably 
something working (may not be perfect) can ease the pain.


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1269
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
Gerrit-PatchSet: 6
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Add user-stream for Twitter Adaptor

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Add user-stream for Twitter Adaptor
..


Patch Set 2:

(1 comment)

The user whose access token is used in the feed creation, is the one that we 
are monitoring.

https://asterix-gerrit.ics.uci.edu/#/c/1272/2/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
File 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java:

Line 68: }
> Can we extract the common code from these three constructors and move it to
Yeah. Basically the only part that could be shrunk is the 
tweetListener.setInputQ(inputQ). But the tweetListener(s) are not the same in 
them, should I cast all to be tweetListener?


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1272
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I99cdd4cb667306d378317616f9811dfce3e6d838
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Steven Jacobs 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Add user-stream for Twitter Adaptor

[Xikui Wang (Code Review)]

Xikui Wang has uploaded a new change for review.

  https://asterix-gerrit.ics.uci.edu/1272

Change subject: Add user-stream for Twitter Adaptor
..

Add user-stream for Twitter Adaptor

1. Add user-stream option for Twitter Adaptor
2. Refactor part of TwitterRecordReaderFactory
3. To create a user-stream feed, using following ddl:
  create feed TwitterFeed using twitter_user_stream(
  ("format"="twitter-status"),
  ("type-name"="Tweet"),
  ...
   // rest is same as push feed

Change-Id: I99cdd4cb667306d378317616f9811dfce3e6d838
---
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/TwitterUtil.java
5 files changed, 278 insertions(+), 93 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/72/1272/1

diff --git 
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
 
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
index 9ead8a9..c296bc6 100644
--- 
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
+++ 
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
@@ -26,6 +26,8 @@
 import org.apache.asterix.external.dataflow.AbstractFeedDataFlowController;
 import org.apache.asterix.external.input.record.GenericRecord;
 import org.apache.asterix.external.util.FeedLogManager;
+import org.apache.asterix.external.util.TwitterUtil;
+import twitter4j.DirectMessage;
 import twitter4j.FilterQuery;
 import twitter4j.StallWarning;
 import twitter4j.Status;
@@ -33,27 +35,43 @@
 import twitter4j.StatusListener;
 import twitter4j.TwitterObjectFactory;
 import twitter4j.TwitterStream;
+import twitter4j.User;
+import twitter4j.UserList;
+import twitter4j.UserStreamListener;
 
 public class TwitterPushRecordReader implements IRecordReader {
 private LinkedBlockingQueue inputQ;
 private TwitterStream twitterStream;
 private GenericRecord record;
+private StatusListener tweetListener;
 private boolean closed = false;
 
-public TwitterPushRecordReader(TwitterStream twitterStream, FilterQuery 
query) {
-record = new GenericRecord<>();
-inputQ = new LinkedBlockingQueue<>();
-this.twitterStream = 
twitterStream;//TwitterUtil.getTwitterStream(configuration);
-this.twitterStream.addListener(new TweetListener(inputQ));
+public TwitterPushRecordReader(TwitterStream twitterStream, 
TwitterUtil.TweetListener tweetListener,
+FilterQuery query) {
+init(twitterStream);
+tweetListener.setInputQ(inputQ);
+this.twitterStream.addListener(tweetListener);
 this.twitterStream.filter(query);
 }
 
-public TwitterPushRecordReader(TwitterStream twitterStream) {
+public TwitterPushRecordReader(TwitterStream twitterStream, 
TwitterUtil.TweetListener tweetListener) {
+init(twitterStream);
+tweetListener.setInputQ(inputQ);
+this.twitterStream.addListener(tweetListener);
+twitterStream.sample();
+}
+
+public TwitterPushRecordReader(TwitterStream twitterStream, 
TwitterUtil.UserTweetsListener tweetListener) {
+init(twitterStream);
+tweetListener.setInputQ(inputQ);
+this.twitterStream.addListener(tweetListener);
+twitterStream.user();
+}
+
+private void init(TwitterStream twitterStream) {
 record = new GenericRecord<>();
 inputQ = new LinkedBlockingQueue<>();
-this.twitterStream = twitterStream;//
-this.twitterStream.addListener(new TweetListener(inputQ));
-twitterStream.sample();
+this.twitterStream = twitterStream;
 }
 
 @Override
@@ -89,46 +107,6 @@
 return false;
 }
 return true;
-}
-
-private class TweetListener implements StatusListener {
-
-private LinkedBlockingQueue inputQ;
-
-public TweetListener(LinkedBlockingQueue inputQ) {
-this.inputQ = inputQ;
-}
-
-@Override
-public void onStatus(Status tweet) {
-String jsonTweet = TwitterObjectFactory.getRawJSON(tweet);
-inputQ.add(jsonTweet);
-}
-
-@Override
-public void 

Change in asterixdb[master]: Introduce XML Reader & Parser

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Introduce XML Reader & Parser
..


Patch Set 6:

(1 comment)

https://asterix-gerrit.ics.uci.edu/#/c/1269/6/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/XMLFileParser.java
File 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/XMLFileParser.java:

Line 48: JSONObject xmlObj = XML.toJSONObject(record.toString());
> Is there a description of this transformation somewhere? A quick google sea
I found this in the org.json.XML documentation.

Convert a well-formed (but not necessarily valid) XML string into a JSONObject. 
Some information may be lost in this transformation because JSON is a data 
format and XML is a document format. XML uses elements, attributes, and content 
text, while JSON uses unordered collections of name/value pairs and arrays of 
values. JSON does not does not like to distinguish between elements and 
attributes. Sequences of similar elements are represented as JSONArrays. 
Content text may be placed in a "content" member. Comments, prologs, DTDs, and 
<[ [ ]]> are ignored.

One more thing is that I skipped the root element so that nested elements 
(mapped into attributes) are exposed in JSON object.


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1269
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
Gerrit-PatchSet: 6
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Add user-stream for Twitter Adaptor

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Add user-stream for Twitter Adaptor
..


Patch Set 2:

(1 comment)

Yes. The user of twitter feed needs to apply for Twitter API using certain 
twitter account. That's the account they are monitoring. :)

https://asterix-gerrit.ics.uci.edu/#/c/1272/2/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
File 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java:

Line 119: ? new 
TwitterPushRecordReader(TwitterUtil.getTwitterStream(configuration),
> I'm guessing this doesn't get closed until the feed is dropped?
Yes. Until the feed is stopped or dropped.


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1272
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I99cdd4cb667306d378317616f9811dfce3e6d838
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Steven Jacobs 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1339

to look at the new patch set (#7).

Change subject: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser
..

Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

1. For ASTERIXDB-1609, add UNION type check in writeField, and add one
more case for orderedList.
2. For OrderedList bug, change UnorderedListBuilder to
OrderedListBuilder.

Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd
---
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
1 file changed, 70 insertions(+), 29 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/39/1339/7
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1339
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd
Gerrit-PatchSet: 7
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Wail Alkowaileet 
Gerrit-Reviewer: Xikui Wang 

Change in asterixdb[master]: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser
..


Patch Set 6:

@Wail,

Yes. You are right. I misunderstood it. :) 

Fixed it in the updated patch. Sorry for the delay.

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1339
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd
Gerrit-PatchSet: 6
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Wail Alkowaileet 
Gerrit-Reviewer: Xikui Wang 
Gerrit-HasComments: No

Change in asterixdb[master]: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1339

to look at the new patch set (#6).

Change subject: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser
..

Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

1. For ASTERIXDB-1609, add UNION type check in writeField, and add one
more case for orderedList.
2. For OrderedList bug, change UnorderedListBuilder to
OrderedListBuilder.

Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd
---
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
1 file changed, 70 insertions(+), 29 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/39/1339/6
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1339
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd
Gerrit-PatchSet: 6
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Wail Alkowaileet 
Gerrit-Reviewer: Xikui Wang 

Change in asterixdb[master]: Add record reader and parser for CAP messages

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Add record reader and parser for CAP messages
..


Patch Set 9:

@Till & Abdullah,

One thing I am not very sure in this patch is ASTERIXDB-1690. If you can help 
me verify that, it will be very helpful. :)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1269
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
Gerrit-PatchSet: 9
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: No

Change in asterixdb[master]: Add record reader and parser for CAP messages

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1269

to look at the new patch set (#10).

Change subject: Add record reader and parser for CAP messages
..

Add record reader and parser for CAP messages

1. Add record reader for CAP message.
2. Add CAP message parser based on JSON library and ADMParser.
3. Fix ASTERIXDB-1690: deadlock between close() and take() in FileSystemWatcher
4. Add test cases for using CAP adaptor in feed and load statement.
5. Add an option in configuration to allow user use CAP adaptor loading
certain XML documents.
- "collection" = "true" (Multiple records in one document, check
"document_with_records_collection.xml" as an example).

Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
---
A asterixdb/asterix-app/data/cap_messages/ER.xml
A asterixdb/asterix-app/data/cap_messages/HSA.xml
A asterixdb/asterix-app/data/cap_messages/STA.xml
A asterixdb/asterix-app/data/cap_messages/bigger.xml
A asterixdb/asterix-app/data/cap_messages/small_ER.xml
A asterixdb/asterix-app/data/document_with_records_collection.xml
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-collection/cap-adaptor-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-no-collection/cap-adaptor-no-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-collection/load-cap-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-messages/load-cap-messages.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/CAPMessageRecordReader.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/CAPMessageParser.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/CAPMessageParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/ParserFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/FileSystemWatcher.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/LocalFileSystemUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
36 files changed, 1,255 insertions(+), 34 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/69/1269/10
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1269
To unsubscribe, visit 

Change in asterixdb[master]: Feed Connection Refactoring

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1259

to look at the new patch set (#2).

Change subject: Feed Connection Refactoring
..

Feed Connection Refactoring

1. The feed subscription network using FeedJoint is removed.
2. FeedConnection metadata dataset is added (pkeys: dataverseName,
feedName, datasetName).
3. Replaced the old intake job + collect job combination with one single
job using SplitOperator.
4. Now one feed can connect to multiple datasets.
5. The disconnect feed job is replaced by ActiveManagerMessage.
6. Twitter adaptor will run on all possible nodes now.
7. The new feed life cycle is:
  - Create feed
  - Connect feed to dataset0, dataset1, dataset2, etc.
  - Start feed
  - Stop feed
  - Disconnect feed

Change-Id: Ic36267eb9a10df21734ce1cc1f38583e23c9e8f0
---
M 
asterixdb/asterix-active/src/main/java/org/apache/asterix/active/message/ActiveManagerMessage.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceRandomPartitioningFeedComputationRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/UnnestToDataScanRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
D 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedJoint.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedOperations.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/big-object/big_object_feed_20M/big_object_feed_20M.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/typed_adapter/typed_adapter.4.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.4.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-in-value/change-feed-with-meta-pk-in-meta-open-index-in-value.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-with-missing/change-feed-with-meta-pk-in-meta-open-index-with-missing.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta/change-feed-with-meta-pk-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-with-mixed-index/change-feed-with-meta-with-mixed-index.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed/change-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.4.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-push-socket/feed-push-socket.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-push-socket/feed-push-socket.3.server.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-push-socket/feed-push-socket.5.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-with-external-function/feed-with-external-function.1.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-with-external-function/feed-with-external-function.3.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-with-external-parser-with-open-index/feed-with-external-parser-with-open-index.4.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-with-external-parser-with-two-open-indexes/feed-with-external-parser-with-two-open-indexes.4.update.aql
M 

Change in asterixdb[master]: Add record reader and parser for CAP messages

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1269

to look at the new patch set (#8).

Change subject: Add record reader and parser for CAP messages
..

Add record reader and parser for CAP messages

1. Add record reader for CAP message.
2. Add CAP message parser based on JSON library and ADMParser.
3. Fix ASTERIX-1690: deadlock between close() and take() in FileSystemWatcher
4. Add test cases for using CAP adaptor in feed and load statement.
5. Add two options in configuration to allow user use CAP adaptor on a
limited set of 'well-formated' XML documents.
- "collection" = "true" (Multiple records in one document, check
"document_with_records_collection.xml" as an example).
- "primary-key-mapping" = "ACCESS-PATH-TO-PRIMARY-KEY"  (Map certain
element in document to Primary key when it cannot be translated
properly. Check "caps-adaptor-collection" test case for more
detail).

Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
---
A asterixdb/asterix-app/data/cap_messages/ER.xml
A asterixdb/asterix-app/data/cap_messages/HSA.xml
A asterixdb/asterix-app/data/cap_messages/STA.xml
A asterixdb/asterix-app/data/cap_messages/bigger.xml
A asterixdb/asterix-app/data/cap_messages/small_ER.xml
A asterixdb/asterix-app/data/document_with_records_collection.xml
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-collection/cap-adaptor-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-no-collection/cap-adaptor-no-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-collection/load-cap-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-messages/load-cap-messages.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/CAPMessageRecordReader.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/CAPMessageParser.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/CAPMessageParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/ParserFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/FileSystemWatcher.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/LocalFileSystemUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
36 

Change in asterixdb[master]: Add record reader and parser for CAP messages

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1269

to look at the new patch set (#7).

Change subject: Add record reader and parser for CAP messages
..

Add record reader and parser for CAP messages

1. Add record reader for CAP message.
2. Add CAP message parser based on JSON library and ADMParser.
3. Fix ASTERIX-1690: deadlock between close() and take() in FileSystemWatcher
4. Add test cases for using CAP adaptor in feed and load statement.
5. Add two options in configuration to allow user use CAP adaptor on a
limited set of 'well-formated' XML documents.
- "collection" = "true" (Multiple records in one document, check
"document_with_records_collection.xml" as an example).
- "primary-key-mapping" = "ACCESS-PATH-TO-PRIMARY-KEY"  (Map certain
element in document to Primary key when it cannot be translated
properly. Check "caps-adaptor-collection" test case for more
detail).

Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
---
A asterixdb/asterix-app/data/cap_messages/ER.xml
A asterixdb/asterix-app/data/cap_messages/HSA.xml
A asterixdb/asterix-app/data/cap_messages/STA.xml
A asterixdb/asterix-app/data/cap_messages/bigger.xml
A asterixdb/asterix-app/data/cap_messages/small_ER.xml
A asterixdb/asterix-app/data/document_with_records_collection.xml
M asterixdb/asterix-app/src/test/resources/runtimets/only.xml
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-collection/cap-adaptor-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-no-collection/cap-adaptor-no-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-collection/load-cap-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-messages/load-cap-messages.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/CAPMessageRecordReader.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/CAPMessageParser.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/CAPMessageParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/ParserFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/FileSystemWatcher.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/LocalFileSystemUtils.java
M 

Change in asterixdb[master]: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

[Xikui Wang (Code Review)]

Xikui Wang has uploaded a new change for review.

  https://asterix-gerrit.ics.uci.edu/1339

Change subject: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser
..

Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

1. For ASTERIXDB-1609, add UNION type check in writeField, and add one
more case for orderedList.
2. For OrderedList bug, change UnorderedListBuilder to
OrderedListBuilder.

Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd
---
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
1 file changed, 43 insertions(+), 23 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/39/1339/1

diff --git 
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
 
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
index 8d483dc..fc69d27 100644
--- 
a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
+++ 
b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
@@ -22,14 +22,16 @@
 import org.apache.asterix.builders.IARecordBuilder;
 import org.apache.asterix.builders.IAsterixListBuilder;
 import org.apache.asterix.builders.ListBuilderFactory;
+import org.apache.asterix.builders.OrderedListBuilder;
 import org.apache.asterix.builders.RecordBuilderFactory;
-import org.apache.asterix.builders.UnorderedListBuilder;
 import org.apache.asterix.external.api.IRawRecord;
 import org.apache.asterix.external.api.IRecordDataParser;
 import org.apache.asterix.om.base.AMutablePoint;
 import org.apache.asterix.om.base.ANull;
+import org.apache.asterix.om.types.AOrderedListType;
 import org.apache.asterix.om.types.ARecordType;
 import org.apache.asterix.om.types.ATypeTag;
+import org.apache.asterix.om.types.AUnionType;
 import org.apache.asterix.om.types.BuiltinType;
 import org.apache.asterix.om.types.IAType;
 import org.apache.asterix.om.util.container.IObjectPool;
@@ -60,46 +62,69 @@
 aPoint = new AMutablePoint(0, 0);
 }
 
-private void parseUnorderedList(JSONArray jArray, DataOutput output) 
throws IOException, JSONException {
+private void parseJSONArray(JSONArray jArray, DataOutput output, 
AOrderedListType orderedListType)
+throws IOException, JSONException {
 ArrayBackedValueStorage itemBuffer = getTempBuffer();
-UnorderedListBuilder unorderedListBuilder = (UnorderedListBuilder) 
getUnorderedListBuilder();
+OrderedListBuilder orderedList = (OrderedListBuilder) 
getOrderedListBuilder();
 
-unorderedListBuilder.reset(null);
+orderedList.reset(orderedListType);
 for (int iter1 = 0; iter1 < jArray.length(); iter1++) {
 itemBuffer.reset();
-if (writeField(jArray.get(iter1), null, 
itemBuffer.getDataOutput())) {
-unorderedListBuilder.addItem(itemBuffer);
+if (writeField(jArray.get(iter1), orderedListType.getItemType(), 
itemBuffer.getDataOutput())) {
+orderedList.addItem(itemBuffer);
 }
 }
-unorderedListBuilder.write(output, true);
+orderedList.write(output, true);
 }
 
 private boolean writeField(Object fieldObj, IAType fieldType, DataOutput 
out) throws IOException, JSONException {
 boolean writeResult = true;
-if (fieldType != null) {
-switch (fieldType.getTypeTag()) {
+IAType chkFieldType;
+if (fieldType instanceof AUnionType) {
+chkFieldType = ((AUnionType) fieldType).getActualType();
+} else {
+chkFieldType = fieldType;
+}
+if (chkFieldType != null) {
+switch (chkFieldType.getTypeTag()) {
 case STRING:
-out.write(BuiltinType.ASTRING.getTypeTag().serialize());
+out.write(fieldType.getTypeTag().serialize());
 utf8Writer.writeUTF8(fieldObj.toString(), out);
 break;
 case INT64:
-aInt64.setValue((long) fieldObj);
+out.write(fieldType.getTypeTag().serialize());
+if (fieldObj instanceof Integer) {
+out.writeLong(((Integer) fieldObj).longValue());
+} else {
+out.writeLong((Long) fieldObj);
+}
 int64Serde.serialize(aInt64, out);
 break;
 case INT32:
-out.write(BuiltinType.AINT32.getTypeTag().serialize());
+out.write(fieldType.getTypeTag().serialize());
 out.writeInt((Integer) fieldObj);
 break;
 case DOUBLE:
-out.write(BuiltinType.ADOUBLE.getTypeTag().serialize());
+

Change in asterixdb[master]: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1339

to look at the new patch set (#2).

Change subject: Fix ASTERIXDB-1609 and OrderedList bug in TweetParser
..

Fix ASTERIXDB-1609 and OrderedList bug in TweetParser

1. For ASTERIXDB-1609, add UNION type check in writeField, and add one
more case for orderedList.
2. For OrderedList bug, change UnorderedListBuilder to
OrderedListBuilder.

Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd
---
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/TweetParser.java
1 file changed, 57 insertions(+), 25 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/39/1339/2
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1339
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia27148cb10206b93dabf7655aed68f3004f96dfd
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: Add record reader and parser for CAP messages

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1269

to look at the new patch set (#9).

Change subject: Add record reader and parser for CAP messages
..

Add record reader and parser for CAP messages

1. Add record reader for CAP message.
2. Add CAP message parser based on JSON library and ADMParser.
3. Fix ASTERIX-1690: deadlock between close() and take() in FileSystemWatcher
4. Add test cases for using CAP adaptor in feed and load statement.
5. Add an option in configuration to allow user use CAP adaptor loading
certain XML documents.
- "collection" = "true" (Multiple records in one document, check
"document_with_records_collection.xml" as an example).

Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
---
A asterixdb/asterix-app/data/cap_messages/ER.xml
A asterixdb/asterix-app/data/cap_messages/HSA.xml
A asterixdb/asterix-app/data/cap_messages/STA.xml
A asterixdb/asterix-app/data/cap_messages/bigger.xml
A asterixdb/asterix-app/data/cap_messages/small_ER.xml
A asterixdb/asterix-app/data/document_with_records_collection.xml
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-collection/cap-adaptor-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-no-collection/cap-adaptor-no-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-collection/load-cap-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-messages/load-cap-messages.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/CAPMessageRecordReader.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/CAPMessageParser.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/CAPMessageParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/ParserFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/FileSystemWatcher.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/LocalFileSystemUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
36 files changed, 1,255 insertions(+), 34 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/69/1269/9
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1269
To unsubscribe, visit 

Change in asterixdb[master]: Introduce XML Adaptor & Parser

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1269

to look at the new patch set (#2).

Change subject: Introduce XML Adaptor & Parser
..

Introduce XML Adaptor & Parser

1. Add a record reader for XML document.
2. Add xml parser based on XML to JSON and ADMParser.
3. Fix ASTERIX-1690: deadlock between close() and take() in FileSystemWatcher

Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
---
A asterixdb/asterix-app/data/xml/ER.xml
A asterixdb/asterix-app/data/xml/HSA.xml
A asterixdb/asterix-app/data/xml/STA.xml
A asterixdb/asterix-app/data/xml/small_ER.xml
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/xml-adaptor/xml-adaptor.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/xml-adaptor/xml-adaptor.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/xml-adaptor/xml-adaptor.3.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/xml-adaptor/xml-adaptor.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/xml-adaptor/xml-adaptor.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/CharArrayRecord.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/XMLFileRecordReader.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/XMLFileParser.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/XMLFileParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/ParserFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/FileSystemWatcher.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/LocalFileSystemUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
20 files changed, 512 insertions(+), 32 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/69/1269/2
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1269
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: Introduce XML Reader & Parser

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1269

to look at the new patch set (#6).

Change subject: Introduce XML Reader & Parser
..

Introduce XML Reader & Parser

1. Add a record reader for XML document.
2. Add xml parser based on XML to JSON and ADMParser.
3. Fix ASTERIX-1690: deadlock between close() and take() in FileSystemWatcher
4. Add test cases for using XML adaptor in feed and load statement.

Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
---
A asterixdb/asterix-app/data/xml/ER.xml
A asterixdb/asterix-app/data/xml/HSA.xml
A asterixdb/asterix-app/data/xml/STA.xml
A asterixdb/asterix-app/data/xml/bigger.xml
A asterixdb/asterix-app/data/xml/small_ER.xml
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/xml-adaptor/xml-adaptor.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/xml-adaptor/xml-adaptor.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/xml-adaptor/xml-adaptor.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/xml-adaptor/xml-adaptor.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/xml-adaptor/xml-adaptor.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-xml-files/load-xml-files.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-xml-files/load-xml-files.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-xml-files/load-xml-files.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/xml-adaptor/xml-adaptor.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-xml-files/load-xml-files.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/XMLFileRecordReader.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/XMLFileParser.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/XMLFileParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/ParserFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/FileSystemWatcher.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/LocalFileSystemUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
25 files changed, 794 insertions(+), 34 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/69/1269/6
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1269
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
Gerrit-PatchSet: 6
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Taewoo Kim 

Change in asterixdb[master]: Introduce XML Reader & Parser

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1269

to look at the new patch set (#4).

Change subject: Introduce XML Reader & Parser
..

Introduce XML Reader & Parser

1. Add a record reader for XML document.
2. Add xml parser based on XML to JSON and ADMParser.
3. Fix ASTERIX-1690: deadlock between close() and take() in FileSystemWatcher

Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
---
A asterixdb/asterix-app/data/xml/ER.xml
A asterixdb/asterix-app/data/xml/HSA.xml
A asterixdb/asterix-app/data/xml/STA.xml
A asterixdb/asterix-app/data/xml/small_ER.xml
M asterixdb/asterix-app/src/test/resources/runtimets/only.xml
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/xml-adaptor/xml-adaptor.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/xml-adaptor/xml-adaptor.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/xml-adaptor/xml-adaptor.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/xml-adaptor/xml-adaptor.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/xml-adaptor/xml-adaptor.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/xml-adaptor/xml-adaptor.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/XMLFileRecordReader.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/XMLFileParser.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/XMLFileParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/ParserFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/FileSystemWatcher.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/LocalFileSystemUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
21 files changed, 529 insertions(+), 34 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/69/1269/4
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1269
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: Add user-stream for Twitter Adaptor

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1272

to look at the new patch set (#2).

Change subject: Add user-stream for Twitter Adaptor
..

Add user-stream for Twitter Adaptor

1. Add user-stream option for Twitter Adaptor
2. Refactor part of TwitterRecordReaderFactory
3. To create a user-stream feed, using following ddl:
  create feed TwitterFeed using twitter_user_stream(
  ("format"="twitter-status"),
  ("type-name"="Tweet"),
  ...
   // rest is same as push feed

Change-Id: I99cdd4cb667306d378317616f9811dfce3e6d838
---
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterPushRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/TwitterUtil.java
5 files changed, 251 insertions(+), 76 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/72/1272/2
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1272
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I99cdd4cb667306d378317616f9811dfce3e6d838
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 

Change in asterixdb[master]: Add record reader and parser for CAP messages

[Xikui Wang (Code Review)]

Hello Till Westmann, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1269

to look at the new patch set (#11).

Change subject: Add record reader and parser for CAP messages
..

Add record reader and parser for CAP messages

1. Add record reader for CAP message.
2. Add CAP message parser based SAXParser. Element attributes are ignored
currently.
3. Fix ASTERIX-1690: deadlock between close() and take() in FileSystemWatcher
4. Add test cases for using CAP adaptor in feed and load statement.
5. Add an option in configuration to allow user use CAP adaptor loading
certain XML documents.
- "collection" = "true" (Multiple records in one document, check
"document_with_records_collection.xml" as an example).

Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
---
A asterixdb/asterix-app/data/cap_messages/CAP_1.xml
A asterixdb/asterix-app/data/cap_messages/CAP_2.xml
A asterixdb/asterix-app/data/cap_messages/CAP_3.xml
A asterixdb/asterix-app/data/cap_messages/CAP_4.xml
A asterixdb/asterix-app/data/document_with_records_collection.xml
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-collection/cap-adaptor-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-no-collection/cap-adaptor-no-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-collection/load-cap-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-messages/load-cap-messages.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/CAPMessageRecordReader.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/CAPMessageParser.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/CAPMessageParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/ParserFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/FileSystemWatcher.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/LocalFileSystemUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
35 files changed, 1,415 insertions(+), 34 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/69/1269/11
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1269
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Change in asterixdb[master]: Runtime ErrorCode fix in external data

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1374

to look at the new patch set (#5).

Change subject: Runtime ErrorCode fix in external data
..

Runtime ErrorCode fix in external data

1. Change all runtime exceptions in external data to use error code.
2. This change only covers HyracksDataExcetion. AsterixException remains
unchanged.

Change-Id: Ida810a56bf4aef1394879f088a6a5e8f82c60b74
---
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/AsterixException.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/library/ILibraryManager.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/AsterixInputStream.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IExternalFunction.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IExternalIndexer.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IExternalScalarFunction.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IFunctionHelper.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/ILookupRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordDataParser.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordReader.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordWithMetadataParser.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordWithPKDataParser.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IStreamDataParser.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/ChangeFeedDataFlowController.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/ChangeFeedWithMetaDataFlowController.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/CounterTimerTupleForwarder.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/FeedRecordDataFlowController.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/FeedWithMetaDataFlowController.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/FrameFullTupleForwarder.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/RateControlledTupleForwarder.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/RecordDataFlowController.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/dataflow/StreamDataFlowController.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/dataset/adapter/LookupAdapter.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/api/IActiveLifecycleEventSubscriber.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/api/ISubscribableRuntime.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/dataflow/DistributeFeedFrameWriter.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/dataflow/FeedExceptionHandler.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/dataflow/FrameDistributor.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/dataflow/FrameSpiller.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/management/ActiveLifecycleEventSubscriber.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/management/FeedEventsListener.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/policy/FeedPolicyEnforcer.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/runtime/AdapterExecutor.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/runtime/AdapterRuntimeManager.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/runtime/CollectionRuntime.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/feed/runtime/IngestionRuntime.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/indexing/ExternalFileIndexAccessor.java
M 

Change in asterixdb[master]: Add record reader and parser for CAP messages

[Xikui Wang (Code Review)]

Hello Till Westmann, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1269

to look at the new patch set (#16).

Change subject: Add record reader and parser for CAP messages
..

Add record reader and parser for CAP messages

1. Add record reader for CAP message.
2. Add CAP message parser based SAXParser. Element attributes are ignored
currently.
3. Fix ASTERIX-1690: deadlock between close() and take() in FileSystemWatcher
4. Add test cases for using CAP adaptor in feed and load statement.
5. Add an option in configuration to allow user use CAP adaptor loading
certain XML documents.
- "collection" = "true" (Multiple records in one document, check
"document_with_records_collection.xml" as an example).
6. Extract message timestamp from the comment. All datetimes in element
content are treated as string due to different timezones.

Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
---
A asterixdb/asterix-app/data/cap_messages/CAP_0.xml
A asterixdb/asterix-app/data/cap_messages/CAP_1.xml
A asterixdb/asterix-app/data/cap_messages/CAP_2.xml
A asterixdb/asterix-app/data/cap_messages/CAP_3.xml
A asterixdb/asterix-app/data/cap_messages/CAP_4.xml
A asterixdb/asterix-app/data/document_with_records_collection.xml
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.4.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-collection/cap-adaptor-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-no-collection/cap-adaptor-no-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-collection/load-cap-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-messages/load-cap-messages.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/CAPMessageRecordReader.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/CAPMessageParser.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/CAPMessageParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/ParserFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/FileSystemWatcher.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/LocalFileSystemUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
35 files changed, 1,721 insertions(+), 36 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/69/1269/16
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1269
To 

Change in asterixdb[master]: Add record reader and parser for CAP messages

[Xikui Wang (Code Review)]

Hello Till Westmann, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1269

to look at the new patch set (#14).

Change subject: Add record reader and parser for CAP messages
..

Add record reader and parser for CAP messages

1. Add record reader for CAP message.
2. Add CAP message parser based SAXParser. Element attributes are ignored
currently.
3. Fix ASTERIX-1690: deadlock between close() and take() in FileSystemWatcher
4. Add test cases for using CAP adaptor in feed and load statement.
5. Add an option in configuration to allow user use CAP adaptor loading
certain XML documents.
- "collection" = "true" (Multiple records in one document, check
"document_with_records_collection.xml" as an example).

Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
---
A asterixdb/asterix-app/data/cap_messages/CAP_0.xml
A asterixdb/asterix-app/data/cap_messages/CAP_1.xml
A asterixdb/asterix-app/data/cap_messages/CAP_2.xml
A asterixdb/asterix-app/data/cap_messages/CAP_3.xml
A asterixdb/asterix-app/data/cap_messages/CAP_4.xml
A asterixdb/asterix-app/data/document_with_records_collection.xml
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.4.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-collection/cap-adaptor-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-no-collection/cap-adaptor-no-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-collection/load-cap-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-messages/load-cap-messages.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/CAPMessageRecordReader.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/CAPMessageParser.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/CAPMessageParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/ParserFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/FileSystemWatcher.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/LocalFileSystemUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
35 files changed, 1,696 insertions(+), 36 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/69/1269/14
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1269
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: 

Change in asterixdb[master]: Add record reader and parser for CAP messages

[Xikui Wang (Code Review)]

Hello Till Westmann, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1269

to look at the new patch set (#15).

Change subject: Add record reader and parser for CAP messages
..

Add record reader and parser for CAP messages

1. Add record reader for CAP message.
2. Add CAP message parser based SAXParser. Element attributes are ignored
currently.
3. Fix ASTERIX-1690: deadlock between close() and take() in FileSystemWatcher
4. Add test cases for using CAP adaptor in feed and load statement.
5. Add an option in configuration to allow user use CAP adaptor loading
certain XML documents.
- "collection" = "true" (Multiple records in one document, check
"document_with_records_collection.xml" as an example).

Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
---
A asterixdb/asterix-app/data/cap_messages/CAP_0.xml
A asterixdb/asterix-app/data/cap_messages/CAP_1.xml
A asterixdb/asterix-app/data/cap_messages/CAP_2.xml
A asterixdb/asterix-app/data/cap_messages/CAP_3.xml
A asterixdb/asterix-app/data/cap_messages/CAP_4.xml
A asterixdb/asterix-app/data/document_with_records_collection.xml
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-collection/caps-adaptor-collection.4.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.1.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.3.sleep.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.4.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/caps-adaptor-no-collection/caps-adaptor-no-collection.5.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-collection/load-cap-collection.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.0.ddl.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.1.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/load/load-cap-messages/load-cap-messages.2.query.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-collection/cap-adaptor-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/feeds/cap-adaptor-no-collection/cap-adaptor-no-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-collection/load-cap-collection.1.adm
A 
asterixdb/asterix-app/src/test/resources/runtimets/results/load/load-cap-messages/load-cap-messages.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/CAPMessageRecordReader.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/CAPMessageParser.java
A 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/CAPMessageParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/ParserFactoryProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/StreamRecordReaderProvider.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/ExternalDataConstants.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/FileSystemWatcher.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/util/LocalFileSystemUtils.java
M 
asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/feeds/FeedMetadataUtil.java
35 files changed, 1,702 insertions(+), 36 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/69/1269/15
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1269
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: 

Change in asterixdb[master]: Add record reader and parser for CAP messages

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Add record reader and parser for CAP messages
..


Patch Set 16:

Sorry for making so many changes in this patch. I found a couple of bugs when I 
was addressing your previous comments. Thanks!

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1269
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
Gerrit-PatchSet: 16
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-HasComments: No

Change in asterixdb[master]: Feed Connection Refactoring

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Feed Connection Refactoring
..


Patch Set 5:

(7 comments)

I added several comments to yours. I will try to do some cleaning and merge the 
updates from master as well. Thanks!

Sorry for the delay. I was preparing for the candidacy exam preparation.

https://asterix-gerrit.ics.uci.edu/#/c/1259/5/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedOperations.java
File 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedOperations.java:

Line 116: private static final ILangCompilationProvider compilationProvider 
= new AqlCompilationProvider();
> get rid of static compilationProvider
I added two local variables in QueryTranslator before calling getConnectJob. Is 
that ok?


Line 162: PrintWriter writer = new PrintWriter(System.err, true);
> MAJOR SonarQube violation:
Any idea about this issue?


Line 197: ingestionOp = new FeedIntakeOperatorDescriptor(jobSpec, 
feed, firstOp.getAdaptorLibraryName(),
> when will the adapterFactory be null?
If it's external adaptor from UDF, this will be null. It will be bind to class 
at execution time.


Line 232: if (opDesc instanceof 
AsterixLSMTreeInsertDeleteOperatorDescriptor
> I believe that we don't need the if or the else here!!
The reason I didn't remove feed collect operator from the workflow is for feed 
policy. I couldn't find a better place to adopt feed policy in current setting, 
so I kept the feed collect operator


https://asterix-gerrit.ics.uci.edu/#/c/1259/5/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
File 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java:

Line 345: case Statement.Kind.CONNECT_FEED:
> add test cases that queries the connection dataset after connect and after 
There is one test case connect-feed added for this case. I will change it to 
query for more detailed connection info.


Line 2143: try {
> what if feed has already been started?
Fixed this, as well as for stopFeed.
I met one tricky case in this though. If I execute 2 stop statement with small 
time gap, the 2nd stop will throw an exception as the 1st stop is not fully 
executed yet. Is that an issue?


Line 2183: // TODO: check feed is actually running.
> do the todo and add a test case?
Done with the start fix.


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1259
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic36267eb9a10df21734ce1cc1f38583e23c9e8f0
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Xikui Wang 
Gerrit-Reviewer: abdullah alamoudi 
Gerrit-HasComments: Yes

Change in asterixdb[master]: Add record reader and parser for CAP messages

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: Add record reader and parser for CAP messages
..


Patch Set 16:

Mike and I had a discussion about moving these research specific patches into 
the BADAsterix branch. So I will resubmit this patch into asterix_bad branch 
instead of master. Thanks for your comments. Please remove this one. :)

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1269
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ia36101a0761973a9edb96b42d3dcc117661301da
Gerrit-PatchSet: 16
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 
Gerrit-HasComments: No

Change in asterixdb[master]: Load parser from classpath in ParserFactoryProvider

[Xikui Wang (Code Review)]

Hello Till Westmann, Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1416

to look at the new patch set (#4).

Change subject: Load parser from classpath in ParserFactoryProvider
..

Load parser from classpath in ParserFactoryProvider

Change the behavior of ParserFactoryProvider to allow it loads parser
from classpath.

Change-Id: I2ac039fe3daaf0636cf004289bd0c8a3229197a9
---
M asterixdb/asterix-external-data/pom.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IDataParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/ADMDataParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/DelimitedDataParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/HiveDataParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/RSSParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/RecordWithMetadataParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/parser/factory/TweetParserFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/ParserFactoryProvider.java
A 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IDataParserFactory
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/library/ClassAdParserFactory.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/factory/TestRecordWithPKParserFactory.java
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/parser/test/ParserFactoryProviderLoadParserTest.java
13 files changed, 172 insertions(+), 30 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/16/1416/4
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1416
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2ac039fe3daaf0636cf004289bd0c8a3229197a9
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang 

Change in asterixdb[master]: WIP-Feed Connection Refactoring

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1259

to look at the new patch set (#6).

Change subject: WIP-Feed Connection Refactoring
..

WIP-Feed Connection Refactoring

1. The feed subscription network using FeedJoint is removed.
2. FeedConnection metadata dataset is added (pkeys: dataverseName,
feedName, datasetName).
3. Replaced the old intake job + collect job combination with one single
job using SplitOperator.
4. Now one feed can connect to multiple datasets.
5. The disconnect feed job is replaced by ActiveManagerMessage.
6. Twitter adaptor will run on all possible nodes now.
7. The new feed life cycle is:
  - Create feed
  - Connect feed to dataset0, dataset1, dataset2, etc.
  - Start feed
  - Stop feed
  - Disconnect feed
8. Fixed one sporadic error described in ASTERIXDB-1709.

Change-Id: Ic36267eb9a10df21734ce1cc1f38583e23c9e8f0
---
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/IntroduceRandomPartitioningFeedComputationRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/UnnestToDataScanRule.java
M 
asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
D 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedJoint.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedOperations.java
D 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/external/FeedWorkCollection.java
M 
asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_dataset/metadata_dataset.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_datatype/metadata_datatype.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_index/metadata_index.1.adm
M 
asterixdb/asterix-app/src/test/resources/metadata/results/basic/metadata_selfjoin/metadata_selfjoin.1.adm
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/big-object/big_object_feed_20M/big_object_feed_20M.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/external-library/typed_adapter/typed_adapter.4.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-after-ingest/change-feed-with-meta-pk-in-meta-index-after-ingest.4.query.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-in-meta/change-feed-with-meta-pk-in-meta-index-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest/change-feed-with-meta-pk-in-meta-index-with-missing-after-ingest.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-in-value/change-feed-with-meta-pk-in-meta-open-index-in-value.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta-open-index-with-missing/change-feed-with-meta-pk-in-meta-open-index-with-missing.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-pk-in-meta/change-feed-with-meta-pk-in-meta.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed-with-meta-with-mixed-index/change-feed-with-meta-with-mixed-index.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/change-feed/change-feed.2.update.aql
A 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.0.ddl.aql
C 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.1.update.aql
R 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.2.update.aql
C 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.3.query.aql
C 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/connect-feed/connect-feed.4.ddl.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-push-socket/feed-push-socket.2.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-push-socket/feed-push-socket.3.server.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-push-socket/feed-push-socket.5.update.aql
M 
asterixdb/asterix-app/src/test/resources/runtimets/queries/feeds/feed-with-external-function/feed-with-external-function.1.ddl.aql
M 

Change in asterixdb[master]: WIP - Modularize feed adaptors

[Xikui Wang (Code Review)]

Xikui Wang has posted comments on this change.

Change subject: WIP - Modularize feed adaptors
..


Patch Set 1:

@Till

I added one package with temporary name for the Twitter4j related classes. I 
tried to move all adaptors into the new package, but that introduces a lot of 
cyclic reference issue. So I upload this one first...

Please help me verify the project configuration... As you may see from the test 
results, the twitter-related test cases are failing. AFAIK we need to add jars 
to classpath for enable extensions. Is there a way to enable that via Maven? I 
am guessing that probably I need to create a villain test executor to handle 
these test cases?

-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1430
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4f95255f5493a813ee1f875b63a62e74bc47602
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Xikui Wang 
Gerrit-HasComments: No

Change in asterixdb[master]: WIP - Modularize feed adaptors

[Xikui Wang (Code Review)]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

https://asterix-gerrit.ics.uci.edu/1430

to look at the new patch set (#2).

Change subject: WIP - Modularize feed adaptors
..

WIP - Modularize feed adaptors

Modularize the feed adaptors and move 'license-unfriendly' adaptors to
separate package.

Change-Id: Ic4f95255f5493a813ee1f875b63a62e74bc47602
---
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite.xml
M 
asterixdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode.java
M asterixdb/asterix-common/src/main/resources/asx_errormsg/en.properties
M asterixdb/asterix-external-data/pom.xml
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/api/IRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/HDFSDataSourceFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/rss/RSSRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/input/record/reader/stream/StreamRecordReaderFactory.java
M 
asterixdb/asterix-external-data/src/main/java/org/apache/asterix/external/provider/DatasourceFactoryProvider.java
A 
asterixdb/asterix-external-data/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/feed/test/DatasourceFactoryProviderTest.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/RecordWithPKTestReaderFactory.java
M 
asterixdb/asterix-external-data/src/test/java/org/apache/asterix/external/input/record/reader/kv/KVTestReaderFactory.java
A asterixdb/asterix-villain/pom.xml
R 
asterixdb/asterix-villain/src/main/java/org/apache/asterix/villain/external/input/record/reader/twitter/TwitterPullRecordReader.java
R 
asterixdb/asterix-villain/src/main/java/org/apache/asterix/villain/external/input/record/reader/twitter/TwitterPushRecordReader.java
R 
asterixdb/asterix-villain/src/main/java/org/apache/asterix/villain/external/input/record/reader/twitter/TwitterRecordReaderFactory.java
R 
asterixdb/asterix-villain/src/main/java/org/apache/asterix/villain/external/util/TwitterUtil.java
A 
asterixdb/asterix-villain/src/main/resources/META-INF/services/org.apache.asterix.external.api.IRecordReaderFactory
A 
asterixdb/asterix-villain/src/test/java/org/apache/asterix/villain/test/VillainExecutionTest.java
A asterixdb/asterix-villain/src/test/resources/runtimets/only.xml
R 
asterixdb/asterix-villain/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.1.ddl.aql
R 
asterixdb/asterix-villain/src/test/resources/runtimets/queries/feeds/revised-tweet-parser/revised-tweet-parser.2.update.aql
R 
asterixdb/asterix-villain/src/test/resources/runtimets/queries/feeds/twitter-feed/twitter-feed.1.ddl.aql
R 
asterixdb/asterix-villain/src/test/resources/runtimets/queries/feeds/twitter-feed/twitter-feed.2.update.aql
A asterixdb/asterix-villain/src/test/resources/runtimets/testsuite.xml
M asterixdb/pom.xml
27 files changed, 482 insertions(+), 95 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb 
refs/changes/30/1430/2
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1430
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic4f95255f5493a813ee1f875b63a62e74bc47602
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Xikui Wang 
Gerrit-Reviewer: Jenkins 
Gerrit-Reviewer: Michael Blow 
Gerrit-Reviewer: Till Westmann 
Gerrit-Reviewer: Xikui Wang