Re: [VOTE] Changing COPY FROM syntax

[Taewoo Kim]

+1

interesting fact - MS has a similar syntax COPY INTO (Transact-SQL) - Azure
Synapse Analytics and Microsoft Fabric | Microsoft Learn


Best,
Taewoo


On Fri, Oct 27, 2023 at 11:11 AM Glenn Justo Galvizo 
wrote:

> +1 from me as well!
>
> > On Oct 27, 2023, at 10:15, Till Westmann  wrote:
> >
> > +1 this is much nicer
> >
> >> On 2023/10/26 05:05:01 Mike Carey wrote:
> >> PS - I assume the semantics will be UPSERT-based? (Vs. one-time or
> >> INSERT-based?)
> >>
> >>> On 10/24/23 10:16 AM, Wail Alkowaileet wrote:
> >>> Hi all,
> >>>
> >>> I'm proposing to change the current syntax for COPY FROM. The current
> >>> syntax looks as follows:
> >>>
>  COPY Customers
>  USING localfs (
>    ("path"="asterix_nc1://data/nontagged/customerData.json"),
>    ("format"="json")
>  );
> 
> >>> This syntax uses the old way of configuring the adapter localfs. In our
> >>> feeds, we use the WITH clause. Another issue is that the current
> syntax is
> >>> missing the keyword FROM, which makes it ambiguous if we add support
> for
> >>> COPY TO.
> >>>
> >>> I propose to change the syntax to be as follows:
> >>>
>  COPY Customers
>  FROM localfs
>  PATH ("asterix_nc1://data/nontagged/customerData.json")
>  WITH {
>  "format": "json"
>  };
> 
> >>> First, the proposed syntax introduces the use of FROM .
> >>> Second, it mandates the use of PATH (instead of having it in the WITH
> >>> clause). Additionally, the proposed syntax will make both COPY FROM and
> >>> COPY TO less different.
> >>>
> >>> Example of COPY TO:
> >>>
>  COPY Customers
>  TO localfs
>  PATH("localhost:///myData/Customers")
>  WITH {
>  "format" : "json"
>  };
> 
>

Re: [VOTE] Release Apache AsterixDB 0.9.8.1 and Hyracks 0.3.8.1

[Taewoo Kim]

+1

- checked hash
- compiled source code
- executed the sample cluster in binary and did a smoke test

Best,
Taewoo


On Wed, May 3, 2023 at 3:44 PM Till Westmann  wrote:

>
> +1
>
> Checked the Hyracks and AsterixDB source artifacts:
> - Signatures and hashes correct
> — LICENSE and NOTICE look ok
> - Source files have Apache header
> - No unexpected binary files (checked expected binary files)
> - Can compile from source
>
> Checked the AsterixDB Server binary artifact:
> - Signatures and hashes correct
> — LICENSE and NOTICE look ok
>
> Till
>
>
> On 2 May 2023, at 2:05, Ian Maxon wrote:
>
> > Hi everyone,
> >
> > Please verify and vote on the latest release of Apache AsterixDB.
> >
> > The change that produced this release is up on Gerrit:
> >
> > https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/17502
> >
> > The release artifacts are as follows:
> >
> > AsterixDB Source
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.8.1-source-release.zip
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.8.1-source-release.zip.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.8.1-source-release.zip.sha512
> >
> > SHA512:
> >
> 35e80f1f2068da10820777b2308a2b5e96727aecc46232beaeff5166dd0a625a8441bbe7961bf70b4796462f7c4dfa004c51685f70702404cde1961cd2e854fc
> >
> >
> > Hyracks Source
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.8.1-source-release.zip
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.8.1-source-release.zip.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.8.1-source-release.zip.sha512
> >
> > SHA512:
> >
> 2c246cc2414ed8f23972e451a49af90483022f69122167a9392b29dcf5bde058bb9e79665abacde3d080b6fe24a9a13cef688c093521f738fd802915ae0bfd5d
> >
> >
> > AsterixDB NCService Installer:
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.8.1-binary-assembly.zip
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.8.1-binary-assembly.zip.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.8.1-binary-assembly.zip.sha512
> >
> > SHA512:
> >
> 4c0b73127d8c33287a3768538094a15aceb905e353ed3e9c7dfb3b1ec553f57ccdcfda9411fa2a2bfb0bc9fa7abe50e9f517bf797326740a65b4605874e9601f
> >
> >
> > The KEYS file containing the PGP keys used to sign the release can be
> > found at
> >
> > https://dist.apache.org/repos/dist/release/asterixdb/KEYS
> >
> > RAT was executed as part of Maven via the RAT maven plugin, but
> > excludes files that are:
> >
> > - data for tests
> > - procedurally generated,
> > - or source files which come without a header mentioning their
> > license,
> >  but have an explicit reference in the LICENSE file.
> >
> >
> > The vote is open for 72 hours, or until the necessary number of votes
> > (3 +1) has been reached.
> >
> > Please vote
> > [ ] +1 release these packages as Apache AsterixDB 0.9.8.1 and
> > Apache Hyracks 0.3.8.1
> > [ ] 0 No strong feeling either way
> > [ ] -1 do not release one or both packages because ...
>

Re: [VOTE] Accept Apache Iceberg support APE

[Taewoo Kim]

+1

On Wed, Apr 12, 2023 at 2:17 PM Ian Maxon  wrote:

> Based on the discussion in
> https://lists.apache.org/thread/w3o0xrth73ml7rxq71c7nytgmt8m2f4y, it seems
> like there is a positive sentiment towards our first APE proposal.
> Therefore I think it is time to vote.
> The vote is the usual procedure, open for 72 hours and passes with either 3
> votes or the time elapsing with a majority of positive votes.
>

Re: [VOTE] AsterixDB Proposed Enhancements (APEs)

[Taewoo Kim]

+1

On Fri, Mar 10, 2023 at 10:58 AM Till Westmann  wrote:

> And here is my +1.
>
> On 10 Mar 2023, at 10:54, Till Westmann wrote:
>
> > Hi,
> >
> >
> > There seems to be a positive sentiment towards the APE proposal [1].
> > Let’s move on to a VOTE.
> >
> >
> > Please vote
> > [ ] +1 adopt the APE proposal
> > [ ]  0 modify the proposal because ...
> > [ ] -1 do not adopt the proposal because ...
> >
> > This procedural vote follows the common format of majority rule, that
> > is if there are more favorable votes than unfavorable ones, the issue
> > is considered to have passed [2].
> >
> > PMC members have formally binding votes, but we encourage all
> > community members to vote.
> >
> > The vote is open for at least 72 hours or until 3 votes have been
> > cast.
> >
> > Cheers,
> > Till
> >
> >
> >
> > [1] https://lists.apache.org/thread/lhhdhvvnkxw8r17z69yvb81l87gzgd9h
> > [2] https://www.apache.org/foundation/voting.html
>

Re: [VOTE] Release Apache AsterixDB JDBC Connector 0.9.8 (RC0)

[Taewoo Kim]

+1 tested the JDBC driver in DbVisualizer

Best,
Taewoo


On Thu, May 19, 2022 at 5:43 PM Murtadha Hubail  wrote:

> +1
>
> Cheers,
> Murtadha
> 
> From: Mike Carey 
> Sent: Friday, May 20, 2022 12:10:16 AM
> To: dev@asterixdb.apache.org 
> Subject: Re: [VOTE] Release Apache AsterixDB JDBC Connector 0.9.8 (RC0)
>
> +1
>
> Tested the JDBC driver in DbVisualizer.
>
> Cheers,
>
> Mike
>
> On 5/18/22 5:08 PM, Dmitry Lychagin wrote:
> > +1.
> >
> > Tested the JDBC driver in DbVisualizer and DBeaver.
> >
> > Thanks,
> > -- Dmitry
> >
> >
> > On 5/17/22, 4:47 PM, "Ian Maxon"  wrote:
> >
> >   EXTERNAL EMAIL:  Use caution when opening attachments or clicking
> on links
> >
> >
> >
> >
> >
> >  Hi everyone,
> >
> >  Please verify and vote on the second release of the AsterixDB JDBC
> >  connector.
> >
> >  The change that produced this release is up on Gerrit:
> >
> >  https://asterix-gerrit.ics.uci.edu/c/asterixdb-clients/+/16383
> >
> >  The release artifacts are as follows:
> >
> >  JDBC Driver Source
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-jdbc-0.9.8-source-release.zip
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-jdbc-0.9.8-source-release.zip.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-jdbc-0.9.8-source-release.zip.sha512
> >
> >  SHA512:
> >
> 9c2a0d85a809def7cf18e6431f61fb1e73a22625b807e57e08e62e7129f1a4a03928876a21ae211800de8582adcf8d86d49bc9c023c4a2f94d64fb48b378fd86⏎⏎
> >
> >  JDBC Driver Distributable Jar
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.8-dist.jar
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.8-dist.jar.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.8-dist.jar.sha512
> >
> >  SHA512:
> >
> 03a96da0d69dbb828a00edd5285e36facea96c63592f3e737c14c681aaaf572e6e769689cbf42203ada345fb469b3d328c3245734d9fb53e14e48ea094fa15f4
> >
> >  Tableau Connector (TACO file)
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterixdb_jdbc.taco
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterixdb_jdbc.taco.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterixdb_jdbc.taco.sha512
> >
> >  SHA512:
> >
> 83a15100d5f6def312eda5a587833256f28ad25f07832b17680d10fa0b72b7434f7888207064c1dbd0cb215f989b391dffc920ce26232ea9a8dc8753e97f1d33
> >
> >
> >  The KEYS file containing the PGP keys used to sign the release can
> be
> >  found at
> >
> >  https://dist.apache.org/repos/dist/release/asterixdb/KEYS
> >
> >  RAT was executed as part of Maven via the RAT maven plugin, but
> >  excludes files that are:
> >
> >  - data for tests
> >  - procedurally generated,
> >  - or source files which come without a header mentioning their
> license,
> >but have an explicit reference in the LICENSE file.
> >
> >
> >  The vote is open for 72 hours, or until the necessary number of
> votes
> >  (3 +1) has been reached.
> >
> >  Please vote
> >  [ ] +1 release these packages as Apache AsterixDB JDBC Driver 0.9.8
> >  [ ] 0 No strong feeling either way
> >  [ ] -1 do not release one or both packages because ...
> >
> >  Thanks!
> >
>

Re: [VOTE] Release Apache AsterixDB 0.9.8 and Apache Hyracks 0.3.8 (RC1)

[Taewoo Kim]

+1

Built from the source codes
Did a smoke test using NCService installer


On Wed, May 4, 2022 at 9:47 PM Mike Carey  wrote:

> +1  Kicked tires of NCService Installer SQL++ and tabular views - latest
> release still WFM!  :-)
>
> On 5/3/22 12:36 AM, Ian Maxon wrote:
> > [X] +1 release these packages as Apache AsterixDB 0.9.8 and
> > Apache Hyracks 0.3.8
>

Re: [VOTE] Release Apache AsterixDB JDBC Connector 0.9.7.1 (RC1)

[Taewoo Kim]

+1
Verified the driver with DbVisualizer


On Fri, Feb 25, 2022 at 10:28 AM Till Westmann 
wrote:

> +1
>
> Checked the source artifacts:
> - Signatures and hashes correct
> — LICENSE and NOTICE look ok
> - Source files have Apache header
> - No unexpected binary files (checked expected binary files)
> - Can compile from source
>
> Checked the Driver jar binary artifact:
> - Signatures and hashes correct
> — LICENSE and NOTICE look ok
>
>
> On 24 Feb 2022, at 23:52, Ian Maxon wrote:
>
> > Hi everyone,
> >
> > Please verify and vote on the first release of the AsterixDB JDBC
> > connector.
> >
> > This release candidate fixes some issues with license files that were
> > present
> > in RC0 as well as simplifying the list of artifacts.
> >
> > The change that produced this release is up on Gerrit:
> >
> > https://asterix-gerrit.ics.uci.edu/c/asterixdb-clients/+/15444
> >
> > The release artifacts are as follows:
> >
> > JDBC Driver Source
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-jdbc-0.9.7.1-source-release.zip
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-jdbc-0.9.7.1-source-release.zip.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-jdbc-0.9.7.1-source-release.zip.sha512
> >
> > SHA512:
> >
> b851a73bfd3da0cd7c17d3c4247a924ecd85c5138bab3f36f813109db38918f3f2993a75f57ae4b7d532719b01d7cb3c9fcfdd0de0dbd47452e9394d01ff5441⏎
> >
> > JDBC Driver Distributable Jar
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1-dist.jar
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1-dist.jar.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1-dist.jar.sha512
> >
> > SHA512:
> >
> c7184054958dffb280de51fc472c4f94e218b2d67435fae73a10663a46d8fc06c607da520facbd4c8ba159b0e7339a23e64079361bff2f9e4ef3c76a2a900df1
> >
> > Tableau Connector (TACO file)
> > https://dist.apache.org/repos/dist/dev/asterixdb/asterixdb_jdbc.taco
> > https://dist.apache.org/repos/dist/dev/asterixdb/asterixdb_jdbc.taco.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterixdb_jdbc.taco.sha512
> >
> > SHA512:
> >
> 2538bd8e5266c280a0ff3de7bdb7e262d7963207181a8bfba3bb852c1c9c3820ea3495e68e4e31038e18542f01891ae2734007e29aea74219e7c1896d0807321
> >
> >
> > The KEYS file containing the PGP keys used to sign the release can be
> > found at
> >
> > https://dist.apache.org/repos/dist/release/asterixdb/KEYS
> >
> > RAT was executed as part of Maven via the RAT maven plugin, but
> > excludes files that are:
> >
> > - data for tests
> > - procedurally generated,
> > - or source files which come without a header mentioning their
> > license,
> >   but have an explicit reference in the LICENSE file.
> >
> >
> > The vote is open for 72 hours, or until the necessary number of votes
> > (3 +1) has been reached.
> >
> > Please vote
> > [ ] +1 release these packages as Apache AsterixDB JDBC Driver 0.9.7.1
> > [ ] 0 No strong feeling either way
> > [ ] -1 do not release one or both packages because ...
> >
> > Thanks!
>

Re: [VOTE] Release Apache AsterixDB JDBC Connector 0.9.7.1 (RC0)

[Taewoo Kim]

+1
 Verified the driver with DbVisualizer

Best,
Taewoo


On Fri, Feb 11, 2022 at 11:39 AM Taewoo Kim  wrote:

> Thanks Dmitry. :-)
>
> Best,
> Taewoo
>
>
> On Fri, Feb 11, 2022 at 10:43 AM Dmitry Lychagin
>  wrote:
>
>> The installation instructions are now available here:
>> https://cwiki.apache.org/confluence/display/ASTERIXDB/JDBC+Driver
>>
>> Thanks,
>> -- Dmitry
>>
>>
>> On 2/11/22, 9:02 AM, "Dmitry Lychagin" 
>> wrote:
>>
>> +1. Verified driver with DbVisualizer and DBeaver.
>>
>> Thanks,
>> -- Dmitry
>>
>>
>> On 2/10/22, 11:31 AM, "Ian Maxon"  wrote:
>>
>>  EXTERNAL EMAIL:  Use caution when opening attachments or
>> clicking on links
>>
>>
>>
>>
>>
>> A small amendment to the list of artifacts:
>>
>> JDBC Driver Distributable Jar
>>
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1-dist.jar
>>
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1-dist.jar.asc
>>
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1-dist.jar.sha512
>>
>> SHA512:
>> cf20d630dc09665a2d2902821f844228c53e869632f49b356741adca47f98b639c29d4a9fef265272ba2fce670c428f63ab64b3692efc3b502ce89286763c097
>>
>>
>> On Thu, Feb 10, 2022 at 11:21 AM Ian Maxon 
>> wrote:
>> >
>> > Hi everyone,
>> >
>> > Please verify and vote on the first release of the AsterixDB
>> JDBC connector.
>> > The change that produced this release is up on Gerrit:
>> >
>> > https://asterix-gerrit.ics.uci.edu/c/asterixdb-clients/+/15224
>> >
>> > The release artifacts are as follows:
>> >
>> > JDBC Driver Source
>> >
>> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-jdbc-0.9.7.1-source-release.zip
>> >
>> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-jdbc-0.9.7.1-source-release.zip.asc
>> >
>> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-jdbc-0.9.7.1-source-release.zip.sha512
>> >
>> > SHA512:
>> 5e88330b4cbcd055120ea74eb66a3f46a67bd84b02c085ad940bf67f22b4714f19c662dd177d324b81868d6717c30aca7e06df30f66697dd59716aae8fc116b6
>> >
>> >
>> > JDBC Driver Jar
>> >
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar
>> >
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar.asc
>> >
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar.sha512
>> >
>> > SHA512:
>> >
>> >
>> bfb79e085bf17fd9fd8f36366f696eb2c1d0aa396763320e595d8b12e3a531433695b2e98852c5d130b9e41243334053c4deb53144c6efdaf70c0a02441983c7
>> >
>> >
>> > JDBC Core Jar
>> >
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar
>> >
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar.asc
>> >
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar.sha512
>> >
>> > SHA512:
>> 989f4e048e29fc46b7414a7854f2c88eb0091e66fdefa07a0b23263d24a6c1c33fa324b386c5d415f70d194b4be51ede352e76176c549f66c344385c058c90e5
>> >
>> > Tableau Connector (TACO file)
>> >
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterixdb_jdbc.taco
>> >
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterixdb_jdbc.taco.asc
>> >
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterixdb_jdbc.taco.sha512
>> >
>> > SHA512:
>> >
>> b6cba1c726526f54f8fdfc1ab5e86b03620230ebaf9127af6a006acbfda5115a4d8ba89221581177a77892a8c25157ad3b61d21447b94133604d224d518efa87
>> >
>> > The KEYS file containing the PGP keys used to sign the release
>> can be
>> > found at
>> >
>> > https://dist.apache.org/repos/dist/release/asterixdb/KEYS
>> >
>> > RAT was executed as part of Maven via the RAT maven plugin, but
>> > excludes files that are:
>> >
>> > - data for tests
>> > - procedurally generated,
>> > - or source files which come without a header mentioning their
>> license,
>> >   but have an explicit reference in the LICENSE file.
>> >
>> >
>> > The vote is open for 72 hours, or until the necessary number of
>> votes
>> > (3 +1) has been reached.
>> >
>> > Please vote
>> > [ ] +1 release these packages as Apache AsterixDB JDBC Driver
>> 0.9.7.1
>> > [ ] 0 No strong feeling either way
>> > [ ] -1 do not release one or both packages because ...
>> >
>> > Thanks!
>>
>>
>>

Re: [VOTE] Release Apache AsterixDB JDBC Connector 0.9.7.1 (RC0)

[Taewoo Kim]

Thanks Dmitry. :-)

Best,
Taewoo


On Fri, Feb 11, 2022 at 10:43 AM Dmitry Lychagin
 wrote:

> The installation instructions are now available here:
> https://cwiki.apache.org/confluence/display/ASTERIXDB/JDBC+Driver
>
> Thanks,
> -- Dmitry
>
>
> On 2/11/22, 9:02 AM, "Dmitry Lychagin" 
> wrote:
>
> +1. Verified driver with DbVisualizer and DBeaver.
>
> Thanks,
> -- Dmitry
>
>
> On 2/10/22, 11:31 AM, "Ian Maxon"  wrote:
>
>  EXTERNAL EMAIL:  Use caution when opening attachments or clicking
> on links
>
>
>
>
>
> A small amendment to the list of artifacts:
>
> JDBC Driver Distributable Jar
>
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1-dist.jar
>
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1-dist.jar.asc
>
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1-dist.jar.sha512
>
> SHA512:
> cf20d630dc09665a2d2902821f844228c53e869632f49b356741adca47f98b639c29d4a9fef265272ba2fce670c428f63ab64b3692efc3b502ce89286763c097
>
>
> On Thu, Feb 10, 2022 at 11:21 AM Ian Maxon  wrote:
> >
> > Hi everyone,
> >
> > Please verify and vote on the first release of the AsterixDB
> JDBC connector.
> > The change that produced this release is up on Gerrit:
> >
> > https://asterix-gerrit.ics.uci.edu/c/asterixdb-clients/+/15224
> >
> > The release artifacts are as follows:
> >
> > JDBC Driver Source
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-jdbc-0.9.7.1-source-release.zip
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-jdbc-0.9.7.1-source-release.zip.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-jdbc-0.9.7.1-source-release.zip.sha512
> >
> > SHA512:
> 5e88330b4cbcd055120ea74eb66a3f46a67bd84b02c085ad940bf67f22b4714f19c662dd177d324b81868d6717c30aca7e06df30f66697dd59716aae8fc116b6
> >
> >
> > JDBC Driver Jar
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar.sha512
> >
> > SHA512:
> >
> >
> bfb79e085bf17fd9fd8f36366f696eb2c1d0aa396763320e595d8b12e3a531433695b2e98852c5d130b9e41243334053c4deb53144c6efdaf70c0a02441983c7
> >
> >
> > JDBC Core Jar
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar.sha512
> >
> > SHA512:
> 989f4e048e29fc46b7414a7854f2c88eb0091e66fdefa07a0b23263d24a6c1c33fa324b386c5d415f70d194b4be51ede352e76176c549f66c344385c058c90e5
> >
> > Tableau Connector (TACO file)
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterixdb_jdbc.taco
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterixdb_jdbc.taco.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterixdb_jdbc.taco.sha512
> >
> > SHA512:
> >
> b6cba1c726526f54f8fdfc1ab5e86b03620230ebaf9127af6a006acbfda5115a4d8ba89221581177a77892a8c25157ad3b61d21447b94133604d224d518efa87
> >
> > The KEYS file containing the PGP keys used to sign the release
> can be
> > found at
> >
> > https://dist.apache.org/repos/dist/release/asterixdb/KEYS
> >
> > RAT was executed as part of Maven via the RAT maven plugin, but
> > excludes files that are:
> >
> > - data for tests
> > - procedurally generated,
> > - or source files which come without a header mentioning their
> license,
> >   but have an explicit reference in the LICENSE file.
> >
> >
> > The vote is open for 72 hours, or until the necessary number of
> votes
> > (3 +1) has been reached.
> >
> > Please vote
> > [ ] +1 release these packages as Apache AsterixDB JDBC Driver
> 0.9.7.1
> > [ ] 0 No strong feeling either way
> > [ ] -1 do not release one or both packages because ...
> >
> > Thanks!
>
>
>

Re: [VOTE] Release Apache AsterixDB JDBC Connector 0.9.7.1 (RC0)

[Taewoo Kim]

@Ian Maxon : Is there an instruction that I can follow to
verify the release? Mike and Dmitry mentioned using DbVisualizer and
DBeaver.

Best,
Taewoo


On Fri, Feb 11, 2022 at 9:02 AM Dmitry Lychagin
 wrote:

> +1. Verified driver with DbVisualizer and DBeaver.
>
> Thanks,
> -- Dmitry
>
>
> On 2/10/22, 11:31 AM, "Ian Maxon"  wrote:
>
>  EXTERNAL EMAIL:  Use caution when opening attachments or clicking on
> links
>
>
>
>
>
> A small amendment to the list of artifacts:
>
> JDBC Driver Distributable Jar
>
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1-dist.jar
>
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1-dist.jar.asc
>
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1-dist.jar.sha512
>
> SHA512:
> cf20d630dc09665a2d2902821f844228c53e869632f49b356741adca47f98b639c29d4a9fef265272ba2fce670c428f63ab64b3692efc3b502ce89286763c097
>
>
> On Thu, Feb 10, 2022 at 11:21 AM Ian Maxon  wrote:
> >
> > Hi everyone,
> >
> > Please verify and vote on the first release of the AsterixDB JDBC
> connector.
> > The change that produced this release is up on Gerrit:
> >
> > https://asterix-gerrit.ics.uci.edu/c/asterixdb-clients/+/15224
> >
> > The release artifacts are as follows:
> >
> > JDBC Driver Source
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-jdbc-0.9.7.1-source-release.zip
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-jdbc-0.9.7.1-source-release.zip.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-jdbc-0.9.7.1-source-release.zip.sha512
> >
> > SHA512:
> 5e88330b4cbcd055120ea74eb66a3f46a67bd84b02c085ad940bf67f22b4714f19c662dd177d324b81868d6717c30aca7e06df30f66697dd59716aae8fc116b6
> >
> >
> > JDBC Driver Jar
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar.sha512
> >
> > SHA512:
> >
> >
> bfb79e085bf17fd9fd8f36366f696eb2c1d0aa396763320e595d8b12e3a531433695b2e98852c5d130b9e41243334053c4deb53144c6efdaf70c0a02441983c7
> >
> >
> > JDBC Core Jar
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-jdbc-driver-0.9.7.1.jar.sha512
> >
> > SHA512:
> 989f4e048e29fc46b7414a7854f2c88eb0091e66fdefa07a0b23263d24a6c1c33fa324b386c5d415f70d194b4be51ede352e76176c549f66c344385c058c90e5
> >
> > Tableau Connector (TACO file)
> > https://dist.apache.org/repos/dist/dev/asterixdb/asterixdb_jdbc.taco
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterixdb_jdbc.taco.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterixdb_jdbc.taco.sha512
> >
> > SHA512:
> >
> b6cba1c726526f54f8fdfc1ab5e86b03620230ebaf9127af6a006acbfda5115a4d8ba89221581177a77892a8c25157ad3b61d21447b94133604d224d518efa87
> >
> > The KEYS file containing the PGP keys used to sign the release can be
> > found at
> >
> > https://dist.apache.org/repos/dist/release/asterixdb/KEYS
> >
> > RAT was executed as part of Maven via the RAT maven plugin, but
> > excludes files that are:
> >
> > - data for tests
> > - procedurally generated,
> > - or source files which come without a header mentioning their
> license,
> >   but have an explicit reference in the LICENSE file.
> >
> >
> > The vote is open for 72 hours, or until the necessary number of votes
> > (3 +1) has been reached.
> >
> > Please vote
> > [ ] +1 release these packages as Apache AsterixDB JDBC Driver 0.9.7.1
> > [ ] 0 No strong feeling either way
> > [ ] -1 do not release one or both packages because ...
> >
> > Thanks!
>
>

Re: [VOTE] Release Apache AsterixDB 0.9.7.1 and Hyracks 0.9.7.1 (RC0)

[Taewoo Kim]

+1

- Verified hashes
- Built from source without an issue
- Ran a sample cluster
- Tried a few sections of the AsterixDB 101 Primer

Best,
Taewoo


On Sat, Dec 11, 2021 at 7:16 PM Wail Alkowaileet  wrote:

> +1 (non-binding)
>
> Verified:
> - Hashes
> - Signatures
> - Build from source
> - Ran a sample cluster:
>
> Ingest and query a "sensors" records
>
>
> On Sat, Dec 11, 2021 at 6:14 PM Till Westmann  wrote:
>
> >
> > +1
> >
> > Checked the Hyracks and AsterixDB source artifacts:
> > - Signatures and hashes correct
> > — LICENSE and NOTICE look ok
> > - Source files have Apache header
> > - No unexpected binary files (checked expected binary files)
> > - Can compile from source
> >
> > Checked the AsterixDB Server binary artifact:
> > - Signatures and hashes correct
> > — LICENSE and NOTICE look ok
> >
> > Till
> >
> > On 11 Dec 2021, at 16:20, Michael Blow wrote:
> >
> > > The log4j-1.2-api is still at an older version (2.13.1), given that
> > > our
> > > tests are passing there likely isn't any compatibility
> > > issue, but if we end up respinning for some other reason, we might
> > > want to
> > > consider advancing log4j-1.2-api to 2.15.0.
> > >
> > > Verified:
> > >
> > >- source builds
> > >- signatures
> > >- checksums
> > >
> > >
> > > [X] +1 release these packages as Apache AsterixDB 0.9.7.1 and
> > > Apache Hyracks 0.3.7.1
> > > [ ] 0 No strong feeling either way
> > > [ ] -1 do not release one or both packages because ...
> > >
> > > On Sat, Dec 11, 2021 at 6:26 PM Ian Maxon  wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> Please verify and vote on the latest release of Apache AsterixDB.
> > >> This
> > >> release is purely a maintenance release to 0.9.7 and contains minimal
> > >> changes.
> > >>
> > >> The change that produced this release is up on Gerrit:
> > >>
> > >> https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/14504
> > >>
> > >> The release artifacts are as follows:
> > >>
> > >> AsterixDB Source
> > >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.7.1-source-release.zip
> > >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.7.1-source-release.zip.asc
> > >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.7.1-source-release.zip.sha256
> > >>
> > >> SHA256:
> > >> a698f6246347592263858af349de206d2636984d040d27bb82770a2a5c6bc0b4
> > >>
> > >>
> > >> Hyracks Source
> > >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.7.1-source-release.zip
> > >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.7.1-source-release.zip.asc
> > >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.7.1-source-release.zip.sha256
> > >>
> > >> SHA256:
> > >> d3b4652aabfee134ea28a92c6fbe5e5ea9091aa623c5ec68d91b0eb5ece755e5
> > >>
> > >>
> > >> AsterixDB NCService Installer:
> > >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.7.1-binary-assembly.zip
> > >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.7.1-binary-assembly.zip.asc
> > >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.7.1-binary-assembly.zip.sha256
> > >>
> > >> SHA256:
> > >> d78c6725eed3386a63ca801e4caa5595bdf2b91c8cf993490e3e429ffc42c163
> > >>
> > >>
> > >> The KEYS file containing the PGP keys used to sign the release can be
> > >> found at
> > >>
> > >> https://dist.apache.org/repos/dist/release/asterixdb/KEYS
> > >>
> > >> RAT was executed as part of Maven via the RAT maven plugin, but
> > >> excludes files that are:
> > >>
> > >> - data for tests
> > >> - procedurally generated,
> > >> - or source files which come without a header mentioning their
> > >> license,
> > >>   but have an explicit reference in the LICENSE file.
> > >>
> > >>
> > >> The vote is open for 72 hours, or until the necessary number of votes
> > >> (3 +1) has been reached.
> > >>
> > >> Please vote
> > >> [ ] +1 release these packages as Apache AsterixDB 0.9.7.1 and
> > >> Apache Hyracks 0.3.7.1
> > >> [ ] 0 No strong feeling either way
> > >> [ ] -1 do not release one or both packages because ...
> > >>
> > >> Thanks!
> > >>
> >
>
>
> --
>
> *Regards,*
> Wail Alkowaileet
>

Re: [VOTE] Accept donation of AsterixDB JDBC Driver

[Taewoo Kim]

+1

Best,
Taewoo


On Mon, Aug 23, 2021 at 4:36 PM Ian Maxon  wrote:

> +1
>
> On Mon, Aug 23, 2021 at 8:37 AM Heri Ramampiaro  wrote:
> >
> > +1!
> > -heri
> >
> > Sent from my iPhone
> >
> > > On Aug 23, 2021, at 16:48, Mike Carey  wrote:
> > >
> > > +1  (!!)
> > >
> > >> On 8/22/21 11:51 PM, Till Westmann wrote:
> > >> Hi,
> > >>
> > >> Couchbase would like to donate a JDBC driver for AsterixDB to the
> Apache
> > >> Software foundation.
> > >>
> > >> [X] +1 accept the donation and add the driver to the AsterixDB code
> base
> > >> [ ] +0 no opinion
> > >> [ ] -1 do not accept the donation because...
> > >>
> > >> The vote will be open for 7 days.
> > >>
> > >> Please vote,
> > >> Till
>

Re: [VOTE] Release Apache AsterixDB 0.9.6 and Hyracks 0.3.6 (RC0)

[Taewoo Kim]

[v] +1 release these packages as Apache AsterixDB 0.9.6 and Apache Hyracks
0.3.6

- Verified signatures and hashes
- Verified that source built correctly
- Smoke test using binary

On Mon, Oct 12, 2020 at 4:26 PM Mike Carey  wrote:

> REMINDER:  Folks should please verify and vote! (That way our Fearless
> Leader can perhaps include the outcome in the report that's due in 2
> days. :-))
>
> On 10/4/20 11:51 PM, Ian Maxon wrote:
> > Hi everyone,
> >
> > Please verify and vote on the latest release of Apache AsterixDB
> >
> > The change that produced this release is up for review on Gerrit:
> >
> > https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/8225
> >
> > The release artifacts are as follows:
> >
> > AsterixDB Source
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.6-source-release.zip
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.6-source-release.zip.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.6-source-release.zip.sha256
> >
> > SHA256:98443ff5a8bb5b25b38fa81b1a4fb43aeb1522742164462909062d1cdf7cd88d
> >
> > Hyracks Source
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.6-source-release.zip
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.6-source-release.zip.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.6-source-release.zip.sha256
> >
> > SHA256:40546121dab77f49f29d74f9ae8138a0dc94daf8b6e4f6ed42e070d1981efdcb
> >
> > AsterixDB NCService Installer:
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.6-binary-assembly.zip
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.6-binary-assembly.zip.asc
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.6-binary-assembly.zip.sha256
> >
> > SHA256:6dd82a03cfa01891589c8e571892aa45b67548aaf2d1a03e5de179f1a38da5f5
> >
> > The KEYS file containing the PGP keys used to sign the release can be
> > found at
> >
> > https://dist.apache.org/repos/dist/release/asterixdb/KEYS
> >
> > RAT was executed as part of Maven via the RAT maven plugin, but
> > excludes files that are:
> >
> > - data for tests
> > - procedurally generated,
> > - or source files which come without a header mentioning their license,
> >but have an explicit reference in the LICENSE file.
> >
> >
> > The vote is open for 72 hours, or until the necessary number of votes
> > (3 +1) has been reached.
> >
> > Please vote
> > [ ] +1 release these packages as Apache AsterixDB 0.9.6 and
> > Apache Hyracks 0.3.6
> > [ ] 0 No strong feeling either way
> > [ ] -1 do not release one or both packages because ...
> >
> > Thanks!
>

Re: Substitute SQL++ sub-query

[Taewoo Kim]

Agreed with Mike regarding a native approach to spatial joins. :-) I just
tried to point to a location for FuzzyJoinRule.

Best,
Taewoo


On Mon, Aug 24, 2020 at 12:10 PM Mike Carey  wrote:

> Tin,
>
> 1. There is no support for SQL++ in templates. Just AQL.  And AQL is now
> deprecated, in a mode where it is only supported to keep that one fuzzy
> join rule alive via AQL+.  The most recent public release of AsterixDB
> is the last one where AQL+ will work - and then AQL will completely
> disappear from the code base (as will the scripting framework) going
> forward, as it is totally unsupported.  (Its inventor left with his PhD
> in 2011.  Taewoo Kim, below, did a really nice refactoring revision of
> the framework more recently, to remove lots of replicated code between
> AQL and AQL+, but he is also no longer focused on AsterixDB work - his
> post-PhD day job seems to be a distraction. :-))
>
> 2. More information about the scripting framework and its use for the
> fuzzy join rule is in Chapter 5 of the aforementioned PhD student's
> thesis:
> http://asterix.ics.uci.edu//thesis/Rares_Vernica_PhD_thesis_2011.pdf.
>
> 3. I suspect it would be significantly messy work to create a SQL++
> alternative to AQL as the basis for the template framework...  If you
> want to explore a template-based approach, I'd stick with AQL+ and just
> be aware that it's kind of a "dead end" approach long-term in the
> AsterixDB code base - but could be explored as a research branch using
> 0.9.5 as the branching point.
>
> I think a native approach to spatial joins would be better than a
> template-based approach - along the lines of what Preston Carmon et al
> are doing for interval joins up at Walla Walla U in Washington state.
>
> Cheers,
>
> Mike
>
> On 8/24/20 11:11 AM, Taewoo Kim wrote:
> > Hi Tin,
> >
> > Here is a paper that explains FuzzyJoinRule in general. Hope this helps.
> >
> >
> https://scholar.google.com/scholar?hl=en_sdt=0%2C5=Similarity+query+support+in+big+data+management+systems=
> >
> > 1. What is the syntax to make placeholders for input operators and
> > variables in a SQL++ template? In the FuzzyJoinRule
> > <
> >
> https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
> > implementation
> > <
> >
> https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
> >> ,
> > I saw that it is using ##LEFT_0 for left input operator and $$LEFT_0 for
> > left input variable. Is it similar in SQL++?
> > You need to implement an extended version of SQL++ (maybe SQL+++) that
> > recognizes ##LEFT_0, etc. FuzzyJoinRule uses AQL+.
> >
> https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-algebra/src/main/javacc/AQLPlusExtension.jj
> >
> https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java
> >
> > 2. Does the query template for substitution must be a (full) compilable
> > query? Or it can be just a part of a full query?
> > It has to be a complete query that is executable in SQL+++ (if you create
> > one).
> >
> > 3. Can the substitute query contain UDF functions?
> > Whatever support that SQL++ has, SQL+++ inherits since SQL++ is an
> extended
> > version.
> >
> > Best,
> > Taewoo
> >
> >
> > On Mon, Aug 24, 2020 at 10:55 AM Tin Vu  wrote:
> >
> >> Hi all,
> >>
> >> I'm working on a project which aims to improve performance of spatial
> join
> >> query in AsterixDB.
> >>
> >> The problem can be described as follows:
> >>
> >> Assume that we have 2 dataset ParkSet(id, geom) and LakeSet(id, geom)
> with
> >> geom is a spatial data type (point, rectangle, polygon). This is a join
> >> query to list all intersected pairs of these two datasets:
> >> ```
> >>
> >> *SELECT COUNT(*) FROM ParkSet AS ps, LakeSet AS lsWHERE
> >> spatial_intersect(ps.geom, ls.geom);*
> >> *```*
> >>
> >> I rewrote this query in a more complicated SQL++ query(*) but has a
> better
> >> performance.
> >>
> >> I took a look at FuzzyJoinRule
> >> <
> >>
> https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
> >> implementation
> >> <
> >>
> https://github.com/apache/asterixdb/blob/master/

Re: Substitute SQL++ sub-query

[Taewoo Kim]

Hi Tin,

Here is a paper that explains FuzzyJoinRule in general. Hope this helps.

https://scholar.google.com/scholar?hl=en_sdt=0%2C5=Similarity+query+support+in+big+data+management+systems=

1. What is the syntax to make placeholders for input operators and
variables in a SQL++ template? In the FuzzyJoinRule
<
https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
>
implementation
<
https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
>,
I saw that it is using ##LEFT_0 for left input operator and $$LEFT_0 for
left input variable. Is it similar in SQL++?
You need to implement an extended version of SQL++ (maybe SQL+++) that
recognizes ##LEFT_0, etc. FuzzyJoinRule uses AQL+.
https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-algebra/src/main/javacc/AQLPlusExtension.jj
https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/AqlPlusExpressionToPlanTranslator.java

2. Does the query template for substitution must be a (full) compilable
query? Or it can be just a part of a full query?
It has to be a complete query that is executable in SQL+++ (if you create
one).

3. Can the substitute query contain UDF functions?
Whatever support that SQL++ has, SQL+++ inherits since SQL++ is an extended
version.

Best,
Taewoo


On Mon, Aug 24, 2020 at 10:55 AM Tin Vu  wrote:

> Hi all,
>
> I'm working on a project which aims to improve performance of spatial join
> query in AsterixDB.
>
> The problem can be described as follows:
>
> Assume that we have 2 dataset ParkSet(id, geom) and LakeSet(id, geom) with
> geom is a spatial data type (point, rectangle, polygon). This is a join
> query to list all intersected pairs of these two datasets:
> ```
>
> *SELECT COUNT(*) FROM ParkSet AS ps, LakeSet AS lsWHERE
> spatial_intersect(ps.geom, ls.geom);*
> *```*
>
> I rewrote this query in a more complicated SQL++ query(*) but has a better
> performance.
>
> I took a look at FuzzyJoinRule
> <
> https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
> >
> implementation
> <
> https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
> >
> (a
> rule for AQL+) in AsterixDB and I'm trying to make a similar implementation
> called SpatalJoinRule
> <
> https://github.com/tinvukhac/asterixdb/blob/cartilage/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/SpatialJoinRule.java
> >
> (a
> rule for SQL++). Now I'm stuck with some questions and I would
> really appreciate if you can help me to answer them:
>
> 1. What is the syntax to make placeholders for input operators and
> variables in a SQL++ template? In the FuzzyJoinRule
> <
> https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
> >
> implementation
> <
> https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/FuzzyJoinRule.java
> >,
> I saw that it is using ##LEFT_0 for left input operator and $$LEFT_0 for
> left input variable. Is it similar in SQL++?
>
> 2. Does the query template for substitution must be a (full) compilable
> query? Or it can be just a part of a full query?
>
> 3. Can the substitute query contain UDF functions?
>
> Thanks,
>
> Tin
>
> (*) Rewitten query: in short, it partitioned the datasets by a grid (red
> statement) then we compute the intersection pairs for each cell of the
> grid.
> - 'mytileids' UDF function returns the corresponding cell ID of a spatial
> object.
> - 'referencepointtileid' UDF function is used to ignore the duplications in
> the final result.
> - (-180.0,83.0,180.0,90.0) is space MBR.
> - (100,100) is grid size (rows x columns).
>
> ```
>
>
>
>
>
> *SELECT COUNT(*) FROM (SELECT parksPartitioned.tile as tile1,
> lakesPartitioned.tile as tile2,
> test#referencepointtileid(parksPartitioned.geom,
> lakesPartitioned.geom,-180.0,83.0,180.0,90.0,100,100) as
> ref_tileFROM(SELECT tile, p.ps.id , p.ps.geom FROM
> (SELECT
> test#mytileids(ps.geom,-180.0,-83.0,180.0,90.0,100,100) AS tileids, ps FROM
> ParkSet as ps) AS p UNNEST p.tileids AS tile) AS parksPartitioned,(SELECT
> tile, p.ps.id , p.ps.geom FROM (SELECT
> test#mytileids(ps.geom,-180.0,-83.0,180.0,90.0,100,100) AS tileids, ps FROM
> LakeSet as ps) AS p UNNEST p.tileids AS tile) AS lakesPartitionedWHERE
> parksPartitioned.tile = lakesPartitioned.tileAND
> spatial_intersect(parksPartitioned.geom, lakesPartitioned.geom)) AS result
> WHERE result.tile1 = result.ref_tile;*
> *```*
>

Re: [VOTE] Release Apache AsterixDB 0.9.5 and Hyracks 0.3.5 (RC4)

[Taewoo Kim]

[ X] +1 release these packages as Apache AsterixDB 0.9.5 and Apache Hyracks
0.3.5

Followed the directions on the following page.
https://cwiki.apache.org/confluence/display/ASTERIXDB/Release+Verification

[v] Verify signatures and hashes
[v] Verify that source builds correctly
[v] Smoke test

Best,
Taewoo


On Wed, Jul 8, 2020 at 6:04 AM Michael Blow  wrote:

> [ X ] +1 release these packages as Apache AsterixDB 0.9.5 and
> Apache Hyracks 0.3.5
>
> Checked:
> - keys, signatures on all packages
> - SHAs
> - sanity check of LICENSE / NOTICEs
> - functional build of source packages
> - all versions advanced from SNAPSHOT
>
>
> On Mon, Jul 6, 2020 at 6:51 PM Ian Maxon  wrote:
>
> > Hi everyone,
> >
> > Please verify and vote on the latest release of Apache AsterixDB
> >
> > The change that produced this release is up for review on Gerrit:
> >
> > https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/7124
> >
> > The release artifacts are as follows:
> >
> > AsterixDB Source
> >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.5-source-release.zip
> >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.5-source-release.zip.asc
> >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.5-source-release.zip.sha256
> >
> > SHA256:09affe9ce5aa75add6c5a75c51505e619f85cb7a87eb3b9d977ac472d5387bd1
> >
> > Hyracks Source
> >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.5-source-release.zip
> >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.5-source-release.zip.asc
> >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.5-source-release.zip.sha256
> >
> > SHA256:577d2b3da91ebfa37c113bae18561dcbfae0bdd526edee604b747f6044f4a03b
> >
> > AsterixDB NCService Installer:
> >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.5-binary-assembly.zip
> >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.5-binary-assembly.zip.asc
> >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.5-binary-assembly.zip.sha256
> >
> > SHA256:6854e71fc78f9cfb68b0dc3c61edb5f5c94b09b41f4a8deaf4c2fc9d804abcac
> >
> > The KEYS file containing the PGP keys used to sign the release can be
> > found at
> >
> > https://dist.apache.org/repos/dist/release/asterixdb/KEYS
> >
> > RAT was executed as part of Maven via the RAT maven plugin, but
> > excludes files that are:
> >
> > - data for tests
> > - procedurally generated,
> > - or source files which come without a header mentioning their license,
> >   but have an explicit reference in the LICENSE file.
> >
> >
> > The vote is open for 72 hours, or until the necessary number of votes
> > (3 +1) has been reached.
> >
> > Please vote
> > [ ] +1 release these packages as Apache AsterixDB 0.9.5 and
> > Apache Hyracks 0.3.5
> > [ ] 0 No strong feeling either way
> > [ ] -1 do not release one or both packages because ...
> >
> > Thanks!
> >
>

Re: Subplan: ExtractCommonExporessions

[Taewoo Kim]

I see. I think this is related to isomorphism. As the comment says, we need
to find a way to resolve "complication of figuring out whether the firstOp
in an equivalence class is within a subplan".

Best,
Taewoo


On Sat, Oct 19, 2019 at 12:40 PM Wail Alkowaileet 
wrote:

> Hi Taewoo,
>
> I might be got confused by the comment. But still did not get why it
> is commented?
> Sub-plans are missing a few optimizations. For example after
> uncommenting this code and fixing InlineSingleReferenceVariableRule
> for sub-plans we get:
>
> Before:
>
> 
> subplan {
>   aggregate [$$10] <- [listify($$9)]
>   -- AGGREGATE  |LOCAL|
> assign [$$9] <- [eq($$22, "Tom")]
> -- ASSIGN  |LOCAL|
>   assign [$$22] <-
> [$$30.getField("firstName")]
>   -- ASSIGN  |LOCAL|
> assign [$$30] <-
> [$$29.getField("name")]
> -- ASSIGN  |LOCAL|
>   assign [$$29] <-
> [$$x.getField("names")]
>   -- ASSIGN  |LOCAL|
> select (eq($$21, "1"))
> -- STREAM_SELECT  |LOCAL|
>   nested tuple source
>   -- NESTED_TUPLE_SOURCE  |LOCAL|
>}
> -- SUBPLAN  |PARTITIONED|
>   project ([$$x, $$21, $$25])
>   -- STREAM_PROJECT  |PARTITIONED|
> assign [$$21, $$25] <-
> [$$28.getField("count"), $$28.getField("name")]
> -- ASSIGN  |PARTITIONED|
>   assign [$$28] <- [$$x.getField("names")]
>   -- ASSIGN  |PARTITIONED|
> 
>
> After:
>
> 
> subplan {
>   aggregate [$$10] <- [listify($$9)]
>   -- AGGREGATE  |LOCAL|
> assign [$$9] <-
> [eq($$26.getField("firstName"), "Tom")]
> -- ASSIGN  |LOCAL|
>   select (eq($$22, "1"))
>   -- STREAM_SELECT  |LOCAL|
> nested tuple source
> -- NESTED_TUPLE_SOURCE  |LOCAL|
>}
> -- SUBPLAN  |PARTITIONED|
>   project ([$$x, $$22, $$26])
>   -- STREAM_PROJECT  |PARTITIONED|
>     assign [$$26, $$22] <- [$$20.getField("name"),
> $$20.getField("count")]
> -- ASSIGN  |PARTITIONED|
>   assign [$$20] <- [$$x.getField("names")]
>   -- ASSIGN  |PARTITIONED|
> 
> On Sat,
> Oct 19, 2019 at 11:15 AM Taewoo Kim wangs...@gmail.com
> wrote:Hi Wail,
> 
> I think that's what the comment implies (I could not produce an
> expression
> where it used in a sub-plan and not visible to upper operators?). If
> you
> want to make it happen, a workaround might be introducing a project
> operator within the subplan of the subplan operator? Actually, if a
> variable is not used, isn't project operator supposed to remove them
> automatically by IntroduceProjectsRule?
> 
> Best,
> Taewoo
> 
> 
> On Sat, Oct 19, 2019 at 10:05 AM Wail Alkowaileet  href="mailto:wael@gmail.com;
> target="_blank">wael@gmail.com
> wrote:
> 
>  Hi Dev,
> 
>  I'm not sure about the commented code in [1]. I could not produce
> an
>  expression where it used in a sub-plan and not visible to upper
> operators?
>  Also, all SQL++ runtime integration tests seem to work just fine.
> 
> 
>  [1] ExtractCommonExpressionsRule.java#L192
>  
>  https://github.com/apache/asterixdb/blob/f2c18aa9646238ab2487ce3a964edfe3e61dd6e1/hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/ExtractCommonExpressionsRule.java#L192
> "
> rel="noreferrer"
> target="_blank">https://github.com/apache/aste
> rixdb/blob/f2c18aa9646238ab2487ce3a964edfe3e61dd6e1/hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/ExtractCommonExpressionsRule.java#L192
>  
> 
>  --
> 
>  *Regards,*
>  Wail Alkowaileet
> 
> --  dir="ltr" class="gmail_signature">Regards,Wail
> Alkowaileet
>

Re: Subplan: ExtractCommonExporessions

[Taewoo Kim]

Hi Wail,

I think that's what the comment implies (I could not produce an expression
where it used in a sub-plan and not visible to upper operators?). If you
want to make it happen, a workaround might be introducing a project
operator within the subplan of the subplan operator? Actually, if a
variable is not used, isn't project operator supposed to remove them
automatically by IntroduceProjectsRule?

Best,
Taewoo


On Sat, Oct 19, 2019 at 10:05 AM Wail Alkowaileet 
wrote:

> Hi Dev,
>
> I'm not sure about the commented code in [1]. I could not produce an
> expression where it used in a sub-plan and not visible to upper operators?
> Also, all SQL++ runtime integration tests seem to work just fine.
>
>
> [1] ExtractCommonExpressionsRule.java#L192
> <
> https://github.com/apache/asterixdb/blob/f2c18aa9646238ab2487ce3a964edfe3e61dd6e1/hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/ExtractCommonExpressionsRule.java#L192
> >
>
> --
>
> *Regards,*
> Wail Alkowaileet
>

Re: [VOTE] Release Apache AsterixDB 0.9.5 and Hyracks 0.3.5 (RC2)

[Taewoo Kim]

-1

- Checked the SHA256 of each zip file: SHA256 of the binary file (
https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.5-binary-assembly.zip)
doesn't match. Could you check?
Expected: d80ff63ea5796022f6ce58676d3954438ce703a1da06c5f382b8ace3d4719445
Actual:   d395bf454e5b53e9f0f6e826e790b0d3a1369308ed6703d76436d0e13cd54c59

- Builds were successful without any error.

- Smoke test using the binary was successful.

Best,
Taewoo


On Fri, Jul 12, 2019 at 12:49 AM Ian Maxon  wrote:

> Hi everyone,
>
> Please verify and vote on the latest release of Apache AsterixDB
>
> The change that produced this release and the change to advance the
> version are
> up for review on Gerrit:
>
>
> https://asterix-gerrit.ics.uci.edu/#/q/status:open+owner:%22Jenkins+%253Cjenkins%2540fulliautomatix.ics.uci.edu%253E%22
>
> The release artifacts are as follows:
>
> AsterixDB Source
>
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.5-source-release.zip
>
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.5-source-release.zip.asc
>
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.5-source-release.zip.sha256
>
> SHA256:1eecef9152ec2e39083383070456a38876e4bf6127cb6800b1e2e365f207
>
> Hyracks Source
>
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.5-source-release.zip
>
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.5-source-release.zip.asc
>
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.5-source-release.zip.sha256
>
> SHA256:a90e1f8e06ed78bfc9d2d3dd3bcdf996d738d8193a4d94fe3f5cd8b4c447ee2e
>
> AsterixDB NCService Installer:
>
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.5-binary-assembly.zip
>
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.5-binary-assembly.zip.asc
>
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.5-binary-assembly.zip.sha256
>
> SHA256:d80ff63ea5796022f6ce58676d3954438ce703a1da06c5f382b8ace3d4719445
>
> The KEYS file containing the PGP keys used to sign the release can be
> found at
>
> https://dist.apache.org/repos/dist/release/asterixdb/KEYS
>
> RAT was executed as part of Maven via the RAT maven plugin, but
> excludes files that are:
>
> - data for tests
> - procedurally generated,
> - or source files which come without a header mentioning their license,
>   but have an explicit reference in the LICENSE file.
>
>
> The vote is open for 72 hours, or until the necessary number of votes
> (3 +1) has been reached.
>
> Please vote
> [ ] +1 release these packages as Apache AsterixDB 0.9.5 and
> Apache Hyracks 0.3.5
> [ ] 0 No strong feeling either way
> [ ] -1 do not release one or both packages because ...
>
> Thanks!
>

Re: [VOTE] Release Apache AsterixDB 0.9.5 and Hyracks 0.3.5 (RC1)

[Taewoo Kim]

[V] +1 release these packages as Apache AsterixDB 0.9.5 and Apache Hyracks
0.3.5

- Checked the SHA256 of the zip files.
- Builds were successful without any error.
- Smoke test using the binary was successful.

Best,
Taewoo


On Tue, Jul 2, 2019 at 12:25 AM Ian Maxon  wrote:

> Hi everyone,
>
> Please verify and vote on the latest release of Apache AsterixDB
>
> The change that produced this release and the change to advance the
> version are
> up for review on Gerrit:
>
>
> https://asterix-gerrit.ics.uci.edu/#/q/status:open+owner:%22Jenkins+%253Cjenkins%2540fulliautomatix.ics.uci.edu%253E%22
>
> The release artifacts are as follows:
>
> AsterixDB Source
>
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.5-source-release.zip
>
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.5-source-release.zip.asc
>
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.5-source-release.zip.sha256
>
> SHA256:35c0249bf7d8bb5868016589018eefb91c0dfde1f6b06001e859ffb3d9144638
>
> Hyracks Source
>
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.5-source-release.zip
>
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.5-source-release.zip.asc
>
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.5-source-release.zip.sha256
>
> SHA256:ee2eda7e9ff03978e21b4fc0db33854475d6dba70b5346b5e78308bf6d8efc72
>
> AsterixDB NCService Installer:
>
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.5-binary-assembly.zip
>
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.5-binary-assembly.zip.asc
>
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.5-binary-assembly.zip.sha256
>
> SHA256:e03f48e410e2ff84fc4b63f32bb50171747c5e0088845b4b462693f239c7d794
>
> The KEYS file containing the PGP keys used to sign the release can be
> found at
>
> https://dist.apache.org/repos/dist/release/asterixdb/KEYS
>
> RAT was executed as part of Maven via the RAT maven plugin, but
> excludes files that are:
>
> - data for tests
> - procedurally generated,
> - or source files which come without a header mentioning their license,
>   but have an explicit reference in the LICENSE file.
>
>
> The vote is open for 72 hours, or until the necessary number of votes
> (3 +1) has been reached.
>
> Please vote
> [ ] +1 release these packages as Apache AsterixDB 0.9.5 and
> Apache Hyracks 0.3.5
> [ ] 0 No strong feeling either way
> [ ] -1 do not release one or both packages because ...
>
> Thanks!
>

Re: [VOTE] Release Apache AsterixDB 0.9.5 and Hyracks 0.3.5 (RC0)

[Taewoo Kim]

Thanks Ian for preparing this release candidate.

[V] Download files
[V] Verify hashes of downloaded files.
[X] Verify that source builds correctly
 - Hyracks build was successful.
 - AsterixDB build was not successful.

@Ian Maxon : I could not build AsterixDB. Could you check
the following error message?

[INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @
asterix-common ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 194 source files to
/Users/waans11/Downloads/asterixdb/apache-asterixdb-0.9.5/asterix-common/target/classes
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/Users/waans11/Downloads/asterixdb/apache-asterixdb-0.9.5/asterix-common/src/main/java/org/apache/asterix/common/transactions/LogRecord.java:[31,51]
cannot find symbol
  symbol:   class SimpleTupleReferenceV0
  location: package org.apache.hyracks.storage.am.common.tuples
[ERROR]
/Users/waans11/Downloads/asterixdb/apache-asterixdb-0.9.5/asterix-common/src/main/java/org/apache/asterix/common/api/INcApplicationContext.java:[43,37]
package org.apache.hyracks.util.cache does not exist
[ERROR]
/Users/waans11/Downloads/asterixdb/apache-asterixdb-0.9.5/asterix-common/src/main/java/org/apache/asterix/common/api/INcApplicationContext.java:[135,5]
cannot find symbol
  symbol:   class ICacheManager
  location: interface org.apache.asterix.common.api.INcApplicationContext
[ERROR]
/Users/waans11/Downloads/asterixdb/apache-asterixdb-0.9.5/asterix-common/src/main/java/org/apache/asterix/common/config/OptimizationConfUtil.java:[26,69]
cannot find symbol
  symbol:   class AbstractGroupByPOperator
  location: package
org.apache.hyracks.algebricks.core.algebra.operators.physical
[ERROR]
/Users/waans11/Downloads/asterixdb/apache-asterixdb-0.9.5/asterix-common/src/main/java/org/apache/asterix/common/utils/WarningUtil.java:[25,41]
cannot find symbol
  symbol:   class Warning
  location: package org.apache.hyracks.api.exceptions
[ERROR]
/Users/waans11/Downloads/asterixdb/apache-asterixdb-0.9.5/asterix-common/src/main/java/org/apache/asterix/common/utils/WarningUtil.java:[33,19]
cannot find symbol
  symbol:   class Warning
  location: class org.apache.asterix.common.utils.WarningUtil
[INFO] 6 errors
[INFO] -
[INFO]

[INFO] Reactor Summary:
[INFO]
[INFO] apache-asterixdb ... SUCCESS [
 4.652 s]
[INFO] asterix-common . FAILURE [
22.420 s]
[INFO] asterix-om . SKIPPED
[INFO] asterix-maven-plugins .. SKIPPED
[INFO] record-manager-generator-maven-plugin .. SKIPPED
[INFO] asterix-transactions ... SKIPPED
[INFO] asterix-runtime  SKIPPED
[INFO] asterix-active . SKIPPED
[INFO] asterix-hivecompat . SKIPPED
[INFO] lexer-generator-maven-plugin ... SKIPPED
[INFO] asterix-external-data .. SKIPPED
[INFO] asterix-metadata ... SKIPPED
[INFO] asterix-lang-common  SKIPPED
[INFO] asterix-lang-aql ... SKIPPED
[INFO] asterix-lang-sqlpp . SKIPPED
[INFO] asterix-grammar-extension-maven-plugin . SKIPPED
[INFO] asterix-algebra  SKIPPED
[INFO] asterix-test-framework . SKIPPED
[INFO] asterix-replication  SKIPPED
[INFO] asterix-fuzzyjoin .. SKIPPED
[INFO] asterix-geo  SKIPPED
[INFO] asterix-test-datagenerator-maven-plugin  SKIPPED
[INFO] asterix-app  SKIPPED
[INFO] asterix-dashboard .. SKIPPED
[INFO] asterix-tools .. SKIPPED
[INFO] asterix-examples ... SKIPPED
[INFO] asterix-client-helper .. SKIPPED
[INFO] asterix-server . SKIPPED
[INFO] asterix-docker . SKIPPED
[INFO] asterix-doc  SKIPPED
[INFO] asterix-benchmark .. SKIPPED
[INFO] asterix-coverage ... SKIPPED
[INFO] asterix-license  SKIPPED
[INFO] asterix-spidersilk . SKIPPED
[INFO]

[INFO] BUILD FAILURE
[INFO]

Re: Code review steps

[Taewoo Kim]

I forgot whether one can do a second +1 if he/she already did +1. If that
is possible, yeah, your approach is more clear and easy to understand.

Best,
Taewoo


On Sun, Jun 9, 2019 at 7:28 PM Till Westmann  wrote:

> Indeed. That clearly helps.
> However, if there’s more than one round of review we’d need to remove
> reviewers and re-add them.
> A 2nd +1 seems easier (and less error-prone) in that case.
>
> Cheers,
> Till
>
> > On Jun 7, 2019, at 22:04, Taewoo Kim  wrote:
> >
> > That clearly works. Or why don't we add reviewers only if the self-review
> > is done? :-)
> >
> > Best,
> > Taewoo
> >
> >
> >> On Fri, Jun 7, 2019 at 12:51 AM Mike Carey  wrote:
> >>
> >> +1 !
> >>
> >>> On 6/6/19 6:07 PM, Till Westmann wrote:
> >>> Hi,
> >>>
> >>> I've proposed an addition to our code review steps on
> >>> https://cwiki.apache.org/confluence/display/ASTERIXDB/Code+Reviews
> >>>
> >>> The only change from the steps we've been doing so far is the use of a
> >>> "+1" code review by the author of a change to indicate that the
> >>> self-review is done (avoiding the guessing game if a change is
> >>> awaiting review).
> >>>
> >>> Please let me know what you think about this approach.
> >>>
> >>> Cheers,
> >>> Till
> >>
>
>

Re: Code review steps

[Taewoo Kim]

That clearly works. Or why don't we add reviewers only if the self-review
is done? :-)

Best,
Taewoo


On Fri, Jun 7, 2019 at 12:51 AM Mike Carey  wrote:

> +1 !
>
> On 6/6/19 6:07 PM, Till Westmann wrote:
> > Hi,
> >
> > I've proposed an addition to our code review steps on
> > https://cwiki.apache.org/confluence/display/ASTERIXDB/Code+Reviews
> >
> > The only change from the steps we've been doing so far is the use of a
> > "+1" code review by the author of a change to indicate that the
> > self-review is done (avoiding the guessing game if a change is
> > awaiting review).
> >
> > Please let me know what you think about this approach.
> >
> > Cheers,
> > Till
>

Re: Firewall Blocked Local Host Site

[Taewoo Kim]

In addition to Chen Luo's comment:

If we decide to open ports:

Open firewall ports in Windows 10
https://www.tomshardware.com/news/how-to-open-firewall-ports-in-windows-10,36451.html

Ports that need to be opened based on the following doc:
https://ci.apache.org/projects/asterixdb/ncservice.html
19001, 19002, 19003, 19006
1098, 1099, 9090
16001

Best,
Taewoo


On Fri, May 31, 2019 at 9:14 AM Chen Luo  wrote:

> Microsoft has tutorial on this topic:
>
> https://support.microsoft.com/en-us/help/4028544/windows-10-turn-windows-defender-firewall-on-or-off
> (Allow
> an app through the firewall)
>
> The student needs to follow this step and allow AsterixDB (or
> Java/NCService/CCService, depending on the name of the AsterixDB process)
> throughput the firewall.
>
> Best regards,
> Chen Luo
>
>
>
> On Fri, May 31, 2019 at 8:47 AM Michael Carey  wrote:
>
> > This sounds vaguely familiar - thoughts?  (Anyone?) This is one of a
> > couple of hundred temporary Windows customers of AsterixDB... :-)
> >
> >
> >  Forwarded Message 
> > Subject:Firewall Blocked Local Host Site
> > Date:   Fri, 31 May 2019 15:42:03 + (UTC)
> > From:   CS 122A on Piazza 
> > Reply-To:   re...@piazza.com
> > To: mjca...@ics.uci.edu
> >
> >
> >
> > *-- Reply directly to this email above this line to create a new follow
> > up. Or Click here
> > <
> >
> https://piazza.com/class?cid=jwc99uq2w2a2zc=jtvr1nor3pg25g=9NThYXn2mS3
> >
> >
> > to view.--*
> > A new Question was posted by Jeremy Anderson.
> >
> > *Firewall Blocked Local Host Site*
> >
> > One of the scripts prompted a windows firewall popup and I accidentally
> > skipped over it. Now, when I try to use the local host website, it gets
> > blocked for me. Does someone know what exactly I have to change on my
> > firewall to make it work? If not, can someone post a picture of the
> > popup if they come across it?
> >
> >
> >
> > Search or link to this question with @807. Follow it
> >  to get
> > notified when a response comes in.
> >
> > Sign up for more classes at http://piazza.com/uci.
> >
> >
> > Tell a colleague about Piazza. It's free, after all.
> >
> > Thanks,
> > The Piazza Team
> > --
> > Contact us at t...@piazza.com
> >
> >
> > You're receiving this email because mjca...@ics.uci.edu is enrolled in
> > CS 122A at University of California, Irvine. Sign in
> >  to manage your email preferences or un-enroll
> >  from this class.
> > Email id: jtvr1nor3pg25g|jwc99uq2w2a2zc|9NThYXn2mS3
> >
>

Re: [VOTE] Release Apache AsterixDB 0.9.4.1 and Hyracks 0.3.4.1 (RC2)

[Taewoo Kim]

[X] +1 release these packages as Apache AsterixDB 0.9.4.1 and
Apache Hyracks 0.3.4.1

- Verified the SHA256 signatures
- Verified the source builds
- Verified the binary by executing a metadata query on the Web interface

Best,
Taewoo


On Thu, Feb 21, 2019 at 7:24 PM Xikui Wang  wrote:

> [X] +1 release these packages as Apache AsterixDB 0.9.4.1 and
> Apache Hyracks 0.3.4.1
>
> - Verified the sha256
> - Tested Twitter feed with drop-in dependencies
>
> On Sat, Feb 16, 2019 at 12:23 PM Mike Carey  wrote:
>
> > [X] +1 release these packages as Apache AsterixDB 0.9.4.1 and Apache
> > Hyracks 0.3.4.1
> >
> > (I downloaded and verified the NCService puzzle piece and it worked like
> a
> > charm.)
> >
> > On 2/15/19 12:03 PM, Ian Maxon wrote:
> > > Hi everyone,
> > >
> > > Please verify and vote on the latest release of Apache AsterixDB
> > >
> > > The change that produced this release and the change to advance the
> > version
> > > are
> > > up for review on Gerrit:
> > >
> > >
> >
> https://asterix-gerrit.ics.uci.edu/#/q/status:open+owner:%22Jenkins+%253Cjenkins%2540fulliautomatix.ics.uci.edu%253E%22
> > >
> > > The release artifacts are as follows:
> > >
> > > AsterixDB Source
> > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.4.1-source-release.zip
> > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.4.1-source-release.zip.asc
> > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.4.1-source-release.zip.sha256
> > >
> > > SHA1:8bdb79294f20ff0140ea46b4a6acf5b787ac1ff3423ec41d5c5c8cdec275000c
> > >
> > > Hyracks Source
> > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.4.1-source-release.zip
> > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.4.1-source-release.zip.asc
> > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.4.1-source-release.zip.sha256
> > >
> > > SHA1:163a879031a270b0a1d5202247d478c7788ac0a5c704c7fb87d515337df54610
> > >
> > > AsterixDB NCService Installer:
> > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.4.1-binary-assembly.zip
> > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.4.1-binary-assembly.zip.asc
> > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.4.1-binary-assembly.zip.sha256
> > >
> > > SHA1:a3961f32aed8283af3cd7b66309770a5cabff426020c9c4a5b699273ad1fa820
> > >
> > > The KEYS file containing the PGP keys used to sign the release can be
> > > found at
> > >
> > > https://dist.apache.org/repos/dist/release/asterixdb/KEYS
> > >
> > > RAT was executed as part of Maven via the RAT maven plugin, but
> > > excludes files that are:
> > >
> > > - data for tests
> > > - procedurally generated,
> > > - or source files which come without a header mentioning their license,
> > >but have an explicit reference in the LICENSE file.
> > >
> > >
> > > The vote is open for 72 hours, or until the necessary number of votes
> > > (3 +1) has been reached.
> > >
> > > Please vote
> > > [ ] +1 release these packages as Apache AsterixDB 0.9.4.1 and
> > > Apache Hyracks 0.3.4.1
> > > [ ] 0 No strong feeling either way
> > > [ ] -1 do not release one or both packages because ...
> > >
> > > Thanks!
> > >
> >
>

Number of lines of the current master branch

[Taewoo Kim]

For some reason, I wanted to get the number of lines of the current master
branch. :-)

   91112 text files.
   50239 unique files.
   63478 files ignored.

http://cloc.sourceforge.net v 1.64  T=1429.40 s (20.7 files/s, 3277.3
lines/s)
---
Language files  blankcomment
code
---
Javascript   15087 168009 582151
 2244236
Java  4962  68839 130511
  429377
TypeScript4376  31470 130691
  293865
JSON  2929696  0
  260519
XML193   3433   5140
   81572
HTML   387   3527797
   39989
Python  70   7530  12604
   32978
CSS229780   2697
   32741
C++ 69   3718   3474
   22092
RobotFramework   1  0  4
   17213
Maven  102348   1869
   11937
LESS   485   1598400
   11247
C/C++ Header80   1612   1510
9140
Bourne Shell   107   1113   2188
5604
YAML   238343   1023
5371
SASS59   1069488
4573
DOS Batch   46658  0
2503
CoffeeScript33638 89
1666
make41359 71
1056
SQL 23 26414
 661
Windows Module Definition6111  0
 608
DTD  1179177
 514
C#   4113 15
 447
Bourne Again Shell  16 71 71
 304
Lisp 3 42 38
 264
m4   2 39  2
 260
XSD  2 58 71
 245
JavaServer Faces 5  3  0
 167
C2 24 25
 113
Perl 1 16 30
  57
Handlebars   3 13  0
  55
ASP.Net  3  9  0
  53
Windows Resource File1  1  1
  33
XSLT 1  5  0
  32
Racket   8  1  0
   8
Ruby 1  0  2
   4
---
SUM: 29576 296451 876553
 3511504
---

Re: Requesting node X to shutdown to ensure failure?

[Taewoo Kim]

The CC node runs an ingestion from Twitter. It inserts about 20 tweets per
sec. Other than that, it does the same amount of work like other NCs.

Best,
Taewoo

On Sun, Sep 30, 2018 at 9:52 PM, Till Westmann  wrote:

> So then it seems that all non-CC nodes have died.
> Do you have an idea of the load on the CC?
> Is is possible that it was too busy to receive the heartbeats?
>
> Cheers,
> Till
>
>
> On 30 Sep 2018, at 13:48, Taewoo Kim wrote:
>
> OK. I got that part. There are five nodes (separate NUC machines) and the
>> first node runs CC and NC.
>>
>> Best,
>> Taewoo
>>
>> On Sun, Sep 30, 2018 at 1:15 PM, Till Westmann  wrote:
>>
>> Yes, and after 10 * 8000 ms = 80s have passed without a heartbeat the node
>>> is considered dead. This seems to align with the log messages:
>>>
>>> 21:51:54.203 [Worker:ClusterController] INFO
>>> org.apache.hyracks.control.cc
>>> .cluster.NodeManager - Requesting node 2 to shutdown to ensure failure
>>> 21:51:54.206 [Worker:ClusterController] INFO
>>> org.apache.hyracks.control.cc
>>> .cluster.NodeManager - Request to shutdown failed node 2 succeeded. false
>>> positive heartbeat miss indication
>>> 21:51:54.206 [Worker:ClusterController] INFO
>>> org.apache.hyracks.control.cc
>>> .cluster.NodeManager - 2 considered dead. Last heartbeat received 83799ms
>>> ago. Max miss period: 8ms
>>>
>>> So that looks consistent.
>>> The question seems to be (unless I missed something) why no heartbeats
>>> were received for 80s.
>>>
>>> And it also looks like this happened for more than one node at the same
>>> time. Are these nodes running on a single box or on multiple boxes?
>>>
>>> Cheers,
>>> Till
>>>
>>>
>>> On 30 Sep 2018, at 13:03, Taewoo Kim wrote:
>>>
>>> P.S. We set the heartbeat setting as follows.
>>>
>>>>
>>>> [cc]
>>>> heartbeat.max.misses=10
>>>> heartbeat.period=8000
>>>>
>>>> Best,
>>>> Taewoo
>>>>
>>>> On Sun, Sep 30, 2018 at 12:50 PM, Taewoo Kim 
>>>> wrote:
>>>>
>>>> Hi Till,
>>>>
>>>>>
>>>>> These files are indeed the same log file. I just extracted one line
>>>>> from
>>>>> the original log and placed the line at the beginning. At 9:50 pm, the
>>>>> CC
>>>>> initiates a task cluster and all NCs were executing. Node 2 sent the
>>>>> taskcomplete message, and somehow the CC decides to shut down the node
>>>>> 2.
>>>>> That's my understanding.
>>>>>
>>>>> Best,
>>>>> Taewoo
>>>>>
>>>>> On Sun, Sep 30, 2018 at 12:40 PM, Till Westmann 
>>>>> wrote:
>>>>>
>>>>> Hi Taewoo,
>>>>>
>>>>>>
>>>>>> I’m little confused as we seem to have 2 cc logs.
>>>>>>
>>>>>> The "Requesting node 2 to shutdown …" message is logged, when nodes
>>>>>> are
>>>>>> considered to be dead as they have missed too many heartbeats. And you
>>>>>> can
>>>>>> see in the later logs that indeed more than 80s passed without
>>>>>> receiving a
>>>>>> heartbeat.
>>>>>> Could it be that the machines were too busy to send/receive
>>>>>> heartbeats?
>>>>>>
>>>>>> Cheers,
>>>>>> Till
>>>>>>
>>>>>> On 30 Sep 2018, at 10:52, Taewoo Kim wrote:
>>>>>>
>>>>>> Hi devs,
>>>>>>
>>>>>>
>>>>>>> I saw the following log messages in the cc.log of the Cloudberrry
>>>>>>> cluster.
>>>>>>> Yesterday around 9:50 pm, a task cluster was created and executed.
>>>>>>> And
>>>>>>> around 9:51 pm, the cluster shows the following message. In the
>>>>>>> nc.log
>>>>>>> of
>>>>>>> the node 2, there were no noticeable log messages. I wonder what
>>>>>>> could
>>>>>>> cause this situation.
>>>>>>>
>>>>>>> -- cc.log at 9:51 pm
>>>>>>> *21:51:54.203 [Worker:ClusterController] INFO
>>>>>>>  org.apache.hyracks.cont

Re: Requesting node X to shutdown to ensure failure?

[Taewoo Kim]

Hi Till,

These files are indeed the same log file. I just extracted one line from
the original log and placed the line at the beginning. At 9:50 pm, the CC
initiates a task cluster and all NCs were executing. Node 2 sent the
taskcomplete message, and somehow the CC decides to shut down the node 2.
That's my understanding.

Best,
Taewoo

On Sun, Sep 30, 2018 at 12:40 PM, Till Westmann  wrote:

> Hi Taewoo,
>
> I’m little confused as we seem to have 2 cc logs.
>
> The "Requesting node 2 to shutdown …" message is logged, when nodes are
> considered to be dead as they have missed too many heartbeats. And you can
> see in the later logs that indeed more than 80s passed without receiving a
> heartbeat.
> Could it be that the machines were too busy to send/receive heartbeats?
>
> Cheers,
> Till
>
> On 30 Sep 2018, at 10:52, Taewoo Kim wrote:
>
> Hi devs,
>>
>> I saw the following log messages in the cc.log of the Cloudberrry cluster.
>> Yesterday around 9:50 pm, a task cluster was created and executed. And
>> around 9:51 pm, the cluster shows the following message. In the nc.log of
>> the node 2, there were no noticeable log messages. I wonder what could
>> cause this situation.
>>
>> -- cc.log at 9:51 pm
>> *21:51:54.203 [Worker:ClusterController] INFO
>>  org.apache.hyracks.control.cc.cluster.NodeManager - Requesting node 2 to
>> shutdown to ensure failure*
>>
>> *-- nc-2.log around 9:50 pm*
>>
>> 21:50:30.314 [Worker:2] INFO
>>  org.apache.hyracks.control.nc.work.StartTasksWork - input: 0: CDID:2
>> 21:50:30.375 [Worker:2] INFO
>>  org.apache.hyracks.control.common.work.WorkQueue - Executing:
>> NotifyTaskCompleteWork:TAID:TID:ANID:ODID:7:0:3:0
>> 21:50:30.382 [Worker:2] INFO
>>  org.apache.hyracks.control.common.work.WorkQueue - Executing:
>> NotifyTaskCompleteWork:TAID:TID:ANID:ODID:9:0:3:0
>> 21:50:30.390 [Worker:2] INFO
>>  org.apache.hyracks.control.common.work.WorkQueue - Executing:
>> NotifyTaskCompleteWork:TAID:TID:ANID:ODID:9:0:2:0
>> 21:50:30.391 [Worker:2] INFO
>>  org.apache.hyracks.control.common.work.WorkQueue - Executing:
>> NotifyTaskCompleteWork:TAID:TID:ANID:ODID:7:0:2:0
>> 21:50:30.392 [Worker:2] INFO
>>  org.apache.hyracks.control.common.work.WorkQueue - Executing:
>> NotifyTaskCompleteWork:TAID:TID:ANID:ODID:1:1:3:0
>> 21:50:30.392 [Worker:2] INFO
>>  org.apache.hyracks.control.common.work.WorkQueue - Executing:
>> NotifyTaskCompleteWork:TAID:TID:ANID:ODID:1:1:2:0
>> 21:50:30.398 [Worker:2] INFO
>>  org.apache.hyracks.control.common.work.WorkQueue - Executing:
>> NotifyTaskCompleteWork:TAID:TID:ANID:ODID:1:1:3:0
>> 21:50:30.398 [Worker:2] INFO
>>  org.apache.hyracks.control.common.work.WorkQueue - Executing:
>> NotifyTaskCompleteWork:TAID:TID:ANID:ODID:1:1:2:0
>> 22:02:30.290 [JVM exit thread] INFO  org.apache.hyracks.util.ExitUtil -
>> JVM
>> exiting with status 0; bye!
>>
>>
>> -- *cc.log from 9:51 pm*
>>
>> 21:50:30.313 [Worker:ClusterController] INFO
>>  org.apache.hyracks.control.cc.executor.ActivityClusterPlanner - Built 1
>> Task Clusters
>> 21:50:30.313 [Worker:ClusterController] INFO
>>  org.apache.hyracks.control.cc.executor.ActivityClusterPlanner - Tasks:
>> [TID:ANID:ODID:1:1:0, TID:ANID:ODID:1:1:1, TID:ANID:ODID:1:1:2,
>> TID:ANID:ODID:1:1:3, TID:ANID:ODID:1:1:4, TID:ANID:ODID:1:1:5,
>> TID:ANID:ODID:1:1:6, TID:ANID:ODID:1:1:7, TID:ANID:ODID:1:1:8,
>> TID:ANID:ODID:1:1:9, TID:ANID:ODID:2:0:0, TID:ANID:ODID:2:0:1,
>> TID:ANID:ODID:2:0:2, TID:ANID:ODID:2:0:3, TID:ANID:ODID:2:0:4,
>> TID:ANID:ODID:2:0:5, TID:ANID:ODID:2:0:6, TID:ANID:ODID:2:0:7,
>> TID:ANID:ODID:2:0:8, TID:ANID:ODID:2:0:9, TID:ANID:ODID:9:0:0,
>> TID:ANID:ODID:9:0:1, TID:ANID:ODID:9:0:2, TID:ANID:ODID:9:0:3,
>> TID:ANID:ODID:9:0:4, TID:ANID:ODID:9:0:5, TID:ANID:ODID:9:0:6,
>> TID:ANID:ODID:9:0:7, TID:ANID:ODID:9:0:8, TID:ANID:ODID:9:0:9]
>> 21:50:30.313 [Worker:ClusterController] INFO
>>  org.apache.hyracks.control.cc.executor.JobExecutor - Runnable TC roots:
>> [TC:[TID:ANID:ODID:1:1:0, TID:ANID:ODID:1:1:1, TID:ANID:ODID:1:1:2,
>> TID:ANID:ODID:1:1:3, TID:ANID:ODID:1:1:4, TID:ANID:ODID:1:1:5,
>> TID:ANID:ODID:1:1:6, TID:ANID:ODID:1:1:7, TID:ANID:ODID:1:1:8,
>> TID:ANID:ODID:1:1:9, TID:ANID:ODID:2:0:0, TID:ANID:ODID:2:0:1,
>> TID:ANID:ODID:2:0:2, TID:ANID:ODID:2:0:3, TID:ANID:ODID:2:0:4,
>> TID:ANID:ODID:2:0:5, TID:ANID:ODID:2:0:6, TID:ANID:ODID:2:0:7,
>> TID:ANID:ODID:2:0:8, TID:ANID:ODID:2:0:9, TID:ANID:ODID:9:0:0,
>> TID:ANID:ODID:9:0:1, TID:ANID:ODID:9:0:2, TID:ANID:ODID:9:0:3,
>> TID:ANID:ODID:9:0:4, TID

Requesting node X to shutdown to ensure failure?

[Taewoo Kim]

Hi devs,

I saw the following log messages in the cc.log of the Cloudberrry cluster.
Yesterday around 9:50 pm, a task cluster was created and executed. And
around 9:51 pm, the cluster shows the following message. In the nc.log of
the node 2, there were no noticeable log messages. I wonder what could
cause this situation.

-- cc.log at 9:51 pm
*21:51:54.203 [Worker:ClusterController] INFO
 org.apache.hyracks.control.cc.cluster.NodeManager - Requesting node 2 to
shutdown to ensure failure*

*-- nc-2.log around 9:50 pm*
21:50:30.314 [Worker:2] INFO
 org.apache.hyracks.control.nc.work.StartTasksWork - input: 0: CDID:2
21:50:30.375 [Worker:2] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
NotifyTaskCompleteWork:TAID:TID:ANID:ODID:7:0:3:0
21:50:30.382 [Worker:2] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
NotifyTaskCompleteWork:TAID:TID:ANID:ODID:9:0:3:0
21:50:30.390 [Worker:2] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
NotifyTaskCompleteWork:TAID:TID:ANID:ODID:9:0:2:0
21:50:30.391 [Worker:2] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
NotifyTaskCompleteWork:TAID:TID:ANID:ODID:7:0:2:0
21:50:30.392 [Worker:2] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
NotifyTaskCompleteWork:TAID:TID:ANID:ODID:1:1:3:0
21:50:30.392 [Worker:2] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
NotifyTaskCompleteWork:TAID:TID:ANID:ODID:1:1:2:0
21:50:30.398 [Worker:2] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
NotifyTaskCompleteWork:TAID:TID:ANID:ODID:1:1:3:0
21:50:30.398 [Worker:2] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
NotifyTaskCompleteWork:TAID:TID:ANID:ODID:1:1:2:0
22:02:30.290 [JVM exit thread] INFO  org.apache.hyracks.util.ExitUtil - JVM
exiting with status 0; bye!


-- *cc.log from 9:51 pm*
21:50:30.313 [Worker:ClusterController] INFO
 org.apache.hyracks.control.cc.executor.ActivityClusterPlanner - Built 1
Task Clusters
21:50:30.313 [Worker:ClusterController] INFO
 org.apache.hyracks.control.cc.executor.ActivityClusterPlanner - Tasks:
[TID:ANID:ODID:1:1:0, TID:ANID:ODID:1:1:1, TID:ANID:ODID:1:1:2,
TID:ANID:ODID:1:1:3, TID:ANID:ODID:1:1:4, TID:ANID:ODID:1:1:5,
TID:ANID:ODID:1:1:6, TID:ANID:ODID:1:1:7, TID:ANID:ODID:1:1:8,
TID:ANID:ODID:1:1:9, TID:ANID:ODID:2:0:0, TID:ANID:ODID:2:0:1,
TID:ANID:ODID:2:0:2, TID:ANID:ODID:2:0:3, TID:ANID:ODID:2:0:4,
TID:ANID:ODID:2:0:5, TID:ANID:ODID:2:0:6, TID:ANID:ODID:2:0:7,
TID:ANID:ODID:2:0:8, TID:ANID:ODID:2:0:9, TID:ANID:ODID:9:0:0,
TID:ANID:ODID:9:0:1, TID:ANID:ODID:9:0:2, TID:ANID:ODID:9:0:3,
TID:ANID:ODID:9:0:4, TID:ANID:ODID:9:0:5, TID:ANID:ODID:9:0:6,
TID:ANID:ODID:9:0:7, TID:ANID:ODID:9:0:8, TID:ANID:ODID:9:0:9]
21:50:30.313 [Worker:ClusterController] INFO
 org.apache.hyracks.control.cc.executor.JobExecutor - Runnable TC roots:
[TC:[TID:ANID:ODID:1:1:0, TID:ANID:ODID:1:1:1, TID:ANID:ODID:1:1:2,
TID:ANID:ODID:1:1:3, TID:ANID:ODID:1:1:4, TID:ANID:ODID:1:1:5,
TID:ANID:ODID:1:1:6, TID:ANID:ODID:1:1:7, TID:ANID:ODID:1:1:8,
TID:ANID:ODID:1:1:9, TID:ANID:ODID:2:0:0, TID:ANID:ODID:2:0:1,
TID:ANID:ODID:2:0:2, TID:ANID:ODID:2:0:3, TID:ANID:ODID:2:0:4,
TID:ANID:ODID:2:0:5, TID:ANID:ODID:2:0:6, TID:ANID:ODID:2:0:7,
TID:ANID:ODID:2:0:8, TID:ANID:ODID:2:0:9, TID:ANID:ODID:9:0:0,
TID:ANID:ODID:9:0:1, TID:ANID:ODID:9:0:2, TID:ANID:ODID:9:0:3,
TID:ANID:ODID:9:0:4, TID:ANID:ODID:9:0:5, TID:ANID:ODID:9:0:6,
TID:ANID:ODID:9:0:7, TID:ANID:ODID:9:0:8, TID:ANID:ODID:9:0:9]],
inProgressTaskClusters: []
21:50:30.366 [Worker:ClusterController] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
TaskComplete: [3[JID:0.700:TAID:TID:ANID:ODID:7:0:5:0]
21:50:30.374 [Worker:ClusterController] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
TaskComplete: [5[JID:0.700:TAID:TID:ANID:ODID:7:0:9:0]
21:50:30.376 [Worker:ClusterController] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
TaskComplete: [2[JID:0.700:TAID:TID:ANID:ODID:7:0:3:0]
21:50:30.376 [Worker:ClusterController] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
TaskComplete: [5[JID:0.700:TAID:TID:ANID:ODID:7:0:8:0]
21:50:30.377 [Worker:ClusterController] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
TaskComplete: [1[JID:0.700:TAID:TID:ANID:ODID:7:0:0:0]
21:50:30.378 [Worker:ClusterController] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
TaskComplete: [3[JID:0.700:TAID:TID:ANID:ODID:7:0:4:0]
21:50:30.381 [Worker:ClusterController] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
TaskComplete: [1[JID:0.700:TAID:TID:ANID:ODID:7:0:1:0]
21:50:30.382 [Worker:ClusterController] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
TaskComplete: [5[JID:0.701:TAID:TID:ANID:ODID:9:0:8:0]
21:50:30.383 [Worker:ClusterController] INFO
 org.apache.hyracks.control.common.work.WorkQueue - Executing:
TaskComplete: 

Re: [VOTE] Release AsterixDB 0.9.4 and Hyracks 0.3.4 (RC2)

[Taewoo Kim]

+1

[v] Verify the signatures and hashes.
[v] Verify that source builds correctly.
[v] Verify that a query works.

Only one minor comment:
https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.4-source-release.zip.sha25
https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.4.zip.256
https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb_0.9.4_all.deb.256

The extension of these files should be .sha256.





On Thu, Sep 20, 2018 at 4:13 PM Wail Alkowaileet  wrote:

> Then it looks good for me +1
>
> On Wed, Sep 19, 2018 at 4:36 PM Ian Maxon  wrote:
>
> > I moved the file, sorry about that, didn't notice. The tests should be
> > OK; it all passes on Gerrit right now.
> > On Tue, Sep 18, 2018 at 3:54 PM Wail Alkowaileet 
> > wrote:
> > >
> > >- mvn verify reports two issues (not sure about severity of them):
> > >
> > >
> > >1. SqlppExecutionWithCancellationTest.tearDown:54 There are 15
> leaked
> > >run files.
> > >2. DiskIsFullTest.testDiskIsFull:179 Expected exception
> > >(org.apache.hyracks.api.exceptions.HyracksDataException: HYR0088:
> > Cannot
> > >modify index (Disk is full)) was not thrown
> > >
> > > The latter does not seem to skip the test for (macOS High Sierra) and
> it
> > > seems it has been removed from the current master.
> > >
> > >- signatures and hashes looks good.
> > >
> > > One thing is the asc file for the AsterixDB Installer should be
> > > renamed: from asterix-server-0.9.4.zip.asc to
> > apache-asterixdb-0.9.4.zip.asc
> > >
> > >
> > > On Fri, Sep 7, 2018 at 1:55 PM Ian Maxon  wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Please verify and vote on the latest release of Apache AsterixDB
> > > >
> > > > The change that produced this release and the change to advance the
> > > > version are
> > > > up for review on Gerrit:
> > > >
> > > >
> > > >
> >
> https://asterix-gerrit.ics.uci.edu/#/q/status:open+owner:%22Jenkins+%253Cjenkins%2540fulliautomatix.ics.uci.edu%253E%22
> > > >
> > > > The release artifacts are as follows:
> > > >
> > > > AsterixDB Source
> > > >
> > > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.4-source-release.zip
> > > >
> > > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.4-source-release.zip.asc
> > > >
> > > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.4-source-release.zip.sha25
> > > >
> > > > SHA256:
> > 2bedc3e30bdebdc26ae7fdbe4ce9b2ec8d546a195ee8bc05f7e0e516e747bfe8
> > > >
> > > > Hyracks Source
> > > >
> > > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.4-source-release.zip
> > > >
> > > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.4-source-release.zip.asc
> > > >
> > > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.4-source-release.zip.sha256
> > > >
> > > > SHA256:
> > 8d3d8c734d0e49b145619d8e083aea4cd599adb2b9fe148b05eac8550caf1764
> > > >
> > > > AsterixDB Installer:
> > > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.4.zip
> > > >
> > > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.4.zip.asc
> > > >
> > > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.4.zip.256
> > > >
> > > > SHA256:
> > 0b939231635f0c2328018f7064df9a4fa4b05b36835127a12eae4543141aecd9
> > > >
> > > > AsterixDB Debian/Ubuntu Package:
> > > >
> > > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb_0.9.4_all.deb
> > > >
> > > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb_0.9.4_all.deb.asc
> > > >
> > > >
> >
> https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb_0.9.4_all.deb.256
> > > >
> > > > SHA256:
> > c41fc765f04cb335c5fb728af625217289f64886d05d724e3ef6aa140d4437f5
> > > >
> > > > The KEYS file containing the PGP keys used to sign the release can be
> > > > found at
> > > >
> > > > https://dist.apache.org/repos/dist/release/asterixdb/KEYS
> > > >
> > > > RAT was executed as part of Maven via the RAT maven plugin, but
> > > > excludes files that are:
> > > >
> > > > - data for tests
> > > > - procedurally generated,
> > > > - or source files which come without a header mentioning their
> license,
> > > >   but have an explicit reference in the LICENSE file.
> > > >
> > > >
> > > > The vote is open for 72 hours, or until the necessary number of votes
> > > > (3 +1) has been reached.
> > > >
> > > > Please vote
> > > > [ ] +1 release these packages as Apache AsterixDB 0.9.4 and
> > > > Apache Hyracks 0.3.4
> > > > [ ] 0 No strong feeling either way
> > > > [ ] -1 do not release one or both packages because ...
> > > >
> > > > Thanks!
> > > >
> > >
> > >
> > > --
> > >
> > > *Regards,*
> > > Wail Alkowaileet
> >
>
>
> --
>
> *Regards,*
> Wail Alkowaileet
>

Re: Ansible script - starting the cluster error?

[Taewoo Kim]

Thanks to Ian's suggestion, I was able to resolve the issue. This happened
because somehow the script can't recognize the user who is running the
script. I am not sure why it suddenly happened for an existing cluster that
I used for 20 days. For the future reference, create a file named
ansible.cfg in the opt/ansible/bin. Put the following contents in the file.

[defaults]
remote_user = YOURACCOUNTNAME



Best,
Taewoo

On Thu, Jun 14, 2018 at 4:08 PM, Taewoo Kim  wrote:

> Hello devs,
>
> Has anyone seen this error when starting the cluster using
> opt/ansible/bin/start.sh?
>
> TASK [Copy cluster config to CC] **
> 
> ***
> fatal: [128.195.11.42]: FAILED! => {"msg": "failed to transfer file to
> ~None/.ansible/tmp/ansible-tmp-1529017615.83-64263400588477/synchronize.py:
> [Errno 2] No such file or directory: '~None/.ansible/tmp/ansible-
> tmp-1529017615.83-64263400588477/synchronize.py'"}
> to retry, use: --limit @/lv_scratch/scratch/taewok2/
> server1/opt/ansible/yaml/instance_start.retry
>
>
>
> Best,
> Taewoo
>

Ansible script - starting the cluster error?

[Taewoo Kim]

Hello devs,

Has anyone seen this error when starting the cluster using
opt/ansible/bin/start.sh?

TASK [Copy cluster config to CC]
*
fatal: [128.195.11.42]: FAILED! => {"msg": "failed to transfer file to
~None/.ansible/tmp/ansible-tmp-1529017615.83-64263400588477/synchronize.py:
[Errno 2] No such file or directory:
'~None/.ansible/tmp/ansible-tmp-1529017615.83-64263400588477/synchronize.py'"}
to retry, use: --limit
@/lv_scratch/scratch/taewok2/server1/opt/ansible/yaml/instance_start.retry



Best,
Taewoo

Re: Stability Q

[Taewoo Kim]

+1

Best,
Taewoo

On Tue, May 1, 2018 at 8:49 AM, Akshay Manchale Sridhar 
wrote:

> I was running into the same issue while running replication experiments. A
> quick fix is to increase the default value of HEARTBEAT_MAX_MISSES. There
> are times in a loaded cluster when some nodes become unresponsive for a few
> seconds and the CC marks them as dead because the defaults are too low.
>
> On Tue, May 1, 2018 at 1:23 AM, Murtadha Hubail 
> wrote:
>
> > Indeed :-)
> >
> > On 05/01/2018, 11:03 AM, "Mike Carey"  wrote:
> >
> > (And several sleep cycles and network changes were involved in my
> case
> > between runs.  Typical enterprise use case, right? :-))
> >
> >
> > On 5/1/18 12:31 AM, Murtadha Hubail wrote:
> > > This is most likely caused by missing heartbeat from the NC to the
> > CC. Some macOS versions had issues with reestablishing connected sockets
> > after waking up from sleep.
> > > But it could also be some unexpected exception that caused the NC
> to
> > shut down. If you could share the logs with me, I can tell you for sure.
> > >
> > > Cheers,
> > > Murtadha
> > >
> > > On 05/01/2018, 9:06 AM, "Michael Carey" 
> > wrote:
> > >
> > >  Q:  Do we maybe have a stability regression in recent versions
> > (e.g.,
> > >  the one leading to the UW snapshot)?  They have occasionally
> > seen things
> > >  like this and I just did too.  (The system had been running
> for
> > awhile
> > >  in the background on my Mac - e.g., for a day or so.)
> > >
> > >  Error: Cluster is in UNUSABLE state.
> > >One or more Node Controllers have left or haven't joined
> yet.
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>

Re: Optimizer Tests for SQLPP

[Taewoo Kim]

That also makes sense. Since the similarity join branch is not merged yet,
I will add them.

On Sat, Apr 14, 2018 at 16:05 Mike Carey <dtab...@gmail.com> wrote:

> Question:  Aren't there similarity join tests as well, though?  (I.e.,
> is it not sufficient to let it defend itself, rather than also testing
> its component parts?)
>
>
> On 4/14/18 10:42 AM, Taewoo Kim wrote:
> > For now, please do not remove AQL optimizer test cases since AQL+
> > functionality that is used for the similarity join depends on them. I
> will
> > go through runtime test cases and optimizer test cases and remove test
> > cases that are covered by SQL++ test cases and that are not directly
> > related to AQL+ functionality.
> >
> > Best,
> > Taewoo
> >
> > On Sat, Apr 14, 2018 at 9:47 AM, Xikui Wang <xik...@uci.edu> wrote:
> >
> >> Hi Devs,
> >>
> >> As I mentioned in the weekly meeting, I found that our OptimizerTest
> >> actually doesn't run the SQLPP tests. Although there is a separate
> >> directory 'queries_sqlpp' which contains all the legacy optimizer tests
> >> translated into SQLPP, they are not picked up by the OptimizerTest, and
> the
> >> new SQLPP tests are still being added to the old directory and mixed up
> >> with AQL tests.
> >>
> >> I tried to run those SQLPP tests. More than half of them are failed.
> There
> >> are syntax error (query-issue838.sqlpp), variable name changes, join
> >> algorithm changes (word-jaccard.sqlpp) and other changes
> (issue730.sqlpp).
> >> I submitted one patch that fixed the test cases with variable name
> changes.
> >> For the rests, I think we need to decide, between the two versions of
> the
> >> results, which ones are the expected plans and fix the errors. There are
> >> some obvious patterns in the plan changes, so I think we only need to
> fix a
> >> few things to cover the rest 450 test cases...
> >>
> >> Best,
> >> Xikui
> >>
>
>

Re: Is there an easier way to wrap/unwrap the entire tuple as a ByteBuffer?

[Taewoo Kim]

Hello Ahmed,

This doc might help.
https://code.google.com/archive/p/asterixdb/wikis/Serialization.wiki

Best,
Taewoo

On Tue, Apr 10, 2018 at 5:30 PM, Ahmed Eldawy  wrote:

> Mike,
>
> What you're suggesting makes more sense. We just don't know how to do it :)
> BTW, is there any document that describes the binary format of the
> frame/tuple/fields? I was able to find out some information myself by
> digging into the code but if there is a document or page that describes
> this it can be of a great help.
>
> On Tue, Apr 10, 2018 at 12:01 PM, Mike Carey  wrote:
>
> > Naive (me as a stupid observer :-)) question:  Is there a reason to
> > wrap/unwrap instead of extend/unextend? (I.e., couldn't you add an
> > additional Hyracks tuple field and then project it away - i.e., expand
> and
> > contract the tuple horizontally rather than nesting and unnesting it?)
> >
> >
> >
> > On 4/10/18 11:10 AM, Chen Luo wrote:
> >
> >> Hi,
> >>
> >> You can try IFrameFieldAppender (and its implementation
> >> FrameFixedFieldAppender) to directly append wrapped tuple (field by
> field)
> >> to the output buffer, without going through the array tuple builder. But
> >> in
> >> general, because of the tuple format, I'm not sure there is a more
> >> efficient way to wrap/unwrap tuples directly.
> >>
> >> Best regards,
> >> Chen Luo
> >>
> >> On Tue, Apr 10, 2018 at 10:33 AM, Muhammad Abu Bakar Siddique <
> >> msidd...@ucr.edu> wrote:
> >>
> >> Hi Dev,
> >>> I'm working on a Hyracks application for parallel random sampling which
> >>> consists of two operators. The first operator generates and appends a
> new
> >>> field to each tuple while the second operator processes that additional
> >>> field and removes it before writing the final output. So, the output of
> >>> the
> >>> second operator should have the same format of the input of the first
> >>> operator. In other words, I want the first operator to wrap the tuple
> >>> as-is
> >>> and add an additional field while the second operator should remove and
> >>> unwrap the tuple. Currently, I use the FrameTupleAppender and
> >>> ArrayTupleAppender where I have to add each field in the input record
> >>> separately but it seems to be an overhead in the code. Is there an
> easier
> >>> way to wrap/unwrap the entire tuple as a ByteBuffer without having to
> >>> worry
> >>> about the individual fields inside it?
> >>>
> >>>
> >
>
>
> --
>
> Ahmed Eldawy
> Assistant Professor
> http://www.cs.ucr.edu/~eldawy
> Tel: +1 (951) 827-5654
>

Re: URGENT: Please shorten test filenames!

[Taewoo Kim]

@Chris:

I have renamed the current index-only plan related test cases to short
ones. The changes are reflected on the master. Let me know if the issue
still remains. Thanks.

Best,
Taewoo

On Fri, Mar 9, 2018 at 2:33 PM, Xikui Wang  wrote:

> I see... I was able to avoid the issue by this some time ago... I guess the
> filenames weren't this long back then. :(
>
> Best,
> Xikui
>
> On Fri, Mar 9, 2018 at 2:18 PM, Chris Hillery 
> wrote:
>
> > On Fri, Mar 9, 2018 at 8:17 AM, Xikui Wang  wrote:
> >
> >> One quick workaround is to put the project under the root, e.g., C:/.
> >> This is not a good solution but it will enable you to at least build the
> >> project...
> >>
> >
> > It doesn't, though. As I said, AsterixDB all by itself cannot fit in 260
> > characters, even when checked out directly into C:\...
> >
> > Ceej
> > aka Chris Hillery
> >
>

Inverted-index search is constrained by the budget.

[Taewoo Kim]

Hi all,

A new config parameter is added and I would like to share this with you.
The inverted-index-search in AsterixDB is different from B-Tree or R-Tree
search (they only require a certain number of frames at a time - no budget
is necessary). However, the inverted-index-search needs to read inverted
list(s) (collection of PKs for the given SK) from disk and generates
intermediate results. Previously, an inverted-index-search read the entire
inverted list into the buffer cache at once and generated intermediate
results in memory without considering any memory budget. Thus, it can
generate an OOM exception during these processes and can terminate an
instance. This problem has been fixed now. And one configuration parameter
is introduced - *compiler.textsearchmemory*. The default size is 32MB just
like other operations such as sort, group or join. If you observe an
inverted-index search is slow, try to allocate more budget to this
parameter. Thanks.

Best,
Taewoo

Index-only plan has been merged.

[Taewoo Kim]

Hi All,

Finally, the index-only plan branch has been merged. Thanks to all who have
reviewed this patch set. Here are a few things that I would like to post.

- The Index-only plan will be triggered on a B+ tree or an R tree (point or
rectangle field only) index. The field should be a closed-type field. A
field cannot be an open-type field.

- Only the secondary key and/or the primary key are used in a selection or
join predicate. Also, only those fields (SK or PK) are used afterward
(e.g., returning only those fields).

- If you see an error and see "UNION_ALL" operator and two paths in the
plan, it might be related to the index-only plan. For this case, before a
query, you can put " set noindexonly 'true'; " to bypass the index-only
plan as a temporary measure. Of course, please file an issue. :-)

Best,
Taewoo

Re: Comparison semantics for complex types

[Taewoo Kim]

I have two questions. How would you want to compare two complex objects?
And why do we need to do a hash?

On Fri, Dec 29, 2017 at 20:31 Wail Alkowaileet <wael@gmail.com> wrote:

> I think we should not call deep_equal implicitly when comparing objects,
> arrays or multisets.
> One reason is that we don't want to do hash join where the key is a complex
> type (i.e what would be the hash function?).
>
> On Fri, Dec 29, 2017 at 10:24 AM, Taewoo Kim <wangs...@gmail.com> wrote:
>
> > @Heri: I'm sorry for not mentioning your deep_equal function. Yeah,
> indeed,
> > we have your function. I checked BuiltinFunctions and found the function
> > named "deep-equal". So, we need to explicitly use that function to
> conduct
> > such comparison? If so, could you revise Wail's query? And it would be
> nice
> > if AsterixDB can call that function when it tries to compare arrays.
> >
> > Best,
> > Taewoo
> >
> > On Fri, Dec 29, 2017 at 8:59 AM, Heri Ramampiaro <heri...@gmail.com>
> > wrote:
> >
> > > Is this similar to the “deep_equal” function I implemented a while ago?
> > >
> > > -heri
> > >
> > > Sent from my iPhone
> > >
> > > > On Dec 29, 2017, at 17:23, Mike Carey <dtab...@gmail.com> wrote:
> > > >
> > > > Indeed - we need it someday!  (Sooner rather than later would be
> nice.)
> > > It basically needs to work like it does in languages like Python, I
> > think.
> > > (Cardinality and element by element equality for arrays, cardinality
> and
> > > order-independent equality for bags, field by field equality for
> records,
> > > and recursively through all of them.)
> > > >
> > > >
> > > >> On 12/28/17 11:14 PM, Taewoo Kim wrote:
> > > >> If I remember correctly, we don't support deep equality comparison
> in
> > > >> AsterixDB yet.
> > > >>
> > > >> Best,
> > > >> Taewoo
> > > >>
> > > >> On Thu, Dec 28, 2017 at 9:19 PM, Wail Alkowaileet <
> wael@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >>> Hi Devs,
> > > >>>
> > > >>> Currently we have an inconsistent behavior regarding the
> comparators:
> > > >>>
> > > >>> In join, we allow such operation
> > > >>>
> > > >>> SELECT *
> > > >>> FROM [[1],[2],[3]] array1, [[1],[2],[3]] array2
> > > >>> WHERE array1 = array2
> > > >>>
> > > >>> In select, an exception is thrown
> > > >>> SELECT *
> > > >>> FROM [[1],[2],[3]] array1
> > > >>> WHERE array1 = [1]
> > > >>>
> > > >>> Error ASX0004: Unsupported type: comparison operations (>, >=, <,
> and
> > > <=)
> > > >>> cannot process input type array
> > > >>>
> > > >>> What should be the semantics for such operations?
> > > >>>
> > > >>>
> > > >>> --
> > > >>>
> > > >>> *Regards,*
> > > >>> Wail Alkowaileet
> > > >>>
> > > >
> > >
> >
>
>
>
> --
>
> *Regards,*
> Wail Alkowaileet
>

Re: Comparison semantics for complex types

[Taewoo Kim]

@Heri: I'm sorry for not mentioning your deep_equal function. Yeah, indeed,
we have your function. I checked BuiltinFunctions and found the function
named "deep-equal". So, we need to explicitly use that function to conduct
such comparison? If so, could you revise Wail's query? And it would be nice
if AsterixDB can call that function when it tries to compare arrays.

Best,
Taewoo

On Fri, Dec 29, 2017 at 8:59 AM, Heri Ramampiaro <heri...@gmail.com> wrote:

> Is this similar to the “deep_equal” function I implemented a while ago?
>
> -heri
>
> Sent from my iPhone
>
> > On Dec 29, 2017, at 17:23, Mike Carey <dtab...@gmail.com> wrote:
> >
> > Indeed - we need it someday!  (Sooner rather than later would be nice.)
> It basically needs to work like it does in languages like Python, I think.
> (Cardinality and element by element equality for arrays, cardinality and
> order-independent equality for bags, field by field equality for records,
> and recursively through all of them.)
> >
> >
> >> On 12/28/17 11:14 PM, Taewoo Kim wrote:
> >> If I remember correctly, we don't support deep equality comparison in
> >> AsterixDB yet.
> >>
> >> Best,
> >> Taewoo
> >>
> >> On Thu, Dec 28, 2017 at 9:19 PM, Wail Alkowaileet <wael@gmail.com>
> >> wrote:
> >>
> >>> Hi Devs,
> >>>
> >>> Currently we have an inconsistent behavior regarding the comparators:
> >>>
> >>> In join, we allow such operation
> >>>
> >>> SELECT *
> >>> FROM [[1],[2],[3]] array1, [[1],[2],[3]] array2
> >>> WHERE array1 = array2
> >>>
> >>> In select, an exception is thrown
> >>> SELECT *
> >>> FROM [[1],[2],[3]] array1
> >>> WHERE array1 = [1]
> >>>
> >>> Error ASX0004: Unsupported type: comparison operations (>, >=, <, and
> <=)
> >>> cannot process input type array
> >>>
> >>> What should be the semantics for such operations?
> >>>
> >>>
> >>> --
> >>>
> >>> *Regards,*
> >>> Wail Alkowaileet
> >>>
> >
>

Re: Comparison semantics for complex types

[Taewoo Kim]

If I remember correctly, we don't support deep equality comparison in
AsterixDB yet.

Best,
Taewoo

On Thu, Dec 28, 2017 at 9:19 PM, Wail Alkowaileet 
wrote:

> Hi Devs,
>
> Currently we have an inconsistent behavior regarding the comparators:
>
> In join, we allow such operation
>
> SELECT *
> FROM [[1],[2],[3]] array1, [[1],[2],[3]] array2
> WHERE array1 = array2
>
> In select, an exception is thrown
> SELECT *
> FROM [[1],[2],[3]] array1
> WHERE array1 = [1]
>
> Error ASX0004: Unsupported type: comparison operations (>, >=, <, and <=)
> cannot process input type array
>
> What should be the semantics for such operations?
>
>
> --
>
> *Regards,*
> Wail Alkowaileet
>

Re: Logging.properties

[Taewoo Kim]

Update:

Murthada's method works like a charm. For those folks who want to see
Algebricks optimization details (before and after), the following is what
you need to add in  section of log4j2-test.xml file. Please note
that the name is "org.apache.hyracks.algebricks", not
"org.apache.hyracks.algebricks.level". Thanks again @Murthada.



  





Best,
Taewoo

On Wed, Dec 20, 2017 at 5:22 PM, Taewoo Kim <wangs...@gmail.com> wrote:

> @Murtadha: forgot to reply. Thank you so much!
>
> Best,
> Taewoo
>
> On Tue, Dec 19, 2017 at 11:53 PM, Murtadha Hubail <hubail...@gmail.com>
> wrote:
>
>> Hi Taewoo,
>>
>> The new argument to set is -Dlog4j.configurationFile and you need to
>> provide a log4j2 compatible configuration file. It is more or less similar
>> to logging.properties.
>> You can check [1] for more details about the configuration. We already
>> have a configuration file that you can use and modify under
>> asterix-app/src/test/resources/log4j2-test.xml. Changing the
>> configuration there should reflect the changes
>> on the tests. One thing to note is that log4j2 log levels are different
>> than java logging. You can check [2] for the mapping between the old and
>> the new levels.
>>
>> Cheers,
>> Murtadha
>>
>> [1] https://logging.apache.org/log4j/2.0/manual/configuration.html
>> [2] https://logging.apache.org/log4j/2.0/log4j-jul/index.html
>>
>> On 12/20/2017, 10:38 AM, "Taewoo Kim" <wangs...@gmail.com> wrote:
>>
>> Hello All,
>>
>> Not long time ago, for each test suite (e.g., OptimizerTest), we can
>> provide a custom log level property file (logging.properties) as a VM
>> option
>> (e.g., -Djava.util.logging.config.file=/.../asterixdb/asterixdb/ast
>> erix-app/src/test/resources/logging.properties)
>> and customize logging level for each phase (Hyracks, Algebricks, and
>> so
>> on). For example, if I set "org.apache.hyracks.algebricks.level =
>> FINE",
>> then only logging level for Algebricks is changed to FINE. It seems
>> that
>> this method doesn't work anymore. Could somebody tell me how we could
>> set a
>> custom logging level for each phase? Thanks!
>>
>> Best,
>> Taewoo
>>
>>
>>
>>
>

Re: Logging.properties

[Taewoo Kim]

@Murtadha: forgot to reply. Thank you so much!

Best,
Taewoo

On Tue, Dec 19, 2017 at 11:53 PM, Murtadha Hubail <hubail...@gmail.com>
wrote:

> Hi Taewoo,
>
> The new argument to set is -Dlog4j.configurationFile and you need to
> provide a log4j2 compatible configuration file. It is more or less similar
> to logging.properties.
> You can check [1] for more details about the configuration. We already
> have a configuration file that you can use and modify under
> asterix-app/src/test/resources/log4j2-test.xml. Changing the
> configuration there should reflect the changes
> on the tests. One thing to note is that log4j2 log levels are different
> than java logging. You can check [2] for the mapping between the old and
> the new levels.
>
> Cheers,
> Murtadha
>
> [1] https://logging.apache.org/log4j/2.0/manual/configuration.html
> [2] https://logging.apache.org/log4j/2.0/log4j-jul/index.html
>
> On 12/20/2017, 10:38 AM, "Taewoo Kim" <wangs...@gmail.com> wrote:
>
> Hello All,
>
> Not long time ago, for each test suite (e.g., OptimizerTest), we can
> provide a custom log level property file (logging.properties) as a VM
> option
> (e.g., -Djava.util.logging.config.file=/.../asterixdb/asterixdb/
> asterix-app/src/test/resources/logging.properties)
> and customize logging level for each phase (Hyracks, Algebricks, and so
> on). For example, if I set "org.apache.hyracks.algebricks.level =
> FINE",
> then only logging level for Algebricks is changed to FINE. It seems
> that
> this method doesn't work anymore. Could somebody tell me how we could
> set a
> custom logging level for each phase? Thanks!
>
> Best,
> Taewoo
>
>
>
>

Re: Primary key lookup plan

[Taewoo Kim]

Got it.

Best,
Taewoo

On Sun, Dec 3, 2017 at 5:15 PM, Wail Alkowaileet <wael@gmail.com> wrote:

> INT to BIGINT seems to work fine.
>
> The primary key type I have is a string (I was testing my rewrite rules) so
> i didn't pay attention to the type difference, and I was wondering why the
> select op is there.
>
> On Dec 3, 2017 16:36, "Taewoo Kim" <wangs...@gmail.com> wrote:
>
> > Hm... type-casting should happen in that case. You are saying providing
> an
> > INT to BIGINT primary index?
> >
> > Best,
> > Taewoo
> >
> > On Sun, Dec 3, 2017 at 4:20 PM, Wail Alkowaileet <wael@gmail.com>
> > wrote:
> >
> > > Got the issue...
> > > if the primary key type is not compatible with the predicate type ...
> it
> > > turns into a scan.
> > >
> > > Thanks Taewoo!
> > >
> > > On Sun, Dec 3, 2017 at 4:08 PM, Taewoo Kim <wangs...@gmail.com> wrote:
> > >
> > > > From Line 531
> > > > https://github.com/apache/asterixdb/blob/master/
> > > > asterixdb/asterix-algebra/src/main/java/org/apache/asterix/
> > > > optimizer/rules/am/BTreeAccessMethod.java
> > > >
> > > >
> > > > Best,
> > > > Taewoo
> > > >
> > > > On Sun, Dec 3, 2017 at 4:05 PM, Taewoo Kim <wangs...@gmail.com>
> wrote:
> > > >
> > > > > My understanding is that if a select condition can be covered by
> the
> > > > > primary key (i.e., only contains the primary key condition and
> B+Tree
> > > can
> > > > > be utilized), then only unnest-map should survive.
> > > > >
> > > > >
> > > > > Best,
> > > > > Taewoo
> > > > >
> > > > > On Sun, Dec 3, 2017 at 4:03 PM, Chen Luo <cl...@uci.edu> wrote:
> > > > >
> > > > >> I don't think it's the case...I tried on my local env, and it's
> > using
> > > a
> > > > >> primary index lookup instead of scan. Can you make sure the
> spelling
> > > of
> > > > >> the
> > > > >> primary key is correct?
> > > > >>
> > > > >> On Sun, Dec 3, 2017 at 3:49 PM, Wail Alkowaileet <
> > wael@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Devs,
> > > > >> >
> > > > >> > *For the given query:*
> > > > >> >
> > > > >> > SELECT VALUE t.text
> > > > >> > FROM ITweets as t
> > > > >> > WHERE t.tid = 100
> > > > >> >
> > > > >> > *The optimized plan:*
> > > > >> >
> > > > >> > distribute result [$$6]
> > > > >> > -- DISTRIBUTE_RESULT  |PARTITIONED|
> > > > >> >   exchange
> > > > >> >   -- ONE_TO_ONE_EXCHANGE  |PARTITIONED|
> > > > >> > project ([$$6])
> > > > >> > -- STREAM_PROJECT  |PARTITIONED|
> > > > >> >   assign [$$6] <- [$$t.getField("text")]
> > > > >> >   -- ASSIGN  |PARTITIONED|
> > > > >> > project ([$$t])
> > > > >> > -- STREAM_PROJECT  |PARTITIONED|
> > > > >> >   select (eq($$7, 100))
> > > > >> >   -- STREAM_SELECT  |PARTITIONED|
> > > > >> > exchange
> > > > >> > -- ONE_TO_ONE_EXCHANGE  |PARTITIONED|
> > > > >> >   data-scan []<-[$$7, $$t] <- FlatDataverse.ITweets
> > > > >> >   -- DATASOURCE_SCAN  |PARTITIONED|
> > > > >> > exchange
> > > > >> > -- ONE_TO_ONE_EXCHANGE  |PARTITIONED|
> > > > >> >   empty-tuple-source
> > > > >> >   -- EMPTY_TUPLE_SOURCE  |PARTITIONED|
> > > > >> >
> > > > >> > Do we always do a scan and then filter the result, even though
> the
> > > > query
> > > > >> > predicate is on the primary key?
> > > > >> > --
> > > > >> >
> > > > >> > *Regards,*
> > > > >> > Wail Alkowaileet
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > *Regards,*
> > > Wail Alkowaileet
> > >
> >
>

Re: Primary key lookup plan

[Taewoo Kim]

>From Line 531
https://github.com/apache/asterixdb/blob/master/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/am/BTreeAccessMethod.java


Best,
Taewoo

On Sun, Dec 3, 2017 at 4:05 PM, Taewoo Kim <wangs...@gmail.com> wrote:

> My understanding is that if a select condition can be covered by the
> primary key (i.e., only contains the primary key condition and B+Tree can
> be utilized), then only unnest-map should survive.
>
>
> Best,
> Taewoo
>
> On Sun, Dec 3, 2017 at 4:03 PM, Chen Luo <cl...@uci.edu> wrote:
>
>> I don't think it's the case...I tried on my local env, and it's using a
>> primary index lookup instead of scan. Can you make sure the spelling of
>> the
>> primary key is correct?
>>
>> On Sun, Dec 3, 2017 at 3:49 PM, Wail Alkowaileet <wael@gmail.com>
>> wrote:
>>
>> > Hi Devs,
>> >
>> > *For the given query:*
>> >
>> > SELECT VALUE t.text
>> > FROM ITweets as t
>> > WHERE t.tid = 100
>> >
>> > *The optimized plan:*
>> >
>> > distribute result [$$6]
>> > -- DISTRIBUTE_RESULT  |PARTITIONED|
>> >   exchange
>> >   -- ONE_TO_ONE_EXCHANGE  |PARTITIONED|
>> > project ([$$6])
>> > -- STREAM_PROJECT  |PARTITIONED|
>> >   assign [$$6] <- [$$t.getField("text")]
>> >   -- ASSIGN  |PARTITIONED|
>> > project ([$$t])
>> > -- STREAM_PROJECT  |PARTITIONED|
>> >   select (eq($$7, 100))
>> >   -- STREAM_SELECT  |PARTITIONED|
>> > exchange
>> > -- ONE_TO_ONE_EXCHANGE  |PARTITIONED|
>> >   data-scan []<-[$$7, $$t] <- FlatDataverse.ITweets
>> >   -- DATASOURCE_SCAN  |PARTITIONED|
>> > exchange
>> > -- ONE_TO_ONE_EXCHANGE  |PARTITIONED|
>> >   empty-tuple-source
>> >   -- EMPTY_TUPLE_SOURCE  |PARTITIONED|
>> >
>> > Do we always do a scan and then filter the result, even though the query
>> > predicate is on the primary key?
>> > --
>> >
>> > *Regards,*
>> > Wail Alkowaileet
>> >
>>
>
>

Re: Primary key lookup plan

[Taewoo Kim]

My understanding is that if a select condition can be covered by the
primary key (i.e., only contains the primary key condition and B+Tree can
be utilized), then only unnest-map should survive.


Best,
Taewoo

On Sun, Dec 3, 2017 at 4:03 PM, Chen Luo  wrote:

> I don't think it's the case...I tried on my local env, and it's using a
> primary index lookup instead of scan. Can you make sure the spelling of the
> primary key is correct?
>
> On Sun, Dec 3, 2017 at 3:49 PM, Wail Alkowaileet 
> wrote:
>
> > Hi Devs,
> >
> > *For the given query:*
> >
> > SELECT VALUE t.text
> > FROM ITweets as t
> > WHERE t.tid = 100
> >
> > *The optimized plan:*
> >
> > distribute result [$$6]
> > -- DISTRIBUTE_RESULT  |PARTITIONED|
> >   exchange
> >   -- ONE_TO_ONE_EXCHANGE  |PARTITIONED|
> > project ([$$6])
> > -- STREAM_PROJECT  |PARTITIONED|
> >   assign [$$6] <- [$$t.getField("text")]
> >   -- ASSIGN  |PARTITIONED|
> > project ([$$t])
> > -- STREAM_PROJECT  |PARTITIONED|
> >   select (eq($$7, 100))
> >   -- STREAM_SELECT  |PARTITIONED|
> > exchange
> > -- ONE_TO_ONE_EXCHANGE  |PARTITIONED|
> >   data-scan []<-[$$7, $$t] <- FlatDataverse.ITweets
> >   -- DATASOURCE_SCAN  |PARTITIONED|
> > exchange
> > -- ONE_TO_ONE_EXCHANGE  |PARTITIONED|
> >   empty-tuple-source
> >   -- EMPTY_TUPLE_SOURCE  |PARTITIONED|
> >
> > Do we always do a scan and then filter the result, even though the query
> > predicate is on the primary key?
> > --
> >
> > *Regards,*
> > Wail Alkowaileet
> >
>

Re: Strange error trying to run Asterix master

[Taewoo Kim]

What I did was:

1) Stop Eclipse
2) mvn clean install -DskipTests
3) Start Eclipse
4) Select all projects -> Maven -> Update Project

After this, it seems working fine at least for now.

Best,
Taewoo

On Tue, Oct 24, 2017 at 9:03 PM, Steven Jacobs <sjaco...@ucr.edu> wrote:

> I have never found a solution. I switched to IntelliJ :)
> Glad to learn I’m not crazy. It has to do with the unnamed generated
> classes that Eclipse creates (it went away when I created named java files
> for each class) but that’s as far as I got. If you find a solution let me
> know!
> Steven
>
> On Tue, Oct 24, 2017 at 8:10 PM Taewoo Kim <wangs...@gmail.com> wrote:
>
> > Have you found a solution? I just have encountered this issue. I'm trying
> > to rebuild.
> >
> > Best,
> > Taewoo
> >
> > On Wed, Oct 4, 2017 at 9:02 PM, Steven Jacobs <sjaco...@ucr.edu> wrote:
> >
> > > Hi Wail, I've tried this many times. This does solve another strange
> > > Eclipse error that I've had in the past, but it doesn't solve the one
> > that
> > > I'm seeing now (bad return type).
> > > Steven
> > >
> > > On Wed, Oct 4, 2017 at 8:44 PM, Wail Alkowaileet <wael@gmail.com>
> > > wrote:
> > >
> > > > If someone got into this again, the solution is:
> > > >
> > > > 1- Go to asterix-runtime
> > > > 2- mvn clean <-- must clean first.
> > > > 3- mvn -DskipTests install
> > > >
> > > > On Fri, Sep 29, 2017 at 3:26 PM, Chen Luo <cl...@uci.edu> wrote:
> > > >
> > > > > Hi Steven,
> > > > >
> > > > > I was using Eclipse (on mac) to debug AsterixDB's code, and things
> > work
> > > > > well for me. After switch to another branch, I think we need to run
> > > "mvn
> > > > > clean install" to rebuild class files, and in the meanwhile need to
> > > > refresh
> > > > > the workspace in Eclipse to rebuild things in Eclipse.
> > > > >
> > > > > Best regards,
> > > > > Chen Luo
> > > > >
> > > > > On Fri, Sep 29, 2017 at 11:56 AM, Steven Jacobs <sjaco...@ucr.edu>
> > > > wrote:
> > > > >
> > > > > > I'm on build build 1.8.0_65-b17. I've switched to Intellij and
> the
> > > > > problem
> > > > > > doesn't occur there so it seems to be related to Eclipse
> > > specifically.
> > > > > > Steven
> > > > > >
> > > > > > On Thu, Sep 28, 2017 at 10:22 PM, Michael Blow <
> > > mblow.apa...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > What JVM is this?  Try Oracle latest Java 8 if not already
> using.
> > > > > > >
> > > > > > > -MDB
> > > > > > >
> > > > > > > On Fri, Sep 29, 2017 at 12:37 AM Steven Jacobs <
> sjaco...@ucr.edu
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > If only that worked for me :( I have even tried deleting the
> m2
> > > > > > > repository
> > > > > > > > cache completely.
> > > > > > > > Steven
> > > > > > > >
> > > > > > > > On Thu, Sep 28, 2017 at 8:19 PM Wail Alkowaileet <
> > > > wael@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I got the same issue before. I did "clean project" and the
> > > issue
> > > > > > seems
> > > > > > > to
> > > > > > > > > be resolved.
> > > > > > > > >
> > > > > > > > > On Thu, Sep 28, 2017 at 2:26 PM, Steven Jacobs <
> > > sjaco...@ucr.edu
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > > I've been stuck for some time now trying to get master
> > > Asterix
> > > > to
> > > > > > > debug
> > > > > > > > > > correctly for me in Eclipse on my machine. It seems to
> deal
> > > > with
> > > > > > the
> > > > > > > > >

Re: Strange error trying to run Asterix master

[Taewoo Kim]

Have you found a solution? I just have encountered this issue. I'm trying
to rebuild.

Best,
Taewoo

On Wed, Oct 4, 2017 at 9:02 PM, Steven Jacobs  wrote:

> Hi Wail, I've tried this many times. This does solve another strange
> Eclipse error that I've had in the past, but it doesn't solve the one that
> I'm seeing now (bad return type).
> Steven
>
> On Wed, Oct 4, 2017 at 8:44 PM, Wail Alkowaileet 
> wrote:
>
> > If someone got into this again, the solution is:
> >
> > 1- Go to asterix-runtime
> > 2- mvn clean <-- must clean first.
> > 3- mvn -DskipTests install
> >
> > On Fri, Sep 29, 2017 at 3:26 PM, Chen Luo  wrote:
> >
> > > Hi Steven,
> > >
> > > I was using Eclipse (on mac) to debug AsterixDB's code, and things work
> > > well for me. After switch to another branch, I think we need to run
> "mvn
> > > clean install" to rebuild class files, and in the meanwhile need to
> > refresh
> > > the workspace in Eclipse to rebuild things in Eclipse.
> > >
> > > Best regards,
> > > Chen Luo
> > >
> > > On Fri, Sep 29, 2017 at 11:56 AM, Steven Jacobs 
> > wrote:
> > >
> > > > I'm on build build 1.8.0_65-b17. I've switched to Intellij and the
> > > problem
> > > > doesn't occur there so it seems to be related to Eclipse
> specifically.
> > > > Steven
> > > >
> > > > On Thu, Sep 28, 2017 at 10:22 PM, Michael Blow <
> mblow.apa...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > What JVM is this?  Try Oracle latest Java 8 if not already using.
> > > > >
> > > > > -MDB
> > > > >
> > > > > On Fri, Sep 29, 2017 at 12:37 AM Steven Jacobs 
> > > wrote:
> > > > >
> > > > > > If only that worked for me :( I have even tried deleting the m2
> > > > > repository
> > > > > > cache completely.
> > > > > > Steven
> > > > > >
> > > > > > On Thu, Sep 28, 2017 at 8:19 PM Wail Alkowaileet <
> > wael@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I got the same issue before. I did "clean project" and the
> issue
> > > > seems
> > > > > to
> > > > > > > be resolved.
> > > > > > >
> > > > > > > On Thu, Sep 28, 2017 at 2:26 PM, Steven Jacobs <
> sjaco...@ucr.edu
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > > I've been stuck for some time now trying to get master
> Asterix
> > to
> > > > > debug
> > > > > > > > correctly for me in Eclipse on my machine. It seems to deal
> > with
> > > > the
> > > > > > > class
> > > > > > > > generation being done by maven, but that's as far as I can
> see
> > so
> > > > > far.
> > > > > > No
> > > > > > > > one I've talked to has a similar issue, so I was wondering if
> > > > anyone
> > > > > > from
> > > > > > > > the community at large has had such an issue. It manifests
> > itself
> > > > > when
> > > > > > > > tying to create the evaluator for a function. The stack trace
> > is
> > > > > > attached
> > > > > > > > below. If anyone has seen such an issue, I would love to get
> > any
> > > > > advice
> > > > > > > you
> > > > > > > > may have.
> > > > > > > >
> > > > > > > >
> > > > > > > > java.lang.VerifyError: Bad return type
> > > > > > > >
> > > > > > > > Exception Details:
> > > > > > > >
> > > > > > > >   Location:
> > > > > > > >
> > > > > > > >
> > > > > > > > org/apache/asterix/runtime/evaluators/functions/records/
> > > > > > > > RecordMergeDescriptor$_EvaluatorFactoryGen.access$0(
> > > > > > > > Lorg/apache/asterix/runtime/evaluators/functions/records/
> > > > > > > >
> > > > > > RecordMergeDescriptor$_EvaluatorFactoryGen;)Lorg/
> > > > apache/asterix/runtime/
> > > > > > > > evaluators/functions/records/RecordMergeDescriptor;
> > > > > > > > @4: areturn
> > > > > > > >
> > > > > > > >   Reason:
> > > > > > > >
> > > > > > > > Type
> > > > > > > > 'org/apache/asterix/runtime/evaluators/functions/records/
> > > > > > > > RecordMergeDescriptor$_Gen'
> > > > > > > > (current frame, stack[0]) is not assignable to
> > > > > > > > 'org/apache/asterix/runtime/evaluators/functions/records/
> > > > > > > > RecordMergeDescriptor'
> > > > > > > > (from method signature)
> > > > > > > >
> > > > > > > >   Current Frame:
> > > > > > > >
> > > > > > > > bci: @4
> > > > > > > >
> > > > > > > > flags: { }
> > > > > > > >
> > > > > > > > locals: {
> > > > > > > > 'org/apache/asterix/runtime/evaluators/functions/records/
> > > > > > > > RecordMergeDescriptor$_EvaluatorFactoryGen'
> > > > > > > > }
> > > > > > > >
> > > > > > > > stack: {
> > > > > > > > 'org/apache/asterix/runtime/evaluators/functions/records/
> > > > > > > > RecordMergeDescriptor$_Gen'
> > > > > > > > }
> > > > > > > >
> > > > > > > >   Bytecode:
> > > > > > > >
> > > > > > > > 0x000: 2ab4 0063 b0
> > > > > > > >
> > > > > > > >
> > > > > > > > at
> > > > > > > > org.apache.asterix.runtime.evaluators.functions.records.
> > > > > > > > RecordMergeDescriptor$_Gen.createEvaluatorFactory(
> > > > > > > > RecordMergeDescriptor.java:86)
> > > > > > > >
> > > > > > > > at
> > > > 

Re: Is it OK for Hyracks to access the config option?

[Taewoo Kim]

Never mind about the first question. The problem becomes simpler (but still
couldn't find the solution) after talking to Chen Luo. Thanks.

Let's say the inverted index search module Blah1 (Actually SearchResult and
FixedSizeElementInvertedListCursor in case you are interested in). For this
Blah1 module to generate intermediate and final result work properly, the
caller needs to pass three parameters (frameSize, how to
allocate/deallocate buffers, and the maximum number of frames that Blah1
can have). Blah1 can be called in two ways.

1) Search using HyracksTaskCotext (class name: Task)
2) Search initiated by merge (class name: AsynchronousScheduler)

1) and 2) are independent processes so it looks like hard to associate 1)
in 2) or 2) in 1). So, for Blah1, it needs to consider 1) as default and 2)
as a special case or vice versa. Any suggestions? Again, for B+ tree, R
tree, this problem doesn't exist since there is no intermediate result
calculation.



Best,
Taewoo

On Tue, Oct 24, 2017 at 9:21 AM, Taewoo Kim <wangs...@gmail.com> wrote:

> Hello All,
>
> Here is the background. I'm currently fixing an inverted-index search OOM
> issue. This could happen because the core inverted-index search module does
> not have any budget limit. One might ask that B+-tree and R-tree search
> doesn't have any budget limit either. However, a difference is that an
> inverted-index has an inverted list (list of primary keys per secondary
> key). That is, for a single PK, there can be multiple secondary keys. So,
> dealing with this requires additional frames. So, let's assume that we need
> to have multiple frames to answer a text query that utilizes an inverted
> index.
>
> Here is my issue. We will add one more parameter to the config (e.g.,
> compiler.textsearchmemory = 32MB) to control this behavior. The
> index-search core module needs to receive and conform to this budget
> information. For inverted-index search query case, it's relatively easy. I
> can use HyracksTaskContext to pass that since there's a pipeline from
> compiler to the search module (IndexSearchOperatorNodePushable). However,
> I found an issue regarding the merge case. Merge of an LSM index happens in
> Hyracks codebase and index merge code doesn't know about
> HyracksTaskContext. So, I talked to Abdullah and he shared a hint about
> ServiceContext. In the ServiceContext, there is AppConfig interface and
> this contains all config information. And when creating an LSM index
> instance, this ServiceContext is accessible.
>
> So, my final question is: is it OK to access AppConfig from Hyracks code
> by specifying the exact parameter (e.g., compiler.textsearchmemory) to get
> some information? Technically, it's doable since the code exists in
> Hyracks. However, most similar parameters are processed in Asterix
> codebase, not in  Hyracks codebase. If not, what could be a way to ensure
> this?
>
> Thanks for reading!
>
> Best,
> Taewoo
>

Is it OK for Hyracks to access the config option?

[Taewoo Kim]

Hello All,

Here is the background. I'm currently fixing an inverted-index search OOM
issue. This could happen because the core inverted-index search module does
not have any budget limit. One might ask that B+-tree and R-tree search
doesn't have any budget limit either. However, a difference is that an
inverted-index has an inverted list (list of primary keys per secondary
key). That is, for a single PK, there can be multiple secondary keys. So,
dealing with this requires additional frames. So, let's assume that we need
to have multiple frames to answer a text query that utilizes an inverted
index.

Here is my issue. We will add one more parameter to the config (e.g.,
compiler.textsearchmemory = 32MB) to control this behavior. The
index-search core module needs to receive and conform to this budget
information. For inverted-index search query case, it's relatively easy. I
can use HyracksTaskContext to pass that since there's a pipeline from
compiler to the search module (IndexSearchOperatorNodePushable). However, I
found an issue regarding the merge case. Merge of an LSM index happens in
Hyracks codebase and index merge code doesn't know about
HyracksTaskContext. So, I talked to Abdullah and he shared a hint about
ServiceContext. In the ServiceContext, there is AppConfig interface and
this contains all config information. And when creating an LSM index
instance, this ServiceContext is accessible.

So, my final question is: is it OK to access AppConfig from Hyracks code by
specifying the exact parameter (e.g., compiler.textsearchmemory) to get
some information? Technically, it's doable since the code exists in
Hyracks. However, most similar parameters are processed in Asterix
codebase, not in  Hyracks codebase. If not, what could be a way to ensure
this?

Thanks for reading!

Best,
Taewoo

Re: Integration tests failing

[Taewoo Kim]

I simply skimmed errors. It looks like some of RebalanceWithCancellationIT
tests generate errors [1].  This patch set  [2] would be a good starting
point to understand. At least somebody who reviewed the patch set could
give more idea about this integration test to give some idea for a possible
volunteer? :-)

[1]
https://asterix-jenkins.ics.uci.edu/job/asterix-integration-tests/org.apache.asterix$asterix-app/905/testReport/
[2] https://asterix-gerrit.ics.uci.edu/#/c/1821/

Best,
Taewoo

On Thu, Oct 12, 2017 at 8:53 AM, Till Westmann  wrote:

> Hi,
>
> it seems that the integration tests [1] have been failing forever (with
> "forever" being the history that Jenkins keeps - right now the last run
> is from Sep 25).
>
> I was wondering if somebody would volunteer to look at the failures in
> more detail and maybe categorize the failures to formulate a plan for
> fixing them?
>
> Cheers,
> Till
>
> [1] https://asterix-jenkins.ics.uci.edu/job/asterix-integration-tests
>

Re: Time to deprecate AQL?

[Taewoo Kim]

For similarity join, we use AQL+ that is based on AQL. I think deprecating
(not removing) AQL is OK. Ultimately, AQL+ should be converted to SQL++ :-)

Best,
Taewoo

On Thu, Sep 7, 2017 at 9:04 PM, Steven Jacobs  wrote:

> I’ll give the BADest +1 I can :)
> Steven
>
> On Thu, Sep 7, 2017 at 8:50 PM Gerald Sangudi  wrote:
>
> > :-)
> >
> > On Sep 7, 2017 11:44 AM, "Michael Carey"  wrote:
> >
> > As AsterixDB evolves, and additional features are added - e.g., DISTINCT
> > aggregate support, or properly implemented query-bodied functions,
> > supporting two query languages is hugely expensive:  Updating two
> grammars,
> > parsers, rules, tests, ... IMO it is time to let go of AQL as an
> externally
> > supported interface to AsterixDB and only have SQL++ going forward.  I
> > think "everyone" has migrated - and if not we should force that
> migration.
> > (Cloudberry is on SQL++ nowadays, BAD is on SQL++ nowadays, ...)  Any
> > objections?  If not, I think we should make this decision officially and
> > stop putting energy into carrying the AQL legacy around with us.
> Thoughts?
> >
>

Re: ILogicalOperator.isMap

[Taewoo Kim]

My understanding is that isMap() is set to true when the cardinality of the
input may not be equal to the the cardinality of the output. To settle this
issue, I come up with one explicit property in my Limit Push down branch.
The attribute is called canDecreaseCardinality.

On Sun, Sep 3, 2017 at 19:46 abdullah alamoudi  wrote:

> Dear devs,
> I was going over implementations of ILogicalOperator.isMap in our code
> base and here is the breakdown:
>
> isMap()
>
> true:
> -Assign
> -AbstractReplicateOperator
> -AbstractUnnestOperator
> -ExchangeOperator
> -InsertDeleteUpsertOperator
> -LimitOperator
> -ProjectOperator
> -SelectOperator
>
>
>
> false:
> -Order
> -Aggregate
> -RunningAggregate
> -AbstractBinaryJoin
> -AbstractOperatorWithNestedPlans
> -DataSourceScanOperator
> -DistinctOperator
> -DistributeResultOperator
> -EmptyTupleSourceOperator
> -IndexInsertDeleteUpsertOperator
> -IntersectOperator
> -MaterializeOperator
> -NestedTupleSourceOperator
> -ScriptOperator
> -SinkOperator
> -TokenizeOperator
> -UnionAllOperator
> -WriteOperator
> -WriteResultOperator
>
> As I understood at some point is that isMap means (does this operator map
> input tuples to output tuples without blocking?). I am not sure how
> accurate this description is but it clearly introduces a problem as:
>
> * InsertDeleteUpsertOperator.isMap() -> true
> * IndexInsertDeleteUpsertOperator.isMap() -> false
>
> Moreover,
> TokenizeOperator.isMap() -> false
> AbstractUnnestOperator.isMap() -> true
>
> while clearly, as per the definition above, these two operators should
> have the same output: true
>
> Can someone solve the mystery?
> Once this is solved, I will add a wiki entry to explain this flag.
>
> Cheers,
> Abdullah.

Re: Entity locks in asterixdb

[Taewoo Kim]

Sure. I will take a look after my break. Thanks.

On Sun, Sep 3, 2017 at 19:38 abdullah alamoudi  wrote:

> Hi All,
> I have created a wiki entry for entity level locks in asterixdb at
> https://cwiki.apache.org/confluence/display/ASTERIXDB/Entity+locks+in+AsterixDB
> <
> https://cwiki.apache.org/confluence/display/ASTERIXDB/Entity+locks+in+AsterixDB
> >
> It is pretty primitive at this stage but with your contribution, it can be
> improved quite a bit.
> The goal of this entry is to act as a reference to our locking for
> developers, testers, and users as well. We would like to use it as the
> basis as we're adding unit tests to locking within different cursors as
> we're adding unit tests to those.
>
> Special mention to @Taewoo to take a look :)
>
> Cheers,
> ~Abdullah.

Re: [VOTE] Release Apache AsterixDB 0.9.2 and Hyracks 0.3.2 (RC1)

[Taewoo Kim]

+1

- Verified the signatures
- Compiled the source codes
- Executed the sample cluster and sent some queries


Best,
Taewoo

On Fri, Aug 18, 2017 at 5:03 PM, Ildar Absalyamov <
ildar.absalya...@gmail.com> wrote:

> Sorry, that was a problem in my local dev version.
> +1 in this case.
>
> > On Aug 18, 2017, at 16:35, Ildar Absalyamov 
> wrote:
> >
> > -1
> >
> > Source artifacts:
> > - Verified signatures and hashes
> > - Verified LICENSE and NOTICE
> > - Verified compilation
> > - Verified license headers
> >
> > Binary artifacts:
> > - Verified Managix installer
> > - Went thought SQL++ Primer and noticed that the wed UI does not print
> results correctly with in JSON Output format (which is a default option!).
> The result is empty with "application/json” Accept header using
> http://localhost:19002/query  endpoint as
> well as http://localhost:19002/query/service <
> http://localhost:19002/query/service>.
> > Turns out we already have an issue for that [1]. Feels like this issue
> is a blocker for the RC.
> >
> > Also the source Asterix artifact has pom.xml.versionsBackup for each
> project (leftover from SNAPSHOT version pom), but I guess that does not
> affect he overall validity.
> >
> > [1] https://issues.apache.org/jira/browse/ASTERIXDB-1977 <
> https://issues.apache.org/jira/browse/ASTERIXDB-1977>
> >> On Aug 18, 2017, at 13:15, Ian Maxon > wrote:
> >>
> >> I've replaced the zip in the dist with the one in the maven repository
> >> and the signature checks out now for me, at least. Also note the link
> >> for the repository is off, it should be
> >> https://repository.apache.org/content/repositories/
> orgapacheasterix-1038/  org/content/repositories/orgapacheasterix-1038/>
> >> , sorry about that.
> >>
> >> On Thu, Aug 17, 2017 at 12:42 PM, Ian Maxon  wrote:
> >>> Hmm. Very interesting. I'll have to investigate. Somehow the zip must
> >>> have become corrupt.
> >>>
> >>> On Thu, Aug 17, 2017 at 11:24 AM, Preston Carman 
> wrote:
>  -1
> 
>  I went through checking the signature and SHA1 for all the files. All
>  but one are correct.
> 
>  Asterix Installer (asterix-installer-0.9.2-binary-assembly.zip) has a
>  BAD signature and does not match the SHA1.
> 
>  On Fri, Aug 11, 2017 at 2:56 PM, Mike Carey 
> wrote:
> > +1 from me on the release.  I grabbed the NCService version and
> > installed/ran it and went through the SQL++ Primer on it to make
> sure it all
> > works as advertised (including checking that the hinted queries
> indeed had
> > the hinted query plans).
> >
> >
> >
> > On 8/7/17 5:50 PM, Ian Maxon wrote:
> >>
> >> Hi everyone,
> >>
> >> Please verify and vote on the 3rd release of Apache AsterixDB
> >>
> >> The change that produced this release and the change to advance the
> >> version are
> >> up for review here:
> >>
> >> https://asterix-gerrit.ics.uci.edu/#/c/1924/
> >> https://asterix-gerrit.ics.uci.edu/#/c/1925/
> >>
> >> To check out the release, simply fetch the review and check out the
> >> fetch head like so:
> >>
> >> git fetch https://asterix-gerrit.ics.uci.edu:29418/asterixdb
> >> refs/changes/24/1924/1 && git checkout FETCH_HEAD
> >>
> >>
> >> AsterixDB Source
> >>
> >> https://dist.apache.org/repos/dist/dev/asterixdb/apache-
> asterixdb-0.9.2-source-release.zip
> >>
> >> https://dist.apache.org/repos/dist/dev/asterixdb/apache-
> asterixdb-0.9.2-source-release.zip.asc
> >>
> >> https://dist.apache.org/repos/dist/dev/asterixdb/apache-
> asterixdb-0.9.2-source-release.zip.sha1
> >>
> >> SHA1:36fae3394755e86d97540b892cda6b80ee02a770
> >>
> >> Hyracks Source
> >>
> >> https://dist.apache.org/repos/dist/dev/asterixdb/apache-
> hyracks-0.3.2-source-release.zip
> >>
> >> https://dist.apache.org/repos/dist/dev/asterixdb/apache-
> hyracks-0.3.2-source-release.zip.asc
> >>
> >> https://dist.apache.org/repos/dist/dev/asterixdb/apache-
> hyracks-0.3.2-source-release.zip.sha1
> >>
> >> SHA1:1984e031570ece528c7bf860c19246870b01b431
> >>
> >> AsterixDB NCService Installer:
> >>
> >> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-
> server-0.9.2-binary-assembly.zip
> >>
> >> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-
> server-0.9.2-binary-assembly.zip.asc
> >>
> >> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-
> server-0.9.2-binary-assembly.zip.sha1
> >>
> >> SHA1: f9e78dbad5a4ae3626c9ada7ae9c2c3160174fc1
> >>
> >> AsterixDB Managix Installer
> >>
> >> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-
> installer-0.9.2-binary-assembly.zip
> >>
> >> 

Re: Nested type + open-enforced-index question.

[Taewoo Kim]

Agreed.

Best,
Taewoo

On Fri, Jul 14, 2017 at 4:09 PM, Yingyi Bu <buyin...@gmail.com> wrote:

> >> When we are encounter a field (“nested”) for which the is no
> compile-time information
> >> we should assume that the type of this field is completely open, i.e.,
> {}, and pass it down the chain.
>
> Correct, since it's enforced.
> The augmented enforced type maps should be recursively added into those
> nested ARecordType.
>
> Best,
> Yingyi
>
>
> On Fri, Jul 14, 2017 at 12:13 AM, Ildar Absalyamov <
> ildar.absalya...@gmail.com> wrote:
>
> > However, there should be a way to deal with this issue when the top-level
> > type is open.
> >
> > create type DBLPType as open {id: int32}
> > create index title_index_DBLP on DBLP(nested.one.title: string?)
> enforced;
> >
> > When we are encounter a field (“nested”) for which the is no compile-time
> > information we should assume that the type of this field is completely
> > open, i.e., {}, and pass it down the chain.
> >
> > > On Jul 14, 2017, at 00:09, Ildar Absalyamov <
> ildar.absalya...@gmail.com>
> > wrote:
> > >
> > > Taewoo,
> > >
> > > You’ve correctly identified the issue here: to make use of an enforced
> > index we must cast the record to a particular type, which is imposed by
> the
> > index.
> > >
> > > So, using your example, if we have an index on path “nested.one.title”
> > the indexed record must be castable to {…, “nested”: {…,”one”:
> {…,”title”:
> > string, …}, ...},…}.
> > > As you have observed a case when there is no “nested” field in the
> > top-level type leads to exception, because it relies of a fact that there
> > is a compile-time type information for a field “nested”. This type
> > information is used to build a type for aforementioned cast operator.
> > > Form the perspective of current implementation a runtime exception is a
> > bug, instead it should have caught this issue during compile time.
> > >
> > >> On Jul 13, 2017, at 23:10, Taewoo Kim <wangs...@gmail.com> wrote:
> > >>
> > >> @Yingyi: thanks.
> > >>
> > >> @Mike: Yeah. My problem is how to associate the field type
> information.
> > >> Ideally, the leaf level has the field to type hash map and the parent
> > of it
> > >> has that hashmap in its record type. And its parent needs to have the
> > >> necessary information to reach to this record type. If we don't need
> any
> > >> pre-defined type at each level to create a multi-level enforced index,
> > then
> > >> things will become more complex to me. :-) Anyway, we can discuss
> > further
> > >> to finalize the field type propagation implementation.
> > >>
> > >> Best,
> > >> Taewoo
> > >>
> > >> On Thu, Jul 13, 2017 at 11:02 PM, Mike Carey <dtab...@gmail.com>
> wrote:
> > >>
> > >>> Taewoo,
> > >>>
> > >>> To clarify further what should work:
> > >>> - We should support nested indexes that go down multiple levels.
> > >>> - We should (ideally) support their use in index-NL joins.
> > >>>
> > >>> Reflecting on our earlier conversation(s), I think I can see why
> you're
> > >>> asking this. :-) The augmented type information that'll be needed to
> do
> > >>> this completely/properly will actually have to associate types with
> > field
> > >>> paths (not just with fields by name) - which is a slightly more
> > complicated
> > >>> association.
> > >>>
> > >>> Cheers,
> > >>> Mike
> > >>>
> > >>>
> > >>> On 7/13/17 10:54 PM, Yingyi Bu wrote:
> > >>>
> > >>>> Hi Taewoo,
> > >>>>
> > >>>> The first query shouldn't fail because indexnl is just a hint.
> > >>>> The second query should succeed because it's a valid indexing
> > statement.
> > >>>> High nesting levels in open record like JSON is not uncommon.
> > >>>>
> > >>>> Best,
> > >>>> Yingyi
> > >>>>
> > >>>>
> > >>>> On Thu, Jul 13, 2017 at 10:51 PM, Taewoo Kim <wangs...@gmail.com>
> > wrote:
> > >>>>
> > >>>> @Mike: In order to properly deal with the enforced index on a
> > nested-type
> > >>>&

Re: Nested type + open-enforced-index question.

[Taewoo Kim]

@Ildar: Yes. The current implementation requires that. So, I asked whether
which one makes sense.

Best,
Taewoo

On Fri, Jul 14, 2017 at 12:09 AM, Ildar Absalyamov <
ildar.absalya...@gmail.com> wrote:

> Taewoo,
>
> You’ve correctly identified the issue here: to make use of an enforced
> index we must cast the record to a particular type, which is imposed by the
> index.
>
> So, using your example, if we have an index on path “nested.one.title” the
> indexed record must be castable to {…, “nested”: {…,”one”: {…,”title”:
> string, …}, ...},…}.
> As you have observed a case when there is no “nested” field in the
> top-level type leads to exception, because it relies of a fact that there
> is a compile-time type information for a field “nested”. This type
> information is used to build a type for aforementioned cast operator.
> Form the perspective of current implementation a runtime exception is a
> bug, instead it should have caught this issue during compile time.
>
> > On Jul 13, 2017, at 23:10, Taewoo Kim <wangs...@gmail.com> wrote:
> >
> > @Yingyi: thanks.
> >
> > @Mike: Yeah. My problem is how to associate the field type information.
> > Ideally, the leaf level has the field to type hash map and the parent of
> it
> > has that hashmap in its record type. And its parent needs to have the
> > necessary information to reach to this record type. If we don't need any
> > pre-defined type at each level to create a multi-level enforced index,
> then
> > things will become more complex to me. :-) Anyway, we can discuss further
> > to finalize the field type propagation implementation.
> >
> > Best,
> > Taewoo
> >
> > On Thu, Jul 13, 2017 at 11:02 PM, Mike Carey <dtab...@gmail.com> wrote:
> >
> >> Taewoo,
> >>
> >> To clarify further what should work:
> >> - We should support nested indexes that go down multiple levels.
> >> - We should (ideally) support their use in index-NL joins.
> >>
> >> Reflecting on our earlier conversation(s), I think I can see why you're
> >> asking this. :-) The augmented type information that'll be needed to do
> >> this completely/properly will actually have to associate types with
> field
> >> paths (not just with fields by name) - which is a slightly more
> complicated
> >> association.
> >>
> >> Cheers,
> >> Mike
> >>
> >>
> >> On 7/13/17 10:54 PM, Yingyi Bu wrote:
> >>
> >>> Hi Taewoo,
> >>>
> >>> The first query shouldn't fail because indexnl is just a hint.
> >>> The second query should succeed because it's a valid indexing
> statement.
> >>> High nesting levels in open record like JSON is not uncommon.
> >>>
> >>> Best,
> >>> Yingyi
> >>>
> >>>
> >>> On Thu, Jul 13, 2017 at 10:51 PM, Taewoo Kim <wangs...@gmail.com>
> wrote:
> >>>
> >>> @Mike: In order to properly deal with the enforced index on a
> nested-type
> >>>> field, I need to make sure that whether my understanding (each nested
> >>>> type
> >>>> (except the leaf level0 has a record type for the next level) is
> correct
> >>>> or
> >>>> not. Which one is a bug? The first one (without index) should fail? Or
> >>>> the
> >>>> second one (with an index) should succeed?
> >>>>
> >>>> Best,
> >>>> Taewoo
> >>>>
> >>>> On Thu, Jul 13, 2017 at 9:58 PM, Yingyi Bu <buyin...@gmail.com>
> wrote:
> >>>>
> >>>> Indeed, it's a bug!
> >>>>>
> >>>>> Best,
> >>>>> Yingyi
> >>>>>
> >>>>> On Thu, Jul 13, 2017 at 9:52 PM, Mike Carey <dtab...@gmail.com>
> wrote:
> >>>>>
> >>>>> Sounds like a bug to me.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 7/13/17 7:59 PM, Taewoo Kim wrote:
> >>>>>>
> >>>>>> Currently, I am working on a field type propagation without using
> >>>>>>> initializing the OptimizableSubTree in the current index access
> >>>>>>>
> >>>>>> method.
> >>>>
> >>>>> I
> >>>>>
> >>>>>> am encountering an issue with an open-type enforced index. So, I
> just
> >>>>>>>
> >>>>>

Re: Nested type + open-enforced-index question.

[Taewoo Kim]

@Ildar: you can change "create type CSXType as closed {id: int32}" to "create
type CSXType as *open* {id: int32}". My intention was that.


Best,
Taewoo

On Fri, Jul 14, 2017 at 12:06 AM, Ildar Absalyamov <
ildar.absalya...@gmail.com> wrote:

> Maybe I missed something, but how nested access on a closed type without a
> proper nested field is ever valid?
>
> create type CSXType as closed {id: int32}
> create index title_index_CSX on CSX(nested.one.title: string?) enforced;
>
> Will this index every be anything but empty?
>
> for $a in dataset('DBLP')
> for $b in dataset('CSX')
> where $a.nested.one.title /*+ indexnl */ = $b.nested.one.title
> return {"arec": $a, "brec": $b}
>
> Will this query return anything, but empty result?
>
> To me it feels like that should be compile time error in both cases:
> during index DDL and during the query.
>
> > On Jul 13, 2017, at 22:51, Taewoo Kim <wangs...@gmail.com> wrote:
> >
> > @Mike: In order to properly deal with the enforced index on a nested-type
> > field, I need to make sure that whether my understanding (each nested
> type
> > (except the leaf level0 has a record type for the next level) is correct
> or
> > not. Which one is a bug? The first one (without index) should fail? Or
> the
> > second one (with an index) should succeed?
> >
> > Best,
> > Taewoo
> >
> > On Thu, Jul 13, 2017 at 9:58 PM, Yingyi Bu <buyin...@gmail.com> wrote:
> >
> >> Indeed, it's a bug!
> >>
> >> Best,
> >> Yingyi
> >>
> >> On Thu, Jul 13, 2017 at 9:52 PM, Mike Carey <dtab...@gmail.com> wrote:
> >>
> >>> Sounds like a bug to me.
> >>>
> >>>
> >>>
> >>> On 7/13/17 7:59 PM, Taewoo Kim wrote:
> >>>
> >>>> Currently, I am working on a field type propagation without using
> >>>> initializing the OptimizableSubTree in the current index access
> method.
> >> I
> >>>> am encountering an issue with an open-type enforced index. So, I just
> >> want
> >>>> to make sure that my understanding is correct. It looks like we can't
> >> have
> >>>> an enforced-index on a completely schemaless nested field. For
> example,
> >>>> the
> >>>> following doesn't generate any issue.
> >>>>
> >>>> //
> >>>> create type DBLPType as open {id: int32}
> >>>> create type CSXType as closed {id: int32}
> >>>>
> >>>> create dataset DBLP(DBLPType) primary key id;
> >>>> create dataset CSX(CSXType) primary key id;
> >>>>
> >>>> for $a in dataset('DBLP')
> >>>> for $b in dataset('CSX')
> >>>> where $a.nested.one.title /*+ indexnl */ = $b.nested.one.title
> >>>> return {"arec": $a, "brec": $b}
> >>>> //
> >>>>
> >>>> However, the following generates an exception. So, can we assume that
> to
> >>>> create an enforced-index, except the leaf level, there should be a
> >> defined
> >>>> record type. For example, for this example, there should be "nested"
> >> type
> >>>> and "one" type.
> >>>>
> >>>> //
> >>>> create type DBLPType as open {id: int32}
> >>>> create type CSXType as closed {id: int32}
> >>>>
> >>>> create dataset DBLP(DBLPType) primary key id;
> >>>> create dataset CSX(CSXType) primary key id;
> >>>>
> >>>> create index title_index_DBLP on DBLP(nested.one.title: string?)
> >> enforced;
> >>>> create index title_index_CSX on CSX(nested.one.title: string?)
> enforced;
> >>>>
> >>>> for $a in dataset('DBLP')
> >>>> for $b in dataset('CSX')
> >>>> where $a.nested.one.title /*+ indexnl */ = $b.nested.one.title
> >>>> return {"arec": $a, "brec": $b}
> >>>> //
> >>>>
> >>>> Best,
> >>>> Taewoo
> >>>>
> >>>>
> >>>
> >>
>
> Best regards,
> Ildar
>
>

Re: Nested type + open-enforced-index question.

[Taewoo Kim]

@Yingyi: thanks.

@Mike: Yeah. My problem is how to associate the field type information.
Ideally, the leaf level has the field to type hash map and the parent of it
has that hashmap in its record type. And its parent needs to have the
necessary information to reach to this record type. If we don't need any
pre-defined type at each level to create a multi-level enforced index, then
things will become more complex to me. :-) Anyway, we can discuss further
to finalize the field type propagation implementation.

Best,
Taewoo

On Thu, Jul 13, 2017 at 11:02 PM, Mike Carey <dtab...@gmail.com> wrote:

> Taewoo,
>
> To clarify further what should work:
>  - We should support nested indexes that go down multiple levels.
>  - We should (ideally) support their use in index-NL joins.
>
> Reflecting on our earlier conversation(s), I think I can see why you're
> asking this. :-) The augmented type information that'll be needed to do
> this completely/properly will actually have to associate types with field
> paths (not just with fields by name) - which is a slightly more complicated
> association.
>
> Cheers,
> Mike
>
>
> On 7/13/17 10:54 PM, Yingyi Bu wrote:
>
>> Hi Taewoo,
>>
>> The first query shouldn't fail because indexnl is just a hint.
>> The second query should succeed because it's a valid indexing statement.
>> High nesting levels in open record like JSON is not uncommon.
>>
>> Best,
>> Yingyi
>>
>>
>> On Thu, Jul 13, 2017 at 10:51 PM, Taewoo Kim <wangs...@gmail.com> wrote:
>>
>> @Mike: In order to properly deal with the enforced index on a nested-type
>>> field, I need to make sure that whether my understanding (each nested
>>> type
>>> (except the leaf level0 has a record type for the next level) is correct
>>> or
>>> not. Which one is a bug? The first one (without index) should fail? Or
>>> the
>>> second one (with an index) should succeed?
>>>
>>> Best,
>>> Taewoo
>>>
>>> On Thu, Jul 13, 2017 at 9:58 PM, Yingyi Bu <buyin...@gmail.com> wrote:
>>>
>>> Indeed, it's a bug!
>>>>
>>>> Best,
>>>> Yingyi
>>>>
>>>> On Thu, Jul 13, 2017 at 9:52 PM, Mike Carey <dtab...@gmail.com> wrote:
>>>>
>>>> Sounds like a bug to me.
>>>>>
>>>>>
>>>>>
>>>>> On 7/13/17 7:59 PM, Taewoo Kim wrote:
>>>>>
>>>>> Currently, I am working on a field type propagation without using
>>>>>> initializing the OptimizableSubTree in the current index access
>>>>>>
>>>>> method.
>>>
>>>> I
>>>>
>>>>> am encountering an issue with an open-type enforced index. So, I just
>>>>>>
>>>>> want
>>>>
>>>>> to make sure that my understanding is correct. It looks like we can't
>>>>>>
>>>>> have
>>>>
>>>>> an enforced-index on a completely schemaless nested field. For
>>>>>>
>>>>> example,
>>>
>>>> the
>>>>>> following doesn't generate any issue.
>>>>>>
>>>>>> //
>>>>>> create type DBLPType as open {id: int32}
>>>>>> create type CSXType as closed {id: int32}
>>>>>>
>>>>>> create dataset DBLP(DBLPType) primary key id;
>>>>>> create dataset CSX(CSXType) primary key id;
>>>>>>
>>>>>> for $a in dataset('DBLP')
>>>>>> for $b in dataset('CSX')
>>>>>> where $a.nested.one.title /*+ indexnl */ = $b.nested.one.title
>>>>>> return {"arec": $a, "brec": $b}
>>>>>> //
>>>>>>
>>>>>> However, the following generates an exception. So, can we assume that
>>>>>>
>>>>> to
>>>
>>>> create an enforced-index, except the leaf level, there should be a
>>>>>>
>>>>> defined
>>>>
>>>>> record type. For example, for this example, there should be "nested"
>>>>>>
>>>>> type
>>>>
>>>>> and "one" type.
>>>>>>
>>>>>> //
>>>>>> create type DBLPType as open {id: int32}
>>>>>> create type CSXType as closed {id: int32}
>>>>>>
>>>>>> create dataset DBLP(DBLPType) primary key id;
>>>>>> create dataset CSX(CSXType) primary key id;
>>>>>>
>>>>>> create index title_index_DBLP on DBLP(nested.one.title: string?)
>>>>>>
>>>>> enforced;
>>>>
>>>>> create index title_index_CSX on CSX(nested.one.title: string?)
>>>>>>
>>>>> enforced;
>>>
>>>> for $a in dataset('DBLP')
>>>>>> for $b in dataset('CSX')
>>>>>> where $a.nested.one.title /*+ indexnl */ = $b.nested.one.title
>>>>>> return {"arec": $a, "brec": $b}
>>>>>> //
>>>>>>
>>>>>> Best,
>>>>>> Taewoo
>>>>>>
>>>>>>
>>>>>>
>

Re: Nested type + open-enforced-index question.

[Taewoo Kim]

@Mike: In order to properly deal with the enforced index on a nested-type
field, I need to make sure that whether my understanding (each nested type
(except the leaf level0 has a record type for the next level) is correct or
not. Which one is a bug? The first one (without index) should fail? Or the
second one (with an index) should succeed?

Best,
Taewoo

On Thu, Jul 13, 2017 at 9:58 PM, Yingyi Bu <buyin...@gmail.com> wrote:

> Indeed, it's a bug!
>
> Best,
> Yingyi
>
> On Thu, Jul 13, 2017 at 9:52 PM, Mike Carey <dtab...@gmail.com> wrote:
>
> > Sounds like a bug to me.
> >
> >
> >
> > On 7/13/17 7:59 PM, Taewoo Kim wrote:
> >
> >> Currently, I am working on a field type propagation without using
> >> initializing the OptimizableSubTree in the current index access method.
> I
> >> am encountering an issue with an open-type enforced index. So, I just
> want
> >> to make sure that my understanding is correct. It looks like we can't
> have
> >> an enforced-index on a completely schemaless nested field. For example,
> >> the
> >> following doesn't generate any issue.
> >>
> >> //
> >> create type DBLPType as open {id: int32}
> >> create type CSXType as closed {id: int32}
> >>
> >> create dataset DBLP(DBLPType) primary key id;
> >> create dataset CSX(CSXType) primary key id;
> >>
> >> for $a in dataset('DBLP')
> >> for $b in dataset('CSX')
> >> where $a.nested.one.title /*+ indexnl */ = $b.nested.one.title
> >> return {"arec": $a, "brec": $b}
> >> //
> >>
> >> However, the following generates an exception. So, can we assume that to
> >> create an enforced-index, except the leaf level, there should be a
> defined
> >> record type. For example, for this example, there should be "nested"
> type
> >> and "one" type.
> >>
> >> //
> >> create type DBLPType as open {id: int32}
> >> create type CSXType as closed {id: int32}
> >>
> >> create dataset DBLP(DBLPType) primary key id;
> >> create dataset CSX(CSXType) primary key id;
> >>
> >> create index title_index_DBLP on DBLP(nested.one.title: string?)
> enforced;
> >> create index title_index_CSX on CSX(nested.one.title: string?) enforced;
> >>
> >> for $a in dataset('DBLP')
> >> for $b in dataset('CSX')
> >> where $a.nested.one.title /*+ indexnl */ = $b.nested.one.title
> >> return {"arec": $a, "brec": $b}
> >> //
> >>
> >> Best,
> >> Taewoo
> >>
> >>
> >
>

Nested type + open-enforced-index question.

[Taewoo Kim]

Currently, I am working on a field type propagation without using
initializing the OptimizableSubTree in the current index access method. I
am encountering an issue with an open-type enforced index. So, I just want
to make sure that my understanding is correct. It looks like we can't have
an enforced-index on a completely schemaless nested field. For example, the
following doesn't generate any issue.

//
create type DBLPType as open {id: int32}
create type CSXType as closed {id: int32}

create dataset DBLP(DBLPType) primary key id;
create dataset CSX(CSXType) primary key id;

for $a in dataset('DBLP')
for $b in dataset('CSX')
where $a.nested.one.title /*+ indexnl */ = $b.nested.one.title
return {"arec": $a, "brec": $b}
//

However, the following generates an exception. So, can we assume that to
create an enforced-index, except the leaf level, there should be a defined
record type. For example, for this example, there should be "nested" type
and "one" type.

//
create type DBLPType as open {id: int32}
create type CSXType as closed {id: int32}

create dataset DBLP(DBLPType) primary key id;
create dataset CSX(CSXType) primary key id;

create index title_index_DBLP on DBLP(nested.one.title: string?) enforced;
create index title_index_CSX on CSX(nested.one.title: string?) enforced;

for $a in dataset('DBLP')
for $b in dataset('CSX')
where $a.nested.one.title /*+ indexnl */ = $b.nested.one.title
return {"arec": $a, "brec": $b}
//

Best,
Taewoo

Re: SqlppExecutionWithCancellationTest fail

[Taewoo Kim]

@Abdullah,
Thank you so much. Let me retrigger the test and see how it goes.

Best,
Taewoo

On Fri, Jul 7, 2017 at 6:57 PM, abdullah alamoudi <bamou...@gmail.com>
wrote:

> Taewoo,
> Here is my "guess" on what is happening.
> The way the cancellation tests work is that it runs regular execution
> tests and tries to cancel queries in those tests (with variable random
> delays). Sometimes queries get cancelled and sometimes not. If no query was
> cancelled, the test fails. It assumes that when it runs the 1000+ test
> case, at least one query will get cancelled.
>
> I think in your case, you tried to run the test with only the failing test
> case and when it wasn't cancelled, the cancellation test complained because
> it didn't see any cancelled query.
> As for the on jenkins case, the encountered exception "Resource not found"
> is not an acceptable failure in case of cancellation and so, it failed.
> Someone should investigate those specific cases and how cancellation lead
> to such error.
>
> Another thing one can do is to create a JIRA, add the stacktraces and
> retrigger. or run the single test 100 times or so locally and hope they get
> to see the failure.
>
> Hope that helps,
> Abdullah.
>
>
> > On Jul 7, 2017, at 6:36 PM, Taewoo Kim <wangs...@gmail.com> wrote:
> >
> > Hi all,
> >
> > I have several failing test cases for the
> > SqlppExecutionWithCancellationTest. I am not sure how to deal with this
> > issue. When I execute this on my local Eclipse, it generates an Assertion
> > error saying that the number of canceled query is not greater than zero.
> > But the Jenkins shows an another error message such as the following. Any
> > information would be appreciated. Thanks.
> >
> > https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-
> cancellation-test/org.apache.asterix$asterix-app/277/
> testReport/junit/org.apache.asterix.test.runtime/
> SqlppExecutionWithCancellationTest/test_SqlppExecutionWithCancellation
> Test_435__index_join__btree_secondary_equi_join_04_/
> >
> >
> > https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-
> cancellation-test/277/#showFailuresLink
>
>

SqlppExecutionWithCancellationTest fail

[Taewoo Kim]

Hi all,

I have several failing test cases for the
SqlppExecutionWithCancellationTest. I am not sure how to deal with this
issue. When I execute this on my local Eclipse, it generates an Assertion
error saying that the number of canceled query is not greater than zero.
But the Jenkins shows an another error message such as the following. Any
information would be appreciated. Thanks.

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/org.apache.asterix$asterix-app/277/testReport/junit/org.apache.asterix.test.runtime/SqlppExecutionWithCancellationTest/test_SqlppExecutionWithCancellationTest_435__index_join__btree_secondary_equi_join_04_/


https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-cancellation-test/277/#showFailuresLink

Re: asterix-gerrit-source-assemblies build error

[Taewoo Kim]

@Xikui: Thanks. I am going to try.

Best,
Taewoo

On Tue, Jul 4, 2017 at 3:46 PM, Xikui Wang <xik...@uci.edu> wrote:

> Hi Taewoo,
>
> I have seen this error from time to time. It seems there are too many open
> files on that machine. Retrigger that build will solve the problem.
>
> Best,
> Xikui
>
> On Tue, Jul 4, 2017 at 2:31 PM, Taewoo Kim <wangs...@gmail.com> wrote:
>
> > Hi all,
> >
> > Has anyone seen the following error in asterix-gerrit-source-assemblies
> > and
> > know how to fix this? The file exists in the directory.
> >
> > https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-
> > source-assemblies/558/
> >
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-assembly-plugin:2.6:single
> > (source-release-assembly) on project apache-asterixdb: Failed to
> > create assembly: Error creating assembly archive source-release:
> > Problem creating zip:
> > /mnt/data/sde/asterix/workspace/asterix-gerrit-
> source-assemblies/checkout/
> > asterixdb/./asterix-app/src/test/resources/optimizerts/
> > results/open-index-non-enforced/btree-index-non-
> enforced/btree-index-non-
> > enforced-09.plan
> > (Too many open files) -> [Help 1]
> >
> > Best,
> > Taewoo
> >
>

asterix-gerrit-source-assemblies build error

[Taewoo Kim]

Hi all,

Has anyone seen the following error in asterix-gerrit-source-assemblies and
know how to fix this? The file exists in the directory.

https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-source-assemblies/558/

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-assembly-plugin:2.6:single
(source-release-assembly) on project apache-asterixdb: Failed to
create assembly: Error creating assembly archive source-release:
Problem creating zip:
/mnt/data/sde/asterix/workspace/asterix-gerrit-source-assemblies/checkout/asterixdb/./asterix-app/src/test/resources/optimizerts/results/open-index-non-enforced/btree-index-non-enforced/btree-index-non-enforced-09.plan
(Too many open files) -> [Help 1]

Best,
Taewoo

Re: Searching for duplicates during feed ingestion.

[Taewoo Kim]

+1 for auto-generated ID case

Best,
Taewoo

On Mon, May 8, 2017 at 8:57 AM, Yingyi Bu  wrote:

> Abdullah has a pending change that disables searches if there's no
> secondary indexes [1].
> Auto-generated ID could be another case for which we can disable searches
> as well.
>
> Best,
> Yingyi
>
> [1] https://asterix-gerrit.ics.uci.edu/#/c/1711/
>
>
> On Mon, May 8, 2017 at 4:30 AM, Wail Alkowaileet 
> wrote:
>
> > Hi Devs,
> >
> > I'm noticing a behavior during the ingestion is that it's getting slower
> by
> > time. I know that is an expected behavior in LSM-indexes. But what I'm
> > seeing is that I can notice the drop in ingestion rate roughly after
> having
> > 10 components (around ~13 GB). That's what I'm not sure if it's expected?
> >
> > I tried multiple setups (increasing Memory component size +
> > max-mergable-component-size). All of which delayed the problem but not
> > solved it. The only part I've never changed is the bloom-filter
> > false-positive rate (1%). Which I want to investigate next.
> >
> > So..
> > What I want to suggest is that when the primary key is auto-generated,
> why
> > AsterixDB looks for duplicates? it seems a wasteful operation to me.
> Also,
> > can we give the user the ability to tell the index that all keys are
> unique
> > ? I know I should not trust the user .. but in certain cases, probably
> the
> > user is certain that the key is unique. Or a more elegant solution can
> > shine in the end :-)
> >
> > --
> >
> > *Regards,*
> > Wail Alkowaileet
> >
>

Re: [VOTE] Release Apache AsterixDB 0.9.1 and Hyracks 0.3.1 (RC2)

[Taewoo Kim]

+1

mvn verify works!

Best,
Taewoo

On Fri, Apr 21, 2017 at 9:24 AM, Till Westmann  wrote:

> +1 to all artifacts.
>
> Checked the source artifacts:
> - Signatures and hashes correct
> — LICENSE and NOTICE look ok
> - Source files have Apache header
> - No unexpected binary files (checked expected binary files)
> - Can compile from source
>
> Checked the binary artifacts:
> - Signatures and hashes correct
> — LICENSE and NOTICE look ok
>
> Till
>
>
> On 13 Apr 2017, at 15:17, Ian Maxon wrote:
>
> Hi everyone,
>>
>> Please verify and vote on the 2nd release of Apache AsterixDB
>>
>> The change that produced this release and the change to advance the
>> version are
>> up for review here:
>>
>> https://asterix-gerrit.ics.uci.edu/#/c/1679/
>> https://asterix-gerrit.ics.uci.edu/#/c/1680/
>>
>> To check out the release, simply fetch the review and check out the
>> fetch head like so:
>>
>> git fetch https://asterix-gerrit.ics.uci.edu:29418/asterixdb
>> refs/changes/79/1679/1 && git checkout FETCH_HEAD
>>
>> The artifacts, sha1's, and signatures are (for each artifact), are at:
>>
>> AsterixDB Source
>> https://dist.apache.org/repos/dist/dev/asterixdb/apache-aste
>> rixdb-0.9.1-source-release.zip
>> https://dist.apache.org/repos/dist/dev/asterixdb/apache-aste
>> rixdb-0.9.1-source-release.zip.asc
>> https://dist.apache.org/repos/dist/dev/asterixdb/apache-aste
>> rixdb-0.9.1-source-release.zip.sha1
>>
>> SHA1: f11eb48b5ed0f033eac5f056398d6543160f1e52
>>
>> Hyracks Source
>> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyra
>> cks-0.3.1-source-release.zip
>> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyra
>> cks-0.3.1-source-release.zip.asc
>> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyra
>> cks-0.3.1-source-release.zip.sha1
>>
>> SHA1: e3d9ac77b8ca02b04fa3c7c64f74c8b2e4150e6f
>>
>> AsterixDB NCService Installer:
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-ser
>> ver-0.9.1-binary-assembly.zip
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-ser
>> ver-0.9.1-binary-assembly.zip.asc
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-ser
>> ver-0.9.1-binary-assembly.zip.sha1
>>
>> SHA1: 7a6bc04c8c2e3d7a31b071d7b4c6e1e34012fb6c
>>
>> AsterixDB Managix Installer
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-ins
>> taller-0.9.1-binary-assembly.zip
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-ins
>> taller-0.9.1-binary-assembly.zip.asc
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-ins
>> taller-0.9.1-binary-assembly.zip.sha1
>>
>> SHA1: df6721bfd9f5ae15463a23cb256ce92d9df19a4c
>>
>> AsterixDB YARN Installer
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-yar
>> n-0.9.1-binary-assembly.zip
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-yar
>> n-0.9.1-binary-assembly.zip.asc
>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-yar
>> n-0.9.1-binary-assembly.zip.sha1
>>
>> SHA1: 5b74c9eaeb0afad6717e4b0107c7dfea3506a479
>>
>> Additionally, a staged maven repository is available at:
>>
>> https://repository.apache.org/content/repositories/orgapacheasterix-1036/
>>
>> The KEYS file containing the PGP keys used to sign the release can be
>> found at
>>
>> https://dist.apache.org/repos/dist/release/asterixdb/KEYS
>>
>> RAT was executed as part of Maven via the RAT maven plugin, but
>> excludes files that are:
>>
>> - data for tests
>> - procedurally generated,
>> - or source files which come without a header mentioning their license,
>>   but have an explicit reference in the LICENSE file.
>>
>>
>> The vote is open for 72 hours, or until the necessary number of votes
>> (3 +1) has been reached.
>>
>> Please vote
>> [ ] +1 release these packages as Apache AsterixDB 0.9.1 and
>> Apache Hyracks 0.3.1
>> [ ] 0 No strong feeling either way
>> [ ] -1 do not release one or both packages because ...
>>
>> Thanks!
>>
>

Re: [VOTE] Release Apache AsterixDB 0.9.1 and Hyracks 0.3.1 (RC1)

[Taewoo Kim]

+1

Best,
Taewoo

On Sat, Apr 8, 2017 at 2:11 PM, Akshay Manchale Sridhar 
wrote:

> +1
>
> - verified hashes
> - source compilation works
> - NCservice works
>
> On Fri, Apr 7, 2017 at 7:51 AM, Mike Carey  wrote:
>
> > +1 to release - I just verified that the NCService Installer package bits
> > (my happy path bits) work!
> >
> >
> >
> > On 4/6/17 7:36 PM, Michael Blow wrote:
> >
> >> [X] +1 release these packages as Apache AsterixDB 0.9.1 and Apache
> Hyracks
> >> 0.3.1
> >>
> >> On Tue, Apr 4, 2017 at 2:17 PM Ian Maxon  wrote:
> >>
> >> Hi everyone,
> >>>
> >>> Please verify and vote on the 2nd release of Apache AsterixDB
> >>>
> >>> The change that produced this release and the change to advance the
> >>> version are
> >>> up for review here:
> >>>
> >>> https://asterix-gerrit.ics.uci.edu/#/c/1660/
> >>> https://asterix-gerrit.ics.uci.edu/#/c/1661/
> >>>
> >>> To check out the release, simply fetch the review and check out the
> >>> fetch head like so:
> >>>
> >>> git fetch https://asterix-gerrit.ics.uci.edu:29418/asterixdb
> >>> refs/changes/54/1660/1 && git checkout FETCH_HEAD
> >>>
> >>> The artifacts, sha1's, and signatures are (for each artifact), are at:
> >>>
> >>> AsterixDB Source
> >>>
> >>> https://dist.apache.org/repos/dist/dev/asterixdb/apache-aste
> >>> rixdb-0.9.1-source-release.zip
> >>>
> >>> https://dist.apache.org/repos/dist/dev/asterixdb/apache-aste
> >>> rixdb-0.9.1-source-release.zip.asc
> >>>
> >>> https://dist.apache.org/repos/dist/dev/asterixdb/apache-aste
> >>> rixdb-0.9.1-source-release.zip.sha1
> >>>
> >>> SHA1: 8fc212b478e1e3ef62865de233e509066dc3445d
> >>>
> >>> Hyracks Source
> >>>
> >>> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyra
> >>> cks-0.3.1-source-release.zip
> >>>
> >>> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyra
> >>> cks-0.3.1-source-release.zip.asc
> >>>
> >>> https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyra
> >>> cks-0.3.1-source-release.zip.sha1
> >>>
> >>> SHA1: 9b004555bb45f8aec234cee1be04b264d603685a
> >>>
> >>> AsterixDB NCService Installer:
> >>>
> >>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-ser
> >>> ver-0.9.1-binary-assembly.zip
> >>>
> >>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-ser
> >>> ver-0.9.1-binary-assembly.zip.asc
> >>>
> >>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-ser
> >>> ver-0.9.1-binary-assembly.zip.sha1
> >>>
> >>> SHA1: 68efd7daa8f8cc758ac618e2acc0923f2896ffd0
> >>>
> >>> AsterixDB Managix Installer
> >>>
> >>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-ins
> >>> taller-0.9.1-binary-assembly.zip
> >>>
> >>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-ins
> >>> taller-0.9.1-binary-assembly.zip.asc
> >>>
> >>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-ins
> >>> taller-0.9.1-binary-assembly.zip.sha1
> >>>
> >>> SHA1: b794d5bcf581cf253aa58dd17f1298256d6210f0
> >>>
> >>> AsterixDB YARN Installer
> >>>
> >>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-yar
> >>> n-0.9.1-binary-assembly.zip
> >>>
> >>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-yar
> >>> n-0.9.1-binary-assembly.zip.asc
> >>>
> >>> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-yar
> >>> n-0.9.1-binary-assembly.zip.sha1
> >>>
> >>> SHA1: 8292657dfb0b81ab1d99a1fd4e5c7f9723803e38
> >>>
> >>> Additionally, a staged maven repository is available at:
> >>>
> >>> https://repository.apache.org/content/repositories/orgapache
> >>> asterix-1035/
> >>>
> >>> The KEYS file containing the PGP keys used to sign the release can be
> >>> found at
> >>>
> >>> https://dist.apache.org/repos/dist/release/asterixdb/KEYS
> >>>
> >>> RAT was executed as part of Maven via the RAT maven plugin, but
> >>> excludes files that are:
> >>>
> >>> - data for tests
> >>> - procedurally generated,
> >>> - or source files which come without a header mentioning their license,
> >>>but have an explicit reference in the LICENSE file.
> >>>
> >>>
> >>> The vote is open for 72 hours, or until the necessary number of votes
> >>> (3 +1) has been reached.
> >>>
> >>> Please vote
> >>> [ ] +1 release these packages as Apache AsterixDB 0.9.1 and
> >>> Apache Hyracks 0.3.1
> >>> [ ] 0 No strong feeling either way
> >>> [ ] -1 do not release one or both packages because ...
> >>>
> >>> Thanks!
> >>>
> >>>
> >
>

Re: New cluster installation options

[Taewoo Kim]

@Yingyi: As far as I know, Gift and Xikui are working on those features.

Best,
Taewoo

On Tue, Apr 4, 2017 at 4:04 PM, Yingyi Bu <buyin...@gmail.com> wrote:

> Thanks for the feedback.
> It hasn't supported yet.  But it should be easy to add.
>
> Is Java UDF support the only missing feature in those two installation
> options?
>
> >> Unless they don't support it, it may be early to deprecate the managix?
> It's deprecated for common use cases -- I'm not sure how many people are
> using the Java UDF?
>
> Best,
> Yingyi
>
>
> On Tue, Apr 4, 2017 at 3:54 PM, Taewoo Kim <wangs...@gmail.com> wrote:
>
> > Sorry for the late response.
> >
> > Do Ansible and AWS support "installing libraries" like the managix does?
> > Just checked the documentation. I'm talking about this:
> > https://ci.apache.org/projects/asterixdb/udf.html. Unless they don't
> > support it, it may be early to deprecate the managix?
> >
> > Best,
> > Taewoo
> >
> > On Thu, Mar 16, 2017 at 9:22 AM, Yingyi Bu <buyin...@gmail.com> wrote:
> >
> > > Hi dev,
> > >
> > >  We recently added two cluster installation options and their
> > > documentation is online now:
> > >
> > >  Ansible: https://ci.apache.org/projects/asterixdb/ansible.html
> > >  AWS: https://ci.apache.org/projects/asterixdb/aws.html
> > >
> > >  With the Ansible option, you can deploy, start, stop, and erase an
> > > AsterixDB cluster from your client with a simple shell scripts.
> > >  With the AWS option, you can start/stop an AsterixDB cluster on
> AWS
> > > with a simple shell scripts. The scripts automate both AWS and
> AsterixDB
> > > setup.
> > >
> > >  Managix is marked as "deprecated" and probably is going to retire
> > > soon, as (1) it's not well maintained and (2) some of its
> functionalities
> > > are already provided by existing tools such as Ansible. It was written
> at
> > > the time when there're not many automation tools that we can rely on,
> > which
> > > is no longer the case now.
> > >
> > >  Please share your thoughts and feedbacks.
> > >  Thanks!
> > >
> > > Best,
> > > Yingyi
> > >
> >
>

Re: New cluster installation options

[Taewoo Kim]

Sorry for the late response.

Do Ansible and AWS support "installing libraries" like the managix does?
Just checked the documentation. I'm talking about this:
https://ci.apache.org/projects/asterixdb/udf.html. Unless they don't
support it, it may be early to deprecate the managix?

Best,
Taewoo

On Thu, Mar 16, 2017 at 9:22 AM, Yingyi Bu  wrote:

> Hi dev,
>
>  We recently added two cluster installation options and their
> documentation is online now:
>
>  Ansible: https://ci.apache.org/projects/asterixdb/ansible.html
>  AWS: https://ci.apache.org/projects/asterixdb/aws.html
>
>  With the Ansible option, you can deploy, start, stop, and erase an
> AsterixDB cluster from your client with a simple shell scripts.
>  With the AWS option, you can start/stop an AsterixDB cluster on AWS
> with a simple shell scripts. The scripts automate both AWS and AsterixDB
> setup.
>
>  Managix is marked as "deprecated" and probably is going to retire
> soon, as (1) it's not well maintained and (2) some of its functionalities
> are already provided by existing tools such as Ansible. It was written at
> the time when there're not many automation tools that we can rely on, which
> is no longer the case now.
>
>  Please share your thoughts and feedbacks.
>  Thanks!
>
> Best,
> Yingyi
>

Re: JSON viewing in our web UI

[Taewoo Kim]

Like this?

http://rawgit.com/abodelot/jquery.json-viewer/master/demo.html



Best,
Taewoo

On Sat, Mar 11, 2017 at 12:14 PM, Mike Carey  wrote:

> Hmmm...  Check out the Chrome JSON handling functionality about 5 minutes
> into https://lagunita.stanford.edu/courses/DB/JSON/SelfPaced/cour
> seware/ch-json_data/seq-vid-json_demo/. Is there some way to "access"
> that from within a browser window, like from our results window?  (Though
> we want to be browser-flexible.)
>
>

Re: IPageManager.createMetadataFrame(): java.lang.NoSuchMethodError

[Taewoo Kim]

@Abdullah: you are right. :-)

Best,
Taewoo

On Sun, Feb 19, 2017 at 9:54 PM, abdullah alamoudi <bamou...@gmail.com>
wrote:

> Taewoo,
> The reason you found the exception in only one is that that one was the
> metadata node as can be seen in the stack trace.
>
> Cheers,
> Abdullah.
>
> > On Feb 19, 2017, at 9:52 PM, Taewoo Kim <wangs...@gmail.com> wrote:
> >
> > @Till: Thanks for the answer. I usually saw this error when I change one
> > binary to a newer binary and there was a meta-data change in-between.
> But,
> > the strange part is that I was using the same binary for 10 days now. I
> > just stopped the instance and restarted instance and saw that error on
> one
> > NC node. The symptom is that the system doesn't accept any query nor
> > generates any errors. That's why I found this exception on one NC. CC and
> > all the other NCs (8 nodes) were fine.
> >
> > Best,
> > Taewoo
> >
> > On Sun, Feb 19, 2017 at 8:47 PM, Till Westmann <ti...@apache.org> wrote:
> >
> >> NoSuchMethodError certainly sounds like something is not right with
> >> the classes on the classpath. Since we’re not playing too much with
> >> the classpath at runtime, it seems that a change of binaries or an
> >> error reading the jars are not unlikely. But maybe the same message was
> >> there before as well.
> >> Does this error affect the system behavior?
> >>
> >> Cheers,
> >> Till
> >>
> >> On 19 Feb 2017, at 15:05, Taewoo Kim wrote:
> >>
> >>> I see. I checked the disk related errors and found nothing special.
> >>>
> >>> Best,
> >>> Taewoo
> >>>
> >>> On Sun, Feb 19, 2017 at 3:01 PM, Mike Carey <dtab...@gmail.com> wrote:
> >>>
> >>>> Got it.  (So not attributable to the all-too-frequent Asterix cluster
> >> I/O
> >>>> misbehavior...)
> >>>>
> >>>>
> >>>>
> >>>> On 2/19/17 2:55 PM, Taewoo Kim wrote:
> >>>>
> >>>>> It happened on the Sensorium cluster.
> >>>>>
> >>>>> Best,
> >>>>> Taewoo
> >>>>>
> >>>>> On Sun, Feb 19, 2017 at 2:54 PM, Mike Carey <dtab...@gmail.com>
> wrote:
> >>>>>
> >>>>> Which cluster hardware?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 2/19/17 2:42 PM, Taewoo Kim wrote:
> >>>>>>
> >>>>>> Nope. I was keeping the same binary for 10 days now.
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Taewoo
> >>>>>>>
> >>>>>>> On Sun, Feb 19, 2017 at 2:20 PM, abdullah alamoudi <
> >> bamou...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> Taewoo,
> >>>>>>>
> >>>>>>>> Did you change the binary in between?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Feb 19, 2017, at 2:18 PM, Taewoo Kim <wangs...@gmail.com>
> wrote:
> >>>>>>>>
> >>>>>>>>> Hi All,
> >>>>>>>>>
> >>>>>>>>> I have a CC and nine NCs. After I stopped the cluster and
> restarted
> >>>>>>>>> the
> >>>>>>>>> instance, I saw the following exception on one NC. Has anybody
> seen
> >>>>>>>>> this
> >>>>>>>>> error?
> >>>>>>>>>
> >>>>>>>>> Feb 19, 2017 2:05:21 PM org.apache.hyracks.control.nc.NCDriver
> >> main
> >>>>>>>>> SEVERE: Setting uncaught exception handler
> >>>>>>>>> org.apache.hyracks.api.lifecycle.LifeCycleComponentManager@
> >> 51565ec2
> >>>>>>>>> Feb 19, 2017 2:05:21 PM org.apache.hyracks.control.nc.
> >>>>>>>>>
> >>>>>>>>> NodeControllerService
> >>>>>>>>
> >>>>>>>> start
> >>>>>>>>> INFO: Starting NodeControllerService
> >>>>>>>>> Feb 19, 2017 2:05:21 PM
> &

Re: IPageManager.createMetadataFrame(): java.lang.NoSuchMethodError

[Taewoo Kim]

@Till: Thanks for the answer. I usually saw this error when I change one
binary to a newer binary and there was a meta-data change in-between. But,
the strange part is that I was using the same binary for 10 days now. I
just stopped the instance and restarted instance and saw that error on one
NC node. The symptom is that the system doesn't accept any query nor
generates any errors. That's why I found this exception on one NC. CC and
all the other NCs (8 nodes) were fine.

Best,
Taewoo

On Sun, Feb 19, 2017 at 8:47 PM, Till Westmann <ti...@apache.org> wrote:

> NoSuchMethodError certainly sounds like something is not right with
> the classes on the classpath. Since we’re not playing too much with
> the classpath at runtime, it seems that a change of binaries or an
> error reading the jars are not unlikely. But maybe the same message was
> there before as well.
> Does this error affect the system behavior?
>
> Cheers,
> Till
>
> On 19 Feb 2017, at 15:05, Taewoo Kim wrote:
>
> > I see. I checked the disk related errors and found nothing special.
> >
> > Best,
> > Taewoo
> >
> > On Sun, Feb 19, 2017 at 3:01 PM, Mike Carey <dtab...@gmail.com> wrote:
> >
> >> Got it.  (So not attributable to the all-too-frequent Asterix cluster
> I/O
> >> misbehavior...)
> >>
> >>
> >>
> >> On 2/19/17 2:55 PM, Taewoo Kim wrote:
> >>
> >>> It happened on the Sensorium cluster.
> >>>
> >>> Best,
> >>> Taewoo
> >>>
> >>> On Sun, Feb 19, 2017 at 2:54 PM, Mike Carey <dtab...@gmail.com> wrote:
> >>>
> >>> Which cluster hardware?
> >>>>
> >>>>
> >>>>
> >>>> On 2/19/17 2:42 PM, Taewoo Kim wrote:
> >>>>
> >>>> Nope. I was keeping the same binary for 10 days now.
> >>>>>
> >>>>> Best,
> >>>>> Taewoo
> >>>>>
> >>>>> On Sun, Feb 19, 2017 at 2:20 PM, abdullah alamoudi <
> bamou...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>> Taewoo,
> >>>>>
> >>>>>> Did you change the binary in between?
> >>>>>>
> >>>>>>
> >>>>>> On Feb 19, 2017, at 2:18 PM, Taewoo Kim <wangs...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hi All,
> >>>>>>>
> >>>>>>> I have a CC and nine NCs. After I stopped the cluster and restarted
> >>>>>>> the
> >>>>>>> instance, I saw the following exception on one NC. Has anybody seen
> >>>>>>> this
> >>>>>>> error?
> >>>>>>>
> >>>>>>> Feb 19, 2017 2:05:21 PM org.apache.hyracks.control.nc.NCDriver
> main
> >>>>>>> SEVERE: Setting uncaught exception handler
> >>>>>>> org.apache.hyracks.api.lifecycle.LifeCycleComponentManager@
> 51565ec2
> >>>>>>> Feb 19, 2017 2:05:21 PM org.apache.hyracks.control.nc.
> >>>>>>>
> >>>>>>> NodeControllerService
> >>>>>>
> >>>>>> start
> >>>>>>> INFO: Starting NodeControllerService
> >>>>>>> Feb 19, 2017 2:05:21 PM
> >>>>>>> org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint start
> >>>>>>> INFO: Starting Asterix node controller: ss1120_nc4
> >>>>>>> Feb 19, 2017 2:05:24 PM
> >>>>>>> org.apache.hyracks.api.lifecycle.LifeCycleComponentManager
> >>>>>>>
> >>>>>>> uncaughtException
> >>>>>>
> >>>>>> SEVERE: Uncaught Exception from thread main message:
> >>>>>>> org.apache.hyracks.storage.am.common.api.IPageManager.
> >>>>>>>
> >>>>>>> createMetadataFrame()Lorg/apache/hyracks/storage/am/common/api/
> >>>>>> ITreeIndexMetaDataFrame;
> >>>>>>
> >>>>>> java.lang.NoSuchMethodError:
> >>>>>>> org.apache.hyracks.storage.am.common.api.IPageManager.
> >>>>>>>
> >>>>>>> createMetadataFrame()Lorg/apache/hyracks/storage/am/common/api/
> >>>>>> ITreeIndexMetaDataFrame;
> >>>>>>
> >>>>>>

Re: IPageManager.createMetadataFrame(): java.lang.NoSuchMethodError

[Taewoo Kim]

I see. I checked the disk related errors and found nothing special.

Best,
Taewoo

On Sun, Feb 19, 2017 at 3:01 PM, Mike Carey <dtab...@gmail.com> wrote:

> Got it.  (So not attributable to the all-too-frequent Asterix cluster I/O
> misbehavior...)
>
>
>
> On 2/19/17 2:55 PM, Taewoo Kim wrote:
>
>> It happened on the Sensorium cluster.
>>
>> Best,
>> Taewoo
>>
>> On Sun, Feb 19, 2017 at 2:54 PM, Mike Carey <dtab...@gmail.com> wrote:
>>
>> Which cluster hardware?
>>>
>>>
>>>
>>> On 2/19/17 2:42 PM, Taewoo Kim wrote:
>>>
>>> Nope. I was keeping the same binary for 10 days now.
>>>>
>>>> Best,
>>>> Taewoo
>>>>
>>>> On Sun, Feb 19, 2017 at 2:20 PM, abdullah alamoudi <bamou...@gmail.com>
>>>> wrote:
>>>>
>>>> Taewoo,
>>>>
>>>>> Did you change the binary in between?
>>>>>
>>>>>
>>>>> On Feb 19, 2017, at 2:18 PM, Taewoo Kim <wangs...@gmail.com> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I have a CC and nine NCs. After I stopped the cluster and restarted
>>>>>> the
>>>>>> instance, I saw the following exception on one NC. Has anybody seen
>>>>>> this
>>>>>> error?
>>>>>>
>>>>>> Feb 19, 2017 2:05:21 PM org.apache.hyracks.control.nc.NCDriver main
>>>>>> SEVERE: Setting uncaught exception handler
>>>>>> org.apache.hyracks.api.lifecycle.LifeCycleComponentManager@51565ec2
>>>>>> Feb 19, 2017 2:05:21 PM org.apache.hyracks.control.nc.
>>>>>>
>>>>>> NodeControllerService
>>>>>
>>>>> start
>>>>>> INFO: Starting NodeControllerService
>>>>>> Feb 19, 2017 2:05:21 PM
>>>>>> org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint start
>>>>>> INFO: Starting Asterix node controller: ss1120_nc4
>>>>>> Feb 19, 2017 2:05:24 PM
>>>>>> org.apache.hyracks.api.lifecycle.LifeCycleComponentManager
>>>>>>
>>>>>> uncaughtException
>>>>>
>>>>> SEVERE: Uncaught Exception from thread main message:
>>>>>> org.apache.hyracks.storage.am.common.api.IPageManager.
>>>>>>
>>>>>> createMetadataFrame()Lorg/apache/hyracks/storage/am/common/api/
>>>>> ITreeIndexMetaDataFrame;
>>>>>
>>>>> java.lang.NoSuchMethodError:
>>>>>> org.apache.hyracks.storage.am.common.api.IPageManager.
>>>>>>
>>>>>> createMetadataFrame()Lorg/apache/hyracks/storage/am/common/api/
>>>>> ITreeIndexMetaDataFrame;
>>>>>
>>>>> at
>>>>>> org.apache.hyracks.storage.am.btree.impls.BTreeOpContext.<
>>>>>>
>>>>>> init>(BTreeOpContext.java:104)
>>>>>
>>>>> at
>>>>>> org.apache.hyracks.storage.am.btree.impls.BTree.
>>>>>>
>>>>>> createOpContext(BTree.java:835)
>>>>>
>>>>> at org.apache.hyracks.storage.am.btree.impls.BTree.access$100(
>>>>>>
>>>>>> BTree.java:74)
>>>>>
>>>>> at
>>>>>> org.apache.hyracks.storage.am.btree.impls.BTree$
>>>>>>
>>>>>> BTreeAccessor.(BTree.java:915)
>>>>>
>>>>> at
>>>>>> org.apache.hyracks.storage.am.btree.impls.BTree.
>>>>>>
>>>>>> createAccessor(BTree.java:896)
>>>>>
>>>>> at
>>>>>> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeOpCont
>>>>>> ext.(
>>>>>>
>>>>>> LSMBTreeOpContext.java:93)
>>>>>
>>>>> at
>>>>>> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.
>>>>>>
>>>>>> createOpContext(LSMBTree.java:770)
>>>>>
>>>>> at
>>>>>> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.
>>>>>>
>>>>>> createAccessor(LSMBTree.java:777)
>>>>>
>>>>> at
>>>>>> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.
>>>>>>
>>>>>> createAccessor(LSMBTree.java:86)
>>>>>
>>>>> at
>>>>>> org.apache.asterix.metadata.MetadataNode.initializeDatasetIdFactory(
>>>>>>
>>>>>> MetadataNode.java:1193)
>>>>>
>>>>> at
>>>>>> org.apache.asterix.metadata.MetadataManager.initializeDatase
>>>>>> tIdFactory(
>>>>>>
>>>>>> MetadataManager.java:634)
>>>>>
>>>>> at
>>>>>> org.apache.asterix.metadata.bootstrap.MetadataBootstrap.star
>>>>>> tUniverse(
>>>>>>
>>>>>> MetadataBootstrap.java:183)
>>>>>
>>>>> at
>>>>>> org.apache.asterix.app.nc.NCAppRuntimeContext.initializeMetadata(
>>>>>>
>>>>>> NCAppRuntimeContext.java:449)
>>>>>
>>>>> at
>>>>>> org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint.
>>>>>>
>>>>>> notifyStartupComplete(NCApplicationEntryPoint.java:227)
>>>>>
>>>>> at
>>>>>> org.apache.hyracks.control.nc.NodeControllerService.start(
>>>>>>
>>>>>> NodeControllerService.java:310)
>>>>>
>>>>> at org.apache.hyracks.control.nc.NCDriver.main(NCDriver.java:47)
>>>>>> Feb 19, 2017 2:05:24 PM
>>>>>> org.apache.hyracks.api.lifecycle.LifeCycleComponentManager stopAll
>>>>>> SEVERE: Lifecycle management not
>>>>>> configuredorg.apache.hyracks.api.lifecycle.LifeCycleComponentManager@
>>>>>>
>>>>>> 51565ec2
>>>>>
>>>>> Best,
>>>>>> Taewoo
>>>>>>
>>>>>>
>>>>>
>

Re: IPageManager.createMetadataFrame(): java.lang.NoSuchMethodError

[Taewoo Kim]

It happened on the Sensorium cluster.

Best,
Taewoo

On Sun, Feb 19, 2017 at 2:54 PM, Mike Carey <dtab...@gmail.com> wrote:

> Which cluster hardware?
>
>
>
> On 2/19/17 2:42 PM, Taewoo Kim wrote:
>
>> Nope. I was keeping the same binary for 10 days now.
>>
>> Best,
>> Taewoo
>>
>> On Sun, Feb 19, 2017 at 2:20 PM, abdullah alamoudi <bamou...@gmail.com>
>> wrote:
>>
>> Taewoo,
>>> Did you change the binary in between?
>>>
>>>
>>> On Feb 19, 2017, at 2:18 PM, Taewoo Kim <wangs...@gmail.com> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> I have a CC and nine NCs. After I stopped the cluster and restarted the
>>>> instance, I saw the following exception on one NC. Has anybody seen this
>>>> error?
>>>>
>>>> Feb 19, 2017 2:05:21 PM org.apache.hyracks.control.nc.NCDriver main
>>>> SEVERE: Setting uncaught exception handler
>>>> org.apache.hyracks.api.lifecycle.LifeCycleComponentManager@51565ec2
>>>> Feb 19, 2017 2:05:21 PM org.apache.hyracks.control.nc.
>>>>
>>> NodeControllerService
>>>
>>>> start
>>>> INFO: Starting NodeControllerService
>>>> Feb 19, 2017 2:05:21 PM
>>>> org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint start
>>>> INFO: Starting Asterix node controller: ss1120_nc4
>>>> Feb 19, 2017 2:05:24 PM
>>>> org.apache.hyracks.api.lifecycle.LifeCycleComponentManager
>>>>
>>> uncaughtException
>>>
>>>> SEVERE: Uncaught Exception from thread main message:
>>>> org.apache.hyracks.storage.am.common.api.IPageManager.
>>>>
>>> createMetadataFrame()Lorg/apache/hyracks/storage/am/common/api/
>>> ITreeIndexMetaDataFrame;
>>>
>>>> java.lang.NoSuchMethodError:
>>>> org.apache.hyracks.storage.am.common.api.IPageManager.
>>>>
>>> createMetadataFrame()Lorg/apache/hyracks/storage/am/common/api/
>>> ITreeIndexMetaDataFrame;
>>>
>>>> at
>>>> org.apache.hyracks.storage.am.btree.impls.BTreeOpContext.<
>>>>
>>> init>(BTreeOpContext.java:104)
>>>
>>>> at
>>>> org.apache.hyracks.storage.am.btree.impls.BTree.
>>>>
>>> createOpContext(BTree.java:835)
>>>
>>>> at org.apache.hyracks.storage.am.btree.impls.BTree.access$100(
>>>>
>>> BTree.java:74)
>>>
>>>> at
>>>> org.apache.hyracks.storage.am.btree.impls.BTree$
>>>>
>>> BTreeAccessor.(BTree.java:915)
>>>
>>>> at
>>>> org.apache.hyracks.storage.am.btree.impls.BTree.
>>>>
>>> createAccessor(BTree.java:896)
>>>
>>>> at
>>>> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeOpContext.(
>>>>
>>> LSMBTreeOpContext.java:93)
>>>
>>>> at
>>>> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.
>>>>
>>> createOpContext(LSMBTree.java:770)
>>>
>>>> at
>>>> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.
>>>>
>>> createAccessor(LSMBTree.java:777)
>>>
>>>> at
>>>> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.
>>>>
>>> createAccessor(LSMBTree.java:86)
>>>
>>>> at
>>>> org.apache.asterix.metadata.MetadataNode.initializeDatasetIdFactory(
>>>>
>>> MetadataNode.java:1193)
>>>
>>>> at
>>>> org.apache.asterix.metadata.MetadataManager.initializeDatasetIdFactory(
>>>>
>>> MetadataManager.java:634)
>>>
>>>> at
>>>> org.apache.asterix.metadata.bootstrap.MetadataBootstrap.startUniverse(
>>>>
>>> MetadataBootstrap.java:183)
>>>
>>>> at
>>>> org.apache.asterix.app.nc.NCAppRuntimeContext.initializeMetadata(
>>>>
>>> NCAppRuntimeContext.java:449)
>>>
>>>> at
>>>> org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint.
>>>>
>>> notifyStartupComplete(NCApplicationEntryPoint.java:227)
>>>
>>>> at
>>>> org.apache.hyracks.control.nc.NodeControllerService.start(
>>>>
>>> NodeControllerService.java:310)
>>>
>>>> at org.apache.hyracks.control.nc.NCDriver.main(NCDriver.java:47)
>>>> Feb 19, 2017 2:05:24 PM
>>>> org.apache.hyracks.api.lifecycle.LifeCycleComponentManager stopAll
>>>> SEVERE: Lifecycle management not
>>>> configuredorg.apache.hyracks.api.lifecycle.LifeCycleComponentManager@
>>>>
>>> 51565ec2
>>>
>>>>
>>>> Best,
>>>> Taewoo
>>>>
>>>
>>>
>

Re: IPageManager.createMetadataFrame(): java.lang.NoSuchMethodError

[Taewoo Kim]

Nope. I was keeping the same binary for 10 days now.

Best,
Taewoo

On Sun, Feb 19, 2017 at 2:20 PM, abdullah alamoudi <bamou...@gmail.com>
wrote:

> Taewoo,
> Did you change the binary in between?
>
>
> > On Feb 19, 2017, at 2:18 PM, Taewoo Kim <wangs...@gmail.com> wrote:
> >
> > Hi All,
> >
> > I have a CC and nine NCs. After I stopped the cluster and restarted the
> > instance, I saw the following exception on one NC. Has anybody seen this
> > error?
> >
> > Feb 19, 2017 2:05:21 PM org.apache.hyracks.control.nc.NCDriver main
> > SEVERE: Setting uncaught exception handler
> > org.apache.hyracks.api.lifecycle.LifeCycleComponentManager@51565ec2
> > Feb 19, 2017 2:05:21 PM org.apache.hyracks.control.nc.
> NodeControllerService
> > start
> > INFO: Starting NodeControllerService
> > Feb 19, 2017 2:05:21 PM
> > org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint start
> > INFO: Starting Asterix node controller: ss1120_nc4
> > Feb 19, 2017 2:05:24 PM
> > org.apache.hyracks.api.lifecycle.LifeCycleComponentManager
> uncaughtException
> > SEVERE: Uncaught Exception from thread main message:
> > org.apache.hyracks.storage.am.common.api.IPageManager.
> createMetadataFrame()Lorg/apache/hyracks/storage/am/common/api/
> ITreeIndexMetaDataFrame;
> > java.lang.NoSuchMethodError:
> > org.apache.hyracks.storage.am.common.api.IPageManager.
> createMetadataFrame()Lorg/apache/hyracks/storage/am/common/api/
> ITreeIndexMetaDataFrame;
> > at
> > org.apache.hyracks.storage.am.btree.impls.BTreeOpContext.<
> init>(BTreeOpContext.java:104)
> > at
> > org.apache.hyracks.storage.am.btree.impls.BTree.
> createOpContext(BTree.java:835)
> > at org.apache.hyracks.storage.am.btree.impls.BTree.access$100(
> BTree.java:74)
> > at
> > org.apache.hyracks.storage.am.btree.impls.BTree$
> BTreeAccessor.(BTree.java:915)
> > at
> > org.apache.hyracks.storage.am.btree.impls.BTree.
> createAccessor(BTree.java:896)
> > at
> > org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeOpContext.(
> LSMBTreeOpContext.java:93)
> > at
> > org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.
> createOpContext(LSMBTree.java:770)
> > at
> > org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.
> createAccessor(LSMBTree.java:777)
> > at
> > org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.
> createAccessor(LSMBTree.java:86)
> > at
> > org.apache.asterix.metadata.MetadataNode.initializeDatasetIdFactory(
> MetadataNode.java:1193)
> > at
> > org.apache.asterix.metadata.MetadataManager.initializeDatasetIdFactory(
> MetadataManager.java:634)
> > at
> > org.apache.asterix.metadata.bootstrap.MetadataBootstrap.startUniverse(
> MetadataBootstrap.java:183)
> > at
> > org.apache.asterix.app.nc.NCAppRuntimeContext.initializeMetadata(
> NCAppRuntimeContext.java:449)
> > at
> > org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint.
> notifyStartupComplete(NCApplicationEntryPoint.java:227)
> > at
> > org.apache.hyracks.control.nc.NodeControllerService.start(
> NodeControllerService.java:310)
> > at org.apache.hyracks.control.nc.NCDriver.main(NCDriver.java:47)
> > Feb 19, 2017 2:05:24 PM
> > org.apache.hyracks.api.lifecycle.LifeCycleComponentManager stopAll
> > SEVERE: Lifecycle management not
> > configuredorg.apache.hyracks.api.lifecycle.LifeCycleComponentManager@
> 51565ec2
> >
> >
> > Best,
> > Taewoo
>
>

IPageManager.createMetadataFrame(): java.lang.NoSuchMethodError

[Taewoo Kim]

Hi All,

I have a CC and nine NCs. After I stopped the cluster and restarted the
instance, I saw the following exception on one NC. Has anybody seen this
error?

Feb 19, 2017 2:05:21 PM org.apache.hyracks.control.nc.NCDriver main
SEVERE: Setting uncaught exception handler
org.apache.hyracks.api.lifecycle.LifeCycleComponentManager@51565ec2
Feb 19, 2017 2:05:21 PM org.apache.hyracks.control.nc.NodeControllerService
start
INFO: Starting NodeControllerService
Feb 19, 2017 2:05:21 PM
org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint start
INFO: Starting Asterix node controller: ss1120_nc4
Feb 19, 2017 2:05:24 PM
org.apache.hyracks.api.lifecycle.LifeCycleComponentManager uncaughtException
SEVERE: Uncaught Exception from thread main message:
org.apache.hyracks.storage.am.common.api.IPageManager.createMetadataFrame()Lorg/apache/hyracks/storage/am/common/api/ITreeIndexMetaDataFrame;
java.lang.NoSuchMethodError:
org.apache.hyracks.storage.am.common.api.IPageManager.createMetadataFrame()Lorg/apache/hyracks/storage/am/common/api/ITreeIndexMetaDataFrame;
at
org.apache.hyracks.storage.am.btree.impls.BTreeOpContext.(BTreeOpContext.java:104)
at
org.apache.hyracks.storage.am.btree.impls.BTree.createOpContext(BTree.java:835)
at org.apache.hyracks.storage.am.btree.impls.BTree.access$100(BTree.java:74)
at
org.apache.hyracks.storage.am.btree.impls.BTree$BTreeAccessor.(BTree.java:915)
at
org.apache.hyracks.storage.am.btree.impls.BTree.createAccessor(BTree.java:896)
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeOpContext.(LSMBTreeOpContext.java:93)
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.createOpContext(LSMBTree.java:770)
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.createAccessor(LSMBTree.java:777)
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTree.createAccessor(LSMBTree.java:86)
at
org.apache.asterix.metadata.MetadataNode.initializeDatasetIdFactory(MetadataNode.java:1193)
at
org.apache.asterix.metadata.MetadataManager.initializeDatasetIdFactory(MetadataManager.java:634)
at
org.apache.asterix.metadata.bootstrap.MetadataBootstrap.startUniverse(MetadataBootstrap.java:183)
at
org.apache.asterix.app.nc.NCAppRuntimeContext.initializeMetadata(NCAppRuntimeContext.java:449)
at
org.apache.asterix.hyracks.bootstrap.NCApplicationEntryPoint.notifyStartupComplete(NCApplicationEntryPoint.java:227)
at
org.apache.hyracks.control.nc.NodeControllerService.start(NodeControllerService.java:310)
at org.apache.hyracks.control.nc.NCDriver.main(NCDriver.java:47)
Feb 19, 2017 2:05:24 PM
org.apache.hyracks.api.lifecycle.LifeCycleComponentManager stopAll
SEVERE: Lifecycle management not
configuredorg.apache.hyracks.api.lifecycle.LifeCycleComponentManager@51565ec2


Best,
Taewoo

Re: Choosing defaults for AsterixDB

[Taewoo Kim]

Regarding this, I have a question.

Does the new revised HTTP API - Query Service (/query/service) support AQL?
I am asking this since inside the code, it gets the SQLPP compilation
provider.

public class CCApplicationEntryPoint implements ICCApplicationEntryPoint {


protected IServlet createServLet(HttpServer server, Lets key, String...
paths) {

switch (key) {

case QUERY_SERVICE:

return new QueryServiceServlet(server.ctx(), paths,
ccExtensionManager.getSqlppCompilationProvider(),

ccExtensionManager.getQueryTranslatorFactory(),
componentProvider);

Best,
Taewoo

On Fri, Feb 3, 2017 at 6:34 PM, Jianfeng Jia  wrote:

> @Yingyi, I’m not saying learning SQL++ is difficult.
> Currently, we have a class called AQLGenerator that can translate the
> Cloudberry request syntax to AQL.  It took us several weeks finishing it.
> I guess it will take similar time to write a SQLPPGenerator to achieve the
> same goal.
>
> As long as the RESTFul API can accept AQL, we don’t need to spend time to
> implement a new generator.
>
> > On Feb 3, 2017, at 6:02 PM, Yingyi Bu  wrote:
> >
> >>> It will be a hard work to switch to SQL++.
> > Why translating to SQL++ is harder than AQL?  I wonder if the current
> SQL++
> > language design and implementation misses some key pieces.
>
>

Re: Google Summer of Code

[Taewoo Kim]

I would like to suggest one feature: adding a security feature to the
AsterixDB: setting DB accounts and privileges. Also, use them when access
the Web Interface or API call.

Best,
Taewoo

On Tue, Jan 31, 2017 at 9:52 PM, Yingyi Bu  wrote:

> Yeah, sounds good to me.
>
> Best,
> Yingyi
>
> On Tue, Jan 31, 2017 at 9:31 PM, Steven Jacobs  wrote:
>
> > Makes sense, based on a past of not quite finishing. There has been some
> > discussion from the Summer of Code Apache threads, and it sounds like
> they
> > want to do a smaller number of better vetted projects this year, but I
> > think we could still try for at least one spot.
> > Steven
> >
> > On Tue, Jan 31, 2017 at 9:21 PM Yingyi Bu  wrote:
> >
> > > IMO, each of them could be a two-three month project.
> > >
> > > The project itself could be conceptually simple and has a relatively
> > small
> > > scope, but we probably should lift the exit bar.
> > > Here are my thoughts for the exit criteria:
> > > 1.  the feature reliably works for all edge cases;
> > > 2.  the implementation does not hack things, which can result in
> > > general-purpose refactoring of other modules;
> > > 3.  there are regression unit tests and integration tests along with
> the
> > > code;
> > > 4.  the code is carefully reviewed by 1-2 committers.
> > >
> > > Getting sth. done as a proof-of-concept and then disappearing doesn't
> > seem
> > > a very effective approach..:-)
> > >
> > > Best,
> > > Yingyi
> > >
> > >
> > > On Tue, Jan 31, 2017 at 9:07 PM, Steven Jacobs 
> wrote:
> > >
> > > > Do you think these could be a single project (dataset improvements)?
> > I'm
> > > > wondering about whether one alone would be a full two-three month
> > > project.
> > > > Steven
> > > >
> > > > On Tue, Jan 31, 2017 at 9:00 PM Yingyi Bu 
> wrote:
> > > >
> > > > > I have two proposals for data loaders:
> > > > >
> > > > > 1. add AWS S3 as an external data source;
> > > > > 2. support POSTing a user file on the client side through HTTP into
> > an
> > > > > internal dataset.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Best,
> > > > > Yingyi
> > > > >
> > > > >
> > > > > On Thu, Jan 19, 2017 at 2:06 PM, Steven Jacobs 
> > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > > The Google Summer of Code official start email was sent out
> today.
> > > This
> > > > > can
> > > > > > be a useful opportunity for us as an Apache software. We will
> > > probably
> > > > > want
> > > > > > to look at potential Asterix projects to suggest for student
> > > proposals.
> > > > > > Steven
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: An exception: Input/output error?

[Taewoo Kim]

For the future references, I am sending this message.

As Yingyi mentioned, "Input/Output Error" happened because of a disk error.
I checked "/var/log/messages" as Yingyi suggested. For this time, it didn't
say anything special. I tried the "kmesg" command that were suggested by
Ian and found that there was a disk error. The message is as follows:

INFO: task flush-8:16:14342 blocked for more than 120 seconds.
  Not tainted 2.6.32-642.1.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
flush-8:16D  0 14342  2 0x0080
 8802093e33e0 0046  88003de4d8e0
  88015e063520 0044e48cad574dea 88005e399840
 88015e063520 000583cf9f6a 88015e063ad8 8802093e3fd8
Call Trace:
 [] io_schedule+0x73/0xc0
... ... ... ... (omitted) ... ... ... ...
 [] ? child_rip+0x0/0x20
ata2: lost interrupt (Status 0x51)
ata2.00: limiting speed to UDMA/33:PIO4
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x4000 action 0x6 frozen
ata2: SError: { }
ata2.00: failed command: READ DMA EXT
ata2.00: cmd 25/00:08:30:ed:af/00:00:25:00:00/e0 tag 0 dma 4096 in
 res 40/00:93:35:ed:af/00:00:25:00:00/05 Emask 0x4 (timeout)
ata2.00: status: { DRDY }
ata2: hard resetting link
ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata2.00: configured for UDMA/33
sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 1:0:0:0: [sdb]  Sense Key : Aborted Command [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
25 af ed 35
sd 1:0:0:0: [sdb]  Add. Sense: No additional sense information
sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 25 af ed 30 00 00 08 00
*end_request: I/O error, dev sdb, sector 632286512*


Best,
Taewoo

On Sun, Jan 29, 2017 at 6:14 PM, Taewoo Kim <wangs...@gmail.com> wrote:

> @Yingyi: Thanks. I will look into that error.
>
> Best,
> Taewoo
>
> On Sun, Jan 29, 2017 at 5:50 PM, Yingyi Bu <buyin...@gmail.com> wrote:
>
>> Hi Taewoo,
>>
>> This error message usually indicates that there are some disk
>> failures.
>> Please check /var/log/messages on the machine to see if that's the
>> case.
>>
>> Best,
>> Yingyi
>>
>> On Sun, Jan 29, 2017 at 4:00 PM, Taewoo Kim <wangs...@gmail.com> wrote:
>>
>> > Hi all,
>> >
>> > I'm trying to load json files into a dataset. I have nine nodes so that
>> I
>> > divide a 250GB file into nine files. After one hour, suddenly, I see the
>> > following exception on one node. Has anyone experienced this symptom?
>> >
>> > org.apache.hyracks.api.exceptions.HyracksDataException:
>> > org.apache.hyracks.api.exceptions.HyracksDataException: Input/output
>> error
>> > at
>> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
>> > ble.runInParallel(SuperActivityOperatorNodePushable.java:228)
>> > at
>> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
>> > ble.initialize(SuperActivityOperatorNodePushable.java:84)
>> > at org.apache.hyracks.control.nc.Task.run(Task.java:273)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(
>> > ThreadPoolExecutor.java:1142)
>> > at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> > ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> > Caused by: java.util.concurrent.ExecutionException:
>> > org.apache.hyracks.api.exceptions.HyracksDataException: Input/output
>> error
>> > at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>> > at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>> > at
>> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
>> > ble.runInParallel(SuperActivityOperatorNodePushable.java:222)
>> > ... 5 more
>> > Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
>> > Input/output error
>> > at
>> > org.apache.asterix.external.operators.ExternalDataScanOperatorDescri
>> > ptor$1.initialize(ExternalDataScanOperatorDescriptor.java:65)
>> > at
>> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
>> > ble.lambda$initialize$0(SuperActivityOperatorNodePushable.java:86)
>> > at
>> > org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePusha
>> > ble.lambda$runInParallel$2(SuperActivityOperatorNodePushable.java:216)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> &g

Re: [VOTE] Release Apache AsterixDB 0.9.0 and Hyracks 0.3.0 (RC2)

[Taewoo Kim]

+1

Followed the instruction that Ian wrote at the following and confirmed that
everything works as expected.
https://cwiki.apache.org/confluence/display/ASTERIXDB/Release+Verification

Best,
Taewoo

On Thu, Jan 19, 2017 at 4:24 PM, Xikui Wang  wrote:

> +1
>
> Verified twitter adaptor by dropping twitter4j library into the repo
> directory and repack
> server-assembly with twitter4j library.
>
> Best,
> Xikui
>
> On Thu, Jan 19, 2017 at 3:00 PM, Mike Carey  wrote:
>
> > +1 for this release
> >
> > Successfully downloaded and started the system and did some SQL++
> tutorial
> > examples using the NCService binary installer.  Worked like a charm!
> >
> > Cheers,
> >
> > Mike
> >
> >
> >
> > On 1/18/17 7:50 PM, Ian Maxon wrote:
> >
> >> Hi again everyone,
> >>
> >> Please verify and vote on the first non-incubating Apache AsterixDB
> >> Release!
> >> This 2nd RC addresses build issues noticed in the previous RC, along
> with
> >> some minor license tweaks.
> >> This release utilizes a series of improvements around the actual release
> >> process that will hopefully shorten the interval between releases. A
> >> further email detailing the features contained in this release as
> compared
> >> to the previous incubating release will be forthcoming once a suitable
> RC
> >> passes voting.
> >>
> >> The tags to be voted on are:
> >>
> >> apache-asterixdb-0.9.0-rc2
> >> commit: 4383bdde78c02d597be65ecf467c5a7df85a2055
> >> link:
> >> https://git-wip-us.apache.org/repos/asf?p=asterixdb.git;a=ta
> >> g;h=refs/tags/apache-asterixdb-0.9.0-rc2
> >>
> >> and
> >>
> >> apache-hyracks-0.3.0-rc2
> >> commit: def643d586b62b2616b8ab8e6fc3ba598cf5ad67
> >> link:
> >> https://git-wip-us.apache.org/repos/asf?p=asterixdb.git;a=ta
> >> g;h=refs/tags/apache-hyracks-0.3.0-rc2
> >>
> >> The artifacts, sha1's, and signatures are (for each artifact), are at:
> >>
> >> AsterixDB Source
> >> https://dist.apache.org/repos/dist/dev/asterixdb/apache-
> >> asterixdb-0.9.0-source-release.zip
> >> https://dist.apache.org/repos/dist/dev/asterixdb/apache-
> >> asterixdb-0.9.0-source-release.zip.asc
> >> https://dist.apache.org/repos/dist/dev/asterixdb/apache-
> >> asterixdb-0.9.0-source-release.zip.sha1
> >>
> >> SHA1: 49f8df822c6273a310027d3257a79afb45c8d446
> >>
> >> Hyracks Source
> >> https://dist.apache.org/repos/dist/dev/asterixdb/apache-
> >> hyracks-0.3.0-source-release.zip
> >> https://dist.apache.org/repos/dist/dev/asterixdb/apache-
> >> hyracks-0.3.0-source-release.zip.asc
> >> https://dist.apache.org/repos/dist/dev/asterixdb/apache-
> >> hyracks-0.3.0-source-release.zip.sha1
> >>
> >> SHA1: 4d042cab164347f0cc5cc1cfb3da8d4f02eea1de
> >>
> >> AsterixDB NCService Installer:
> >> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-
> >> server-0.9.0-binary-assembly.zip
> >> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-
> >> server-0.9.0-binary-assembly.zip.asc
> >> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-
> >> server-0.9.0-binary-assembly.zip.sha1
> >>
> >> SHA1: 46c4cc3dc09e915d4b1bc6f912faef389488fdb6
> >>
> >> AsterixDB Managix Installer
> >> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-
> >> installer-0.9.0-binary-assembly.zip
> >> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-
> >> installer-0.9.0-binary-assembly.zip.asc
> >> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-
> >> installer-0.9.0-binary-assembly.zip.sha1
> >>
> >> SHA1: 41497dbadb0ad281ba0a10ee87eaa5f7afa78cef
> >>
> >> AsterixDB YARN Installer
> >> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-
> >> yarn-0.9.0-binary-assembly.zip
> >> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-
> >> yarn-0.9.0-binary-assembly.zip.asc
> >> https://dist.apache.org/repos/dist/dev/asterixdb/asterix-
> >> yarn-0.9.0-binary-assembly.zip.sha1
> >>
> >> SHA1: 3ade0d2957e7f3e465e357aced6712ef72598613
> >>
> >> Additionally, a staged maven repository is available at:
> >>
> >> https://repository.apache.org/content/repositories/
> orgapacheasterix-1024/
> >>
> >> The KEYS file containing the PGP keys used to sign the release can be
> >> found at
> >>
> >> https://dist.apache.org/repos/dist/release/asterixdb/KEYS
> >>
> >> RAT was executed as part of Maven via the RAT maven plugin, but
> >> excludes files that are:
> >>
> >> - data for tests
> >> - procedurally generated,
> >> - or source files which come without a header mentioning their license,
> >>but have an explicit reference in the LICENSE file.
> >>
> >>
> >> The vote is open for 72 hours, or until the necessary number of votes
> >> (3 +1) has been reached.
> >>
> >> Please vote
> >> [ ] +1 release these packages as Apache AsterixDB 0.9.0 and
> >> Apache Hyracks 0.3.0
> >> [ ] 0 No strong feeling either way
> >> [ ] -1 do not release one or both packages because ...
> >>
> >> Thanks!
> >>
> >>
> >
>

Re: Line limit (120) is not enforced for an assignment statement.

[Taewoo Kim]

Finally, two things have changed in the code style format. You can download
new code file.

 - Do not remove blank lines in the java-doc comment section.
 - Wrap an assignment statement when necessary (currently, it doesn't).

https://cwiki.apache.org/confluence/display/ASTERIXDB/Formatting

Best,
Taewoo

On Tue, Oct 25, 2016 at 4:34 PM, Yingyi Bu <buyin...@gmail.com> wrote:

> >> @Yingyi: I wonder whether your proposed change (do not remove blank
> lines
> >> in the comment section) is already applied to the format file
>
> It looks that I haven't...
> You can update this row:
>
>  "org.eclipse.jdt.core.formatter.comment.clear_blank_
> lines_in_javadoc_comment"
> value="false"/>
>
> Thanks!
>
> Yingyi
>
>
> On Tue, Oct 25, 2016 at 4:26 PM, Till Westmann <ti...@apache.org> wrote:
>
> > +1
> >
> >
> > On 25 Oct 2016, at 13:29, Taewoo Kim wrote:
> >
> > There is a formatter issue regarding an assignment statement whose length
> >> is over 120. An example is as follows:
> >> private final IBinaryComparator strLowerCaseTokenCmp =
> >> AqlBinaryComparatorFactoryProvider.UTF8STRING_LOWERCASE_TOKE
> >> N_POINTABLE_INSTANCE.createBinaryComparator();
> >>
> >> Currently, the formatter doesn't wrap this line into two lines because
> "No
> >> Wrap" policy is applied. So, I would like to propose the following
> change.
> >> After applying this change, we can keep the hard limit (120) well.
> >>
> >> AsterixCodeFormatProfile.xml
> >>
> >> Line Wrapping -> Expressions -> Assignments
> >>
> >> (Before) Line Wrapping Policy: Do not Wrap
> >> (After) Wrap where necessary
> >>
> >> @Yingyi: I wonder whether your proposed change (do not remove blank
> lines
> >> in the comment section) is already applied to the format file. Please
> let
> >> me know. Then, I will download the file from
> >> https://cwiki.apache.org/confluence/download/attachments/
> >> 61322291/AsterixCodeFormatProfile.xml
> >> and apply my change and upload the file.
> >>
> >
>

Re: Exception refactoring proposal

[Taewoo Kim]

+1

Best,
Taewoo

On Fri, Jan 6, 2017 at 3:27 PM, Till Westmann  wrote:

> Hi,
>
> I like the proposal and I’d like to propose on more change:
> Let’s not start the names of ErrorCodes with "ERROR_".
> In the code they are usually used as
>
> ErrorCode.ERROR_THIS_IS_THE_PROBLEM
>
> and it seems that
>
> ErrorCode.THIS_IS_THE_PROBLEM
>
> would work just as well.
>
> Thoughts?
>
> Cheers,
> Till
>
>
>
> On 5 Jan 2017, at 11:12, Yingyi Bu wrote:
>
> Hi dev,
>>
>> I recent made a change that moved some exception error message
>> templates from the source code into properties files [1]. Here is an
>> exception refactoring proposal [2].
>>
>> I propose that:
>> --  from now on, we all follow the proposal when we want to throw
>> exceptions, and enforce the key design during our code reviews.
>> --  for existing code that does not follow the proposal, we refactor
>> them gradually according to the proposal.
>>
>> Feel free to share your thoughts.
>> Thanks!
>>
>> Best,
>> Yingyi
>>
>> [1] https://asterix-gerrit.ics.uci.edu/#/c/1410/
>> [2] https://cwiki.apache.org/confluence/display/ASTERIXDB/Except
>> ion+Handling
>>
>

Re: Anybody seen this error? HyracksDataException: The file with absolute path: XXX is outside all io devices

[Taewoo Kim]

Thanks Abdullah. The patch is already on the Jenkins:
https://asterix-gerrit.ics.uci.edu/#/c/1388/  - patch set 9. I am not
trying to touch the internal of indexing part. What I'm trying to do is
just creating an inverted index type index. :-) Anyway, thanks for the
information.

Best,
Taewoo

On Tue, Dec 20, 2016 at 11:05 AM, abdullah alamoudi <bamou...@gmail.com>
wrote:

> Taewoo,
> If you can push the branch somewhere online where I can see the diff from
> master, I will take a look. This could be due to incorrect setup of
> location constraints.
> Be careful and ensure that indexes are built in the right place and things
> should be good to go. you can also take a look at how FileSplits and
> location constraints
> are selected for other indexes and do the same. Then the failures should
> go away.
>
> Cheers,
> Abdullah.
>
>
> > On Dec 20, 2016, at 10:57 AM, Taewoo Kim <wangs...@gmail.com> wrote:
> >
> > @Abdullah: Thanks for the info. It's triggered on my branch. Could you
> > think of a reason for your comment - "it seems like it is inside an IO
> > device but could be in the wrong node?"?
> >
> > Best,
> > Taewoo
> >
> > On Tue, Dec 20, 2016 at 10:44 AM, abdullah alamoudi <bamou...@gmail.com>
> > wrote:
> >
> >> Hi Taewoo,
> >> So recently we did a refactoring change to FileSplit and FileReference
> >> classes. Part of the change was that an index that is managed by
> asterixdb
> >> can only be inside an IO device. Looking at the error trace, seems like
> a
> >> new incorrect use of the FileSplit where
> >> a FileSplit is defined as an absolute file split and the path points to
> a
> >> place outside all IO devices, hence the IO manager throws an exception.
> >>
> >> Looking at the file path, it seems like it is inside an IO device but
> >> could be in the wrong node?
> >>
> >>
> >> Ultimately, we should not use full paths and instead use a relative
> path:
> >> instead of ../asterix-server/target/tmp/asterix_nc1/iodevice1/storage/
> >> partition_0/test/MyData_idx_fulltext_index_title/2016-12-
> >> 19-22-32-36-543_2016-12-19-22-32-36-543_i, we should use
> >> partition_0/test/MyData_idx_fulltext_index_title/2016-12-
> >> 19-22-32-36-543_2016-12-19-22-32-36-543_i and we can select the IO
> device
> >> number or leave it to the IOManager to select one of the IO devices.
> >>
> >> Q. Was this triggered on master???
> >>
> >>
> >>> On Dec 20, 2016, at 10:08 AM, Taewoo Kim <wangs...@gmail.com> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> I am now working on index-based version of full-text search. The
> >> full-text
> >>> index is actually SINGLE_PARTITION_WORD_INVIX type, rather than
> >>> LENGTH_PARTITIONED_WORD_INVIX type that we are currently using. I know
> >> that
> >>> we are not using this index type quite a long time. But, the strange
> >> thing
> >>> is that the build is always successful. That means, execution test
> >> succeeds
> >>> without any issue.  But, for the NCServiceExecutionIT test, the
> following
> >>> error is generated. Is there anybody who saw this error before? Any
> >>> comments would be appreciated.
> >>>
> >>> https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-
> >> integration-tests/org.apache.asterix$asterix-server/1352/
> >> testReport/junit/org.apache.asterix.server.test/
> NCServiceExecutionIT/test_
> >> NCServiceExecutionTest_30__fulltext__fulltext_index_01_/
> >>>
> >>> Best,
> >>> Taewoo
> >>
> >>
>
>

Re: BAD flag on code reviews

[Taewoo Kim]

@Steven: can we check this locally before uploading a patch set?

Best,
Taewoo

On Thu, Dec 15, 2016 at 9:37 AM, Ian Maxon  wrote:

> As a sidenote, it doesn't block, the flag acts the same as SonarQube. If it
> does break though, please do add a BAD fellow to the review so there's
> visibility as to what broke and when.
>
> On Thu, Dec 15, 2016 at 9:32 AM, Steven Jacobs  wrote:
>
> > Hi all,
> > Those of you with reviews going on may have noticed that there is a new
> > column on gerrit, called bad. This flag is because we now have an apache
> > codebase for the BAD extension (You can see it here if you are
> interested:
> > https://git-wip-us.apache.org/repos/asf/asterixdb-bad.git) which enables
> > channels and soon stored procedures.
> >
> > The BAD extension is dependent on the core of Asterix itself. The new
> flag
> > is to check whether new changes to Asterix will break the BAD extension.
> In
> > general most changes should be getting a +1 here. The main thing we are
> > trying to avoid is a change to Asterix intended as an enhancement or
> > cleanup that inadvertently removes functionality needed for Asterix
> > extensions.
> >
> > "What should I do if I see a -1?"
> >
> > If you do happen to get a -1 from BAD, the easy thing to do is to add me
> > (or another extension expert such as Till or Abdullah) to the code
> review.
> > That way I can look at the affecting part of the change and see how to
> > handle it.
> >
> > Please let me know if you have any questions or concerns.
> > Thanks,
> > Steven
> >
>

Re: One of NC node is not stopping for a hash join.

[Taewoo Kim]

Some additional info to Mike's comment:

I discussed this "skew-resistant parallel join" with Mike yesterday and
checked some papers briefly. It looks like the common strategy is splitting
the block of tuples that has the same join key from one side (e.g., R) to
several nodes and let the tuple from the other side (e.g., S) to be
broadcasted to those nodes if they have the same value. Teradata utilizes
the stats to apply this plan over hash plan.

*Teradata*: if the statistics is available, then the optimizer chooses
"broadcasting" strategy on skewed values over "hash" strategy.
http://community.teradata.com/t5/Database/tuning-the-Skewed-joins/td-p/13305
http://www.info.teradata.com/htmlpubs/DB_TTU_14_00/index.html#page/Database_Management/B035_1094_111A/ch09.050.139.html

*Pig*: use "skewed" keyword to let Pig to do skewed join handling.
https://pig.apache.org/docs/r0.9.2/perf.html#skewed-joins


Best,
Taewoo

On Wed, Dec 7, 2016 at 11:27 PM, Mike Carey <dtab...@gmail.com> wrote:

> To all, just to clarify:  This is a self-join (equijoin) query on a
> non-key attribute using real data (Amazon reviews, key is reviewer id)
> which has a non-uniform value distribution in terms of the number of
> entries per join key value, and in this case we really (someday...) need a
> more sophisticated parallel join method to have a balanced load.  (This
> case needs several nodes to work together on joining the biggest value with
> itself, actually - we need to look at how Teradata, Pig, and other systems
> with skew-resistant join algorithms handle this case.  Could be a good
> immigration project for someone joining the team, perhaps.  Or maybe it
> would be something of interest to Mingda+Tyson, since they've been joining
> things recently?)
>
> Cheers,
>
> Mike
>
>
>
> On 12/7/16 3:08 PM, Taewoo Kim wrote:
>
>> In short, the reason on why a specific one node among 9 nodes didn't stop
>> its hash-join job was due to a skewness (out of 9M records, 40,000 records
>> was containing the same join key) as Abdullah suggested. Thanks all for
>> the
>> information. Our system works fine as expected for this matter! Along the
>> way, I found some optimization chances and excessive byte[] allocation
>> issues and fixing this will increase the performance. :-)
>>
>> Best,
>> Taewoo
>>
>> On Mon, Dec 5, 2016 at 9:39 PM, Taewoo Kim <wangs...@gmail.com> wrote:
>>
>> @Abdullah: Thanks. I missed your e-mail and just checked that. Will try.
>>>
>>> Best,
>>> Taewoo
>>>
>>> On Fri, Dec 2, 2016 at 10:32 AM, abdullah alamoudi <bamou...@gmail.com>
>>> wrote:
>>>
>>> Taewoo,
>>>> You can use the diagnostics end point (/admin/diagnostics) to look at
>>>> all
>>>> the stack traces from a single interface when that happens. This could
>>>> give
>>>> an idea on what is happening in such case.
>>>> Although, from what you described, it could be that we have some
>>>> skewness
>>>> during query execution? (could be nulls,missing? any special values?).
>>>> That
>>>> is also worth considering.
>>>>
>>>> Trying to help without enough context :-). Cheers,
>>>> Abdullah.
>>>>
>>>> On Dec 2, 2016, at 10:22 AM, Taewoo Kim <wangs...@gmail.com> wrote:
>>>>>
>>>>> Additional note: @Till: Yes. It happened again for the same hash-join
>>>>> query. As we can see in the bold part of the following CC.log, one node
>>>>> alone was executing for two hours.
>>>>>
>>>>>
>>>>> Dec 01, 2016 10:41:56 PM
>>>>> org.apache.hyracks.control.cc.scheduler.ActivityClusterPlanner
>>>>> planActivityCluster
>>>>> INFO: Plan for org.apache.hyracks.api.job.ActivityCluster@383ecfdd
>>>>> Dec 01, 2016 10:41:56 PM
>>>>> org.apache.hyracks.control.cc.scheduler.ActivityClusterPlanner
>>>>> planActivityCluster
>>>>> INFO: Built 1 Task Clusters
>>>>> Dec 01, 2016 10:41:56 PM
>>>>> org.apache.hyracks.control.cc.scheduler.ActivityClusterPlanner
>>>>> planActivityCluster
>>>>> INFO: Tasks: [TID:ANID:ODID:1:1:0, TID:ANID:ODID:1:1:1,
>>>>> TID:ANID:ODID:1:1:2, TID:ANID:ODID:1:1:3, TID:ANID:ODID:1:1:4,
>>>>> TID:ANID:ODID:1:1:5, TID:ANID:ODID:1:1:6, TID:ANID:ODID:1:1:7,
>>>>> TID:ANID:ODID:1:1:8, TID:ANID:ODID:1:1:9, TID:ANID:ODID:1:1:10,
>>>>> TID:ANID:ODID:1:1:11, TID:

Re: One of NC node is not stopping for a hash join.

[Taewoo Kim]

In short, the reason on why a specific one node among 9 nodes didn't stop
its hash-join job was due to a skewness (out of 9M records, 40,000 records
was containing the same join key) as Abdullah suggested. Thanks all for the
information. Our system works fine as expected for this matter! Along the
way, I found some optimization chances and excessive byte[] allocation
issues and fixing this will increase the performance. :-)

Best,
Taewoo

On Mon, Dec 5, 2016 at 9:39 PM, Taewoo Kim <wangs...@gmail.com> wrote:

> @Abdullah: Thanks. I missed your e-mail and just checked that. Will try.
>
> Best,
> Taewoo
>
> On Fri, Dec 2, 2016 at 10:32 AM, abdullah alamoudi <bamou...@gmail.com>
> wrote:
>
>> Taewoo,
>> You can use the diagnostics end point (/admin/diagnostics) to look at all
>> the stack traces from a single interface when that happens. This could give
>> an idea on what is happening in such case.
>> Although, from what you described, it could be that we have some skewness
>> during query execution? (could be nulls,missing? any special values?). That
>> is also worth considering.
>>
>> Trying to help without enough context :-). Cheers,
>> Abdullah.
>>
>> > On Dec 2, 2016, at 10:22 AM, Taewoo Kim <wangs...@gmail.com> wrote:
>> >
>> > Additional note: @Till: Yes. It happened again for the same hash-join
>> > query. As we can see in the bold part of the following CC.log, one node
>> > alone was executing for two hours.
>> >
>> >
>> > Dec 01, 2016 10:41:56 PM
>> > org.apache.hyracks.control.cc.scheduler.ActivityClusterPlanner
>> > planActivityCluster
>> > INFO: Plan for org.apache.hyracks.api.job.ActivityCluster@383ecfdd
>> > Dec 01, 2016 10:41:56 PM
>> > org.apache.hyracks.control.cc.scheduler.ActivityClusterPlanner
>> > planActivityCluster
>> > INFO: Built 1 Task Clusters
>> > Dec 01, 2016 10:41:56 PM
>> > org.apache.hyracks.control.cc.scheduler.ActivityClusterPlanner
>> > planActivityCluster
>> > INFO: Tasks: [TID:ANID:ODID:1:1:0, TID:ANID:ODID:1:1:1,
>> > TID:ANID:ODID:1:1:2, TID:ANID:ODID:1:1:3, TID:ANID:ODID:1:1:4,
>> > TID:ANID:ODID:1:1:5, TID:ANID:ODID:1:1:6, TID:ANID:ODID:1:1:7,
>> > TID:ANID:ODID:1:1:8, TID:ANID:ODID:1:1:9, TID:ANID:ODID:1:1:10,
>> > TID:ANID:ODID:1:1:11, TID:ANID:ODID:1:1:12, TID:ANID:ODID:1:1:13,
>> > TID:ANID:ODID:1:1:14, TID:ANID:ODID:1:1:15, TID:ANID:ODID:1:1:16,
>> > TID:ANID:ODID:1:1:17, TID:ANID:ODID:4:0:0]
>> > Dec 01, 2016 10:43:18 PM
>> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
>> > INFO: Executing: TaskComplete: [ss1120_nc3[JID:5:TAID:TID:ANI
>> D:ODID:1:1:5:0]
>> > Dec 01, 2016 10:43:22 PM
>> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
>> > INFO: Executing: TaskComplete: [ss1120_nc4[JID:5:TAID:TID:ANI
>> D:ODID:1:1:7:0]
>> > Dec 01, 2016 10:43:23 PM
>> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
>> > INFO: Executing: TaskComplete:
>> > [ss1120_nc9[JID:5:TAID:TID:ANID:ODID:1:1:16:0]
>> > Dec 01, 2016 10:43:28 PM
>> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
>> > INFO: Executing: TaskComplete: [ss1120_nc2[JID:5:TAID:TID:ANI
>> D:ODID:1:1:2:0]
>> > Dec 01, 2016 10:43:31 PM
>> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
>> > INFO: Executing: TaskComplete: [ss1120_nc2[JID:5:TAID:TID:ANI
>> D:ODID:1:1:3:0]
>> > Dec 01, 2016 10:43:34 PM
>> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
>> > INFO: Executing: TaskComplete: [ss1120_nc5[JID:5:TAID:TID:ANI
>> D:ODID:1:1:8:0]
>> > Dec 01, 2016 10:43:40 PM
>> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
>> > INFO: Executing: TaskComplete:
>> > [ss1120_nc9[JID:5:TAID:TID:ANID:ODID:1:1:17:0]
>> > Dec 01, 2016 10:43:41 PM
>> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
>> > INFO: Executing: TaskComplete: [ss1120_nc4[JID:5:TAID:TID:ANI
>> D:ODID:1:1:6:0]
>> > Dec 01, 2016 10:43:49 PM
>> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
>> > INFO: Executing: TaskComplete:
>> > [ss1120_nc7[JID:5:TAID:TID:ANID:ODID:1:1:12:0]
>> > Dec 01, 2016 10:43:51 PM
>> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
>> > INFO: Executing: TaskComplete: [ss1120_nc1[JID:5:TAID:TID:ANI
>> D:ODID:1:1:1:0]
>> > Dec 01, 2016 10:43:53 PM
>> > org.apache.hyracks

Re: One of NC node is not stopping for a hash join.

[Taewoo Kim]

@Abdullah: Thanks. I missed your e-mail and just checked that. Will try.

Best,
Taewoo

On Fri, Dec 2, 2016 at 10:32 AM, abdullah alamoudi <bamou...@gmail.com>
wrote:

> Taewoo,
> You can use the diagnostics end point (/admin/diagnostics) to look at all
> the stack traces from a single interface when that happens. This could give
> an idea on what is happening in such case.
> Although, from what you described, it could be that we have some skewness
> during query execution? (could be nulls,missing? any special values?). That
> is also worth considering.
>
> Trying to help without enough context :-). Cheers,
> Abdullah.
>
> > On Dec 2, 2016, at 10:22 AM, Taewoo Kim <wangs...@gmail.com> wrote:
> >
> > Additional note: @Till: Yes. It happened again for the same hash-join
> > query. As we can see in the bold part of the following CC.log, one node
> > alone was executing for two hours.
> >
> >
> > Dec 01, 2016 10:41:56 PM
> > org.apache.hyracks.control.cc.scheduler.ActivityClusterPlanner
> > planActivityCluster
> > INFO: Plan for org.apache.hyracks.api.job.ActivityCluster@383ecfdd
> > Dec 01, 2016 10:41:56 PM
> > org.apache.hyracks.control.cc.scheduler.ActivityClusterPlanner
> > planActivityCluster
> > INFO: Built 1 Task Clusters
> > Dec 01, 2016 10:41:56 PM
> > org.apache.hyracks.control.cc.scheduler.ActivityClusterPlanner
> > planActivityCluster
> > INFO: Tasks: [TID:ANID:ODID:1:1:0, TID:ANID:ODID:1:1:1,
> > TID:ANID:ODID:1:1:2, TID:ANID:ODID:1:1:3, TID:ANID:ODID:1:1:4,
> > TID:ANID:ODID:1:1:5, TID:ANID:ODID:1:1:6, TID:ANID:ODID:1:1:7,
> > TID:ANID:ODID:1:1:8, TID:ANID:ODID:1:1:9, TID:ANID:ODID:1:1:10,
> > TID:ANID:ODID:1:1:11, TID:ANID:ODID:1:1:12, TID:ANID:ODID:1:1:13,
> > TID:ANID:ODID:1:1:14, TID:ANID:ODID:1:1:15, TID:ANID:ODID:1:1:16,
> > TID:ANID:ODID:1:1:17, TID:ANID:ODID:4:0:0]
> > Dec 01, 2016 10:43:18 PM
> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
> > INFO: Executing: TaskComplete: [ss1120_nc3[JID:5:TAID:TID:
> ANID:ODID:1:1:5:0]
> > Dec 01, 2016 10:43:22 PM
> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
> > INFO: Executing: TaskComplete: [ss1120_nc4[JID:5:TAID:TID:
> ANID:ODID:1:1:7:0]
> > Dec 01, 2016 10:43:23 PM
> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
> > INFO: Executing: TaskComplete:
> > [ss1120_nc9[JID:5:TAID:TID:ANID:ODID:1:1:16:0]
> > Dec 01, 2016 10:43:28 PM
> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
> > INFO: Executing: TaskComplete: [ss1120_nc2[JID:5:TAID:TID:
> ANID:ODID:1:1:2:0]
> > Dec 01, 2016 10:43:31 PM
> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
> > INFO: Executing: TaskComplete: [ss1120_nc2[JID:5:TAID:TID:
> ANID:ODID:1:1:3:0]
> > Dec 01, 2016 10:43:34 PM
> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
> > INFO: Executing: TaskComplete: [ss1120_nc5[JID:5:TAID:TID:
> ANID:ODID:1:1:8:0]
> > Dec 01, 2016 10:43:40 PM
> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
> > INFO: Executing: TaskComplete:
> > [ss1120_nc9[JID:5:TAID:TID:ANID:ODID:1:1:17:0]
> > Dec 01, 2016 10:43:41 PM
> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
> > INFO: Executing: TaskComplete: [ss1120_nc4[JID:5:TAID:TID:
> ANID:ODID:1:1:6:0]
> > Dec 01, 2016 10:43:49 PM
> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
> > INFO: Executing: TaskComplete:
> > [ss1120_nc7[JID:5:TAID:TID:ANID:ODID:1:1:12:0]
> > Dec 01, 2016 10:43:51 PM
> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
> > INFO: Executing: TaskComplete: [ss1120_nc1[JID:5:TAID:TID:
> ANID:ODID:1:1:1:0]
> > Dec 01, 2016 10:43:53 PM
> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
> > INFO: Executing: TaskComplete: [ss1120_nc5[JID:5:TAID:TID:
> ANID:ODID:1:1:9:0]
> > Dec 01, 2016 10:43:58 PM
> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
> > INFO: Executing: TaskComplete:
> > [ss1120_nc7[JID:5:TAID:TID:ANID:ODID:1:1:13:0]
> > Dec 01, 2016 10:44:25 PM
> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
> > INFO: Executing: TaskComplete:
> > [ss1120_nc8[JID:5:TAID:TID:ANID:ODID:1:1:14:0]
> > Dec 01, 2016 10:44:29 PM
> > org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
> > INFO: Executing: TaskComplete:
> > [ss1120_nc6[JID:5:TAID:TID:ANID:ODID:1:1:11:0]
> > Dec 01, 2016 10:44:51 PM
> > org.apache.hyracks.control.common

Re: One of NC node is not stopping for a hash join.

[Taewoo Kim]

leanup for JobRun with id: JID:5
Dec 02, 2016 12:30:19 AM
org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
INFO: Executing: JobletCleanupNotification
Dec 02, 2016 12:30:19 AM
org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
INFO: Executing: JobletCleanupNotification
Dec 02, 2016 12:30:19 AM
org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
INFO: Executing: JobletCleanupNotification
Dec 02, 2016 12:30:19 AM
org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
INFO: Executing: JobletCleanupNotification
Dec 02, 2016 12:30:19 AM
org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
INFO: Executing: JobletCleanupNotification
Dec 02, 2016 12:30:19 AM
org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
INFO: Executing: JobletCleanupNotification
Dec 02, 2016 12:30:19 AM
org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
INFO: Executing: JobletCleanupNotification
Dec 02, 2016 12:30:19 AM
org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
INFO: Executing: JobletCleanupNotification
Dec 02, 2016 12:30:19 AM
org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
INFO: Executing: JobletCleanupNotification
Dec 02, 2016 12:30:19 AM org.apache.asterix.active.ActiveLifecycleListener
notifyJobFinish
INFO: NO NEED TO NOTIFY JOB FINISH!
Dec 02, 2016 12:30:19 AM
org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
INFO: Executing: GetResultPartitionLocations: JobId@JID:5 ResultSetId@RSID:0
Known@null

Best,
Taewoo

On Thu, Dec 1, 2016 at 10:43 PM, Taewoo Kim <wangs...@gmail.com> wrote:

> @Ian: I have a separate CC on one node that doesn't have a NC. Yourkit
> might be a good way to find the reason. Thanks.
>
> @Till: I think so. I am sending the same query now to see what happens
> this time.
>
> Best,
> Taewoo
>
> On Thu, Dec 1, 2016 at 10:41 PM, Till Westmann <ti...@apache.org> wrote:
>
>> Hi Taewoo,
>>
>> is this behavior reproducible?
>>
>> Cheers,
>> Till
>>
>>
>> On 1 Dec 2016, at 22:14, Taewoo Kim wrote:
>>
>> PS: It took 2 more hours to finish the job on one NC. I wonder why this
>>> happens.
>>>
>>> Dec 01, 2016 7:19:35 PM
>>> org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
>>> INFO: Executing: NotifyTaskComplete
>>> Dec 01, 2016 9:11:23 PM
>>> org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
>>> INFO: Executing: CleanupJoblet
>>> Dec 01, 2016 9:11:23 PM
>>> org.apache.hyracks.control.nc.work.CleanupJobletWork run
>>> INFO: Cleaning up after job: JID:4
>>> Dec 01, 2016 9:11:23 PM org.apache.hyracks.control.nc.Joblet close
>>> WARNING: Freeing leaked 54919521 bytes
>>>
>>> Best,
>>> Taewoo
>>>
>>> On Thu, Dec 1, 2016 at 8:39 PM, Taewoo Kim <wangs...@gmail.com> wrote:
>>>
>>> Hi All,
>>>>
>>>> Have you experienced this case?
>>>>
>>>> I have 9 NCs and the CPU utilization of one NC shows 100% for 1 hour and
>>>> 30 minutes while other NCs have finished their job about 1 hour ago.
>>>> Even
>>>> the problematic NC shows the following log at the end. So, looks like
>>>> it's
>>>> done but I'm not sure why this job never finishes. It's a simple hash
>>>> join
>>>> for 9M records on 9 nodes.
>>>>
>>>> Dec 01, 2016 7:18:02 PM org.apache.hyracks.control.com
>>>> mon.work.WorkQueue$WorkerThread
>>>> run
>>>> INFO: Executing: NotifyTaskComplete
>>>>
>>>> Best,
>>>> Taewoo
>>>>
>>>>
>

Re: One of NC node is not stopping for a hash join.

[Taewoo Kim]

@Ian: I have a separate CC on one node that doesn't have a NC. Yourkit
might be a good way to find the reason. Thanks.

@Till: I think so. I am sending the same query now to see what happens this
time.

Best,
Taewoo

On Thu, Dec 1, 2016 at 10:41 PM, Till Westmann <ti...@apache.org> wrote:

> Hi Taewoo,
>
> is this behavior reproducible?
>
> Cheers,
> Till
>
>
> On 1 Dec 2016, at 22:14, Taewoo Kim wrote:
>
> PS: It took 2 more hours to finish the job on one NC. I wonder why this
>> happens.
>>
>> Dec 01, 2016 7:19:35 PM
>> org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
>> INFO: Executing: NotifyTaskComplete
>> Dec 01, 2016 9:11:23 PM
>> org.apache.hyracks.control.common.work.WorkQueue$WorkerThread run
>> INFO: Executing: CleanupJoblet
>> Dec 01, 2016 9:11:23 PM
>> org.apache.hyracks.control.nc.work.CleanupJobletWork run
>> INFO: Cleaning up after job: JID:4
>> Dec 01, 2016 9:11:23 PM org.apache.hyracks.control.nc.Joblet close
>> WARNING: Freeing leaked 54919521 bytes
>>
>> Best,
>> Taewoo
>>
>> On Thu, Dec 1, 2016 at 8:39 PM, Taewoo Kim <wangs...@gmail.com> wrote:
>>
>> Hi All,
>>>
>>> Have you experienced this case?
>>>
>>> I have 9 NCs and the CPU utilization of one NC shows 100% for 1 hour and
>>> 30 minutes while other NCs have finished their job about 1 hour ago. Even
>>> the problematic NC shows the following log at the end. So, looks like
>>> it's
>>> done but I'm not sure why this job never finishes. It's a simple hash
>>> join
>>> for 9M records on 9 nodes.
>>>
>>> Dec 01, 2016 7:18:02 PM org.apache.hyracks.control.com
>>> mon.work.WorkQueue$WorkerThread
>>> run
>>> INFO: Executing: NotifyTaskComplete
>>>
>>> Best,
>>> Taewoo
>>>
>>>

Fwd: [jira] [Created] (ASTERIXDB-1736) Grace Hash Join and Hybrid Hash Join are not being used.

[Taewoo Kim]

Hi all,

Please share your thought on this issue. In short, Grace Hash Join and
Hybrid Hash Join are not being used. We only use Optimized Hybrid Hash
Join. Therefore, I think it would be better to remove them.
https://issues.apache.org/jira/browse/ASTERIXDB-1736
<https://issues.apache.org/jira/browse/ASTERIXDB-1736>
-- Forwarded message --
From: Taewoo Kim (JIRA) <j...@apache.org>
Date: Fri, Nov 18, 2016 at 5:06 PM
Subject: [jira] [Created] (ASTERIXDB-1736) Grace Hash Join and Hybrid Hash
Join are not being used.
To: notificati...@asterixdb.incubator.apache.org


Taewoo Kim created ASTERIXDB-1736:
-

 Summary: Grace Hash Join and Hybrid Hash Join are not being
used.
 Key: ASTERIXDB-1736
 URL: https://issues.apache.org/jira/browse/ASTERIXDB-1736
 Project: Apache AsterixDB
  Issue Type: Improvement
    Reporter: Taewoo Kim
    Assignee: Taewoo Kim


As the title says, Grace Hash Join and Hybrid Hash Join are not being used.
I suggest that we remove these two join methods. Here are my findings for
these two joins.

1) Grace Hash Join
GraceHashJoinOperatorDescriptor is only called from two places:
org.apache.hyracks.examples.tpch.client.join and
TPCHCustomerOrderHashJoinTest.
One is a Hyracks example (tpch.client) and the other is a unit test. This
join is not used currently (not chosen during the compilation).

2) Hybrid Hash Join
During the compilation, the optimizer decides whether it will use Hybrid
Hash Join or Optimized Hybrid Hash Join.
If the hash function family for each key variable is set, then we use the
optimized hybrid hash join.
If not, we use the hybrid hash join. However, in fact, this path - hybrid
hash join path will never be chosen. Let's check the code.

{code:title=HybridHashJoinPOperator.java|borderStyle=solid}
IBinaryHashFunctionFamily[] hashFunFamilies = JobGenHelper.
variablesToBinaryHashFunctionFamilies(keysLeftBranch,
env, context);

...

boolean optimizedHashJoin = true;
for (IBinaryHashFunctionFamily family : hashFunFamilies) {
if (family == null) {
optimizedHashJoin = false;
break;
}
}

if (optimizedHashJoin) {
opDesc = generateOptimizedHashJoinRuntime(context,
inputSchemas, keysLeft, keysRight, hashFunFamilies,
comparatorFactories, predEvaluatorFactory,
recDescriptor, spec);
} else {
opDesc = generateHashJoinRuntime(context, inputSchemas,
keysLeft, keysRight, hashFunFactories,
comparatorFactories, predEvaluatorFactory,
recDescriptor, spec);
}
{code}

As we can see, optimizedHashJoin is set to false only when the hash family
is null.
Then, how do we assign the hashfamily for each key variable?

{code:title=JobGenHelper.java|borderStyle=solid}
public static IBinaryHashFunctionFamily[] variablesToBinaryHashFunctionF
amilies(
Collection varLogical,
IVariableTypeEnvironment env, JobGenContext context)
throws AlgebricksException {
IBinaryHashFunctionFamily[] funFamilies = new
IBinaryHashFunctionFamily[varLogical.size()];
int i = 0;
IBinaryHashFunctionFamilyProvider bhffProvider = context.
getBinaryHashFunctionFamilyProvider();
for (LogicalVariable var : varLogical) {
Object type = env.getVarType(var);
funFamilies[i++] = bhffProvider.getBinaryHashFunctionFamily(
type);
}
return funFamilies;
}
{code}

For each variable type, we try to get hash function family. In the current
codebase, AqlBinaryHashFunctionFamilyProvider is the only class that
implements IBinaryHashFunctionFamilyProvider.
And for any type, it returns AMurmurHash3BinaryHashFunctionFamily.
So, there is no way that the hash function family is null.

{code:title= AqlBinaryHashFunctionFamilyProvider.java|borderStyle=solid}
public class AqlBinaryHashFunctionFamilyProvider implements
IBinaryHashFunctionFamilyProvider, Serializable {

private static final long serialVersionUID = 1L;
public static final AqlBinaryHashFunctionFamilyProvider INSTANCE = new
AqlBinaryHashFunctionFamilyProvider();

private AqlBinaryHashFunctionFamilyProvider() {

}

@Override
public IBinaryHashFunctionFamily getBinaryHashFunctionFamily(Object
type) throws AlgebricksException {
// AMurmurHash3BinaryHashFunctionFamily converts numeric type to
double type before doing hash()
return AMurmurHash3BinaryHashFunctionFamily.INSTANCE;
}

}
{code}







--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Line limit (120) is not enforced for an assignment statement.

[Taewoo Kim]

There is a formatter issue regarding an assignment statement whose length
is over 120. An example is as follows:
private final IBinaryComparator strLowerCaseTokenCmp =
AqlBinaryComparatorFactoryProvider.UTF8STRING_LOWERCASE_TOKEN_POINTABLE_INSTANCE.createBinaryComparator();

Currently, the formatter doesn't wrap this line into two lines because "No
Wrap" policy is applied. So, I would like to propose the following change.
After applying this change, we can keep the hard limit (120) well.

AsterixCodeFormatProfile.xml

Line Wrapping -> Expressions -> Assignments

(Before) Line Wrapping Policy: Do not Wrap
(After) Wrap where necessary

@Yingyi: I wonder whether your proposed change (do not remove blank lines
in the comment section) is already applied to the format file. Please let
me know. Then, I will download the file from
https://cwiki.apache.org/confluence/download/attachments/61322291/AsterixCodeFormatProfile.xml
and apply my change and upload the file.

Status

[Taewoo Kim]

I will not be able to attend the Weekly meeting. So, I have attached my
status here:

[Paper]
- Conducting a preliminary experiment on similarity search paper (preparing
similar search queries and sending them to a small dataset)

[Individual]
- Addressed Ian's comments on the full-text implementation step 1 branch
- Reviewed Steven's non-pure branch
- Preparing another index-only plan chunks (select and join rule change)
- Fixed ASTERIXDB-1700
  (if we have multiple same type of indexes on the same field, all of
indexes will be used for a select query.
   Only picking one is enough. E.g., 2-gram and 3-gram index on the same
string field)

Re: Eclipse - Plugin execution not covered by lifecycle configuration error

[Taewoo Kim]

It looks like that there is no m2e connector for analyze-only plugin. I
think we need to add this plugin to the plugin management section of
pom.xml file so that this could be ignored on Eclipse only.

Best,
Taewoo

On Wed, Oct 19, 2016 at 5:06 PM, Taewoo Kim <wangs...@gmail.com> wrote:

> Thanks Till. Let me check that.
>
> Best,
> Taewoo
>
> On Wed, Oct 19, 2016 at 4:45 PM, Till Westmann <ti...@apache.org> wrote:
>
>> Hi,
>>
>> I haven’t seen this (as I don’t use eclipse), but I can tell you where
>> it’s coming from. We’ve added the maven-dependency-plugin to validate
>> that the dependencies that are declared in the POM actually match what
>> it being used by the module. The build will fail if dependencies are
>> missing or if we declare dependencies that are not being used in the
>> module. The goal is to have precise dependency information to be able
>> to produce correct LICENSE/NOTICE files for our binary artifacts.
>>
>> I would guess that there's a way to integrate that plugin with the
>> eclipse build using m2e, but it's just a guess ...
>>
>> Maybe somebody else took a look at this and knows more?
>>
>> Cheers,
>> Till
>>
>> [1] http://www.eclipse.org/m2e/
>>
>> On 19 Oct 2016, at 16:33, Taewoo Kim wrote:
>>
>> > Hi all,
>> >
>> > Since a few days ago, I have seen the following error. Has anyone seen
>> this
>> > error, too?
>> >
>> > Plugin execution not covered by lifecycle configuration:
>> > org.apache.maven.plugins:maven-dependency-plugin:2.10:*analyze-only*
>> > (execution: default, phase: process-test-classes)
>> >
>> > Best,
>> > Taewoo
>>
>
>

Function name and format change in the codebase

[Taewoo Kim]

Hi All,

I would like to talk about 1) Function name convention and Format change in
the codebase.

1) Function name
When we invent a new function name, would it be better if we avoid using
'-'(dash) in a new function name and use "_" instead to avoid two versions
for the same function in AQL and SQL++?

2) Format change in the codebase
When we modify a java source file, is it better to apply format changes to
the whole file or the lines that we only touch? For history perspective, I
think the latter is desirable.

Best,
Taewoo

Re: Physical Operator Tag Change (SPLIT -> REPLICATE)

[Taewoo Kim]

Thanks all. Now, the change has been merged to the master. So,
ReplicatePOperator uses REPLICATE tag, instead of SPLIT tag. SPLIT tag will
be used by real split operator.


Best,
Taewoo

On Tue, Sep 27, 2016 at 10:42 PM, Mike Carey <dtab...@gmail.com> wrote:

> +1
>
> On Sep 27, 2016 10:11 PM, "Preston Carman" <prest...@apache.org> wrote:
>
> > I always thought it was confusing that the ReplicatePOperator used the
> > SPLIT tag. I think making the name consistent would be nice.
> >
> > On Tue, Sep 27, 2016 at 5:08 PM, Yingyi Bu <buyin...@gmail.com> wrote:
> > > Taewoo,
> > >
> > > I agree with you.
> > > However, can you have two separate changes:
> > > 1. Your current change https://asterix-gerrit.ics.uci.edu/#/c/1196/.
> > Call
> > > your "Split" "PartitionSplit" for now.
> > > 2. Do what you proposed in the next change which only does the
> renaming.
> > >
> > > That makes reviews less overwhelming and easier.
> > >
> > > Best,
> > > Yingyi
> > >
> > >
> > > On Tue, Sep 27, 2016 at 5:01 PM, Taewoo Kim <wangs...@gmail.com>
> wrote:
> > >
> > >> Hello All,
> > >>
> > >> Regarding the index-only plan, I am going to introduce an operator
> named
> > >> SplitOperator that propagates each tuple to only one output branch.
> > >> Currently, ReplicateOperator propagates each tuple to all output
> > branches.
> > >> And ReplicatePOperator uses the physical operator tag as SPLIT. I
> > suggest
> > >> we change it to REPLICATE. Thus, SPLIT tag can be used for real Split
> > >> operator. Any thoughts?
> > >>
> > >> Best,
> > >> Taewoo
> > >>
> >
>

Re: Physical Operator Tag Change (SPLIT -> REPLICATE)

[Taewoo Kim]

@Yingyi: Thanks for the suggestion. Actually, I have crated a separate
patch for this change only: https://asterix-gerrit.ics.uci.edu/#/c/1219/.
Once this is merged, the review for the second step would be much easier.

Best,
Taewoo

On Tue, Sep 27, 2016 at 5:08 PM, Yingyi Bu <buyin...@gmail.com> wrote:

> Taewoo,
>
> I agree with you.
> However, can you have two separate changes:
> 1. Your current change https://asterix-gerrit.ics.uci.edu/#/c/1196/.  Call
> your "Split" "PartitionSplit" for now.
> 2. Do what you proposed in the next change which only does the renaming.
>
> That makes reviews less overwhelming and easier.
>
> Best,
> Yingyi
>
>
> On Tue, Sep 27, 2016 at 5:01 PM, Taewoo Kim <wangs...@gmail.com> wrote:
>
> > Hello All,
> >
> > Regarding the index-only plan, I am going to introduce an operator named
> > SplitOperator that propagates each tuple to only one output branch.
> > Currently, ReplicateOperator propagates each tuple to all output
> branches.
> > And ReplicatePOperator uses the physical operator tag as SPLIT. I suggest
> > we change it to REPLICATE. Thus, SPLIT tag can be used for real Split
> > operator. Any thoughts?
> >
> > Best,
> > Taewoo
> >
>

Re: Do we allow optional parameter in a function?

[Taewoo Kim]

@Steven: Thanks for the information. I think two separate function
signatures would be better instead let use say F(a,b,null).

Best,
Taewoo

On Fri, Sep 23, 2016 at 8:38 PM, Steven Jacobs <sjaco...@ucr.edu> wrote:

> The only way we handle this currently is to have two separate function
> signatures. They can share code underneath, but we are strict about
> parameter count. The other option would be for the user to pass some null
> representation for the third value.
> Steven
>
> On Friday, September 23, 2016, Taewoo Kim <wangs...@gmail.com> wrote:
>
> > Hi all,
> >
> > Here is my question. Can a function have an optional parameter? For
> > example, if a function F requires three parameters  - a, b, c, can we
> omit
> > the last one? F(a,b,c) VS F(a,b). I tried and Asterix says that F(a,b) is
> > not defined. If someone knows the workaround, it would be really nice.
> >
> > Actual scenario: Full-text search function name is ftcontains() and it
> has
> > three parameters - expresssion1, expression2, options as a record. An
> > example is ftcontains($o.title, "database", {"mode":"and"}). I would like
> > to give user the option to not put the third parameter. In that case,
> each
> > option can be set using default values.
> >
> > Best,
> > Taewoo
> >
>

Do we allow optional parameter in a function?

[Taewoo Kim]

Hi all,

Here is my question. Can a function have an optional parameter? For
example, if a function F requires three parameters  - a, b, c, can we omit
the last one? F(a,b,c) VS F(a,b). I tried and Asterix says that F(a,b) is
not defined. If someone knows the workaround, it would be really nice.

Actual scenario: Full-text search function name is ftcontains() and it has
three parameters - expresssion1, expression2, options as a record. An
example is ftcontains($o.title, "database", {"mode":"and"}). I would like
to give user the option to not put the third parameter. In that case, each
option can be set using default values.

Best,
Taewoo

Re: Function name change: contains() -> string-contains()

[Taewoo Kim]

@Till: I see. Thanks for the suggestion. It's more clearer now.

Best,
Taewoo

On Thu, Sep 15, 2016 at 5:58 PM, Till Westmann <ti...@apache.org> wrote:

> And as it turns out, we already have some infrastructure to translate a
> constant record constructor expression into a record in
> LangRecordParseUtil.
> So supporting that wouldn’t be too painful.
>
> Cheers,
> Till
>
>
> On 15 Sep 2016, at 17:41, Till Westmann wrote:
>
> One option to express those parameters, would be to pass in a (compile time
>> constant) record/object. E.g.
>>
>> where ftcontains($o.title, ["hello","hi"],
>>  { "combine": "and", "stop list": "default" })
>>
>> That way we could have named optional parameters (please ignore the
>> ugliness of
>> my chosen parameters) which avoid the problem of dealing with positions.
>> We do have a nested datamodel, so we could put it to good use here :)
>>
>> Does this make sense?
>>
>> Cheers,
>> Till
>>
>> On 15 Sep 2016, at 16:26, Taewoo Kim wrote:
>>
>> @Till: we can add whether the given search is AND/OR search, stop list
>>> and/or stemming method. For example, if we use ftcontains(), then it
>>> might
>>> look like:
>>>
>>> 1) where ftcontains($o.title, "hello"): find $o where the title field
>>> contains hello.
>>> 2) where ftcontains($o.title, ["hello","hi"], any): find $o where the
>>> title
>>> field contains hello *and/or* hi.
>>> 3) where ftcontains($o.title, ["hello","hi"], all): find $o where the
>>> title
>>> field contains both hello *and* hi.
>>> 4) where ftcontains($o.title, ["hello","hi"], all, defaultstoplist): find
>>> $o where the title field contains both hello *and* hi. Also apply the
>>> default stoplist to the search. The default stop list contains the number
>>> of English common words that can be filtered.
>>>
>>> The issue here is that the position of each parameter should be observed
>>> (e.g., the third one indicates whether we do disjunctive/conjunctive
>>> search. The fourth one tells us which stop list we use). So, if we have
>>> three parameters, how to specify/omit these becomes a challenge.
>>>
>>> Best,
>>> Taewoo
>>>
>>> On Thu, Sep 15, 2016 at 4:12 PM, Till Westmann <ti...@apache.org> wrote:
>>>
>>> Makes sense to me (especially as I always think about this specific one
>>>> as
>>>> "ftcontains" :) ).
>>>>
>>>> Another thing you mentioned is about the parameters that will get added
>>>> in
>>>> the
>>>> future. Could you provide an example for this?
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>> On 15 Sep 2016, at 15:37, Taewoo Kim wrote:
>>>>
>>>> Maybe we could come up with a function form - *ftcontains*(). Here, ft
>>>> is
>>>>
>>>>>
>>>>> an abbreviation for full-text. This function replaces "contains text"
>>>>> in
>>>>> XQuery spec. An example might be:
>>>>>
>>>>> XQuery spec: where $o.titile contains text "hello"
>>>>> AQL: where ftcontains($o.title, "hello")
>>>>>
>>>>> Best,
>>>>> Taewoo
>>>>>
>>>>> On Thu, Sep 15, 2016 at 3:18 PM, Taewoo Kim <wangs...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> @Till: Got it. I agree to your opinion. The issue here for the
>>>>> full-text
>>>>>
>>>>>> search is that many function parameters that controls the behavior of
>>>>>> full-text search will be added in the future. Maybe this is not the
>>>>>> issue?
>>>>>> :-)
>>>>>>
>>>>>> Best,
>>>>>> Taewoo
>>>>>>
>>>>>> On Thu, Sep 15, 2016 at 3:11 PM, Till Westmann <ti...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>>
>>>>>>> I think that our challenge here is, that XQuery is very liberal in
>>>>>>> the
>>>>>>> introduction of new keywords, as the grammar is keyword free.
>>>>>>> However,
>>