Re: [APE] Unlimited Storage: Local disk caching in cloud deployment

[Glenn Justo Galvizo]

+1 from me, sounds interesting!

> On Apr 3, 2024, at 11:42, Ian Maxon  wrote:
> 
>  +1, this will be a great addition.
> 
>> On Apr 3, 2024 at 09:44:01, Wail Alkowaileet  wrote:
>> 
>> In the current cloud deployment, users are limited by the disk space of the
>> cluster's nodes. However, the blob storage services provided by cloud
>> providers (e.g., S3) can virtually store an "unlimited" amount of data.
>> Thus, AsterixDB can provide the means to store beyond what the cluster's
>> local drives can.
>> 
>> In this proposal, we want to extend AsterixDB's capability to allow the
>> local drives to act as a cache, instead of a mirror image of what's stored
>> in the cloud. By "as a cache" we mean files and pages can be
>> retrieved/persited and removed (evicted) from the local drives, according
>> to some policy.
>> 
>> The aim of this proposal is to describe and implement a mechanism called
>> "*Weep
>> and Sweep*". Those are the names of two phases when the amount of the data
>> in the cloud exceeds the space of the cluster's local disks.
>> Weep
>> 
>> When the disk is pressured (the pressure size can be configured), the
>> system will start to "weep" and devise a plan to what should be "evicted"
>> according to some statistics and policies, *which are not solidified yet
>> and still a work in progress.*
>> Sweep
>> 
>> After "weeping", a sweep operation will take place and start evicting what
>> the weep's plan considers as evictable. Depending on the index type
>> (primary/secondary) and the storage format (row/column), the smallest
>> evictable unit can differ. The following table shows the smallest unit of
>> evictable unit:
>> *Index Type* *Evictable*
>> Metadata Indexes (e.g., Dataset, ..etc) Not evictable
>> Secondary indexes Evicted as a whole
>> Primary Indexes (Row) Evicted as a whole
>> Primary Indexes (Columnar) Columns (or columns’ pages)
>> Featured Considerations
>> 
>>  - For columnar primary index, they will never be downloaded as a whole
>> - Instead, columns will be streamed from the cloud (if accessed for
>> the first time) and persisted to local disk if necessary
>>  - We are considering providing a mechanism to prefetch the next columns
>>  of the next mega-leaf node
>>  <
>> https://urldefense.com/v3/__https://www.vldb.org/pvldb/vol15/p2085-alkowaileet.pdf__;!!CzAuKJ42GuquVTTmVmPViYEvSg!Oah7iQPtzg5ozE3ckKpn-ANVgu_VrdWY_2gO_-HwxeYgrKWj8kmv7ifZQKnf36jne2V_SXXvmITxy_E$
>>> . The hope here
>>  is to mask any latencies when reading columns from the cloud
>>  - Depending on the disk pressure and the operation, the system can
>>  determine if the streamed columns from the cloud are "worthy" to be
>> cached
>>  locally. For example, if columns are read in a merge operation, it might
>>  not be "wise" to persist these columns as their on-disk component is
>> going
>>  to be deleted at the end of the merge operation. Thus, it might be
>> "better"
>>  to dedicate the free space on disk for the newly created/merged
>> component.
>> 
>> 
>> Multiple aspects (such as the evictable units and policies) of this APE are
>> not solidified yet, but the core concepts are in place and are ready for
>> the community's vote :)
>> 
>> EPIC: ASTERIXDB-3373 <
>> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/ASTERIXDB-3373__;!!CzAuKJ42GuquVTTmVmPViYEvSg!Oah7iQPtzg5ozE3ckKpn-ANVgu_VrdWY_2gO_-HwxeYgrKWj8kmv7ifZQKnf36jne2V_SXXv8xZvKPI$
>>> 
>> --
>> 
>> *Regards,*
>> Wail Alkowaileet
>> 

Re: [VOTE] Release Apache AsterixDB 0.9.8.2 and Apache Hyracks 0.9.8.2 (RC0)

[Glenn Justo Galvizo]

+1

- Glenn Galvizo

> On Mar 2, 2024, at 08:23, Murtadha Hubail  wrote:
> 
> +1
> 
> -Murtadha
> 
> From: Wail Alkowaileet mailto:wael@gmail.com>>
> Sent: Saturday, March 2, 2024 5:50:23 PM
> To: dev@asterixdb.apache.org  
> mailto:dev@asterixdb.apache.org>>
> Subject: Re: [VOTE] Release Apache AsterixDB 0.9.8.2 and Apache Hyracks 
> 0.9.8.2 (RC0)
> 
> +1
> 
> 
> *Regards,*
> Wail Alkowaileet
> 
> 
> On Fri, Mar 1, 2024 at 13:50 Ian Maxon  wrote:
> 
>> Hi everyone,
>> 
>> Please verify and vote on the latest stabilization release of Apache
>> AsterixDB.
>> 
>> The change that produced this release is up on Gerrit:
>> 
>> https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18187
>> 
>> The release artifacts are as follows:
>> 
>> AsterixDB Source
>> 
>> https://urldefense.com/v3/__https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.8.2-source-release.zip__;!!CzAuKJ42GuquVTTmVmPViYEvSg!KJUMHCD9RNxXbRAr9rjgw6lkM3brd7kfavDeLKEQxLibMwI6vzvuKu0eUuxH-B-raiJy20xx-jk9HwNcojQ$
>> 
>> https://urldefense.com/v3/__https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.8.2-source-release.zip.asc__;!!CzAuKJ42GuquVTTmVmPViYEvSg!KJUMHCD9RNxXbRAr9rjgw6lkM3brd7kfavDeLKEQxLibMwI6vzvuKu0eUuxH-B-raiJy20xx-jk9xdeJY9Q$
>> 
>> https://urldefense.com/v3/__https://dist.apache.org/repos/dist/dev/asterixdb/apache-asterixdb-0.9.8.2-source-release.zip.sha512__;!!CzAuKJ42GuquVTTmVmPViYEvSg!KJUMHCD9RNxXbRAr9rjgw6lkM3brd7kfavDeLKEQxLibMwI6vzvuKu0eUuxH-B-raiJy20xx-jk9bmTFmB8$
>> 
>> SHA512:
>> 
>> 9424c56301e538639170b2c8064a8cb16cb9278423bfca4b3475cb36ce8067bc49f001a6dc3970e6b45b1b3bd275537ef0a54145ca829e26242f82c7e05d8a58
>> 
>> Hyracks Source
>> 
>> https://urldefense.com/v3/__https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.8.2-source-release.zip__;!!CzAuKJ42GuquVTTmVmPViYEvSg!KJUMHCD9RNxXbRAr9rjgw6lkM3brd7kfavDeLKEQxLibMwI6vzvuKu0eUuxH-B-raiJy20xx-jk9uXqgiCM$
>> 
>> https://urldefense.com/v3/__https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.8.2-source-release.zip.asc__;!!CzAuKJ42GuquVTTmVmPViYEvSg!KJUMHCD9RNxXbRAr9rjgw6lkM3brd7kfavDeLKEQxLibMwI6vzvuKu0eUuxH-B-raiJy20xx-jk9X7zwDbU$
>> 
>> https://urldefense.com/v3/__https://dist.apache.org/repos/dist/dev/asterixdb/apache-hyracks-0.3.8.2-source-release.zip.sha512__;!!CzAuKJ42GuquVTTmVmPViYEvSg!KJUMHCD9RNxXbRAr9rjgw6lkM3brd7kfavDeLKEQxLibMwI6vzvuKu0eUuxH-B-raiJy20xx-jk9w61HuZM$
>> 
>> SHA512:
>> 
>> f17e004ad4b3dcf74db4582ae064189c27ae8f9f3aa352be1d11b936ac1bd679607318f5dfbd86e3d915f9f2e4550271db8ad144b3a41e182738047e398ba477
>> 
>> AsterixDB NCService Installer:
>> 
>> https://urldefense.com/v3/__https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.8.2-binary-assembly.zip__;!!CzAuKJ42GuquVTTmVmPViYEvSg!KJUMHCD9RNxXbRAr9rjgw6lkM3brd7kfavDeLKEQxLibMwI6vzvuKu0eUuxH-B-raiJy20xx-jk9bqonlsw$
>> 
>> https://urldefense.com/v3/__https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.8.2-binary-assembly.zip.asc__;!!CzAuKJ42GuquVTTmVmPViYEvSg!KJUMHCD9RNxXbRAr9rjgw6lkM3brd7kfavDeLKEQxLibMwI6vzvuKu0eUuxH-B-raiJy20xx-jk9bB2eihU$
>> 
>> https://urldefense.com/v3/__https://dist.apache.org/repos/dist/dev/asterixdb/asterix-server-0.9.8.2-binary-assembly.zip.sha512__;!!CzAuKJ42GuquVTTmVmPViYEvSg!KJUMHCD9RNxXbRAr9rjgw6lkM3brd7kfavDeLKEQxLibMwI6vzvuKu0eUuxH-B-raiJy20xx-jk9Lv80Ldc$
>> 
>> SHA512:
>> 
>> cb7a8e941090332f99589257f14913b4dcc9c15a9a3c4395c558fae68a6570c0bf14824be3aa1a672917a61a4c07988cacd705be12012e246d161ca8b69949c5
>> 
>> The KEYS file containing the PGP keys used to sign the release can be
>> found at
>> 
>> https://urldefense.com/v3/__https://dist.apache.org/repos/dist/release/asterixdb/KEYS__;!!CzAuKJ42GuquVTTmVmPViYEvSg!KJUMHCD9RNxXbRAr9rjgw6lkM3brd7kfavDeLKEQxLibMwI6vzvuKu0eUuxH-B-raiJy20xx-jk9FXgBoG4$
>> 
>> RAT was executed as part of Maven via the RAT maven plugin, but
>> excludes files that are:
>> 
>> - data for tests
>> - procedurally generated,
>> - or source files which come without a header mentioning their license,
>> but have an explicit reference in the LICENSE file.
>> 
>> 
>> The vote is open for 72 hours, or until the necessary number of votes
>> (3 +1) has been reached.
>> 
>> Please vote
>> [ ] +1 release these packages as Apache AsterixDB 0.9.8.2 and
>> Apache Hyracks 0.3.8.2
>> [ ] 0 No strong feeling either way
>> [ ] -1 do not release one or both packages because ...

Re: [RESULT][APE] Query Plan Cache

[Glenn Justo Galvizo]

With four +1's, the vote passes.

- A cluster-wide configuration (most likely as a "cc.conf" parameter) should 
definitely be added.
- Agreed on the "compiler.querycache.clear" not being ideal, using the admin 
API is a great alternative.
- Agreed on more research into the memory footprint of the cache to set a 
better default value.

> On Dec 8, 2023, at 12:49, Ian Maxon  wrote:
> 
> +1. This has been a long awaited feature.
> 
> On Dec 7, 2023 at 14:21:31, Glenn Justo Galvizo  <mailto:ggalv...@uci.edu>> wrote:
> 
>> Every time a query is issued to AsterixDB, the query must undergo
>> compilation. If the same query is run repeatedly, this query must be
>> recompiled each and every time. A query plan cache can help AsterixDB
>> achieve a lower floor on the end-to-end time by storing the job
>> specifications for previously compiled queries, ultimately skipping the AST
>> rewriting and Algebricks compilation of a previously executed query.
>> 
>> (APE copied from contributor Sushrut Borkor)
>> 
>> This APE is about adding a query plan cache to AsterixDB. More
>> specifically, this query plan cache acts as a hash table that skips 1) the
>> AST rewriting, 2) the entire Algebricks plan translation to Algebricks
>> optimization, and 3) the Hyracks job generation. The keys of this hash
>> table are:
>>   • AST String. We cache this instead of the original query string before
>> parsing because it is resilient to minor changes in the query, such as
>> adding spaces or empty lines.
>>   • SessionConfig. For example, if the user runs a query, changes part of
>> the session configuration (e.g. the preferred output format), and reruns
>> the query, this prevents the second query from being served from the cache.
>>   • Config, to capture the effects of used SET statements.
>>   • Active Dataverse, e.g., as defined in a USE statement.
>>   • Result Set ID, which distinguishes among queries in multi-statement
>> requests.
>> 
>> While the values of each hash table entry are:
>>   • Hyracks Job Spec to be submitted to Hyracks.
>>   • Cached warnings. Since we skip compilation when serving queries from
>> the cache, we cannot detect compile time warnings. To get around this, we
>> cache warnings issued during rewriting and compilation, and then reissue
>> them for cache hits. As a result, line numbers in warnings may be incorrect
>> for queries answered using the cache.
>>   • Lock. Since running the same job from multiple threads does not work,
>> we include a lock in the cache value. To use a cached job spec, a thread
>> must acquire this lock, and then release it after the job has finished
>> running. If the lock is held by another thread, we recompile the query
>> instead of blocking.
>> 
>> The proposed changes are the following:
>> 
>> Interface:
>> We introduce two new statements for controlling cache access:
>>   • “SET `compiler.querycache.bypass` "true";” forces the current query
>> to ignore the cache.
>>   • “SET `compiler.querycache.clear` "true";” clears all cache entries.
>> The current query may still insert into the cache.
>> We also add a boolean HTTP API parameter bypass_cache which does the same
>> thing as the first SET statement above. Finally, the parameter
>> query.cache.capacity can be configured in the [cc] section of the cc.conf
>> file to control the maximum cache size before replacement.
>> 
>> Changes:
>>   • Compilation logic is changed in the source code since we skip
>> rewriting and compilation for cache hits.
>>   • Hints are now included in the AST string to prevent incorrect cache
>> lookups that would otherwise miss the hints.
>>   • A bug is fixed where the AST string of WINDOW expressions did not
>> include FROM LAST or IGNORE NULLS.
>> 
>> See
>> https://urldefense.com/v3/__https://issues.apache.org/jira/projects/ASTERIXDB/issues/ASTERIXDB-3183__;!!CzAuKJ42GuquVTTmVmPViYEvSg!Jhj6lSdZ_YN_3h2QM-EEPYwthqvlhCZ13nFvx1rMAotNv3UxlZmgXM-q4xCOBR2zE5iaBDGBXD5P-ZBx$
>> for the JIRA issue, as well as
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/ASTERIXDB/APE*2*3A*Query*Plan*Cache__;KyUrKys!!CzAuKJ42GuquVTTmVmPViYEvSg!Jhj6lSdZ_YN_3h2QM-EEPYwthqvlhCZ13nFvx1rMAotNv3UxlZmgXM-q4xCOBR2zE5iaBDGBXDWm8yfg$
>> for more details.
>> 
>> Please vote on this APE. We will keep this open for 72 hours and pass with
>> either 3 votes or a majority of positive votes.

[VOTE][APE] Query Plan Cache

[Glenn Justo Galvizo]

Every time a query is issued to AsterixDB, the query must undergo compilation. 
If the same query is run repeatedly, this query must be recompiled each and 
every time. A query plan cache can help AsterixDB achieve a lower floor on the 
end-to-end time by storing the job specifications for previously compiled 
queries, ultimately skipping the AST rewriting and Algebricks compilation of a 
previously executed query.

(APE copied from contributor Sushrut Borkor)

This APE is about adding a query plan cache to AsterixDB. More specifically, 
this query plan cache acts as a hash table that skips 1) the AST rewriting, 2) 
the entire Algebricks plan translation to Algebricks optimization, and 3) the 
Hyracks job generation. The keys of this hash table are:
• AST String. We cache this instead of the original query string before 
parsing because it is resilient to minor changes in the query, such as adding 
spaces or empty lines.
• SessionConfig. For example, if the user runs a query, changes part of the 
session configuration (e.g. the preferred output format), and reruns the query, 
this prevents the second query from being served from the cache.
• Config, to capture the effects of used SET statements.
• Active Dataverse, e.g., as defined in a USE statement.
• Result Set ID, which distinguishes among queries in multi-statement 
requests.

While the values of each hash table entry are:
• Hyracks Job Spec to be submitted to Hyracks.
• Cached warnings. Since we skip compilation when serving queries from the 
cache, we cannot detect compile time warnings. To get around this, we cache 
warnings issued during rewriting and compilation, and then reissue them for 
cache hits. As a result, line numbers in warnings may be incorrect for queries 
answered using the cache.
• Lock. Since running the same job from multiple threads does not work, we 
include a lock in the cache value. To use a cached job spec, a thread must 
acquire this lock, and then release it after the job has finished running. If 
the lock is held by another thread, we recompile the query instead of blocking.

The proposed changes are the following:

Interface:
We introduce two new statements for controlling cache access:
• “SET `compiler.querycache.bypass` "true";” forces the current query to 
ignore the cache.
• “SET `compiler.querycache.clear` "true";” clears all cache entries. The 
current query may still insert into the cache.
We also add a boolean HTTP API parameter bypass_cache which does the same thing 
as the first SET statement above. Finally, the parameter query.cache.capacity 
can be configured in the [cc] section of the cc.conf file to control the 
maximum cache size before replacement.

Changes:
• Compilation logic is changed in the source code since we skip rewriting 
and compilation for cache hits.
• Hints are now included in the AST string to prevent incorrect cache 
lookups that would otherwise miss the hints.
• A bug is fixed where the AST string of WINDOW expressions did not include 
FROM LAST or IGNORE NULLS.

See https://issues.apache.org/jira/projects/ASTERIXDB/issues/ASTERIXDB-3183 for 
the JIRA issue, as well as 
https://cwiki.apache.org/confluence/display/ASTERIXDB/APE+2%3A+Query+Plan+Cache 
for more details.

Please vote on this APE. We will keep this open for 72 hours and pass with 
either 3 votes or a majority of positive votes.

Re: [VOTE][APE] Compute-Storage Separation (Cloud Mode Deployment)

[Glenn Justo Galvizo]

+1 from me as well.

> On Dec 2, 2023, at 10:27, Ian Maxon  wrote:
> 
> +1
> 
>> On Dec 1, 2023 at 12:28:23, Murtadha Al-Hubail  wrote:
>> 
>> Each AsterixDB cluster today consists of one or more Node Controllers (NC)
>> where the data is stored and processed. Each NC has a predefined set of
>> storage partitions (iodevices). When data is ingested into the system, the
>> data is hash-partitioned across the total number of storage partitions in
>> the cluster. Similarly, when the data is queried, each NC will start as
>> many threads as the number storage partitions it has to read and process
>> the data in parallel. While this shared-nothing architecture has its
>> advantages, it has its drawbacks too. One major drawback is the time needed
>> to scale the cluster. Adding a new NC to an existing cluster of (n) nodes
>> means writing a completely new copy of the data which will now be
>> hash-partitioned to the new total number of storage partitions of (n + 1)
>> nodes. This operation could potentially take several hours or even days
>> which is unacceptable in the cloud age.
>> 
>> This APE is about adding a new deployment (cloud) mode to AsterixDB by
>> implementing compute-storage separation to take advantage of the elasticity
>> of the cloud. This will require the following:
>> 
>> 1. Moving from the dynamic data partitioning described earlier to a static
>> data partitioning based on a configurable, but fixed during a cluster's
>> life, number of storage partitions.
>> 2. Introducing the concept of a "compute partition" where each NC will have
>> a fixed number of compute partitions. This number could potentially be
>> based on the number of CPU cores it has.
>> 
>> This will decouple the number of storage partitions being processed on an
>> NC from the number of its compute partitions.
>> 
>> When an AsterixDB cluster is deployed using the cloud mode, we will do the
>> following:
>> 
>> - The Cluster Controller will maintain a map containing the assignment of
>> storage partitions to compute partitions.
>> - New writes will be written to the NC's local storage and uploaded to an
>> object store (e.g. AWS S3) which will be used as a highly available shared
>> filesystem between NCs.
>> - On queries, each NC will start as many threads as its compute partitions
>> to process its currently assigned storage partitions.
>> - On scaling operations, we will simply update the assignment map and NCs
>> will lazily cache any data of newly assigned storage partitions from the
>> object store.
>> 
>> Improvement tickets:
>> Static data partitioning:
>> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/ASTERIXDB-3144__;!!CzAuKJ42GuquVTTmVmPViYEvSg!OAhVXrR7KC09sldpj5RPLxWAUgdr8MVlQ9bIpT5QK76KPmMlxnjFGChosdZpBbe81Z_KZI7COEEXdi5a$
>> Compute-Storage Separation
>> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/ASTERIXDB-3196__;!!CzAuKJ42GuquVTTmVmPViYEvSg!OAhVXrR7KC09sldpj5RPLxWAUgdr8MVlQ9bIpT5QK76KPmMlxnjFGChosdZpBbe81Z_KZI7COGLN6MWp$
>> 
>> Please vote on this APE. We'll keep this open for 72 hours and pass with
>> either 3 votes or a majority of positive votes.
>> 

[RESULT] Message frame support APE

[Glenn Justo Galvizo]

Hi everyone: a bit late, but with two +1s after the 72 hours, the vote passes. 
The following have voted:

Ian Maxon
Mike Carey

Vote thread: https://lists.apache.org/thread/4o10y19s8kzx0dnq2dfd6prbllp8kkp7 

Thanks for voting!

Glenn

> On Oct 13, 2023, at 11:21, Ian Maxon  wrote:
> 
>  +1, this will be great to have and fix a longstanding issue in Hyracks.
> 
> On Sep 30, 2023 at 13:17:06, Glenn Justo Galvizo  wrote:
> 
>> Hi everyone,
>> 
>> We are proposing a new feature for Hyracks users to communicate using
>> "message frames" to communicate state to downstream operators. This is
>> meant to be an less invasive alternative to using message tuples and using
>> context objects. This is currently being used to support navigational
>> queries in Graphix.
>> 
>> APE:
>> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/ASTERIXDB/APE*3:*Message*Frames*for*In-Band*Communication__;KysrKysr!!CzAuKJ42GuquVTTmVmPViYEvSg!NXfhXbrB6Nst1g1AQxKFpteAcf18t4JdKG2y609yW7Npo679koqWZuDu10iU_qJZOjzZeIq3Ko0Kuwej$
>> Patch: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/17731
>> Issue:
>> https://urldefense.com/v3/__https://issues.apache.org/jira/browse/ASTERIXDB-3241__;!!CzAuKJ42GuquVTTmVmPViYEvSg!NXfhXbrB6Nst1g1AQxKFpteAcf18t4JdKG2y609yW7Npo679koqWZuDu10iU_qJZOjzZeIq3KiHlEpcK$
>> 
>> Following the procedure from the last APE, we'll keep this open for 72
>> hours and pass with either 3 votes or a majority of positive votes.
>> 

[VOTE] Message frame support APE

[Glenn Justo Galvizo]

(Forwarded)

> Begin forwarded message:
> 
> From: Mike Carey 
> Subject: Re: [VOTE] Message frame support APE
> Date: October 1, 2023 at 13:15:07 PDT
> To: Glenn Justo Galvizo 
> 
> +1 for this APE.
> 
> :-)
> 
> On 9/30/23 1:17 PM, Glenn Justo Galvizo wrote:
>> Hi everyone,
>> 
>> We are proposing a new feature for Hyracks users to communicate using 
>> "message frames" to communicate state to downstream operators. This is meant 
>> to be an less invasive alternative to using message tuples and using context 
>> objects. This is currently being used to support navigational queries in 
>> Graphix.
>> 
>> APE: 
>> https://cwiki.apache.org/confluence/display/ASTERIXDB/APE+3:+Message+Frames+for+In-Band+Communication
>>  
>> <https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/ASTERIXDB/APE*3:*Message*Frames*for*In-Band*Communication__;KysrKysr!!CzAuKJ42GuquVTTmVmPViYEvSg!NtUSqhPuOuI1PnVAa4WVastdo4sqFZ_7KC8XmSxA7SpQXt38u6Ze7qsrGKc_cPrINeLIXp_TI8tjb922$>
>> Patch: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/17731
>> Issue: https://issues.apache.org/jira/browse/ASTERIXDB-3241 
>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/ASTERIXDB-3241__;!!CzAuKJ42GuquVTTmVmPViYEvSg!NtUSqhPuOuI1PnVAa4WVastdo4sqFZ_7KC8XmSxA7SpQXt38u6Ze7qsrGKc_cPrINeLIXp_TI4Y5Ir5g$>
>> 
>> Following the procedure from the last APE, we'll keep this open for 72 hours 
>> and pass with either 3 votes or a majority of positive votes.

Re: [VOTE] Changing COPY FROM syntax

[Glenn Justo Galvizo]

+1 from me as well!

> On Oct 27, 2023, at 10:15, Till Westmann  wrote:
> 
> +1 this is much nicer
> 
>> On 2023/10/26 05:05:01 Mike Carey wrote:
>> PS - I assume the semantics will be UPSERT-based? (Vs. one-time or 
>> INSERT-based?)
>> 
>>> On 10/24/23 10:16 AM, Wail Alkowaileet wrote:
>>> Hi all,
>>> 
>>> I'm proposing to change the current syntax for COPY FROM. The current
>>> syntax looks as follows:
>>> 
 COPY Customers
 USING localfs (
   ("path"="asterix_nc1://data/nontagged/customerData.json"),
   ("format"="json")
 );
 
>>> This syntax uses the old way of configuring the adapter localfs. In our
>>> feeds, we use the WITH clause. Another issue is that the current syntax is
>>> missing the keyword FROM, which makes it ambiguous if we add support for
>>> COPY TO.
>>> 
>>> I propose to change the syntax to be as follows:
>>> 
 COPY Customers
 FROM localfs
 PATH ("asterix_nc1://data/nontagged/customerData.json")
 WITH {
 "format": "json"
 };
 
>>> First, the proposed syntax introduces the use of FROM .
>>> Second, it mandates the use of PATH (instead of having it in the WITH
>>> clause). Additionally, the proposed syntax will make both COPY FROM and
>>> COPY TO less different.
>>> 
>>> Example of COPY TO:
>>> 
 COPY Customers
 TO localfs
 PATH("localhost:///myData/Customers")
 WITH {
 "format" : "json"
 };
 

[VOTE] Message frame support APE

[Glenn Justo Galvizo]

Hi everyone,

We are proposing a new feature for Hyracks users to communicate using "message 
frames" to communicate state to downstream operators. This is meant to be an 
less invasive alternative to using message tuples and using context objects. 
This is currently being used to support navigational queries in Graphix.

APE: 
https://cwiki.apache.org/confluence/display/ASTERIXDB/APE+3:+Message+Frames+for+In-Band+Communication
Patch: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/17731
Issue: https://issues.apache.org/jira/browse/ASTERIXDB-3241

Following the procedure from the last APE, we'll keep this open for 72 hours 
and pass with either 3 votes or a majority of positive votes.

RE: [VOTE] AsterixDB Proposed Enhancements (APEs)

[Glenn Justo Galvizo]

+1

On 2023/03/10 18:54:05 Till Westmann wrote:
> Hi,
> 
> 
> There seems to be a positive sentiment towards the APE proposal [1]. 
> Let’s move on to a VOTE.
> 
> 
> Please vote
> [ ] +1 adopt the APE proposal
> [ ]  0 modify the proposal because ...
> [ ] -1 do not adopt the proposal because ...
> 
> This procedural vote follows the common format of majority rule, that is 
> if there are more favorable votes than unfavorable ones, the issue
> is considered to have passed [2].
> 
> PMC members have formally binding votes, but we encourage all community 
> members to vote.
> 
> The vote is open for at least 72 hours or until 3 votes have been cast.
> 
> Cheers,
> Till
> 
> 
> 
> [1] https://lists.apache.org/thread/lhhdhvvnkxw8r17z69yvb81l87gzgd9h
> [2] https://www.apache.org/foundation/voting.html
>