Re: Calcite based SQL query engine. Local queries
Dmitriy, Would be great if you can describe your use-case in more details, might be sharing a code it the best option here. Denis, Yep, the idea of mixing up Compute, SQL, KV APIs in a super weapon sounds as a killer feature. But I have a great deal of doubt that it is not over-complex to use such tool properly in practice. Partition reservation is not obvious with Ignite compute, KV API can be transactional but SQL not and so on. Too many pitfalls. пт, 8 нояб. 2019 г. в 16:50, Denis Magda : > > Take the amount of cashback calculation or payments authorization as > examples of compute tasks with local SQL. In the first case, all > transactions are collocated per account and a bank needs to calculate the > cashback monthly by broadcasting the task that executes special logic > across all accounts and SQL is used by the logic to access the data with > various filters. The same is done for an individual account with an > affinity call. In the second case, a man swipes a card at a shop register, > systems sends a compute task to the node that collocates a lot of data per > the man account and begins calculating hundreds or thousands of variables > retrieving data with both key-value and SQL. > > Also, take drugs discovery and other pharmaceutical examples. Those are > compute-heavy and the users from that space were sharing the stories how > compute, scan, sql and key-value apis are used together with compute. > > At all, each industry has compute-heavy use cases that need to retrieve > local data with local SQL, there are real Ignite users who do this in prod. > Again, we also need to think about our compute as of advanced stored and > complex procedures that can retrieve local/collocated data not only with > key-value and scans but with SQL as well that supports conditions, joins, > etc. > > Denis > > On Thursday, November 7, 2019, Ivan Pavlukhin wrote: > > > Denis, > > > > To make things really clearer we need to provide some concrete example > > of Compute + LocalSQL and reason about it to figure out whether > > "smart" SQL engine can deliver the same (or better) results or not. > > > > пт, 8 нояб. 2019 г. в 01:48, Denis Magda : > > > > > > Folks, > > > > > > See our compute tasks as an advanced version of stored procedures that > > let > > > the users code the logic of various complexity with Java, .NET or C++ > > (and > > > not with PL/SQL). The logic can use a combination of APIs (key-value, > > SQL, > > > etc.) to access data both locally and remotely while being executed on > > > server nodes. The logic can make N key-value requests or run M SQL > > queries. > > > > > > We kept supporting local SQL queries exactly for such scenarios (for our > > > version of stored procedures) to ensure the distributed map-reduce phase > > is > > > canceled if all the data is local. And affinityCalls were improved one > > day > > > to pin the partitions. > > > > > > If the new engine is smart enough to understand that all the partitions > > are > > > available locally during the affinityRun execution then it's totally fine > > > to remove the 'local' flag. Otherwise, we need to instruct the engine > > > manually that a distributed phase is redundant via 'local' flag or by > > other > > > means. > > > > > > Does it make things clearer? > > > > > > > > > - > > > Denis > > > > > > > > > On Thu, Nov 7, 2019 at 3:53 AM Ivan Pavlukhin > > wrote: > > > > > > > Stephen, > > > > > > > > In my understanding we need to do a better job to realize use-cases of > > > > Compute + LocalSQL ourselves. > > > > > > > > Ideally smart optimizer should do the best job of query deployment. > > > > > > > > чт, 7 нояб. 2019 г. в 13:04, Stephen Darlington > > > > : > > > > > > > > > > I made a (bad) assumption that this would also affect queries against > > > > partitions. If “setLocal()” goes away but “setPartitions()” remains I’m > > > > happy. > > > > > > > > > > What I would say is that the “broadcast / local” method is one I see > > > > fairly often. Do we need to do a better job educating people of the > > > > “correct” way? > > > > > > > > > > Regards, > > > > > Stephen > > > > > > > > > > > On 7 Nov 2019, at 08:30, Alexey Goncharuk < > > alexey.goncha...@gmail.com> > > > > wrote: > > > > > > > > > > > > Denis, Stephen, > > > > > > > > > > > > Running a local query in a broadcast closure won't work on changing > > > > > > topology. We specifically added an affinityCall method to the > > compute > > > > API > > > > > > in order to pin a partition to prevent its moving and eviction > > > > throughout > > > > > > the task execution. Therefore, the query inside an affinityCall is > > > > always > > > > > > executed against some partitions (otherwise the query may give > > > > incorrect > > > > > > results when topology is changed). > > > > > > > > > > > > I support Igor's question and think that the 'local' flag for the > > query > > > > > > should be deprecated and eventually removed. A 'local' query can > > > > always be > > > > >
Re: Contribute to Apache Ignite
Hello Stanislav! Is there any specific you're interested to contribute to? - Denis On Tue, Nov 12, 2019 at 8:58 AM Stanislav Bausov wrote: > Hello everyone! > > С уважением/ Yours sincerely/ Mit besten Grüßen > > Stanislav Bausov > stas...@gmail.com >
Re: [DISCUSS] Proposal for Ignite Extensions as a separate Bahir module or Incubator project
Saikat, Thanks for decoupling the task. Please ping the community once there is anything ready for a review or a further discussion. - Denis On Sat, Nov 9, 2019 at 3:58 PM Saikat Maitra wrote: > Hello, > > As discussed I have created the following jira issues for tracking the > migration work for ignite-extensions. > > Main issue : IEP-36: Modularization > https://issues.apache.org/jira/browse/IGNITE-12355 > > Sub-Tasks > > Migrate flink module to ignite-extensions > https://issues.apache.org/jira/browse/IGNITE-12356 > Migrate Twitter module to ignite-extensions > https://issues.apache.org/jira/browse/IGNITE-12357 > Migrate ZeroMQ module to ignite-extensions > https://issues.apache.org/jira/browse/IGNITE-12358 > Migrate RocketMQ module to ignite-extensions > https://issues.apache.org/jira/browse/IGNITE-12359 > Migrate Storm module to ignite-extensions > https://issues.apache.org/jira/browse/IGNITE-12360 > Migrate Flume module to ignite-extensions > https://issues.apache.org/jira/browse/IGNITE-12361 > Migrate MQTT module to ignite-extensions > https://issues.apache.org/jira/browse/IGNITE-12362 > Migrate Camel module to ignite-extensions > https://issues.apache.org/jira/browse/IGNITE-12363 > Migrate JMS module to ignite-extensions > https://issues.apache.org/jira/browse/IGNITE-12364 > > Regards, > Saikat > > On Fri, Nov 1, 2019 at 9:23 PM Saikat Maitra > wrote: > > > Ivan, Thank you for initialising the master branch, much appreciate it. > > > > Denis, Emmanouil, > > > > We use both local server and mockito to write the tests for flink. Incase > > of sink it is easier to use a local server mode and publish the data into > > Ignite Sink where as in case of Ignite used as data source we use mockito > > to mock Ignite events and Ignite clusters. > > > > Regards, > > Saikat > > > > > > > > On Fri, Nov 1, 2019 at 5:28 PM Emmanouil Gkatziouras < > gkatzio...@gmail.com> > > wrote: > > > >> Hi, > >> > >> I just took another look at the Kafka Implementation and I realised that > >> it > >> is a local implementation that is used on the tests (I thought it was a > >> server already existing somewhere). > >> It seems I might be able to spin up the Pub/Sub server locally (I found > an > >> official tool which might work), in the same way it is done for Kafka. > >> I will try to do so and come back with a pull request. > >> Kind regards > >> > >> *Emmanouil Gkatziouras* > >> https://egkatzioura.com/ | > >> https://www.linkedin.com/in/gkatziourasemmanouil/ > >> https://github.com/gkatzioura > >> > >> > >> On Fri, 1 Nov 2019 at 21:31, Denis Magda wrote: > >> > >> > Saikat, Roman, > >> > > >> > What's your thinking? I'm not sure we set any infrastructure to > validate > >> > our Kafka or Flink integrations. For instance, for Flink we use > Mockito > >> > just to ensure that basic tests pass: > >> > https://en.wikipedia.org/wiki/Mockito > >> > > >> > Our IP finders for AWS, GCE or K8 don't start up any environment for > >> > testing. Instead, we create some smoke tests ensuring that the APIs > are > >> not > >> > broken. > >> > > >> > - > >> > Denis > >> > > >> > > >> > On Fri, Nov 1, 2019 at 1:32 PM Emmanouil Gkatziouras < > >> gkatzio...@gmail.com > >> > > > >> > wrote: > >> > > >> > > Hi all, > >> > > > >> > > That's great news! Yes I am ready to do a pull request on the new > >> repo. > >> > > I believe it is ok to create a ticket for the Pub/Sub proposal [1]. > >> > > My next question has to do with the CI and the integration tests. In > >> > order > >> > > for my tests to work, a local Pub/Sub server needs to be up and > >> running. > >> > > I tried to find any deployment scripts that set the infrastructure > >> needed > >> > > up and running (for example Kafka) but I was not successful on this > >> one. > >> > > How should I proceed on that? > >> > > > >> > > Kind regards > >> > > > >> > > [1] > >> > > > >> > > > >> > > >> > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Pub-Sub-Streamer-Implementation-tc43944.html > >> > > > >> > > *Emmanouil Gkatziouras* > >> > > https://egkatzioura.com/ | > >> > > https://www.linkedin.com/in/gkatziourasemmanouil/ > >> > > https://github.com/gkatzioura > >> > > > >> > > > >> > > On Fri, 1 Nov 2019 at 06:52, Ivan Pavlukhin > >> wrote: > >> > > > >> > > > Saikat, > >> > > > > >> > > > I initialized master branch with empty README file [1]. I suppose > >> all > >> > > > ignite committers have rights for push. > >> > > > > >> > > > [1] https://github.com/apache/ignite-extensions > >> > > > > >> > > > пт, 1 нояб. 2019 г. в 08:01, Saikat Maitra < > saikat.mai...@gmail.com > >> >: > >> > > > > > >> > > > > Hello Dmitriy, Denis > >> > > > > > >> > > > > Thank you for help in creating the repo. I can see it has synced > >> in > >> > > > github > >> > > > > as well. > >> > > > > > >> > > > > Can you please add a sample readme.md file as the repo is empty > I > >> am > >> > > > unable > >> > > > > to fork it? > >> > > > > > >> > > > > or let me check if I can directly
Contribute to Apache Ignite
Hello everyone! С уважением/ Yours sincerely/ Mit besten Grüßen Stanislav Bausov stas...@gmail.com
[jira] [Created] (IGNITE-12367) Disable by default bash pipefail/errexit/nounset options from startup scripts
Andrey N. Gura created IGNITE-12367:
---
Summary: Disable by default bash pipefail/errexit/nounset options
from startup scripts
Key: IGNITE-12367
URL: https://issues.apache.org/jira/browse/IGNITE-12367
Project: Ignite
Issue Type: Bug
Reporter: Andrey N. Gura
Fix For: 2.8
A fix for https://issues.apache.org/jira/browse/IGNITE-11216 introduced
following bash options
{noformat}
set -o nounset
set -o errexit
set -o pipefail
set -o errtrace
set -o functrace
{noformat}
to Ignite startup scripts. These options cause scripts to fail immediately
1. on any error
2. on any error in child process
3. on any undeclared variable
However, some, if not all non-zero errorcodes are valid and should not cause
scripts to fail.
Due to huge number of ignite startup scripts and possible configurations, risk
is very high that all possible problems were not found and more to go with
similar symptoms: scripts just silently exit at random.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
