Re: Distributed Closures VS Executor Service

Dear all, I have created a new Topic: "Ignite SQL Indexing Performance problems" since the conversation has deviate a lot form the original subject. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Distributed-Closures-VS-Executor-Service-tp11192p12347.html Sent

Re: Distributed Closures VS Executor Service

Hello, I don't know about H2, I sent you link from apacheignite.readme.io which describes that IN clause does not support Indexes when run on Ignite. Also at my previous post I have sent a simply query using IN clause that has very poor performance on Ignite even run only on local mode with less

Re: Distributed Closures VS Executor Service

Actually H2 can use index for IN operator, but using JOIN instead of IN may be still useful. Sergi 2017-05-01 12:40 GMT+03:00 Nikolai Tikhonov : > Hi, > > The page [1] contains also workaround for IN operation. You can use JOIN > instead of IN. > > Sergi, > > Will this tip

Re: Distributed Closures VS Executor Service

Hi, The page [1] contains also workaround for IN operation. You can use JOIN instead of IN. Sergi, Will this tip actual for 2.0 version yet? 1. https://apacheignite.readme.io/docs/sql-performance-and- debugging#sql-performance-and-usability-considerations On Sun, Apr 30, 2017 at 1:50 PM,

Re: Distributed Closures VS Executor Service

After investigation more I run into the following: https://apacheignite.readme.io/docs/sql-performance-and-debugging#sql-performance-and-usability-considerations So, cause my query has IN clause no indexing will be used anyway. Could you please elaborate more on this topic and if/what will

Re: Distributed Closures VS Executor Service

Hello Andrey, The issue with the indexes for such a plain query seems major IMHO. So at version 2.0 adding hints will be supported? Is this final? However, even with no indexes, the query execution time it too much for in-memory (time=8645 ms). I have streamed to cache a single table of

Re: Distributed Closures VS Executor Service

Hi, 1. Your query uses index "PPSP-IMDG-CACHE"."type_idx" that included single field. Most probably it is due to H2 planner lack. Unfortunately, current H2 version does not support index hints. This should be fixed in ignite-2.0. 2. SQL queries are running in single thread by default. But it is

Re: Distributed Closures VS Executor Service

Hello again, I am testing the approach described in previous posts with REPLICATED cache mode. The performance of the queries is extremely poor. Example: [18:23:33,016][INFO ][grid-timeout-worker-#13%ppsp-cluster-IV%][IgniteKernal%ppsp-cluster-IV] Metrics for local node (to disable set

Re: Distributed Closures VS Executor Service

Moreover, even if I use custom Affinity what will happen if a node fails (or a node is added)? Will I be aware / have control at which node its cache is transferred so I send there co-located jobs? P.S: this table (SimulationInitialValues) is mainly read-only with the exception that new entries

Re: Distributed Closures VS Executor Service

Hello Christo, The problem with the original approach is this: - I have a flat table structure which is is mapped by the following POJO: @Entity @Table(name = "simulation_initial_values") public class SimulationInitialValues extends IdEntity { @Basic @Column(name =

Re: Distributed Closures VS Executor Service

Hi Kyriako, Thinking about your original approach of using compute tasks assigned to nodes with all CATEGORIES required by the task to be local, I believe you could use the affinity interface to figure this out. You'd need to partition your CATEGORIES cache and use the CATEGORY ID as the

Re: Distributed Closures VS Executor Service

Hello Christo, Thank you for your prompt response. I will check your idea and get back to you and the community. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Distributed-Closures-VS-Executor-Service-tp11192p11759.html Sent from the Apache Ignite Users

Re: Distributed Closures VS Executor Service

If I understand correctly, the SQL query is executed within every task on each of the nodes and it is not set to be a local query. Correct? If so then what you are really doing is executing the SQL queries from all the nodes to all the nodes. This is bound to be inefficient. In essence what you

Re: Distributed Closures VS Executor Service

Hi Christo, I have tried the following: - When app starts, I stream into PARTITIONED CACHE one (1) table needed for the distributed jobs - Now when a Task is submitted, each Job takes over some CATEGORYIDs. Performs an Ignite cache-query (I paste it below), gathers the results, runs calculations

Re: Distributed Closures VS Executor Service

Hi Kyriako, Agree with Nikolai on both points. Regarding point number 2, you want to use affinity data / data colocation and data / compute colocation. You basically need to ensure that when you are inserting data into the cache you use the same affinity for all entries that need to be

Re: Distributed Closures VS Executor Service

Hello Nikolai, all About 1: yes, this was my perception as well, thanks for the confirmation About 2: Even if all the nodes provide result sets of local execution to the query initiator, if we are talking about results being Lists containing a couple of thousands POJOs, then wouldn't be a big

Re: Distributed Closures VS Executor Service

Hi, 1) The better way using DataStreamer API. More details you can find there [1]. 2) Internal implementation SQL works in similar fashion. The query will be parsed and split into multiple map queries and a single reduce query. All the map queries are executed on all the data nodes where cache

Re: Distributed Closures VS Executor Service

Hello Christo and thank you for your feedback. For the moment we have a Relational Database that is not distributed or partitioned. We have given some thought for configuring at least some database tables as cache store to Ignite and it not something that we exclude. Just for the moment due to

Re: Distributed Closures VS Executor Service

> - The application server-side queries the DB, finds the products belonging > to the selected categories, performs calculations (applies the metrics on > them etc.) and persists the results in the database for future reference So if I understand this correctly you are not loading any data from

Re: Distributed Closures VS Executor Service

kmandalas wrote* > One question is * > : if some Job fails and other Jobs belonging to the same Task are still > running (in the same or other grid nodes), is it possible to cancel them > and abort the whole Task before waiting for all of them to finish? This will happen if result() method throws

Re: Distributed Closures VS Executor Service

-- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Distributed-Closures-VS-Executor-Service-tp11192p11517.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Distributed Closures VS Executor Service

Hello Christo, After checking in more detail the options we have with Ignite, our first user case and feedback we have so far (e.g. feedback from Dmitry here: http://apache-ignite-users.70518.x6.nabble.com/Distributed-callables-passing-data-and-getting-back-results-td11224.html), we conclude that

Re: Distributed Closures VS Executor Service

Hi Kyriako, - If I want to pass data to callables, for example Lists of objects, small to -medium size (collections from 2000 to 6000 objects maximum, with average object size some ~200KBs) what is the best way to do this: to pass them as argument in callables or put them in the distributed cache

Re: Distributed Closures VS Executor Service

Hi, Ignite has internal implementation ThreadPoolExecutor, which used in ignite compute, and it has BlockingQueue with Integer#MAX_VALUE internal. Lock at in IgniteThreadPoolExecutor [1]. [1]:

Re: Distributed Closures VS Executor Service

Hi, Ignite compute[2] is the general mechanism to execute distributed task. If you are mean Executor Service[1], it is only convenient API. Distributed tasks will be stay at queue if public thread pool is full. If you need to increase thread count in public pull you can do that[3]. ...