Hello Igniters, I wonder if anybody could advise on the following. I am trying to implement simple extract, transform, load (ETL) process using Apache Ignite 2.3 platform. In extract phase local caches are loaded with some data and then passed to transform phase where the SQL SELECT (sometimes quite complex) is executed against these local caches (after SQL execution the local caches are destroyed). Then the result of SQL is uploaded to global partitioned cluster-wide cache.
The problem is that local caches seem to be rather heavy tool for this purpose (we are about to process thousands of local caches per day). They seem not to be 'very local', i.e. involve significant part of cluster infrastructure (exchange worker, grid timeout workers, etc...), and causing memory leaks (like IGNITE-7319, IGNITE-7918). I would like to ask if there is a way to execute SQL queries against some really lightweight cache layer (i.e. somehow replace the underlying infrastructure with simple map implementation or something similar). Also, would the request for real private cache (not visible to other nodes, not involving cluster infrastructure, allowing SQL execution) make sense? thank you in advance, zbyszek -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
