Hi Tracy, In addition to Vlad answer about SQL fields query I would like to mention, that may be you can implement your functionality directly in Ignite without Spark? Ignite has a lot of features: in memory key-value storage, SQL and scan queries and Compute engine for map reduce (and much more). See: https://ignite.apache.org/features.html
On Tue, Oct 11, 2016 at 3:25 PM, Vladislav Pyatkov <vldpyat...@gmail.com> wrote: > Hi Tracy, > > Ignite support SQLFieldQuery for the purpose[1] > SQL with default marshaller (Binary) will be use only needed fields when > evaluation. > > [1]: https://apacheignite.readme.io/docs/sql-queries#fields-queries > > On Mon, Oct 10, 2016 at 8:54 PM, Tracy Liang (BLOOMBERG/ 731 LEX) < > tlian...@bloomberg.net> wrote: > >> Thanks for this clear explanation, Alexey. Basically I want to use Ignite >> as a shared in-memory layer among multiple Spark Server instances. Also I >> have another question: does ignite cache support predicate pushdown or a >> logic view of cache? For example, I only want certain column of the value >> instead of returning the entire universe. How do I do that? >> >> >> From: user@ignite.apache.org >> Subject: Re: Is it possible to enable both REPLICATED and PARTITIONED? >> >> Tracy, >> >> First of all, cache mode and number of backups could be set only once - >> on cache start. >> So, if you know the size of your cluster you could set number of backups >> before cache start. >> But, I think it is not reasonable to set number of backups equals to >> number of nodes. >> If you need 100% high availability, just use replicated cache. But I >> would recommend to think about how many nodes at once can be lost? >> May be it is reasonable to set backups = 2? The more backups you choose - >> the more memory will be consumed by backup partitions and also >> grid will spend time in rebalancing data. >> What is your use case? >> >> >> On Mon, Oct 10, 2016 at 11:18 PM, Tracy Liang (BLOOMBERG/ 731 LEX) < >> tlian...@bloomberg.net> wrote: >> >>> Thanks, and PARTITIONED mode could have any number of backups right? I >>> want backups for high availability and also my dataset is large. I guess I >>> will use PARTITIONED mode and configure number of backups based on actual >>> needs in that case right? >>> >>> From: user@ignite.apache.org >>> Subject: Re: Is it possible to enable both REPLICATED and PARTITIONED? >>> >>> Hi, Tracyl. >>> >>> Actually, REPLICATED cache is a PARTITIONED cache win backups on all >>> nodes. >>> >>> But, why did you need this? >>> >>> On Mon, Oct 10, 2016 at 10:46 AM, Tracyl <tlian...@bloomberg.net> wrote: >>> >>>> As subject shows. >>>> >>>> >>>> >>>> -- >>>> View this message in context: http://apache-ignite-users.705 >>>> 18.x6.nabble.com/Is-it-possible-to-enable-both-REPLICATED- >>>> and-PARTITIONED-tp8167.html >>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com. >>>> >>> >>> >>> >>> -- >>> Alexey Kuznetsov >>> >>> >>> >> >> >> -- >> Alexey Kuznetsov >> >> >> > > > -- > Vladislav Pyatkov > -- Alexey Kuznetsov
