Shawn, What's the purpose of Presto then if you consider holding all the data in RAM? From what I see, Presto is intended for joining the data stored in different storages.
As for the Ignite persistence, here are some performance hints you might need to apply: https://apacheignite.readme.io/docs/durable-memory-tuning -- Denis On Mon, Oct 16, 2017 at 7:42 PM, shawn.du <[email protected]> wrote: > Hi Denis, > > We are evaluating this feature(our production use ignite 1.9 and we are > testing ignite 2.2), this do make things simple. But we don't want to loss > performance. > we need careful testing, seeing from our first round test result, the disk > IO will be the bottleneck. > The load average is higher than ignite 1.9 without this feature. Also i > don't know ignite load data from disk will fast enough > comparing with decode the data in memory. > > Thanks > Shawn > > On 10/17/2017 10:25，Denis Magda<[email protected]> <[email protected]> > wrote： > > Shawn, > > Then my suggestion would be to enable Ignite persistence [ > 1] that will store the whole data set you have. RAM will > keep only a subset for the performance benefits. Ignite > SQL is full supported for the persistence, you can even > join data RAM and disk only data sets. Plus, your compression becomes > optional. > > > [1] https://ignite.apache.org/features/persistence.html <htt > ps://ignite.apache.org/features/persistence.html> > > — > Denis > > > On Oct 16, 2017, at 7:18 PM, shawn.du <[email protected]> wrote: > > > > Hi Denis, > > > > Yes, We do want to limit the RAM to less than 64G. > RAM resource is still an expensive resource. > > If we store our data with ignite SQL queryable format, > our data may use more than 640G. This is too expensive for us. > > So we store data using binary format which works a > bit like orc or parquet. Only several important columns are > SQL queryable and the others are not. In this way, we do > store using less RAMs, but we have to use map-reduce to > query the data, which is a little bit of complex: Query > in client with SQL, then submit jobs to ignite compute, > finally do some post aggregation in client. > > This is why I want to have a try of Presto. We like SQL, > we want all computation on server side. > > > > welcome your comments. > > > > Thanks > > Shawn > > > > On 10/17/2017 07:57，Denis Magda<[email protected]> <mailto: > [email protected]> wrote： > > Hello Shawn, > > > > Do I understand properly that you have scarce RAM > resources and think to exploit Presto as an alternative SQL engine in > Ignite that queries both RAM and disk data sets? If that’s > the case than just enable Ignite native persistence [1] > and you’ll get all the data stored on disk and as much as > you can afford in RAM. The SQL works over both tiers transparently for you. > > > > > [1] https://ignite.apache.org/features/persistence.html < > https://ignite.apache.org/features/persistence.html> <ht > tps://ignite.apache.org/features/persistence.html <htt > ps://ignite.apache.org/features/persistence.html>> > > > > — > > Denis > > > > > On Oct 16, 2017, at 2:19 AM, Alexey Kukushkin <kukushki > [email protected] <mailto:[email protected]>> wrote: > > > > > > Cross-sending to the DEV community. > > > > > > On Mon, Oct 16, 2017 at 12:14 PM, shawn.du <[email protected] > <mailto:[email protected]> <mailto:[email protected] <mailto: > [email protected]>>> wrote: > > > Hi community, > > > > > > I am trying to implement a connector for presto to connect ignite. > > > I think it will be a very interest thing to connect ignite and presto. > > > > > > > In fact, currently we use ignite and it works very well. > but in order to save memory, we build compressed binary data. > > > thus we cannot query them using SQL. We use ignite map-reduce to query > > > the data. > > > > > > > Using presto, we may use SQL again. If it is fast > enough, ignite will be our in memory storage and not > responsible for computing or only for simple query. > > > The only thing I concern about is presto is fast > enough or not like Ignite. For now all ignite query cost > less than 5 seconds and most are hundreds of milliseconds. > > > Also presto provides a connector for redis. I don't > know community has interest to contribute to presto-ignite? > > > > > > Thanks > > > Shawn > > > > > > > > > > > > > > > -- > > > Best regards, > > > Alexey > > > >
