Hi Robert, The architecture [1] suggested here is an ideal one. You deploy Ignite separately as an in-memory store and use one of the replication techniques to update the HDFS cluster: * Hadoop Connector form [1]. * Implement a custom version of a CacheStore [2] interface that will push all the changes from Ignite to Hadoop.
Does this work for you? [1] https://www.gridgain.com/docs/latest/integrations/datalake-accelerator/getting-started [2] https://apacheignite.readme.io/docs/3rd-party-store#section-custom-cachestore - Denis On Fri, Oct 25, 2019 at 8:21 AM Munoz, Robert <[email protected]> wrote: > Hi Denis, > > > > Thank you for responding. The use case is to deploy ignite cluster on our > Hadoop cluster nodes and make use of persistence to HDFS since the data > disks have already been allocated for HDFS. I am new to ignite so am > unfamiliar with its file/folder structure but the goal would be to use > ignite as in memory cache for sql tables and to persist the data to HDFS as > write through / read through . This should not impact the current HDFS > configuration as we have users/applications using HDFS already. The > implementation should be standard sql and not hbase get/put operations. Let > me know if this helps. > > > > > > > > Thanks, > > > > Robert Munoz > > 949-220-5169 > > https://pimco.webex.com/meet/robert.munoz > > > > > > > > *From:* Denis Magda <[email protected]> > *Sent:* Thursday, October 24, 2019 10:08 PM > *To:* [email protected] > *Subject:* Re: Support for hdfs caching > > > > Robert, > > > > Could you please describe your use case? There several options that might > suit your needs. > > > > Denis > > On Thursday, October 24, 2019, Munoz, Robert <[email protected]> > wrote: > > Hello, > > > > With the removal of Hadoop accelerator from ignite binaries, are there any > other options for using HDFS as persistence store? > > > > > > Thanks, > > > > Robert Munoz > > > > This message contains confidential information and is intended only for > the individual named. If you are not the named addressee, you should not > disseminate, distribute, alter or copy this e-mail. Please notify the > sender immediately by e-mail if you have received this e-mail by mistake > and delete this e-mail from your system. E-mail transmissions cannot be > guaranteed to be secure or without error as information could be > intercepted, corrupted, lost, destroyed, arrive late or incomplete, or > contain viruses. The sender, therefore, does not accept liability for any > errors or omissions in the contents of this message which arise during or > as a result of e-mail transmission. If verification is required, please > request a hard-copy version. This message is provided for information > purposes and should not be construed as a solicitation or offer to buy or > sell any securities or related financial instruments in any jurisdiction. > Securities are offered in the U.S. through PIMCO Investments LLC, > distributor and a company of PIMCO LLC. > > The individual providing the information herein is an employee of Pacific > Investment Management Company LLC ("PIMCO"), an SEC-registered investment > adviser. To the extent such individual advises you regarding a PIMCO > investment strategy, he or she does so as an associated person of PIMCO. > To the extent that any information is provided to you related to a > PIMCO-sponsored investment fund ("PIMCO Fund"), it is being provided to you > in the individual's capacity as a registered representative of PIMCO > Investments LLC ("PI"), an SEC-registered broker-dealer. PI is not > registered, and does not intend to register, as a municipal advisor and > therefore does not provide advice with respect to the investment of the > proceeds of municipal securities or municipal escrow investments. In > addition, unless otherwise agreed by PIMCO, this communication and any > related attachments are being provided on the express basis that they will > not cause PIMCO LLC, or its affiliates, to become an investment advice > fiduciary under ERISA or the Internal Revenue Code. > > > > -- > > - > > Denis > > >
