I don’t think there’s anything “out of the box,” but you could write a custom CacheStore to do that.
See here for more details: https://apacheignite.readme.io/docs/3rd-party-store#section-custom-cachestore Regards, Stephen > On 9 Aug 2019, at 21:50, sri hari kali charan Tummala > <[email protected]> wrote: > > one last question, is there an S3 connector for Ignite which can load s3 > objects in realtime to ignite cache and data updates directly back to S3? I > can use spark as one alternative but is there another approach of doing? > > Let's say I want to build in-memory near real-time data lake files which get > loaded to S3 automatically gets loaded to Ignite (I can use spark structured > streaming jobs but is there a direct approach ?) > > On Fri, Aug 9, 2019 at 4:34 PM sri hari kali charan Tummala > <[email protected] <mailto:[email protected]>> wrote: > Thank you, I got it now I have to change the id values to see the same data > as extra results (this is just for testing) amazing. > > val df = spark.sql(SELECT monolitically_id() as id, name, department FROM > json_person) > > df.write(append)... to ignite > > Thanks > Sri > > > On Fri, Aug 9, 2019 at 6:08 AM Andrei Aleksandrov <[email protected] > <mailto:[email protected]>> wrote: > Hi, > > Spark contains several SaveModes that will be applied if the table that you > are going to use exists: > > * Overwrite - with this option you will try to re-create existed table or > create new and load data there using IgniteDataStreamer implementation > * Append - with this option you will not try to re-create existed table or > create new table and just load the data to existed table > * ErrorIfExists - with this option you will get the exception if the table > that you are going to use exists > > * Ignore - with this option nothing will be done in case if the table that > you are going to use exists. If table already exists, the save operation is > expected to not save the contents of the DataFrame and to not change the > existing data. > > According to your question: > > You should use the Append SaveMode for your spark integration in case if you > are going to store new data to cache and save the previous stored data. > > Note, that in case if you will store the data for the same Primary Keys then > with data will be overwritten in Ignite table. For example: > > 1)Add person {id=1, name=Vlad, age=19} where id is the primary key > 2)Add person {id=1, name=Nikita, age=26} where id is the primary key > > In Ignite you will see only {id=1, name=Nikita, age=26}. > > Also here you can see the code sample for you and other information about > SaveModes: > > https://apacheignite-fs.readme.io/docs/ignite-data-frame#section-saving-dataframes > > <https://apacheignite-fs.readme.io/docs/ignite-data-frame#section-saving-dataframes> > > BR, > Andrei > > On 2019/08/08 17:33:39, sri hari kali charan Tummala <[email protected]> > <mailto:[email protected]> wrote: > > Hi All,> > > > > I am new to Apache Ignite community I am testing out ignite for knowledge> > > sake in the below example the code reads a json file and writes to ingite> > > in-memory table is it overwriting can I do append mode I did try spark> > > append mode .mode(org.apache.spark.sql.SaveMode.Append)> > > without stopping one ignite application inginte.stop which keeps the cache> > > alive and tried to insert data to cache twice but I am still getting 4> > > records I was expecting 8 records , what would be the reason ?> > > > > https://github.com/apache/ignite/blob/1f8cf042f67f523e23f795571f609a9c81726258/examples/src/main/spark/org/apache/ignite/examples/spark/IgniteDataFrameWriteExample.scala#L89 > > > > <https://github.com/apache/ignite/blob/1f8cf042f67f523e23f795571f609a9c81726258/examples/src/main/spark/org/apache/ignite/examples/spark/IgniteDataFrameWriteExample.scala#L89>> > > > > > > -- > > > Thanks & Regards> > > Sri Tummala> > > > > > -- > Thanks & Regards > Sri Tummala > > > > -- > Thanks & Regards > Sri Tummala >
