I am reading a parquet file using Spark dataframe as df
I want to write some part of this data to ignite cache
Assume I want to write df2 to the cache
I used df2.write.format('ignite')
Is there a better way to do this or this is the only way??
Regards
Arunima
On Mon, 3 Jul, 2023, 1:19 pm Stephen Darlington, <
stephen.darling...@gridgain.com> wrote:
> Commercial options are available, but otherwise help would generally be
> limited to email lists and Stack Overflow.
>
> On 1 Jul 2023, at 06:59, Arunima Barik <arunimabari...@gmail.com> wrote:
>
> Are there any provisions wherein I can discuss about my project
> implementation with someone from the Ignite team to clarify some doubts?
>
> Preferably through a small online meet?
>
> Regards
> Arunima
>
> On Sat, 1 Jul, 2023, 12:03 am Jeremy McMillan, <
> jeremy.mcmil...@gridgain.com> wrote:
>
>> Python doesn't at this time go anywhere near Ignite CacheStore. You would
>> need to implement the CacheStore in Java or some other language which
>> compiles to JVM runtime/jar. There's a talk from the most recent summit on
>> using Groovy, if you want a higher level language than Java, but
>> theoretically you could use Jython (if you are willing to experiment and
>> can find a compatible JVM that runs both Ignite and Jython).
>>
>> Ignite can operate like a federated query proxy if different caches are
>> implemented with different external persistence for each cache. CacheStore
>> is the interface Ignite would use to send a cache miss to a backend
>> database. In your original question you intended to use Parquet files as a
>> backend database, but Ignite does not (yet) provide one for Parquet. If
>> someone were to donate a supportable Java implementation, I suspect the
>> community would adopt and support it. Since Parquet is columnar, I also
>> suspect it would need to target Ignite 3 to adopt conventions around
>> columnar data, and then might be backported to Ignite 2.
>>
>>
>> On Fri, Jun 30, 2023 at 12:13 PM Arunima Barik <arunimabari...@gmail.com>
>> wrote:
>>
>>> Which do you think would be a better option?
>>>
>>> Federated queries or CacheStore
>>>
>>> And is CacheStore supported in Python?
>>>
>>> On Fri, 30 Jun, 2023, 1:50 pm Stephen Darlington, <
>>> stephen.darling...@gridgain.com> wrote:
>>>
>>>> You’d need to implement your own Cache Store.
>>>> https://ignite.apache.org/docs/latest/persistence/custom-cache-store
>>>>
>>>> On 30 Jun 2023, at 06:46, Arunima Barik <arunimabari...@gmail.com>
>>>> wrote:
>>>>
>>>>
>>>> ---------- Forwarded message ---------
>>>> From: Arunima Barik <arunimabari...@gmail.com>
>>>> Date: Fri, 30 Jun, 2023, 10:52 am
>>>> Subject: Ignite for Parquet files
>>>> To: <user@ignite.apache.org>
>>>>
>>>>
>>>> Hello Team
>>>>
>>>> I have my data stored as parquet files. I want a caching layer on top
>>>> of this existing file system. I am going to use Ignite for that but I do
>>>> not need native persistence for that.
>>>>
>>>> I want that any changes to database should be reflected in both cache
>>>> and file.
>>>> And same for read queries. It should automatically read from disk if
>>>> data is not present in cache.
>>>>
>>>> I want to do all this is python. Please let me know how the same can be
>>>> done.
>>>> Resources if any as well.
>>>>
>>>> Thank you and looking forward to hearing from you.
>>>>
>>>> Regard,
>>>> Arunima Barik
>>>>
>>>>
>>>>
>
