IGFS is alive: proposals of vertical solutions

[Manu]

Hi!

I read in other post that IGFS is going to be discontinued, isn't it?

I have a proposal to come alive again IGFS, it has interesting applications,
this could be one:
 



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Lucene text index persistent ?

[Manu]

Hi! take a look to 
https://github.com/hawkore/examples-apache-ignite-extensions/ they are 
implemented a solution for persisted lucene and spatial indexes 



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Client nodes are unable to discover server nodes using kubernetes IpFinder

[Manu]

Hi!

We are trying to deploy an ignite cluster(s) into kubernetes and found some
issues.

Sample deployed topology into kubernetes (we need each ignite server cluster
has different discovery port)

Ignite Server(s) cluster to store main data (discoPort 49500)
\
  Client App with 2 ignite clients (discoPort 49500, discoPort 48500) to
talk with both clusters
/
Ignite Server(s) cluster to store cached data (discoPort 48500)

Apparently, client nodes are unable to discover server nodes using
kubernetes IpFinder whether non default discovery port, for example 49500,
is used on server side ignite configuration.

TcpDiscoverySpi local port configuration is ignored on client mode, so port
47500 is always used to connect to ignite servers (we don't want to use
forceServerMode=true as it is a deprecated feature and may have negative
performance impact)

In general seems that any IpFinder implementation that has this behaviour
(not provides server port when registered addresses were returned) will have
same issue.

Fix suggestion, modify org.apache.ignite.spi.discovery.tcp.ClientImpl
spiStart, apparently this change has no collateral effects:

Original 
   @Override public void spiStart(@Nullable String igniteInstanceName)
throws IgniteSpiException {
spi.initLocalNode(
0,
true);

Fix
   @Override public void spiStart(@Nullable String igniteInstanceName)
throws IgniteSpiException {
// FIX ignored TcpDiscoverySpi local port configuration on client
mode, takes configured one
spi.initLocalNode(
spi.locPort,
true);
 

How to reproduce issue:

1. Deploy a server node with tcp discovery kubernetes IpFinder and non
default discovery port 49500 into kubernetes

2. Deploy client nodes with tcp discovery kubernetes IpFinder and non
default discovery port 49500 into kubernetes

3. You will see on client nodes logs that will try to connect to servers on
port 47500, instead of configured 49500.

Ignite configuration for both server and client node(s)
[...]

  



  
  



https://kubernetes.default.svc.cluster.local:443"/>

  

  

[...]

Regards!

Manu.






--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: [ML] Machine Learning Pipeline Improvement

[Manu]

Hi, all!

Could be viable to integrate Apache Arrow to improve ML computation using
GPU?
Out of this thread, could be viable to integrate Apache Arrow to improve
Indexing computation using GPU?

Regards

https://rapids.ai   
https://arrow.apache.org   



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

NATIVE PERSISTENCE: Cache data is destroyed after disable WAL and restart

[Manu]

Hi!

I have a question, is it normal that if WAL is deactivated for a persisted
cache when the server node(s) is restarted, the persisted content of the
cache is completely destroyed?

I need to disable WAL for large heavy ingestion processes, but eventually
ingestion may fail (OS, machine crash), so WAL state is not re-enabled after
ingestion. On this situation if I restart a server node, cache persistent
directory is deleted and recreated again, so data is lost.

Thanks!

This is the method that do this hell thing
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.beforeCacheGroupStart

Process to replicate it:

1. Start one or more server nodes with native persistence enabled
2. Create a cache (natively persisted) and store some data
3. Disable WAL for cache - ignite().cluster().disableWal("TheCacheName")
4. Restart server/s nodes
5. Check cache directory was deleted and recreated again, all data was lost.

Call stack on server node start:
*org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.beforeCacheGroupStart*
org.apache.ignite.internal.processors.cache.ClusterCachesInfo.registerCacheGroup
org.apache.ignite.internal.processors.cache.ClusterCachesInfo.registerNewCache
org.apache.ignite.internal.processors.cache.ClusterCachesInfo.processJoiningNode
org.apache.ignite.internal.processors.cache.ClusterCachesInfo.onStart
*org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCachesOnStart*
org.apache.ignite.internal.processors.cache.GridCacheProcessor.onReadyForRead
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead
org.apache.ignite.internal.IgniteKernal.start
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start
org.apache.ignite.internal.IgnitionEx.start0
org.apache.ignite.internal.IgnitionEx.startConfigurations
org.apache.ignite.internal.IgnitionEx.start
org.apache.ignite.internal.IgnitionEx.start
org.apache.ignite.internal.IgnitionEx.start
org.apache.ignite.internal.IgnitionEx.start
org.apache.ignite.Ignition.start
org.apache.ignite.startup.cmdline.CommandLineStartup.main

Ignite version 2.7.0



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Hello Ignite Community

[Manu]

Hello Ignite Community!

My name is Manu. I want to contribute to Apache Ignite. My JIRA username
mnusan. Any help on this will be appreciated.

Thanks!



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Manu]

Hi,

Have you think about implementing TDE per DataRegion instead of per Cache?

And using a transparent encrypted java  file system?  
This GitHub project is interesting  https://github.com/usrflo/encfs4j
  .

Hope it helps!

Regards!



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: [GitHub] ignite pull request #3719: IGNITE-8048 merge query entities for dynamic cach...

[Manu]

Hi,

Sorry, I assumed that QueryEntities merge process was always applied (on
active or inactive cluster) and that it was responsible for auto creating
and populate new indexes and columns on the fly... my fault.

Then, in addition to updating CacheConfiguration and QuerySchema, does it do
something else?

One more question, when the cluster is re-activated, will new columns and
new indexes be created and populated?

Thanks!



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: [GitHub] ignite pull request #3719: IGNITE-8048 merge query entities for dynamic cach...

[Manu]

Hi! 

I have taken a look at this new functionality and it is very interesting.
But I have a couple of doubts about the performance.

For each modification of the same QueryEntitity, a new
SchemaAbstractOperation is generated, in the case of adding columns the
performance is not affected, but in the case of creating new indexes I think
there is a problem:

- If the modification in the QueryEntitity contains the creation of N
indexes, N SchemaIndexCreateOperation are generated for the same table, this
causes the underline cache to be scanned N times to populate each indice (by
SchemaIndexCacheVisitorClosure). For caches with a few data is not a
problem, but for caches with millions of records is not optimal.

A possible solution would be to create a new type of SchemaAbstractOperation
(for example SchemaUpgradeQueryEntityOperation with the new QueryEntity,
boolean forceRebuild, boolean forceMutateQueryEntity), to register on one
block this new QueryEntitity for:

- Create new indices at same time (populate all indices with a single scan)
(auto, like now)
- With on demand forceMutateQueryEntity (ignore QueryEntity patch checks),
ability to eliminate indices (and fully release resources) and columns. Over
time the QueryEntity will evolve and at some point it is possible that
certain columns and indices are not necessary any more... I think it would
be interesting to allow trim indexed data and eliminate unused columns.

Hope it helps!

Regards




--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/