Re: Unreasonable segmentation in Kubernetes on one node reboot

[artem_zin]

What bothers me after looking carefully at the logs is that there are ports
in use that I've not specified,

like this log entry and similar to it:

```
locAddr=/10.42.32.36:42504,
rmtAddr=ignite-0.ignite.buildcache.svc.cluster.local/10.42.40.84:47100
```

Note that local port is `42504` even though it's set to default 47100 for
communication and 47500 for discovery and ranges of both are set to `0`.

If `42504` and other similar ridiculous ports like `43851`, etc I see in
other logs are actually in use and exposed for network communications, *that
is problematic in Kubernetes environment*. In k8s, container cannot expose
range of ports atm, only fixed set of predifened ports can be exposed.

Moreover I don't understand where would a port number like `42504` come
from, it's not in any default range of default Ignite ports as far as I know



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: java.lang.NullPointerException in GridDhtPartitionsExchangeFuture

[Павлухин Иван]

Hi Subash,

Correct me if I am wrong, but in current Ignite version persistence is not
mandatory and it is even disabled by default. With disabled persistence
nothing will be written to disk. 2.x version storage architecture was
developed as an improvement of the previous one. Ignite 2.x uses off-heap
memory, but it is vanilla RAM local to JVM process (managed by Ignite
instead of automatic JVM management). Several benefits of off-heap are
mentioned in [1].

[1]
https://apacheignite.readme.io/docs/durable-memory#section-in-memory-features

2018-09-27 14:45 GMT+03:00 Ilya Kasnacheev :

> Hello!
>
> It's hard to answer these questions without thorough review of logs and
> 1.9 code, and I doubt anyone will volunteer to do that since 1.x branch
> does not see any new development in Apache Ignite.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 26 сент. 2018 г. в 22:36, HEWA WIDANA GAMAGE, SUBASH <
> subash.hewawidanagam...@fmr.com>:
>
>> Hi Kasnacheev,
>>
>> Thank you very much for the response..
>>
>>
>>
>> We use v1.9, because 2.x uses mandatory ignite native persistence(local
>> disk) along with durable memory management(RAM), and no option for a **java
>> heap only** cache storage. We wanted to keep away from storing anything in
>> disk. Hence using 1.9 Ignite.
>>
>> Can you please clarify “I believe that cache in question is no longer
>> consistent between nodes on metadata level” ?
>>
>>
>>
>> 1.   Like what could have caused & under what conditions this
>> inconsistent state between nodes on cache metadata occurred ?
>>
>>
>>
>> 2.   And does 2.x fixes the problem?( Is there a way you can suggest
>> to reproduce this issue, so that we can know for sure 2.x fixes the problem)
>>
>>
>>
>> It appeared from our logs that the cache.put threads hangs forever after
>> this error. Can that be possible with this?
>>
>>
>>
>>
>>
>>
>>
>> *From:* Ilya Kasnacheev [mailto:ilya.kasnach...@gmail.com]
>> *Sent:* Wednesday, September 26, 2018 12:31 PM
>> *To:* user@ignite.apache.org
>> *Subject:* Re: java.lang.NullPointerException in
>> GridDhtPartitionsExchangeFuture
>>
>>
>> This email is from an external source - exercise caution regarding links
>> and attachments.
>>
>> Hello!
>>
>>
>>
>> This is a bad error message. I believe that cache in question is no
>> longer consistent between nodes on metadata level, and you can't fix that
>> without full restart or at least dropping and recreating the cache.
>>
>> I don't think that you will get much support from community on 1.x since
>> the focus has shifted to 2.x. Have you considered upgrading?
>>
>>
>>
>> Regards,
>>
>> --
>>
>> Ilya Kasnacheev
>>
>>
>>
>>
>>
>> ср, 26 сент. 2018 г. в 18:50, HEWA WIDANA GAMAGE, SUBASH <
>> subash.hewawidanagam...@fmr.com>:
>>
>> This is the only single error from ignite happened after JVM startup.
>> Looks like I only posted the exception stack trace. Here’s the message and
>> from which thread it logged.
>>
>>
>>
>> level:ERROR
>>
>>
>>
>>  logger: 
>> org.apache.ignite.internal.processors.cache.GridCacheIoManager
>>
>>
>>
>>
>>  message:Failed processing message
>> [senderId=57ee6544-e0b3-45cc-bdb5-f3fd37d7db1e, 
>> msg=GridDhtPartitionsSingleMessage
>> [parts=null, partCntrs=null, client=false, compress=false, super=
>> GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId
>> [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0],
>> nodeId=57ee6544, evt=NODE_JOINED], lastVer=GridCacheVersion [topVer=0,
>> time=0, order=1536996016410, nodeOrder=0], flags=0, super=GridCacheMessage
>> [msgId=1, depInfo=null, err=null, skipPrepare=false, cacheId=0,
>> cacheId=0 [IM_GROUP=fsy-pi-dt-ssam ]
>>
>>
>>
>>  thread:sys-stripe-2-#3%null%
>>
>>
>>
>> *From:* Ilya Kasnacheev [mailto:ilya.kasnach...@gmail.com]
>> *Sent:* Tuesday, September 25, 2018 11:11 AM
>> *To:* user@ignite.apache.org
>> *Subject:* Re: java.lang.NullPointerException in
>> GridDhtPartitionsExchangeFuture
>>
>>
>> This email is from an external source - exercise caution regarding links
>> and attachments.
>>
>> Hello!
>>
>>
>>
>> It's hard to say without reviewing logs, but it seems that there's some
>> inconsistency with regards to cache metadata on nodes.
>>
>>
>>
>> Regards,
>>
>> --
>>
>> Ilya Kasnacheev
>>
>>
>>
>>
>>
>> вт, 25 сент. 2018 г. в 0:13, HEWA WIDANA GAMAGE, SUBASH <
>> subash.hewawidanagam...@fmr.com>:
>>
>> Hi all,
>>
>> We use Ignite 1.9.
>>
>>
>>
>> We could see this in our logs.  All we do is cache.get() , cache.put()
>> operations. With this log being seen, is it possible for  cache.put or
>> ignite.getOrCreateCache() method calling threads be blocked forever ?
>> (unfortunately we couldn’t get a thread dump to prove that, but from
>> application logs, it looks like it).
>>
>>
>>
>> java.lang.NullPointerException: null
>>
>> at org.apache.ignite.internal.
>> 

Re: Unreasonable segmentation in Kubernetes on one node reboot

[artem_zin]

With timeouts you've recommended we are still observing sudden latency spike
on client -> server communication leading to timeouts and restart after just
2 minutes running *with no production load on service*, literally just
health checks running

Here are logs from client pod:

```
$ kubectl logs -f -n buildcache buildcache-3 --previous

2018-09-28T05:23:31.231Z [main]/I: Starting the server...
2018-09-28T05:23:31.381Z [RxComputationThreadPool-1]/I: JvmState =
JvmState(totalMemoryMb=4096, usedMemoryMb=10, freeMemoryMb=4085)
Sep 28, 2018 5:23:31 AM java.util.logging.LogManager$RootLogger log
SEVERE: Failed to resolve default logging config file:
config/java.util.logging.properties
Sep 28, 2018 5:23:32 AM org.apache.ignite.logger.java.JavaLogger info
INFO: 

>>>__    
>>>   /  _/ ___/ |/ /  _/_  __/ __/  
>>>  _/ // (7 7// /  / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/   
>>> 
>>> ver. 2.6.0#20180710-sha1:669feacc
>>> 2018 Copyright(C) Apache Software Foundation
>>> 
>>> Ignite documentation: http://ignite.apache.org

Sep 28, 2018 5:23:32 AM org.apache.ignite.logger.java.JavaLogger info
INFO: Config URL: n/a
Sep 28, 2018 5:23:32 AM org.apache.ignite.logger.java.JavaLogger info
INFO: IgniteConfiguration [igniteInstanceName=null, pubPoolSize=8,
svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8,
mgmtPoolSize=4, igfsPoolSize=8, dataStreamerPoolSize=8,
utilityCachePoolSize=8, utilityCacheKeepAliveTime=6, p2pPoolSize=2,
qryPoolSize=8, igniteHome=null, igniteWorkDir=/tmp/ignite/work,
mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@363ee3a2,
nodeId=395941f3-a9fd-41c9-8035-06b2fac1b55e,
marsh=org.apache.ignite.internal.binary.BinaryMarshaller@2898ac89,
marshLocJobs=false, daemon=false, p2pEnabled=false, netTimeout=5000,
sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=1,
metricsUpdateFreq=2000, metricsExpTime=9223372036854775807,
discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0,
marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=60,
forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null],
segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true,
allResolversPassReq=true, segChkFreq=1, commSpi=TcpCommunicationSpi
[connectGate=null, connPlc=null, enableForcibleNodeKill=false,
enableTroubleshootingLog=false,
srvLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2@6a2b953e,
locAddr=null, locHost=null, locPort=47100, locPortRange=0, shmemPort=-1,
directBuf=true, directSndBuf=false, idleConnTimeout=60,
connTimeout=5000, maxConnTimeout=60, reconCnt=10, sockSndBuf=32768,
sockRcvBuf=32768, msgQueueLimit=512, slowClientQueueLimit=0, nioSrvr=null,
shmemSrv=null, usePairedConnections=false, connectionsPerNode=1,
tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32,
unackedMsgsBufSize=0, sockWriteTimeout=2000, lsnr=null, boundTcpPort=-1,
boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null,
ctxInitLatch=java.util.concurrent.CountDownLatch@313b2ea6[Count = 1],
stopping=false,
metricsLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationMetricsListener@2f465398],
evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@548e6d58,
colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [lsnr=null],
indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@610f7aa,
addrRslvr=null, clientMode=true, rebalanceThreadPoolSize=1,
txCfg=org.apache.ignite.configuration.TransactionConfiguration@6a03bcb1,
cacheSanityCheckEnabled=true, discoStartupDelay=6, deployMode=SHARED,
p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100,
timeSrvPortRange=100, failureDetectionTimeout=6,
clientFailureDetectionTimeout=3, metricsLogFreq=6, hadoopCfg=null,
connectorCfg=org.apache.ignite.configuration.ConnectorConfiguration@21b2e768,
odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration
[seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null,
grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null,
binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration
[sysRegionInitSize=41943040, sysCacheMaxSize=104857600, pageSize=8192,
concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=Default_Region,
maxSize=52428800, initSize=52428800, swapPath=null,
pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100,
metricsEnabled=false, metricsSubIntervalCount=5,
metricsRateTimeInterval=6, persistenceEnabled=false,
checkpointPageBufSize=0], storagePath=null, checkpointFreq=18,
lockWaitTime=1, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL,
walHistSize=20, walSegments=10, walSegmentSize=67108864, walPath=db/wal,
walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY,
walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000,
walRecordIterBuffSize=67108864, alwaysWriteFullPages=false,

Re: Issues running Ignite with Cassandra and spark.

[Alexey Kuznetsov]

Hi,  Shrey!

Just as idea - Ignite now has persistence (see
https://apacheignite.readme.io/docs/distributed-persistent-store),
 may be you can completely replace  Cassandra with Ignite?

In this case all data always be actual, no need to sync with external db.

-- 
Alexey Kuznetsov

Continuous queries with aggregations

[Francesco di Siena]

We have an ignite system with a table of about 100 columns, and 300,000 rows. 
Of this data around 10 columns are changing frequently, i.e. up to several
times per second on most of the 300K rows.We are able to query this data (e
.g aggregation and grouping) at a reasonable speed (<1s).

We can also use a "continuous query" to listen to all changes to the raw
data, however this only notifies the subscriber of changes to individual
table rows, not the aggregated results.
Is there any mechanism to allow a continuous aggregated query?

e.g. Say my query was: 
 select max(price), min(size), name from Products where price>23 group
by name
e.g. I'd like the client to receive updates at the aggregated level whenever
max(price) changes due to an underlying row causing this to change.

One work-around is to notice when a changed price was the current max, and
if so, re-query the aggregated data, but this is not efficient.
Is there any such a mechanism, or any suggested work-around?
thanks



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Unreasonable segmentation in Kubernetes on one node reboot

[artem_zin]

Sure, I can try that. 

However just to be on same page, high timeouts usually hide errors and let
system go into degraded state while still accepting client requests thus if
something actually goes wrong we can't gracefully recover after say 59
seconds compared to 10 or less.

I'll get back to you with results given timeouts you've suggested 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is ID generator split brain compliant?

[Pavel Pereslegin]

Ankit,
I  tried test your example with the following code:


configure();

System.out.println("Initial value: " + getCurrentValue("sequence"));

for (int i = 0; i < 10_000; i++)
incrementAndGet("sequence");

stop();

configure();

System.out.println("Value after restart: " + getCurrentValue("sequence"));

stop();


And the output is:

Initial value: 150
Value after restart: 1511000


This is the expected result with cleaned persistence directory
(because on last increment was reserved (+1000 by default) values).
Maybe the problem elsewhere?

чт, 27 сент. 2018 г. в 20:14, abatra :
>
> Configuration
>
> private static final String STORAGE_PATH = 
> "/existingDirectory/ignite/data/storage";
> private static final String WAL_PATH = 
> "/existingDirectory/ignite/data/wal";
> private static final String WAL_ARCHIVE_PATH = 
> "/existingDirectory/ignite/data/wal/archive";
>
> @Override
> public void configure() {
> Ignite ignite = Ignition.start(createConfiguration());
> /*
>  * This call is required if ignite persistence is used.
>  */
> ignite.cluster().active(true);
> mIgniteLogger.info("Ignite cluster is configured.");
> }
>
> @Override
> public void stop() {
> Ignition.stopAll(true);
> }
>
> private IgniteConfiguration createConfiguration() {
> IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
> igniteConfiguration.setDiscoverySpi(createDiscoverySpi());
> igniteConfiguration.setGridLogger(mIgniteLogger);
> 
> igniteConfiguration.setDataStorageConfiguration(createDataStorageConfiguration());
> return igniteConfiguration;
> }
>
> private DiscoverySpi createDiscoverySpi() {
> checkMinimumClusterIpAddressCount();
> TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
> TcpDiscoveryVmIpFinder tcpDiscoveryVmIpFinder = new 
> TcpDiscoveryVmIpFinder();
> tcpDiscoveryVmIpFinder.setAddresses(getConfig().getNodeIpAddresses());
> tcpDiscoverySpi.setIpFinder(tcpDiscoveryVmIpFinder);
> return tcpDiscoverySpi;
> }
>
> private DataStorageConfiguration createDataStorageConfiguration() {
> DataStorageConfiguration dataStorageConfiguration = new 
> DataStorageConfiguration();
> 
> dataStorageConfiguration.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
> dataStorageConfiguration.setStoragePath(STORAGE_PATH);
> dataStorageConfiguration.setWalPath(WAL_PATH);
> dataStorageConfiguration.setWalArchivePath(WAL_ARCHIVE_PATH);
> return dataStorageConfiguration;
> }
>
> Usage
>
> @Override
> public long getCurrentValue(String key) {
> return getAtomicSequence(key).get();
> }
>
> @Override
> public long incrementAndGet(String key) {
> return incrementAndGet(getAtomicSequence(key));
> }
>
> @Override
> public List incrementAndGet(String key, int count) {
> List result = new ArrayList<>();
> IgniteAtomicSequence atomicSequence = getAtomicSequence(key);
> for (int i = 0; i < count; i++) {
> result.add(incrementAndGet(atomicSequence));
> }
> return result;
> }
>
> private long incrementAndGet(IgniteAtomicSequence igniteAtomicSequence) {
> return igniteAtomicSequence.incrementAndGet();
> }
>
> private IgniteAtomicSequence getAtomicSequence(String name) {
> return Ignition.ignite().atomicSequence(name, 150, true);
> }
>
> Please let me know if more details are required.
> 
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Ignite Query Slow

[Skollur]

1. I have defined primary key - dwId 
2. Not defined affinity_key since there is no Forgein Key in table. 
3. 
CacheConfiguration ccfg = new CacheConfiguration();
ccfg.setName(cacheName);
ccfg.setCacheMode(CacheMode.PARTITIONED);
ccfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
ccfg.setPeerClassLoadingEnabled(true); 

The code i am using is generated from ignite web console.

Code is below.


CacheConfiguration
ccfg=CacheConfigurationInitialize.initCacheConfiguration(
IdbCacheConstants.GROUP_MEMBER_CACHE, 

jdbcTypeGroupMember(IdbCacheConstants.GROUP_MEMBER_CACHE));

ArrayList qryEntities = new ArrayList<>();

QueryEntity qryEntity = new QueryEntity();

qryEntity.setKeyType("java.lang.Long");
qryEntity.setValueType("com.idb.cache.model.GroupMember");
qryEntity.setTableName("Group_Member");
qryEntity.setKeyFieldName("dwId");

HashSet keyFields = new HashSet<>();

keyFields.add("dwId");

qryEntity.setKeyFields(keyFields);

LinkedHashMap fields = new LinkedHashMap<>();

fields.put("dwId", "java.lang.Long");
fields.put("groupCustomerId", "java.lang.Long");
fields.put("customerId", "java.lang.Long");
fields.put("role", "java.lang.String");
fields.put("isPrimary", "java.lang.String");
fields.put("recordValidFromDate", "java.sql.Date");
fields.put("recordValidToDate", "java.sql.Date");
fields.put("recordIsValid", "java.lang.String");
fields.put("recordCreatedTime", "java.sql.Timestamp");
fields.put("recordCreatedBy", "java.lang.String");
fields.put("recordUpdatedTime", "java.sql.Timestamp");
fields.put("recordUpdatedBy", "java.lang.String");
fields.put("dwBatch", "java.lang.Long");
fields.put("dwSourcecode", "java.lang.String");
fields.put("dwTimestamp", "java.sql.Timestamp");

qryEntity.setFields(fields);

HashMap aliases = new HashMap<>();

aliases.put("dwId", "DW_Id");
aliases.put("groupCustomerId", "Group_Customer_ID");
aliases.put("customerId", "Customer_ID");
aliases.put("recordValidFromDate", "Record_Valid_From_Date");
aliases.put("recordValidToDate", "Record_Valid_To_Date");
aliases.put("recordIsValid", "Record_Is_Valid");
aliases.put("recordCreatedTime", "Record_Created_Time");
aliases.put("recordCreatedBy", "Record_Created_By");
aliases.put("recordUpdatedTime", "Record_Updated_Time");
aliases.put("recordUpdatedBy", "Record_Updated_By");
aliases.put("dwBatch", "DW_Batch");
aliases.put("dwSourcecode", "DW_SourceCode");
aliases.put("dwTimestamp", "DW_TimeStamp");

qryEntity.setAliases(aliases);
qryEntities.add(qryEntity);

ccfg.setQueryEntities(qryEntities);

return ccfg;



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ignite Query Slow

[Skollur]

I have 3 cache store and each has its own table. When I tried to do simple
join, query is taking longer in DBeaver.  Server topology is ver=4,
servers=1, clients=1, CPUs=4, offheap=3.2GB, heap=6.0GB

Cache 1. GroupCache"."[GROUP]" has 100k records
Cache 2. "GroupMemberCache".GROUP_MEMBER  has 200k records
Cache 3. "CustomerCache".CUSTOMER has 200k records

Query is as below
SELECT CUST.Name
FROM "GroupCache"."[GROUP]" GRP
INNER JOIN "GroupMemberCache".GROUP_MEMBER GM
ON GRP.Group_Customer_ID = 44 
INNER JOIN "CustomerCache".CUSTOMER CUST
ON GM.Customer_ID = CUST.Customer_ID
AND GM.Record_Is_Valid = 'Y'
AND GRP.Record_Is_Valid = 'Y'
AND GRP.Group_Customer_ID = GM.Group_Customer_ID
AND GM.Record_Is_Valid = 'Y'

Query is executing longer than 2000 seconds. Any suggestion on this?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is ID generator split brain compliant?

[abatra]

*Configuration*
private static final String STORAGE_PATH =
"/existingDirectory/ignite/data/storage";private static final String
WAL_PATH = "/existingDirectory/ignite/data/wal";private static final
String WAL_ARCHIVE_PATH = "/existingDirectory/ignite/data/wal/archive";   
@Overridepublic void configure() {Ignite ignite =
Ignition.start(createConfiguration());/* * This call is
required if ignite persistence is used. */   
ignite.cluster().active(true);mIgniteLogger.info("Ignite cluster is
configured.");}@Overridepublic void stop() {   
Ignition.stopAll(true);}private IgniteConfiguration
createConfiguration() {IgniteConfiguration igniteConfiguration = new
IgniteConfiguration();   
igniteConfiguration.setDiscoverySpi(createDiscoverySpi());   
igniteConfiguration.setGridLogger(mIgniteLogger);   
igniteConfiguration.setDataStorageConfiguration(createDataStorageConfiguration());
   
return igniteConfiguration;}private DiscoverySpi
createDiscoverySpi() {checkMinimumClusterIpAddressCount();   
TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();   
TcpDiscoveryVmIpFinder tcpDiscoveryVmIpFinder = new
TcpDiscoveryVmIpFinder();   
tcpDiscoveryVmIpFinder.setAddresses(getConfig().getNodeIpAddresses());   
tcpDiscoverySpi.setIpFinder(tcpDiscoveryVmIpFinder);return
tcpDiscoverySpi;}private DataStorageConfiguration
createDataStorageConfiguration() {DataStorageConfiguration
dataStorageConfiguration = new DataStorageConfiguration();   
dataStorageConfiguration.getDefaultDataRegionConfiguration().setPersistenceEnabled(true);
   
dataStorageConfiguration.setStoragePath(STORAGE_PATH);   
dataStorageConfiguration.setWalPath(WAL_PATH);   
dataStorageConfiguration.setWalArchivePath(WAL_ARCHIVE_PATH);return
dataStorageConfiguration;}
*Usage*
@Overridepublic long getCurrentValue(String key) {return
getAtomicSequence(key).get();}@Overridepublic long
incrementAndGet(String key) {return
incrementAndGet(getAtomicSequence(key));}@Overridepublic List
incrementAndGet(String key, int count) {List result = new
ArrayList<>();IgniteAtomicSequence atomicSequence =
getAtomicSequence(key);for (int i = 0; i < count; i++) {   
result.add(incrementAndGet(atomicSequence));}return result;   
}private long incrementAndGet(IgniteAtomicSequence igniteAtomicSequence)
{return igniteAtomicSequence.incrementAndGet();}private
IgniteAtomicSequence getAtomicSequence(String name) {return
Ignition.ignite().atomicSequence(name, 150, true);}
Please let me know if more details are required.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Java 11 and the future of Apache Ignite

[Dmitriy Pavlov]

Hi,

I agree with Ilya, Java 11 is currently under active discussions on
dev.list. You can look at archives using
https://lists.apache.org/list.html?d...@ignite.apache.org

I can see a number of tickets, which created for support/test features of
the newest Java.

Sincerely,
Dmitriy Pavlov

чт, 27 сент. 2018 г. в 18:51, ilya.kasnacheev :

> Hello!
>
> This might be more suited to developers list. I will answers from user
> perspective:
>
> 1) I guess that all LTS releases will be supported (which are every 6th
> release?). Intermediate releases will receive some testing in releases that
> they are released, but will not be a primary target.
>
> 2) Maybe if RedHat and IBM have success in promoting their offerings, there
> will be official support or even migration of development process towards
> that. It's to early to tell.
>
> 3) This was under assumption that JVM remains largely backwards
> compatibility, which is what they're going away from. So you should not
> expect that Ignite will run out of box on later releases than ones it was
> developed for.
>
> 4) It is too early to say.
>
> Regards,
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: ignite .net index group definition

[Pavel Tupitsyn]

Hi,

Yes, attribute-based configuration is limited in this regard.
But, as Ilya said, `CacheConfiguration.QueryEntities` gives you full
control to achieve the same.

Thanks,
Pavel

On Thu, Sep 27, 2018 at 6:46 PM ilya.kasnacheev 
wrote:

> Hello!
>
> Unfortunately this looks like a genuine limitation of .Net's QuerySqlField.
>
> You can try specifying those indexes with QueryEntities or CREATE INDEX
> instead to have more control.
>
> Regards,
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: How to set node Id?

[slava.koptilin]

Hi,

> Any ideas on how we can end up having consistent IDs conflict on restarts?
Well, I see only one reason that can lead to this behavior.
This is the case when a new node is trying to join the cluster with a
`consistentId` which is already in use,
i.e. there is a node with the same `consistentId` (to be more precise, hash
codes of consitentIds are the same).
Perhaps, someone started a node locally, that joined the cluster and has the
same consistentId.

Thanks,
S.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite getAll does not return values randomly

[ilya.kasnacheev]

Hello!

It's hard to say. Do you really have CacheRebalance.NONE? What would you
expect with that? What happens if you take it off your configuration?

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Issues running Ignite with Cassandra and spark.

[ilya.kasnacheev]

Hello!

1) There is no generic way of pulling updates from 3rd party database and
there is no API support for it usually, so it's not obvious how we could
implement that even if we wanted.

2) By default cache store will process data in parallel on all nodes.
However if will not align data distribution with that of cassandra, and I
would say that implementing it will be infeasible. However, you could try to
see if there are ways to speed up loadCache by tuning Ignite and-or cache
configurations.

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: .NET ContinuousQuery lose cache entries

[ilya.kasnacheev]

Hello!

Do you have some kind of reproducer project? It's hard to say otherwise what
is happening in your case.

Regards,




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Java 11 and the future of Apache Ignite

[ilya.kasnacheev]

Hello!

This might be more suited to developers list. I will answers from user
perspective:

1) I guess that all LTS releases will be supported (which are every 6th
release?). Intermediate releases will receive some testing in releases that
they are released, but will not be a primary target.

2) Maybe if RedHat and IBM have success in promoting their offerings, there
will be official support or even migration of development process towards
that. It's to early to tell.

3) This was under assumption that JVM remains largely backwards
compatibility, which is what they're going away from. So you should not
expect that Ignite will run out of box on later releases than ones it was
developed for.

4) It is too early to say.

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: ignite .net index group definition

[ilya.kasnacheev]

Hello!

Unfortunately this looks like a genuine limitation of .Net's QuerySqlField.

You can try specifying those indexes with QueryEntities or CREATE INDEX
instead to have more control.

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Data modeling for segmenting a huge data set: precomputing vs real time computations

[ilya.kasnacheev]

Hello!

It's hard to say for sure which way to go since Apache Ignite offers a very
diverse range of tools.

Note that you can persist arbitrary data structures into Apache Ignite. You
are not in any sense limited to SQL. You can turn sparse row into a tiny
object. While keeping some SQL indexes.
You can also have a service deployed as cluster singleton which will just
keep hot data on-heap and update it in the real time.

1) Nulls take a few bytes. Ignite object store isn't column-aligned if
that's what you mean.
2) It is low since all data is stored in off-heap only by default. So it is
heavily optimized.
3) Yes, data have to be moved from off-heap. However, when using indexes,
they are applied directly from off-heap AFAIK. Why not do two queries when
in doubt? Indexes may help GROUP BY if values are inlined in index. GROUP BY
is especially efficient when done by affinity key.
4) I do not recommend having many small tables since having big table means
you can hold complex objects in cache and do in-place processing and
transformation on them.

My recommendation is to stick more to compute and less to SQL. Of course,
you can combine both when using AI.

Regards,





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite Cache metrics on K8s

[Maxim.Pudov]

Try to call CacheMetricsMXBean.enableMetrics(). 
See also
https://apacheignite.readme.io/docs/cache-metrics#section-using-jmx-bean



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is ID generator split brain compliant?

[Pavel Pereslegin]

Ankit,
I checked sequence and atomic on Ignite 2.6 with the following code:


try (Ignite ignite = Ignition.start("example-persistent-store.xml")) {
ignite.active(true);

IgniteAtomicSequence seq = ignite.atomicSequence("sequence", 0, true);
IgniteAtomicLong atomic = ignite.atomicLong("long", 0, true);

for (int i = 0; i < 100; i++) {
seq.incrementAndGet();
atomic.incrementAndGet();
}
}

try (Ignite ignite = Ignition.start("example-persistent-store.xml")) {
ignite.active(true);

IgniteAtomicSequence seq = ignite.atomicSequence("sequence", 0, true);
IgniteAtomicLong atomic = ignite.atomicLong("long", 0, true);

assert seq.get() == 1000 : seq.get();
assert atomic.get() == 100 : atomic.get();
}


Code works fine with enabled assertions (-ea) and cleaned persistence
directory. We have sequence value = 1000, atomic value = 100 after
node restart.
Could you tell us in more detail how you use "Ignite ID generator"?

чт, 27 сент. 2018 г. в 16:27, abatra :
>
> Hi Anton,
>
> With persistence enabled, if I reboot the nodes in the cluster, ID generator
> starts generating ID from 1 after the nodes are back up. My expectation is
> that that the latest generated ID(s) will be persisted across cluster reboot
> and the ID generator will start from the latest persisted ID.
>
> I am using the latest version of Ignite i.e. 2.6.0.
>
> -Ankit
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Is ID generator split brain compliant?

[abatra]

Hi Anton,

With persistence enabled, if I reboot the nodes in the cluster, ID generator
starts generating ID from 1 after the nodes are back up. My expectation is
that that the latest generated ID(s) will be persisted across cluster reboot
and the ID generator will start from the latest persisted ID.

I am using the latest version of Ignite i.e. 2.6.0.

-Ankit



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: java.lang.NullPointerException in GridDhtPartitionsExchangeFuture

[Ilya Kasnacheev]

Hello!

It's hard to answer these questions without thorough review of logs and 1.9
code, and I doubt anyone will volunteer to do that since 1.x branch does
not see any new development in Apache Ignite.

Regards,
-- 
Ilya Kasnacheev


ср, 26 сент. 2018 г. в 22:36, HEWA WIDANA GAMAGE, SUBASH <
subash.hewawidanagam...@fmr.com>:

> Hi Kasnacheev,
>
> Thank you very much for the response..
>
>
>
> We use v1.9, because 2.x uses mandatory ignite native persistence(local
> disk) along with durable memory management(RAM), and no option for a **java
> heap only** cache storage. We wanted to keep away from storing anything in
> disk. Hence using 1.9 Ignite.
>
> Can you please clarify “I believe that cache in question is no longer
> consistent between nodes on metadata level” ?
>
>
>
> 1.   Like what could have caused & under what conditions this
> inconsistent state between nodes on cache metadata occurred ?
>
>
>
> 2.   And does 2.x fixes the problem?( Is there a way you can suggest
> to reproduce this issue, so that we can know for sure 2.x fixes the problem)
>
>
>
> It appeared from our logs that the cache.put threads hangs forever after
> this error. Can that be possible with this?
>
>
>
>
>
>
>
> *From:* Ilya Kasnacheev [mailto:ilya.kasnach...@gmail.com]
> *Sent:* Wednesday, September 26, 2018 12:31 PM
> *To:* user@ignite.apache.org
> *Subject:* Re: java.lang.NullPointerException in
> GridDhtPartitionsExchangeFuture
>
>
> This email is from an external source - exercise caution regarding links
> and attachments.
>
> Hello!
>
>
>
> This is a bad error message. I believe that cache in question is no longer
> consistent between nodes on metadata level, and you can't fix that without
> full restart or at least dropping and recreating the cache.
>
> I don't think that you will get much support from community on 1.x since
> the focus has shifted to 2.x. Have you considered upgrading?
>
>
>
> Regards,
>
> --
>
> Ilya Kasnacheev
>
>
>
>
>
> ср, 26 сент. 2018 г. в 18:50, HEWA WIDANA GAMAGE, SUBASH <
> subash.hewawidanagam...@fmr.com>:
>
> This is the only single error from ignite happened after JVM startup.
> Looks like I only posted the exception stack trace. Here’s the message and
> from which thread it logged.
>
>
>
> level:ERROR
>
>
>
>  logger:
> org.apache.ignite.internal.processors.cache.GridCacheIoManager
>
>
>
>  message:Failed processing message
> [senderId=57ee6544-e0b3-45cc-bdb5-f3fd37d7db1e,
> msg=GridDhtPartitionsSingleMessage [parts=null, partCntrs=null,
> client=false, compress=false, super=GridDhtPartitionsAbstractMessage
> [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=13, minorTopVer=0], nodeId=57ee6544, evt=NODE_JOINED],
> lastVer=GridCacheVersion [topVer=0, time=0, order=1536996016410,
> nodeOrder=0], flags=0, super=GridCacheMessage [msgId=1, depInfo=null,
> err=null, skipPrepare=false, cacheId=0, cacheId=0
> [IM_GROUP=fsy-pi-dt-ssam ]
>
>
>
>  thread:sys-stripe-2-#3%null%
>
>
>
> *From:* Ilya Kasnacheev [mailto:ilya.kasnach...@gmail.com]
> *Sent:* Tuesday, September 25, 2018 11:11 AM
> *To:* user@ignite.apache.org
> *Subject:* Re: java.lang.NullPointerException in
> GridDhtPartitionsExchangeFuture
>
>
> This email is from an external source - exercise caution regarding links
> and attachments.
>
> Hello!
>
>
>
> It's hard to say without reviewing logs, but it seems that there's some
> inconsistency with regards to cache metadata on nodes.
>
>
>
> Regards,
>
> --
>
> Ilya Kasnacheev
>
>
>
>
>
> вт, 25 сент. 2018 г. в 0:13, HEWA WIDANA GAMAGE, SUBASH <
> subash.hewawidanagam...@fmr.com>:
>
> Hi all,
>
> We use Ignite 1.9.
>
>
>
> We could see this in our logs.  All we do is cache.get() , cache.put()
> operations. With this log being seen, is it possible for  cache.put or
> ignite.getOrCreateCache() method calling threads be blocked forever ?
> (unfortunately we couldn’t get a thread dump to prove that, but from
> application logs, it looks like it).
>
>
>
> java.lang.NullPointerException: null
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.updatePartitionSingleMap(GridDhtPartitionsExchangeFuture.java:1446)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processMessage(GridDhtPartitionsExchangeFuture.java:1199)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$100(GridDhtPartitionsExchangeFuture.java:86)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$1.apply(GridDhtPartitionsExchangeFuture.java:1167)
>
> at
> 

Re: How to Auto Increment ID column

[Maxim.Pudov]

It was already discussed  perviously

 
. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

How to Auto Increment ID column

[Malashree]

How to Auto Increment ID column in Apache Ignite



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Ignite getAll does not return values randomly

[KR Kumar]

Hi Guys - I have 10 node ignite cluster thats used for cache and some
distributed computing as well. Now the problem is when I do get all on the
cache, the cache entries returned are zero though while persisting, I get
success. Does WalMode background or CacheRebalance.NONE have any effect on
how I read the data.

Thanx and Regards,
KR Kumar



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IgniteCache.size() for different cache show the same number

[Stephen Darlington]

They’re different _types_ but you’ve given them both the same _name_.

Try something like:

 public *IgniteCache* comment() { 
   return igniteSpringBean.getOrCreateCache(“COMMENTS_CACHE"); 
   } 

   public *IgniteCache *reply() { 
   return igniteSpringBean.getOrCreateCache(“REPLY_CACHE"); 
   } 

Regards,
Stephen

> On 27 Sep 2018, at 09:33, kcheng.mvp  wrote:
> 
>public *IgniteCache* comment() { 
>return igniteSpringBean.getOrCreateCache(IgDef.DOCS); 
>} 
> 
>public *IgniteCache *reply() { 
>return igniteSpringBean.getOrCreateCache(IgDef.DOCS); 
>} 
> 
> 
> you can find the caches are totally different(key type and value type)
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite Web Agent

[wt]

just an fyi - the data loader for the web agent is very slow. I was getting
10 rows per second throughput and nobody on here could really provide any
assistance so we opted to write our own version of the tool that uses the
data streamer which is loading 15k rows per second with persisted storage
and 100k rows with no persistence. It also means the config file for ignite
is much cleaner with no cache specific or data source settings. 

I really wish the Ignite developers would have opted for a data streamer
load mechanism in the web consoles generated project. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite Web Agent

[Alexey Kuznetsov]

Hi, Daryl !

It seems that you were able to run Web Console from docker.
Did you managed to open it in browser?
If yes, there a link "Download agent" in footer, try to click it.
It should allow you to download a web agent that you can unpack and run
near to your cluster.

Let me know if it helps.

>
> --
Alexey Kuznetsov

Re: Unreasonable segmentation in Kubernetes on one node reboot

[Roman Guseinov]

Hi Artem,

I see quite small timeouts in your configurations. Could you try to increase
them and reproduce the issue?

I suggest to try the following values:
* networkTimeout = 15 000
* failureDetectionTimeout = 60 000
* joinTimeout = 120 000

Best Regards,
Roman



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IgniteCache.size() for different cache show the same number

[kcheng.mvp]

public *IgniteCache* comment() { 
return igniteSpringBean.getOrCreateCache(IgDef.DOCS); 
} 

public *IgniteCache *reply() { 
return igniteSpringBean.getOrCreateCache(IgDef.DOCS); 
} 


you can find the caches are totally different(key type and value type)



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IgniteCache.size() for different cache show the same number

[Mikael]

It's the same cache you are using for both data types so 10 is the 
correct return value, otherwise you need to create two different caches


Mikael


Den 2018-09-27 kl. 09:05, skrev kcheng.mvp:

here is my code

 public IgniteCache comment() {
 return igniteSpringBean.getOrCreateCache(IgDef.DOCS);
 }

 public IgniteCache reply() {
 return igniteSpringBean.getOrCreateCache(IgDef.DOCS);
 }


I put 10 records to  *IgniteCache *

but when I call  IgniteCache.size() it returns 10,
even I did not put any *Reply* to cache.

I suppose it should return 0, as I did not put any *Reply* to cache.

is this correct?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: ignite .net programatically access cluster config (not working)

[wt]

i am building a management tool that creates and manages caches and for this
i wanted to be able to dynamically detect the data regions on the cluster



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

IgniteCache.size() for different cache show the same number

[kcheng.mvp]

here is my code 

public IgniteCache comment() {
return igniteSpringBean.getOrCreateCache(IgDef.DOCS);
}

public IgniteCache reply() {
return igniteSpringBean.getOrCreateCache(IgDef.DOCS);
}


I put 10 records to  *IgniteCache *

but when I call  IgniteCache.size() it returns 10,
even I did not put any *Reply* to cache.

I suppose it should return 0, as I did not put any *Reply* to cache.

is this correct?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/