RE: 2.8.1 : CacheEntryProcessor for insert/update within Transaction supported ?

[VeenaMithare]

Hi Anton, 

Thanks for the reply . 

As per the definition of entry processor - it is used to execute updates on
entries on
the nodes that store it - currently this is not the case. It runs on the
client side as well. 



Will this issue not be resolved if the entry processor ran only on the
server nodes.

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

HI Alexey, 

Thank you for the response, 

The below are the details from our Linux env. test :

The log files from the test are attached here : 
http://apache-ignite-users.70518.x6.nabble.com/2-8-1-INFO-org-apache-ignite-spi-communication-tcp-TcpCommunicationSpi-Accepted-incoming-communicatin-tp33854p35247.html

Version we use is : 2.8.1

Start type :  a shell script that calls java

size of the server nodes cluster : 3 

The IDE reproducer that i used on my windows env is attached here  ( however
i dont use this on the linux env. On Linux I test against the actual server
cluster ): 
http://apache-ignite-users.70518.x6.nabble.com/2-8-1-INFO-org-apache-ignite-spi-communication-tcp-TcpCommunicationSpi-Accepted-incoming-communicatin-tp33854p35021.html

I intend to do some more checks as well in a day or two on this issue. Will
get back on these.

regards,
Veena










--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Hi all, 

Continuation of this post (
Page 1 :
http://apache-ignite-users.70518.x6.nabble.com/2-8-1-INFO-org-apache-ignite-spi-communication-tcp-TcpCommunicationSpi-Accepted-incoming-communicatin-td33854.html

Page 2 :
http://apache-ignite-users.70518.x6.nabble.com/2-8-1-INFO-org-apache-ignite-spi-communication-tcp-TcpCommunicationSpi-Accepted-incoming-communicatin-td33854i20.html#a35238
) . Created a new post , since the old post has got paginated for some
reason. 

I checked, this issue is still reproducible in the 2.10 version. 

I noticed that Alexey has created a issue for this : 

https://issues.apache.org/jira/browse/IGNITE-14104

Is there any workaround till this issue is resolved.

regards,
Veena.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Too many TCP discovery accepted incoming connections

[VeenaMithare]

Hi Ilya, 


I was wondering if it was a case of Node1 trying to connect to the old node
id of Node2 , but the Node2 has got a new node id since it was last
connected.  ( That is the issue we faced in that link shared ) . 

Maybe that is not what is happening here. 
Please ignore if so,

regards,
Veena.








--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Too many TCP discovery accepted incoming connections

[VeenaMithare]

Hi Ilya, 

Could the issue be as mentioned here : 

http://apache-ignite-users.70518.x6.nabble.com/2-8-1-INFO-org-apache-ignite-spi-communication-tcp-TcpCommunicationSpi-Accepted-incoming-communicatin-tp33854p35224.html

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

2.8.1 : CacheEntryProcessor for insert/update within Transaction supported ?

[VeenaMithare]

Hi Team, 

Is the functionality to use CacheEntryProcessor for insert/update within
Transaction supported ?

As mentioned in this post, insert of a record using cacheentryprocessor 
always fails with UnregisteredBinaryTypeException if the insert is done
within a transaction : 

http://apache-ignite-users.70518.x6.nabble.com/Right-Handling-of-UnRegisteredBinaryObjectTypeException-tp35412p35681.html

regards,
Veena.







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Right Handling of UnRegisteredBinaryObjectTypeException

[VeenaMithare]

Hello all, 

Please guide me on the right way to use cacheentryprocessor within a
transaction to update cache records . 

As shown in the example above, insert of a new record always throws
UnregisteredBinaryTypeException - unless a metadata is registered on the
client side before starting the transaction.

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Right Handling of UnRegisteredBinaryObjectTypeException

[VeenaMithare]

Hello, 

Did anyone get a chance to look at this ?

Also I noticed that I can skip the execution of the entry processor if it is
executing on the client side ( could pass a variable that identifies where
it runs ) . 

i.e. at the start of the entry processor put something like 

public Object process(MutableEntry
entry,
Object... arguments) throws EntryProcessorException {
if ( execution  is on a client node )
{
return; 
}


return ;
}

This avoids the time taken for the entry processor to execute on the client
side. 
It ensures that the entry processor runs only the serverside and updates the
desired records. 

Kindly guide if this approach is acceptable or will it have any other side
effects. 

regards,
Veena



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Right Handling of UnRegisteredBinaryObjectTypeException

[VeenaMithare]

Hi Anton, 

Did you get a chance to look at this ?

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Right Handling of UnRegisteredBinaryObjectTypeException

[VeenaMithare]

Hi Anton,

Please find a sample project with one of the
'unregisteredbinarytypeexception' .  The readme.txt is present in the
attached zip.

TestProject.zip
  
As mentioned in the readme.txt , the behaviour noticed is as below :

1. Run the UpdateClientNotWorking - insert of a record through entry
processor doesnt work.
2. Run the UpdateClientWorking - insert of a record through entry processor
works. This is because we do a dummy registration of metadata on the client
side before the entryprocessor is invoked within the transaction.

Questions are as below :
1. Why does the entry processor run in the client side ? As per the
definition of entry processor - it is used to execute updates on entries on
the nodes that store it . TestProject.zip
https://ignite.apache.org/docs/latest/distributed-computing/collocated-computations#entry-processor

2. Because of point 1, when we execute a transaction of say 1000 records, we
need to give a huge timeout value - since most of the time of the
transaction is spent in the client execution of the entry processor . Is
there any workaround for this ?

regards,
Veena.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite PME query

[VeenaMithare]

Hi Kamlesh, 

PME related questions have been answered in this forum earlier as well.
Maybe if you search for these, you might get some of your questions
answered.

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Right Handling of UnRegisteredBinaryObjectTypeException

[VeenaMithare]

Hi Anton, 

>>I read somewhere that this exception happens if registering metadata
happens
while the topology is locked.

Could you kindly help me understand this exception. 

Also what is the right way to handle any runtimeexception within the entry
processor ? Is it by wrapping  it with an EntryProcessorException and
throwing the same, 

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Right Handling of UnRegisteredBinaryObjectTypeException

[VeenaMithare]

Hi Anton ,

Thank you for the reply,

We sync tables/caches in from a source db. i.e. we read all the records of a
table and insert/update records into the table/cache in ignite.

When the sync application runs just for a single table - we dont see the
unregisteredbinarytypeexception .

When we run sync for say 10 tables - where update of the 10 tables happen in
parallel through threads , we see this exception for a few records on some
server nodes.

I am not sure if this issue is happening because of the update load or
because there are multiple tables getting synced at the same time.

Please note each of the 10 tables are independent of each other ( i.e. when
we update one table, it doesnt affect the update of another table )

I read somewhere that this exception happens if registering metadata happens
while the topology is locked. I am not sure what 'topology locked' means.
There is no baseline topology change while these updates are taking place.

regards,
Veena.  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Right Handling of UnRegisteredBinaryObjectTypeException

[VeenaMithare]

Hello Team

Our configuration is as follows :
Ignite Version : 2.8.1
Server nodes : 3
Native Persistence : yes

Cache Mode : Replicated
Write Sync Mode : FULL_SYNC

Updates/Inserts : Through entry processor .

No. of caches : ~ 30
No. of records per cache : ~30k


Our entry processor code looks something  like this :
class ProjectCacheEntryProcessor implements
CacheEntryProcessor {

public Object process(MutableEntry
entry,
Object... arguments) throws EntryProcessorException {
BinaryObject value = entry.getValue();
  BinaryObjectBuilder builder;
 
if (value == null ) {
builder = ignite.binary().builder(tablename);

} else {
builder =value.toBuilder();
   
}

  If ( some logic ){
Builder.setField(columname, )
  }

  entry.setValue(builder.build())

}

}


It is noticed that when we populate the caches for the first time  ( i.e.
sync data in from other db ) , where all the updates are happening in
parallel in separate threads ( around 10 tables at a time ) , update/insert
for some records show 'UnRegisteredBinaryObjectException' on some server
nodes . ( We dont get this exception if we sync any single cache alone )

The detailed message for this says :
class org.apache.ignite.internal.UnregisteredBinaryTypeException: Attempted
to update binary metadata inside a critical synchronization block (will be
automatically retried). This exception must not be wrapped to any other
exception class. If you encounter this exception outside of EntryProcessor,
please report to Apache Ignite dev-list. Debug info [typeId=-1816288802,
binaryMetadata=null, fut=MetadataUpdateResultFuture [key=SyncKey
[typeId=-1816288802, ver=3]]]

Because of that message , we had not caught this exception. But looks like
this causes the data not inserted on that node . Since this exception is not
caught , client doesn't receive any exception for the update/insert of the
record. This causes data inconsistency between nodes for the data of any
given replicated cache.

Kindly let us know the below :
1. What is the correct/recommended way to handle
UnregisteredBinaryTypeException
2. Considering that we are using FULL_SYNC mode, shouldnt this exception
have propagated to the client and prevent the record update rather than
cause data inconsistency between the server nodes for a REPLICATED cache.

What is the right way to handle any RunTimeExceptions ? Should we catch
these and propagate it to the clients ?

regards,
Veena.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : Continuous Query Initial query not returning any result sometimes and UnregisteredBinaryTypeExceptio

[VeenaMithare]

On further debugging, it looks like all the clients that had failed the
count, were executing their query on 
 the same server machinename002. 

When I looked at the server logs, it looks like that the records that were
missing were not present on this server when the query was executed but were
present on the other two servers. ( Kindly note that the cache is 
replicated one and all the records should ideally be present on all the 3
servers ). 

When I look at the exceptions logs, I see unregisteredbinarytypeexception
for these records : 
Attempted to update binary metadata inside a critical synchronization block
(will be automatically retried). This exception must not be wrapped to any
other exception class. If you encounter this exception outside of
EntryProcessor, please report to Apache Ignite dev-list.

Typically we dont do anything when we encounter this exception as per the
message. Please let me know if we need to do anything here .  

In this case, I suspect the reattempt of the entry processor did not work ?
and left the machinename002 out of sync with the other two servers ? How do
we know if the UnregisteredBinaryTypeException has resolved in the next
attempt ?









--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : Continuous Query Initial query not returning any result sometimes

[VeenaMithare]

Hi Andrei, 

Did you get a chance to look at my comments . 

Regarding doing the continuous query as per this example : 
https://github.com/gridgain/gridgain/blob/master/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheContinuousQueryExample.java

Our code is pretty much as shown in the example. I cannot use the
cache.query in a try with resources block since I want the continuous query
to live for the life time of the client. Hence the cursor is closed when the
client is stopped. 

The changed code is there in 
InitialQueryProject-1.zip

  

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Regarding Connection timed out observed during client startup

[VeenaMithare]

Hi Ilya, 

>>Generally a reducer node will need to connect to all map nodes while doing
SQL query.

>>You will not see this message if connection was already up.
*Here are examples of 'outgoing communication' when the count was incorrect
: 
*Example 1 : 
2021-01-15T17:04:35,905 INFO  o.a.i.s.c.t.TcpCommunicationSpi
[grid-nio-worker-tcp-comm-0-#60%Instance%]: Established outgoing
communication connection [locAddr=/a.b.c.153:59010,
rmtAddr=/a.b.c.199:47130]
2021-01-15T17:04:51,682 INFO  o.a.i.s.c.t.TcpCommunicationSpi
[grid-nio-worker-tcp-comm-1-#61%Instance%]: Established outgoing
communication connection [locAddr=/a.b.c.153:61374,
rmtAddr=/a.b.c.201:47130]
/2021-01-15T17:04:51,840 INFO  o.a.i.s.c.t.TcpCommunicationSpi
[grid-nio-worker-tcp-comm-2-#62%Instance%]: Established outgoing
communication connection [locAddr=/x.y.z.153:52558,
rmtAddr=machine003.cmc.local/x.y.z.202:47130]/


Example 2 :
2021-01-15T17:17:25,876 INFO  o.a.i.s.c.t.TcpCommunicationSpi
[grid-nio-worker-tcp-comm-0-#60%Instance%]: Established outgoing
communication connection [locAddr=/a.b.c.153:59786,
rmtAddr=/a.b.c.199:47130]
2021-01-15T17:17:41,720 INFO  o.a.i.s.c.t.TcpCommunicationSpi
[grid-nio-worker-tcp-comm-1-#61%Instance%]: Established outgoing
communication connection [locAddr=/a.b.c.153:62150,
rmtAddr=/a.b.c.201:47130]
/2021-01-15T17:17:41,878 INFO  o.a.i.s.c.t.TcpCommunicationSpi
[grid-nio-worker-tcp-comm-2-#62%Instance%]: Established outgoing
communication connection [locAddr=/x.y.z.153:53334,
rmtAddr=machine003.cmc.local/x.y.z.202:47130]/

*Here is an example of  'outgoing communication' when the count was correct
: 
*Example 3 :
2021-01-15T17:30:12,638 INFO  o.a.i.s.c.t.TcpCommunicationSpi
[grid-nio-worker-tcp-comm-0-#60%Instance%]: Established outgoing
communication connection [locAddr=/a.b.c.153:60534,
rmtAddr=/a.b.c.199:47130]
2021-01-15T17:30:28,403 INFO  o.a.i.s.c.t.TcpCommunicationSpi
[grid-nio-worker-tcp-comm-1-#61%Instance%]: Established outgoing
communication connection [locAddr=/a.b.c.153:62898,
rmtAddr=/a.b.c.201:47130]

1. We are on a 3 node cluster ( a.b.c.199, a.b.c.201, x.y.z.202  ).  When
the count is right it doesn't even try to make a connection to x.y.z.202 .
It is happy with the connections made to a.b.c.199, a.b.c.201.  

As mentioned in the cases where the count is incorrect the connection to
x.y.z.202 is made between cache.query and cursor.getall. 


2. The cache that is used in the query here is a REPLICATED cache. As per
this documentation here - the query on a REPLICATED CACHE would not be run
against multiple server nodes - 
https://apacheignite-sql.readme.io/docs/select
Would the reducer still be used ?

regards,
Veena.






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Regarding Connection timed out observed during client startup

[VeenaMithare]

Hi Ilya, 

>>I don't think it could have affected CQ initial query, since queries/cache
operations are supposed to block until they are successful, but not fail
silently.

I have another observation related to this
http://apache-ignite-users.70518.x6.nabble.com/2-8-1-Continuous-Query-Initial-query-not-returning-any-result-sometimes-td35167.html

On all the nodes where I got the wrong count :
Between the time it executes cache.query and cur.getall, I see this log :
2021-01-15T15:37:15,167 INFO  o.a.i.s.c.t.TcpCommunicationSpi
[grid-nio-worker-tcp-comm-2-#62%InstanceName%]: Established outgoing
communication connection [locAddr=/a.b.c.153:50546,
rmtAddr=machinename003.cmc.local/a.b.c.202:47130 ]


On the nodes where the count has been right, I dont see that line.

Why it is trying to connect to another server from the cluster between the
cache.query and cur.getall ?

>>The main factor is that you expect your transactions to finish
successfully. Setting low TxTimeoutOnPartitionMapExchange will lead to
aborting of all your transactions when PME happens. Why would you want that,
as opposed to the default when transactions finish and PME happens?

Okay. Let me try a few values on this.

regards,
Veena.






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : Continuous Query Initial query not returning any result sometimes

[VeenaMithare]

HI Andrei, 

Some more points :

1. Also the javadocs for the querycursor says : 
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/query/QueryCursor.html#getAll--

List getAll()
Gets all query results and stores them in the collection. Use this method
when you know in advance that query result is relatively small and will not
cause memory utilization issues.
/Since all the results will be fetched, all the resources will be closed
automatically after this call, e.g. there is no need to call close() method
in this case.
/

 We are closing the cursor as mentioned in my previous comment - but
according to this comment, if the cursor is not closed it shouldnt be a
issue since a invocation of the getall method will close it . 

Please let me know if we are missing something. 

2. Some observations-  On all the nodes where I got the wrong count :
Between the time it executes cache.query and cur.getall, I see this log : 
2021-01-15T15:37:15,167 INFO  o.a.i.s.c.t.TcpCommunicationSpi
[grid-nio-worker-tcp-comm-2-#62%InstanceName%]: Established outgoing
communication connection [locAddr=/a.b.c.153:50546,
rmtAddr=machinename003.cmc.local/a.b.c.202:47130 ]

Not sure why it is trying to connect to another server between the
cache.query and cur.getall

On the nodes where the count has been right, I dont see that line. 

3. Could this be linked to this issue here : 

http://apache-ignite-users.70518.x6.nabble.com/Regarding-Connection-timed-out-observed-during-client-startup-td35157.html









--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : Continuous Query Initial query not returning any result sometimes

[VeenaMithare]

Hi Andrei,

Also as I said it is a trimmed down  version of the actual code. In the
actual code, the cursor is taken into a instance variable and closed. 

Made modifications accordingly to the client class . Please find the same
attached,
InitialQueryProject-1.zip

  
regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Hi Alexey, 

Please find the debug logs attached. 

This log "Client creation failed" is present in server2.log

regards,
Veena.
serverlogs.zip
  




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Sure. Let me get back on this.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : Continuous Query Initial query not returning any result sometimes

[VeenaMithare]

Hi Andrei,
>>From your source code, I can see that the QueryCursor was created inside
startContinuousQuery. Then you return it from that method, but don't store
it in any variable:

>>continuousQueryHelper.startContinuousQuery(ignite,qry,testCache,subscribeParameters.callBackList,outputFieldsList);

I dont need to store the cursor that is returned from startContinuousQuery
because the cursor is read within the startContinuousQuery . 

Please find the code within startContinuousQuery in ContinuousQueryHelper
class . The cursor is stored in a variable and read through in the loop . 

// Create new continuous query.
QueryCursor> cur =
cache.query(qry);
LOGGER.info(
"PROJECT LISTENS: >>> Cache continuous query started
successfully. Start Initial Query...{}",
qry.getInitialQuery());

// Iterate through existing data.
// Please note there is a strange behaviour here. The iterator
returns a iterator of Cache.Entry
// However, each of the element of the iterator is actually not
Cache.Entry .
// Each element of the iterator is List of fields in a row. ( i.e.
it matches what is supposed to be returned by
// SQLFieldsQuery.query("query..").getAll();
List addedOrModifiedList = new ArrayList<>();
List removedList = new ArrayList<>();

int count=0;
for (Object e : cur.getAll()) {

BinaryObject
binaryObject=createBinaryObject((List)e,outputFieldsList,ignite,"TEST");
++count;
addedOrModifiedList.add(binaryObject);
LOGGER.debug("PROJECT LISTENS: Initial entry binaryObject : {}
", binaryObject);


}

I am not sure how it can be garbage collected before it is read. 

regards,
Veena.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Hi Alexey,

I am able to reproduce this on my linux env. by doing a kill -9 .  Please
find the logs from my linux env. attached. 
temp.log
  
Also note in our linux env. we have dual ip for our boxes - not sure if that
contributes to this issue. 

Also I do not see this log in the INFO mode in our client logs : 
Sending local node ID (bc6ae633-b643-44ba-afab-321dbf01aaf1 to newly
accepted session ...

I will try shutting down the client with ctrl+c on windows and check if that
helps.

regards,
Veena.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Unfortunately , will be difficult to try this - since we are working from
home and are connected to office network through citrix . 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Regarding Connection timed out observed during client startup

[VeenaMithare]

>>It's possible that you have more than one network/interface and some
combinations are causing problems. Please try to specify localHost property
on every node pointing to a current actual external IP address of the node.

This does seem to help the points 1,2,3,4. 
Do you think this could have affected this issue : 

http://apache-ignite-users.70518.x6.nabble.com/2-8-1-Continuous-Query-Initial-query-not-returning-any-result-sometimes-td35167.html

>>5. Yes, it will do exponential backout based on those values.
Can the connectTimeout be set up for client side as well ? Would you know
why the default value is 600,000. 

>>6. I'm not sure what is the optimal value, it depends on your env.
Thanks. My understanding is that if a partition map exchange is detected
while a transaction is going on , timeout the transaction . In this case
shouldnt the TxTimeoutOnPartitionMapExchange be kept as small as possible ?
What are the other factors to consider here. 






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Regarding Connection timed out observed during client startup

[VeenaMithare]

Thanks Ilya, 

I will check if adding localhost improves the issue mentioned in point 1, 2,
3 , 4.

Could you guide on point 5,6 ,

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : Continuous Query Initial query not returning any result sometimes

[VeenaMithare]

Hi Andrei,

Please find a trimmed down version of the code attached. 
Have a look at the ContinuousQueryHelper -> setInitialQuery to check how the
initial query has been coded.

regards,
Veena.
InitialQueryProject.zip

  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : Continuous Query Initial query not returning any result sometimes

[VeenaMithare]

HI Andrei, 

The initial query(sql query) is failing, not the continuous update. 

Also so far we have had no issues in the initial query.. ( for the past 1
year or so ). I can share a reproducer if that will help, 

regards,
Veena.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : Continuous Query Initial query not returning any result sometimes

[VeenaMithare]

Hi , 

It happened a couple of times on Friday ( 15th ) . Today I started a dummy
client with a continuous query against the same client multiple times
against the same cluster - and I am not able to reproduce it. This cache
contains 58 records , and I see all the 58 records being recorded in the
initial query . 

I have so far looked into the below : 
1. Baseline topology of the cluster on 15th - It looks okay . The topology
log shows servers as 3 all the time.i.e The cluster was not segmented.  
2. Another client which has a continuous query against the same cache -
seemed to fetch only 40 records as against 58 records on friday. 
3. I have checked the cache entry distribution for this cache yesterday /
today and the primary entries for the  58 records seem to be distributed as
17, 18, 23 on the 3 server nodes. However I dont know what the distribution
was on friday. 

How do I debug the below  :
2. Did the replication between the servers not work ? Is it possible that
one of the servers had the wrong value ? How do I debug this ? Please note
when I connect to each of the 3 servers using dbeaver, they all return the
correct row right now.( Since I got to know of this issue after 2 days, I am
not sure what was the row value at the time of this issue )

3. How do I know which server( out of the 3 servers ) was the client's
continuous query -> initial query run on 

regards,
Veena.






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : Continuous Query Initial query not returning any result sometimes

[VeenaMithare]

No . Its a SQL query.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

The reproducer code works fine on my home mac ( I dont see the 100s of logs)
- so not sure if the issue is because of 
a. Change in OS - at home I use mac and office it is windows/linux.
b. Change in the network ( between my home and office )

Any thoughts on how I can debug further  ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

2.8.1 : Continuous Query Initial query not returning any result sometimes

[VeenaMithare]

Hi Team,

Our env : 
Servers : 3
Cache Configuration : REPLICATED

In one of our clients, the continuous query - Initial query did not return
any result during client start whereas it was expected to return a single
row. This happened for about 3-4 times over a period of 2 hours. It did
return the right result after a couple of restarts. 

I am trying to debug what happened when it did not return the result . My
suspicions are as below :

1. Last week we changed the  client communication timeout as below as per
the discussion here ( 
http://apache-ignite-users.70518.x6.nabble.com/IgniteSpiOperationTimeoutException-Operation-timed-out-timeoutStrategy-ExponentialBackoffTimeoutStray-tp34196p34377.html
). Server was already on the below timeout values. 

 : 

 




Could it be possible that TcpCommunicationSpi timed out before the results
were seen on the client side  ?

2. Did the replication between the servers not work ? Is it possible that
one of the servers had the wrong value ? How do I debug this ? Please note
when I connect to each of the 3 servers using dbeaver, they all return the
correct row right now.( Since I got to know of this issue after 2 days, I am
not sure what was the row value at the time of this issue )

3. How do I know which server( out of the 3 servers ) was the client's
continuous query -> initial query run on ? 

How do I debug this further ?

regards,
Veena.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Regarding Connection timed out observed during client startup

[VeenaMithare]

Hello,

We see this behaviour in our client startup :
Client 1 -

Client1.txt
  
Server003 log -

SERVER3.TXT
  

1. The client join the cluster around (2021-01-14T16:16:29) in both
client1.log and server003.log. This started the partition map exchange . 
2. The client1.log shows BLOCKED SYSTEM CRITICAL THREAD : 
2021-01-14T16:16:44,472 ERROR o.a.i.i.u.t.G
[grid-timeout-worker-#21%InstanceName%]: Blocked system-critical thread has
been detected. This can lead to cluster-wide undefined behaviour
[workerName=partition-exchanger,
threadName=exchange-worker-#37%InstanceName%, blockedFor=14s]

After this , we see the connection timesout  to machinename003. 
021-01-14T16:16:45,017 WARN  o.a.i.s.c.t.TcpCommunicationSpi
[exchange-worker-#37%InstanceName%]: Connection timed out (will stop
attempts to perform the connect) [node=7b9d8c6f-814c-4cb6-9822-8fa3d7f79eb7,
connTimeoutStgy=ExponentialBackoffTimeoutStrategy [maxTimeout=1,
totalTimeout=15000, startNanos=14942834332684124, currTimeout=1],
failureDetectionTimeoutEnabled=false, timeout=9938, err=null,
addr=/m.n.o.202:47130]*



3. If I look in to the logs in machinename003, the partition map exchange
finished in 8 seconds. 


2021-01-14T16:16:29,671 INFO 
o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture [exchange-worker-#566]:
Exchange timings [startVer=AffinityTopologyVersion [topVer=47,
minorTopVer=0], resVer=AffinityTopologyVersion [topVer=47, minorTopVer=0],
stage="Waiting in exchange queue" (0 ms), stage="Exchange parameters
initialization" (0 ms), stage="Determine exchange type" (4 ms),
stage="Exchange done" (4 ms), stage="Total time" (8 ms)]

If so, why was the partition-exchanger blocked on the client ?

4. Inspite of showing connection timeout however, it manages to successfully
connect to machinename003. ( Please note that m.n.o.202, x.y.z.202 are ip
addresses of the same server machinename003 ). 

2021-01-14T16:16:45,026 INFO  o.a.i.s.c.t.TcpCommunicationSpi
[grid-nio-worker-tcp-comm-0-#22%InstanceName%]: Established outgoing
communication connection [locAddr=/a.b.c.21:53607,
rmtAddr=machinename003.cmc.local/x.y.z.202:47130]



Kindly guide us what happened here.. 
=
5. Also we have configured TcpCommunicationSpi timeouts as below as per the
recommendation given in : 

http://apache-ignite-users.70518.x6.nabble.com/IgniteSpiOperationTimeoutException-Operation-timed-out-timeoutStrategy-ExponentialBackoffTimeoutStray-tp34196p34377.html




  
Is the timeout observed because of this setting ?
===

6. Our TxTimeoutOnPartitionMapExchange is 1 second


What is the ideal TxTimeoutOnPartitionMapExchange value that should be given
should it be something like 50 milliseconds ?

=
A similar log captured during client 2 startup attached as well. 
Client2.txt
  

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Hi Ilya,

Please find the video recording of the issue I face.

https://drive.google.com/file/d/1fDqxmgjiL3o_eenHoUosdiYlMMBJ4xVj/view?usp=sharing

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Client stuck on startup

[VeenaMithare]

Let me debug this a bit more. Also I will try and capture another set of
thread dumps when this occurs again. This issue occurs on my windows machine
sometimes - doesnt happen all the time. I have not seen this on my linux
env. yet.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Client stuck on startup

[VeenaMithare]

I dont think the server dumps were taken after the client disconnected . They
were taken when the client was in the stuck  state. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Kindly confirm the below : 

a. Are you on :2.8.1
b. Is this cod uncommented : 
 ignite.active(true);
addPersistentCacheConfiguration(
ignite);
c. using both server and client on the reproducer?

I get this on all env. ( even in our linux env. )



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Client stuck on startup

[VeenaMithare]

Hello, 

Thread dump from the server nodes has been provided in the original post, 

regards,
Veena



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Client stuck on startup

[VeenaMithare]

a. If you see the thread dump, it shows these locked synchronizers 
   Locked ownable synchronizers:
- <0x0006da73f540> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006da73f690> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006da84add8> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006da84aee0> (a


If the shutdown is clean, we dont see these locked synchronizers .

Looks like it has registered some state with the server before the shutdown
hook was invoked. Also this registered state is now preventing further
restarts . 

b. Even though the log says 'invoking shutdown hook' , the jvm does not
shutdown - it is blocked on starting the ignite.

I will try and see if I can debug more. 




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Client stuck on startup

[VeenaMithare]

Okay, thanks Ilya .  After it gets this issue, the app doesnt startup till I
restart my server nodes. 

Is there anyway I can ensure clean shutdown when I face issues like this  ?
It looks like some ignite state is not cleaned up . 

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

This issue is also observed if two different clients exist on the same box. 
Steps to reproduce  : 

1. Both the clients are not running
2. Start the client1
3. Stop the client 1 and start client 2 . 
4. The huge set of logs are visible on client 2 logs. 

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Explicit Communication / Discovery port configuration on the client side needed ?

[VeenaMithare]

Hi , 

1. If we deploy more than one client on same machine - should we explicitly
configure communication and discovery port for the client i.e. should we do
the below : 

   





2. The localPortRange for communicationport and discovery port is 100 . Why
is this range needed ?

3. Can the port range be reduced to 5 on the client side configuration ?

regards,
Veena




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Client stuck on startup

[VeenaMithare]

>>Try to use some realistic value such as 64.

I think it hadnt worked earlier - I will try again. client-log.txt
  

Please find the client log file attached. I am seeing 'invoking shutdown
hook..' as an INFO in the startup logs. 
Is that expected ?

regards,
Veena.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Client stuck on startup

[VeenaMithare]

This client is a sync which syncs around 1 million records from our old db to
ignite . When I start the application, it runs the initial sync and I need a
higher striped pool.  I think the client sees starvation in striped thread
pool  when I give a small thread pool. ( I dont have the logs for the lower
thread pool now. ). 

Is the striped pool size linked to client startup issue ?





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Client stuck on startup

[VeenaMithare]

HI , 

One of the clients on my windows box sometimes gets stuck up on startup . I 
see this in the thread dump of the client. 

a. Not sure what those locked synchronizers are  

"main" #1 prio=5 os_prio=0 tid=0x02c9c800 nid=0x4404 waiting on
condition [0x02c8e000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0006fda12d20> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at
org.apache.ignite.spi.discovery.tcp.ClientImpl.spiStart(ClientImpl.java:312)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2099)
at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:299)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:943)
at
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1960)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1276)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2045)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1703)
- locked <0x0006db5ca060> (a
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1117)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:637)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:563)
at org.apache.ignite.Ignition.start(Ignition.java:321)
at
com.company.prophet.projectname.dao.projectnameClientUtil.startIgnite(projectnameClientUtil.java:168)
at
com.company.prophet.projectname.dao.projectnameClientUtil.startIgniteForSingleOrBlueGreen(projectnameClientUtil.java:109)
at
com.company.prophet.projectname.dao.projectnameClientUtil.startIgnite(projectnameClientUtil.java:84)
at
com.company.prophet.projectname.dao.projectnameClient.start(projectnameClient.java:126)
at
com.company.prophet.projectname.sync.main.Sync.startIgnite(Sync.java:264)
at com.company.prophet.projectname.sync.main.Sync.start(Sync.java:133)
at
com.company.prophet.milliways.shared.common.launcher.MilliwaysGenericLauncher.run(MilliwaysGenericLauncher.java:69)
at
com.company.prophet.milliways.shared.common.launcher.MilliwaysGenericLauncher$1.run(MilliwaysGenericLauncher.java:150)
at
com.company.prophet.milliways.shared.common.launcher.MilliwaysGenericLauncher.initService(MilliwaysGenericLauncher.java:120)
at com.company.prophet.projectname.sync.main.Sync.main(Sync.java:108)

   Locked ownable synchronizers:
- <0x0006da73f540> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006da73f690> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006da84add8> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006da84aee0> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006da8825d8> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006da8c1198> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006da912440> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006da95eac0> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006da95eb98> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006daa93360> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006daa93470> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006dab3b070> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006dab3b0c0> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006daba1990> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006daba19e0> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006dabdb580> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006dabdb628> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x0006dac3df68> (a

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Hi Ilya,

Did you get a chance to look in to this ..?

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Hi Ilya,

Please find the reproducer project attached. 

Steps to reproduce : 
1. Run the server.java 
2. Run the ListenerClient.java. Wait till it starts fully.
3. Stop and start the client again ( within the failuredetectiontimeout of
10 secs. )

You will see these logs on the console.

Another thing I noticed today is that , the logs seem to be connected to
this code in Server.java
ignite.active(true);
addPersistentCacheConfiguration(
ignite);

If I comment out the above lines and try the steps above, I dont get those
logs.

regards,
Veena. ClientConnectProject.zip

  







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Regarding Partition Map exchange Triggers

[VeenaMithare]

Hi Ilya,

thank you !

regards,
Veena



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Regarding Partition Map exchange Triggers

[VeenaMithare]

>>Looking at your stack traces it seems that the cache you accessed in the
SQL query was stopped right before the new client node is joined.
>>All topology events are processed one-by-one, so the initial blocking time
was caused by cache stop PME rather than client node join.


What is the guidance regarding cleanly shutting down an client or server (
for deployment etc. )? Is it a good practice to call cache.close when a
client or a server node is being shutdown ? 

Cache closed but not destroyed means it can be still accessed by other nodes
by calling ignite.cache - what is the purpose of calling PME when
cache.close is invoked ?

Is there any difference in calling cache.close on the client( non baselined
node ) or server( baselined node ) node ?

regards,
Veena.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Regarding Partition Map exchange Triggers

[VeenaMithare]

HI Pavel,

Thank you for the reply. 

>>> *for the end user , is this invoked when we do ignite.getOrCreate( xx )
and ignite.cache(xx )*

>>Yes.


getOrCreateCache would create a cache if doesnt exist . I would guess this
would have the effect of starting the cache if doesnt exist . And I think
this would start the PME.

And ignite.cache would only get an instance of the existing cache. In this
case, I would imagine the cache already exists and a reference to the cache
is returned. I would have thought this WOULD NOT have the effect of starting
the cache and hence should not start the PME. 

Please guide. Below is the documentation of getOrCreateCache and
ignite.cache from the javadocs.

getOrCreateCache(CacheConfiguration cacheCfg)
Gets existing cache with the given name or creates new one with the given
configuration.

 IgniteCache cache(String name)
  throws javax.cache.CacheException
Gets an instance of IgniteCache API. IgniteCache is a fully-compatible
implementation of JCache (JSR 107) specification.

regards,
Veena.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Regarding Partition Map exchange Triggers

[VeenaMithare]

Hi ,


I can see the triggers for PME initiation here :
https://cwiki.apache.org/confluence/display/IGNITE/%28Partition+Map%29+Exchange+-+under+the+hood

Triggers
Events which causes exchange

Topology events:

Node Join (EVT_NODE_JOINED) - new node discovered and joined topology
(exchange is done after a node is included into the ring). This event
doesn't trigger the PME if a thick client connects the cluster and an Ignite
version is 2.8 or later.


--> *This means in ignite 2.8 or higher, this is triggered only if nodes
that participate in the baseline topology are added ?*


Node Left (EVT_NODE_LEFT) - correct shutdown with call ignite.close. This
event doesn't trigger the PME in Ignite 2.8 and later versions if a node
belonging to an existing baseline topology leaves.

--> *This means this is not triggered at all 2.8.1 or higher if shutdown
cleanly ? i.e. if this is called : Ignition.stop(false) *


Node Failed (EVT_NODE_FAILED) - detected unresponsive node, probably crashed
and is considered failed

--> *This means this is  triggered at all 2.8.1 or higher for baseline nodes
or any thick client node ?*

Custom events:

Activation / Deactivation / Baseline topology set - ChangeGlobalStateMessage
Dynamic cache start / Dynamic cache stop - DynamicCacheChangeBatch

--> *for the end user , is this invoked when we do ignite.getOrCreate( xx )
and ignite.cache(xx )*


Snapshot create / restore - SnapshotDiscoveryMessage
Global WAL enable / disable - WalStateAbstractMessage
Late affinity assignment - CacheAffinityChangeMessage


regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Unable to free disk space after keys are removed in ignite 2.7.0

[VeenaMithare]

Thank you .. !



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Unable to free disk space after keys are removed in ignite 2.7.0

[VeenaMithare]

Hi Ilya,

I have a question related to this : 
>>Then, WAL will usually only grow and db/ will grow too, they will not
shrink from the deletion of data from caches. Dropping a cache will free
space.

>>This is tolerable because modern server instances are usually
single-purpose. You expect an instance's storage to be used for Ignite data
only and the amount of data is expected to be stable not grow monotonously.

We store audit data for our cache modifications. And audit data will grow
monotonously over a period of time. What is the best way of housekeeping
audit records ?

regards,
Veena.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Regarding Partition Map exchange Triggers

[VeenaMithare]

Hi ,


I can see the triggers for PME initiation here :
https://cwiki.apache.org/confluence/display/IGNITE/%28Partition+Map%29+Exchange+-+under+the+hood

Triggers
Events which causes exchange

Topology events:

Node Join (EVT_NODE_JOINED) - new node discovered and joined topology
(exchange is done after a node is included into the ring). This event
doesn't trigger the PME if a thick client connects the cluster and an Ignite
version is 2.8 or later.
*/-- This means in ignite 2.8 or higher, this is triggered only if nodes
that participate in the baseline topology are added ?/*
Node Left (EVT_NODE_LEFT) - correct shutdown with call ignite.close. This
event doesn't trigger the PME in Ignite 2.8 and later versions if a node
belonging to an existing baseline topology leaves.
*/-- This means this is not triggered at all 2.8.1 or higher if shutdown
cleanly ? i.e. if this is called : Ignition.stop(false) /*
Node Failed (EVT_NODE_FAILED) - detected unresponsive node, probably crashed
and is considered failed
*/-- This means this is  triggered at all 2.8.1 or higher for baseline nodes
or any thick client node ?/*

Custom events:

Activation / Deactivation / Baseline topology set - ChangeGlobalStateMessage
Dynamic cache start / Dynamic cache stop - DynamicCacheChangeBatch
*/ for the end user , is this invoked when we do ignite.getOrCreate( xx )
and ignite.cache(xx )/*

Snapshot create / restore - SnapshotDiscoveryMessage
Global WAL enable / disable - WalStateAbstractMessage
Late affinity assignment - CacheAffinityChangeMessage


regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Regarding partition map exchange triggers

[VeenaMithare]

Hi , 


I can see the triggers for PME initiation here : 
https://cwiki.apache.org/confluence/display/IGNITE/%28Partition+Map%29+Exchange+-+under+the+hood

Triggers
Events which causes exchange

Topology events:

Node Join (EVT_NODE_JOINED) - new node discovered and joined topology
(exchange is done after a node is included into the ring). This event
doesn't trigger the PME if a thick client connects the cluster and an Ignite
version is 2.8 or later. 
/*-- This means in ignite 2.8 or higher, this is triggered only if nodes
that participate in the baseline topology are added ?*/
Node Left (EVT_NODE_LEFT) - correct shutdown with call ignite.close. This
event doesn't trigger the PME in Ignite 2.8 and later versions if a node
belonging to an existing baseline topology leaves.
/*-- This means this is not triggered at all 2.8.1 or higher if shutdown
cleanly ? i.e. if this is called : Ignition.stop(false) */
Node Failed (EVT_NODE_FAILED) - detected unresponsive node, probably crashed
and is considered failed
/*-- This means this is  triggered at all 2.8.1 or higher for baseline nodes
or any thick client node ?*/

Custom events:

Activation / Deactivation / Baseline topology set - ChangeGlobalStateMessage 
Dynamic cache start / Dynamic cache stop - DynamicCacheChangeBatch
/* for the end user , is this invoked when we do ignite.getOrCreate( xx )
and ignite.cache(xx )*/

Snapshot create / restore - SnapshotDiscoveryMessage
Global WAL enable / disable - WalStateAbstractMessage
Late affinity assignment - CacheAffinityChangeMessage


regards,
Veena.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Regarding partition map exchange triggers

[VeenaMithare]

Hi , 


I can see the triggers for PME initiation here : 
https://cwiki.apache.org/confluence/display/IGNITE/%28Partition+Map%29+Exchange+-+under+the+hood

Triggers
Events which causes exchange

Topology events:

Node Join (EVT_NODE_JOINED) - new node discovered and joined topology
(exchange is done after a node is included into the ring). This event
doesn't trigger the PME if a thick client connects the cluster and an Ignite
version is 2.8 or later. 
/*-- This means in ignite 2.8 or higher, this is triggered only if nodes
that participate in the baseline topology are added ?*/
Node Left (EVT_NODE_LEFT) - correct shutdown with call ignite.close. This
event doesn't trigger the PME in Ignite 2.8 and later versions if a node
belonging to an existing baseline topology leaves.
/*-- This means this is not triggered at all 2.8.1 or higher if shutdown
cleanly ? i.e. if this is called : Ignition.stop(false) */
Node Failed (EVT_NODE_FAILED) - detected unresponsive node, probably crashed
and is considered failed
/*-- This means this is  triggered at all 2.8.1 or higher for baseline nodes
or any thick client node ?*/

Custom events:

Activation / Deactivation / Baseline topology set - ChangeGlobalStateMessage 
Dynamic cache start / Dynamic cache stop - DynamicCacheChangeBatch
/* for the end user , is this invoked when we do ignite.getOrCreate( xx )
and ignite.cache(xx )*/

Snapshot create / restore - SnapshotDiscoveryMessage
Global WAL enable / disable - WalStateAbstractMessage
Late affinity assignment - CacheAffinityChangeMessage


regards,
Veena.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Regarding long running transaction

[VeenaMithare]

Hi , 

I see this in my transaction log : 
2020-12-10T07:32:59,525 WARN  o.a.i.i.diagnostic
[grid-timeout-worker-#1035%config-sync-GREEN-igniteclient-SINGLE%]: First 10
long running transactions [total=1]
2020-12-10T07:32:59,525 WARN  o.a.i.i.diagnostic
[grid-timeout-worker-#1035%config-sync-GREEN-igniteclient-SINGLE%]: >>>
Transaction [startTime=07:31:22.861, curTime=07:32:59.392, systemTime=0,
userTime=96531, tx=GridNearTxLocal [mappings=IgniteTxMappingsImpl [],
nearLocallyMapped=false, colocatedLocallyMapped=false, needCheckBackup=null,
hasRemoteLocks=false, trackTimeout=true, systemTime=0, systemStartTime=0,
prepareStartTime=0, prepareTime=0, commitOrRollbackStartTime=0,
commitOrRollbackTime=0,
txDumpsThrottling=o.a.i.i.processors.cache.transactions.IgniteTxManager$TxDumpsThrottling@f46affc,
lb=null, mvccOp=null, qryId=-1, crdVer=0, thread=, mappings=IgniteTxMappingsImpl [], super=GridDhtTxLocalAdapter
[nearOnOriginatingNode=false, nearNodes=KeySetView [], dhtNodes=KeySetView
[], explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null,
sndTransformedVals=false, depEnabled=false, txState=IgniteTxStateImpl
[activeCacheIds=[], recovery=null, mvccEnabled=null, mvccCachingCacheIds=[],
txMap=EmptySet []], super=IgniteTxAdapter [xidVer=GridCacheVersion
[topVer=218929849, order=1607454902519, nodeOrder=6], writeVer=null,
implicit=false, loc=true, threadId=39198, startTime=1607585482861,
nodeId=e25ecc8b-66e4-4856-9da5-5eebeec67a45, startVer=GridCacheVersion
[topVer=218929849, order=1607454902519, nodeOrder=6], endVer=null,
isolation=READ_COMMITTED, concurrency=PESSIMISTIC, timeout=90,
sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE,
invalidParts=null, state=ACTIVE, timedOut=false,
topVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0],
mvccSnapshot=null, skipCompletedVers=false, parentTx=null, duration=96531ms,
onePhaseCommit=false], size=0

How do I know the cause for this long running transaction ?

I see this in the log above : thread=.
What does this mean. 

I am updating around 50k records in a transaction and have given the tx
timeout as 120 secs. 
What is the threshold for ignite to decide that a transaction is long
running ?I can see that this is showing a usertime of 96531. 

I dont see any other WARN or ERROR messages after this log , ( there is one
more warn message about 'dumping the stack trace of near node that started
this transaction' , but that is not helpful )

regards,
Veena.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy

[VeenaMithare]

Thank you for the suggestions.

I will try the timeout settings. It looks like the server has connected to
the client on the new client id using the discovery SPI but it is trying to
send the cache update to the old client id using the communication SPI.

What impact does the IgniteAsyncCallback annotation on the remote filter
have ? What are the pros and cons of having this annotation on the remote
filter ?

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

Hi Ilya,

This is easy to reproduce. Have a server node and a client node in a
cluster. Stop and start the client immediately so that the start happens
within the failure detection timeout ( 10 sec typically ). You will see
these messages in the client log as it is starting up the second time.

Let me know if you still need a reproducer.

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy

[VeenaMithare]

Hi Ilya, The node communication issue is because one of the node is being
restarted - and not due to network failure . The original issue is as below
: Our setup :Servers - 3 node clusterReader clients : wait for an update on
an entry of a cache ( around 20 ofthem )Writer Client : 1If one of the
reader client restarts while the writer is writing into theentry of the
cache , the server attempts to send the update to the failedclient's local
listener . It keeps attempting to communicate with the failedclient (
client's continous query local listener ? ) till it timesout as
perconnTimeoutStrategy=ExponentialBackoffTimeoutStrategy . ( Please find
thesnippet of the exception below. The complete log is attached as
anattachment ) This delays the completion of the transaction that was
startedby the writer client.Is there any way the writer client could
complete the transaction withoutgetting impacted by the reader client
restarts ?regards,Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy

[VeenaMithare]

Hi Ilya, 

The node communication issue is because one of the node is being restarted -
and not due to network failure . The original issue is as below : 

Our setup :
Servers - 3 node cluster
Reader clients : wait for an update on an entry of a cache ( around 20 of
them )
Writer Client : 1

If one of the reader client restarts while the writer is writing into the
entry of the cache , the server attempts to send the update to the failed
client's local listener . It keeps attempting to communicate with the failed
client ( client's continous query local listener ? ) till it timesout as per
connTimeoutStrategy=ExponentialBackoffTimeoutStrategy . ( Please find the
snippet of the exception below. The complete log is attached as an
attachment ) This delays the completion of the transaction that was started
by the writer client.

Is there any way the writer client could complete the transaction without
getting impacted by the reader client restarts ?

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Continuous query not transactional ?

[VeenaMithare]

Hi Ilya,

That is what I assume too, could someone from the developers community help
confirm/comment on  this ?

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Continuous query not transactional ?

[VeenaMithare]

Hi ,

This is in continuation of the below statement on this post : 
http://apache-ignite-users.70518.x6.nabble.com/Lag-before-records-are-visible-after-transaction-commit-tp33787p33861.html

>>Continuous Query itself is not transactional and it looks like it can't be
used for this at the moment. So, it gets notification before other entries
were committed.

Does this mean we could get dirty reads as updates in continuous query ?
i.e. for eg if the code is as below:
1. Start transaction  
2. update records of cache a
3. update records of cache b
4. update records for cache c
5. commit

if update of cache a succeeds , but update of cache b fails, will the local
listener for continuous query for 'cache a' get an update ?

regards,
Veena.


regards
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Lag before records are visible after transaction commit

[VeenaMithare]

Hi , 

>>Continuous Query itself is not transactional and it looks like it can't be
used for this at the moment. So, it gets notification before other entries
were committed.

Does this mean we could get dirty reads as updates in continuous query ?
i.e. for eg if the code is as below:
1. Start transaction  
2. update records of cache a
3. update records of cache b 
4. update records for cache c
5. commit

if update of cache a succeeds , but update of cache b fails, will the local
listener for continuous query for 'cache a' get an update ?

regards,
Veena.


regards
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

[VeenaMithare]

Thanks Ilya,

>>I don't think so. Node can reconnect to a *different* cluster, or it may
be disconnected for a while and return to the same cluster. For client node,
segmented/disconnected difference is not relevant. Neither it is for server
node, in fact.

In  that case, we will have the same handler for segmented and reconnected
events. The handler will restart ignite and deploy any continuous queries as
needed.
Thank you for this guidance.

Does the server or the client need to have a handler for any other 'cluster
specific' events ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

[VeenaMithare]

Thanks Ilya,

>>I don't think so. Node can reconnect to a *different* cluster, or it may
be disconnected for a while and return to the same cluster. For client node,
segmented/disconnected difference is not relevant. Neither it is for server
node, in fact.

In  that case, we will have the same handler for segmented and reconnected
events. The handler will restart ignite and deploy any continuous queries as
needed.
Thank you for this guidance.

Does the server or the client need to have a handler for any other 'cluster
specific' events ?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy

[VeenaMithare]

Hi Team,

Kindly awaiting a reply on this one,

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

[VeenaMithare]

Hi Team,


In terms of the client, do we need to have the below 2 EVENT handlers? :
1. EVT_NODE_SEGMENTED - to restart the node and deploy and continuous
queries if needed.
2. EVT_CLIENT_NODE_RECONNECTED - to just deploy the continuous queries ( no
restart of ignite needed, since the node is already connected )

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

IgniteSpiOperationTimeoutException: Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy

[VeenaMithare]

Hi , 

Our setup : 
Servers - 3 node cluster
Reader clients : wait for an update on an entry of a cache ( around 20 of
them )
Writer Client : 1

If one of the reader client restarts while the writer is writing into the
entry of the cache , the server attempts to send the update to the failed
client's local listener . It keeps attempting to communicate with the failed
client ( client's continous query local listener ? ) till it timesout as per
connTimeoutStrategy=ExponentialBackoffTimeoutStrategy . ( Please find the
snippet of the exception below. The complete log is attached as an
attachment ) This delays the completion of the transaction that was started
by the writer client. 

Is there any way the writer client could complete the transaction without
getting impacted by the reader client restarts ? 





2020-10-08 14:35:21,465 [sys-stripe-26-#27] WARN 
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Handshake
timed out (will stop attempts to perform the handshake)
[node=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b,
connTimeoutStrategy=ExponentialBackoffTimeoutStrategy [maxTimeout=60,
totalTimeout=3, startNanos=223772180706400, currTimeout=60],
err=Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=60, totalTimeout=3, startNanos=223772180706400,
currTimeout=60]], addr=MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103,
failureDetectionTimeoutEnabled=true, timeout=0]
2020-10-08 14:35:21,465 [sys-stripe-26-#27] ERROR
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Failed to
send message to remote node [node=TcpDiscoveryNode
[id=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b,
consistentId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, addrs=ArrayList
[0:0:0:0:0:0:0:1, 1.2.3.4, 127.0.0.1], sockAddrs=HashSet
[MACHINENAME.COMPANY.LOCAL/1.2.3.4:0, /0:0:0:0:0:0:0:1:0, /127.0.0.1:0],
discPort=0, order=12, intOrder=8, lastExchangeTime=1602163619453, loc=false,
ver=2.8.1#20200521-sha1:86422096, isClient=true], msg=GridIoMessage [plc=2,
topic=T4 [topic=TOPIC_CACHE, id1=94370aa1-e970-37ae-9471-fd583d923522,
id2=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b, id3=0], topicOrd=-1, ordered=true,
timeout=0, skipOnTimeout=true, msg=GridContinuousMessage
[type=MSG_EVT_NOTIFICATION, routineId=7dac3ff4-3460-4dc4-8324-a1ebe4561854,
data=null, futId=null]]]
org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node
still alive?). Make sure that each ComputeTask and cache Transaction has a
timeout set in order to prevent parties from waiting forever in case of
network issues [nodeId=3311a67b-bfcb-41af-8c09-b2e8f2fbde9b,
addrs=[/127.0.0.1:47103, /0:0:0:0:0:0:0:1:47103,
MACHINENAME.COMPANY.LOCAL/1.2.3.4:47103]]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3698)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3198)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3078)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2918)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2877)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2035)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:2280)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1963)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1934)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1916)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:1323)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:1260)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:1045)
~[ignite-core-2.8.1.jar:2.8.1]
at

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

[VeenaMithare]

Hi Anton,

Thank you for the reply .

>>I can confirm that there is no SEGMENTED event thrown on 2.8.1.

I guess you meant that there is no SEGMENTED event thrown  in 2.8.1 client 
when the servers are brought down and started again. Because SEGMENTED event
could be thrown in 2.8.1 in other scenarios as well I guess, like a long GC
pause .

I have captured the logs for 2.7.6 client , where it receives a SEGMENTED
event. On closer look, it looks like the client's attempt to reconnect
failed in this case with 'Authentication Failure' and this has caused the
SEGMENTED event. 

Check this part of the log in the attached log : 
2020-10-08 09:59:33,327
[tcp-client-disco-msg-worker-#4%ProjectClient-null-igniteclient-SINGLE%]
ERROR
com.company.projectname.project.common.plugin.discoveryspi.CustomTcpDiscoverySpi
[] - Failed to reconnect, segment local node.
org.apache.ignite.spi.IgniteSpiException: Authentication failed
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:1946)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1888)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:304)
~[ignite-core-2.7.6.jar:2.7.6]
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
~[ignite-core-2.7.6.jar:2.7.6]
2020-10-08 09:59:33,328
[tcp-client-disco-msg-worker-#4%ProjectClient-null-igniteclient-SINGLE%]
DEBUG 

In 2.8.1, we have changed the security plugin , hence looks like the client
is able to reconnect successfully. 

In terms of the client, do we need to have the below 2 EVENT handlers? :
1. EVT_NODE_SEGMENTED - to restart the node and deploy and continuous
queries if needed.
2. EVT_CLIENT_NODE_RECONNECTED - to just deploy the continuous queries ( no
restart of node needed, since the node is already connected )

SegmentedClientLogs.txt

  

regards,
Veena






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

[VeenaMithare]

Hi Anton,

Thank you for the reply .

>>I can confirm that there is no SEGMENTED event thrown on 2.8.1.

I guess you meant that there is no SEGMENTED event thrown  in 2.8.1 client 
when the servers are brought down and started again. Because SEGMENTED event
could be thrown in 2.8.1 in other scenarios as well I guess, like a long GC
pause .

I have captured the logs for 2.7.6 client , where it receives a SEGMENTED
event. On closer look, it looks like the client's attempt to reconnect
failed in this case with 'Authentication Failure' and this has caused the
SEGMENTED event. 

Check this part of the log in the attached log : 
2020-10-08 09:59:33,327
[tcp-client-disco-msg-worker-#4%ProjectClient-null-igniteclient-SINGLE%]
ERROR
com.company.projectname.project.common.plugin.discoveryspi.CustomTcpDiscoverySpi
[] - Failed to reconnect, segment local node.
org.apache.ignite.spi.IgniteSpiException: Authentication failed
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.checkFailedError(TcpDiscoverySpi.java:1946)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1888)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
~[ignite-core-2.7.6.jar:2.7.6]
at
org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:304)
~[ignite-core-2.7.6.jar:2.7.6]
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
~[ignite-core-2.7.6.jar:2.7.6]
2020-10-08 09:59:33,328
[tcp-client-disco-msg-worker-#4%ProjectClient-null-igniteclient-SINGLE%]
DEBUG 

In 2.8.1, we have changed the security plugin , hence looks like the client
is able to reconnect successfully. 

In terms of the client, do we need to have the below 2 EVENT handlers? :
1. EVT_NODE_SEGMENTED - to restart the node and deploy and continuous
queries if needed.
2. EVT_CLIENT_NODE_RECONNECTED - to just deploy the continuous queries ( no
restart of node needed, since the node is already connected )

2.txt   


regards,
Veena






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

[VeenaMithare]

Hi ,

Kindly awaiting a reply on this ,

regards,
Veena



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite thread pool configuration

[VeenaMithare]

Thanks Denis,

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite thread pool configuration

[VeenaMithare]

Thanks Denis,

I am thinking of setting the below thread pools as this on both client and
server since we dont use data streamer, IGFS or Peer Class loading: 





Also our thick clients dont connect using REST . So thinking of adding the
below configuration on our thick client configuration.







Hope that is okay ?

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite thread pool configuration

[VeenaMithare]

I noticed that the DataStreamer thread pool size seems to be around 32 in my
local machine. In our cluster setup we dont stream data from any external db
. We use native persistence. Do you think it makes sense to reduce this pool
size for both server and client configuraiton. I am not sure if this pool
will ever be used in our project.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

[VeenaMithare]

I rechecked this. My cluster setup is : 
servers : 3
client : 1

Bring down all nodes in the cluster for about 2 mins (
failuredetectiontimeout is 10 seconds ). The client receives
EVT_NODE_RECONNECTED and does not receive EVT_NODE_SEGMENTED.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : MutableEntry.getValue().toBuilder throws NullPointerException

[VeenaMithare]

I have observed it on a subset of caches .  We do not have assertions
enabled.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Webinar: Apache Ignite Management & Monitoring

[VeenaMithare]

Hi , 

I am trying to configure cluster in my machine with the control center agent
to understand it. The servers are started using java so I tried to add the
maven dependency or add the library as a external dependency to the project
as mentioned in these slides  

https://www.gridgain.com/sites/default/files/technical-presentations/Using-Distributed-Tracing-to-Resolve-Performance-Issues-in-Apache-Ignite.pdf

https://www.gridgain.com/docs/control-center/latest/connect-ignite-cluster
( Maven artifact section )

Tried below : 
1. Added Library as an external dependency into the project. 
But control center agent did not start when I started the server. 

I dont see the following printed in the startup logs : 
Found control center

2.  Maven dependency : looks like we are not allowed to access it from the
gridgain maven repository: 
  > Could not resolve org.gridgain:control-center-agent:2.8.1.2.
 > Could not get resource
'http://mvnrepository.com/artifact/com.experlog/xapool/org/gridgain/control-center-agent/2.8.1.2/control-center-agent-2.8.1.2.pom'.
> Could not HEAD
'http://mvnrepository.com/artifact/com.experlog/xapool/org/gridgain/control-center-agent/2.8.1.2/control-center-agent-2.8.1.2.pom'.
Received status code 403 from server: Forbidden


Could you kindly let me know what I am missing ?

Please note I am able to start the server with the control center agent when
I start it using ignite.bat script on the command prompt.


regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

2.8.1 : MutableEntry.getValue().toBuilder throws NullPointerException

[VeenaMithare]

Hello,

In some rare scenarios , we get this exception when we update records . 


java.lang.NullPointerException: null
at
org.apache.ignite.internal.binary.builder.BinaryBuilderReader.(BinaryBuilderReader.java:72)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.binary.builder.BinaryObjectBuilderImpl.(BinaryObjectBuilderImpl.java:122)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.binary.builder.BinaryObjectBuilderImpl.wrap(BinaryObjectBuilderImpl.java:617)
~[ignite-core-2.8.1.jar:2.8.1]
at
org.apache.ignite.internal.binary.BinaryObjectExImpl.toBuilder(BinaryObjectExImpl.java:159)
~[ignite-core-2.8.1.jar:2.8.1]
at
com.company.project.projectname.helper.ProjectCacheEntryBinaryObjectProcessor.process(ProjectCacheEntryBinaryObjectProcessor.java:134)
~[dao-2.9.14.jar:?]
at
com.company.project.projectname.helper.ProjectCacheEntryBinaryObjectProcessor.process(ProjectCacheEntryBinaryObjectProcessor.java:34)
~[dao-2.9.14.jar:?]

===

As shown, the exception happens at this point in the project code : 
at
org.apache.ignite.internal.binary.BinaryObjectExImpl.toBuilder(BinaryObjectExImpl.java:159)
~[ignite-core-2.8.1.jar:2.8.1]
at
com.company.project.projectname.helper.CacheEntryBinaryObjectProcessor.process(ProjectCacheEntryBinaryObjectProcessor.java:134)
~[dao-2.9.14.jar:?]


Line 134 of ProjectCacheEntryBinaryObjectProcessor says : 

132:BinaryObject currentValueObject = mutableEntry.getValue();
133:if (currentValueObject !=null) {
134:binaryObjectBuilder = currentValueObject.toBuilder();


The exception happens at this position in the BinaryBuilderReader.java

reader = new BinaryReaderExImpl(ctx,
BinaryHeapInputStream.create(arr, pos),
ctx.configuration().getClassLoader(),
false);

=

The exception log in the try/catch prints the mutableentry.getValue as : 
2020-09-30T16:58:23,989 ERROR
c.c.p.c.h.ProjectCacheEntryBinaryObjectProcessor [sys-stripe-12-#13]: CACHE
ENTRY PROCESSOR EXCEPTION for record MutableEntry.keyPRODUCTKEY
[idHash=219577898, hash=-978873982,
NAME=xxx],MutableEntry.valueBinaryObjectImpl [arr= true, ctx=false, start=0]



Please note we dont get this exception always. Kindly request your guidance
on how to find out the root cause for this exception.

regards,
Veena.







--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

[VeenaMithare]

Hi Team, 

Kindly awaiting a reply on this,

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : INFO org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi [] - Accepted incoming communication connection

[VeenaMithare]

HI Anton,

The setup I am working on is as below : 
server nodes : 3
client : 1

I get these logs when I start the client node within 10 secs of stopping it.
There is no clue on the server nodes as to why these logs are printed. 

It looks as if the client node that was killed, is trying to join back ,
since it was not marked as DEAD yet by the servers.

It is very easy to reproduce this. Stop and start any client node within 10
seconds of stopping it.

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : Server cluster node startup issue

[VeenaMithare]

Thanks Alex,
Will try and see if I can create a reproducer.

>>If Ignite hasn't fully started, it will wait( U.awaitQuiet(startLatch); )
until it has, before returning the instance.

Does the Ignition.start wait for the storage spi to return from record, to
start ? since the storage spi is waiting on startLatch and the main thread
which is waiting for storage spi to return, there is a deadlock ?

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : Server cluster node startup issue

[VeenaMithare]

Hi Alex,



1. How is the storage spi record method getting invoked on this node if the
ignite has not even fully started on this node ? Which ignite instance is it
using to invoke the record method ?

I have enabled only these events :












2. >>  Is there a reason you are using Igintion.ignite(..) inside the
eventstorage api?

I use it in the eventstorage spi to get a handle to the audit cache( which
is a ignite cache ) , where i can store the events generated from a cache
modification that was done on dbeaver.


3. >> It is best to redesign your event storage system to not call these
methods.

What is the recommended design to capture and store the event generated from
dbeaver cache update.

regards,
Veena



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

2.8.1 : Server cluster node startup issue

[VeenaMithare]

Hello,

Our set up : 
Server nodes : 3
Writer application : 1 

A event storage spi that writes the cache_put events that have been
generated due to a dbeaver update into a ignite cache.

We restarted one of the servers yesterday during regular business hours and
noticed that the server was not starting properly. I took a thread dump and
noticed that the following two threads were in locked state . 

1. event storage spi : The thread that gets the ignite using
Ignition.ignite(ignitenamedinstance ) 
2. The server : The thread that starts ignite using Ignition.start(..).

Am I right in assuming that the above two events have locked against each
other ?

How can I avoid this ?In the event storage spi before I do Ignition.ignite ,
is there any way I can check if the ignite in the current server has been
started ?

The thread dump has been attached.
threaddump.txt
  

regards,
Veena.





  




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 - Loading Plugin Provider - Conflicting documentation

[VeenaMithare]

HI , 

Raised a Documentation Improvement jira : 
IGNITE-13356
Documentation Change needed: PluginProvider loading changed from 2.8.1
regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 - Loading Plugin Provider - Conflicting documentation

[VeenaMithare]

Hi Ilya,

I have used the new style. 

It will also mean that this document is no longer the best way to load the
plugin provider:
https://apacheignite.readme.io/docs/plugins 

And this might need a documentation update.

regards,
Veena.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

[VeenaMithare]

Hi Anton,

There are a few thoughts on this : 

1. I found this issue after I upgraded from 2.7.6 to 2.8.1 ( No timeout
values etc has been changed in our application ). So for the same scenario
and the same application configuration 
a. in 2.7.6 , the client receives a evt_node_segmented
b. in 2.8.1 , the client receives a evt_node_reconnected.

2. it logs the event of client state updated
from DISCONNECTED to RECONNECTED because the node succeeded to join the
topology back within some time, the node was not segmented from the
topology

>From the logs I posted, it looks like client tries to reconnect using a new
client id 
>>[tcp-client-disco-msg-worker-#4%client-null-igniteclient-SINGLE%] INFO
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi [] - Client node
disconnected from cluster, will try to reconnect with new id

3. The logs also say 
>>Client node was
reconnected after it was already considered failed by the server topology
(this could happen after all servers restarted or due to a long network
outage between the client and servers). All continuous queries and remote
event listeners created by this client will be unsubscribed, consider
listening to EVT_CLIENT_NODE_RECONNECTED event to restore them.

If a client node reconnects after it was considered failed by server,
shouldnt it receive a EVT_NODE_SEGMENTED ?

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 - Loading Plugin Provider - Conflicting documentation

[VeenaMithare]

Okay. If I set pluginconfiguration using setPluginConfiguration, how do I get
it in the plugins ?

getPluginConfigurations is deprecated in 2.8.1.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

[VeenaMithare]

HI Anton,

Checked again -In 2.8.1 it receives the EVT_NODE_RECONNECTED with a new node
id .  It receives Eventtype 17 ( EVT_NODE_RECONNECTED  ) and not Eventtype
14 ( EVT_NODE_SEGMENTED ) . 
Also it connects with a new node id. Please find the log with relevant parts
highlighted.


2020-07-31 06:32:17,150
[tcp-client-disco-msg-worker-#4%client-null-igniteclient-SINGLE%] INFO 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi [] - */Client node
disconnected from cluster, will try to reconnect with new id/*
[newId=bcbaa5ad-07aa-4118-802e-433dcfcb83b4,
prevId=bca9b2da-bca5-4eac-817b-c989a072b50a, locNode=TcpDiscoveryNode
[id=bca9b2da-bca5-4eac-817b-c989a072b50a,
consistentId=bca9b2da-bca5-4eac-817b-c989a072b50a, addrs=ArrayList
[0:0:0:0:0:0:0:1, a.b.c.d, 127.0.0.1], sockAddrs=HashSet
[machinename/a.b.c.d:0, /0:0:0:0:0:0:0:1:0, /127.0.0.1:0], discPort=0,
order=4, intOrder=0, lastExchangeTime=1596173504995, loc=true,
ver=2.8.1#20200521-sha1:86422096, isClient=true]]

2020-07-31 06:32:46,197
[tcp-client-disco-msg-worker-#4%client-null-igniteclient-SINGLE%] INFO 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi [] - Router node:
TcpDiscoveryNode [id=4ca9ed4f-416b-43f2-ada9-d5569e7fe8ce,
consistentId=d46c9ddd-d97c-4d07-b857-fc4281d9f495, addrs=ArrayList
[0:0:0:0:0:0:0:1, a.b.c.d, 127.0.0.1], sockAddrs=HashSet
[/0:0:0:0:0:0:0:1:47501, /127.0.0.1:47501, machinename/a.b.c.d:47501],
discPort=47501, order=3, intOrder=3, lastExchangeTime=1596173566191,
loc=false, ver=2.8.1#20200521-sha1:86422096, isClient=false]

2020-07-31 06:32:46,232
[disco-notifier-worker-#96%client-null-igniteclient-SINGLE%] INFO 
org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl [] -
Assigned mvcc coordinator [crd=MvccCoordinator
[topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0],
nodeId=d0b9191e-18ea-42fe-8148-b7d28a75d9f8, ver=1596173562700, local=false,
initialized=false]]
2020-07-31 06:32:46,233
[exchange-worker-#136%client-null-igniteclient-SINGLE%] INFO 
org.apache.ignite.internal.processors.cache.GridCacheProcessor [] - Stopped
cache [cacheName=ignite-sys-cache]
2020-07-31 06:32:46,233
[tcp-client-disco-msg-worker-#4%client-null-igniteclient-SINGLE%] WARN 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi [] - /*Client node was
reconnected after it was already considered failed by the server topology
(this could happen after all servers restarted or due to a long network
outage between the client and servers). All continuous queries and remote
event listeners created by this client will be unsubscribed, consider
listening to EVT_CLIENT_NODE_RECONNECTED event to restore them.*/
[06:32:46] Client node was reconnected after it was already considered
failed by the server topology (this could happen after all servers restarted
or due to a long network outage between the client and servers). All
continuous queries and remote event listeners created by this client will be
unsubscribed, consider listening to EVT_CLIENT_NODE_RECONNECTED event to
restore them.
2020-07-31 06:32:46,237
[disco-event-worker-#98%client-null-igniteclient-SINGLE%] INFO 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager [] -
Client node reconnected to topology: TcpDiscoveryNode
[id=bcbaa5ad-07aa-4118-802e-433dcfcb83b4,
consistentId=bca9b2da-bca5-4eac-817b-c989a072b50a, addrs=ArrayList
[0:0:0:0:0:0:0:1, a.b.c.d, 127.0.0.1], sockAddrs=HashSet
[machinename/a.b.c.d:0, /0:0:0:0:0:0:0:1:0, /127.0.0.1:0], discPort=0,
order=4, intOrder=0, lastExchangeTime=1596173504995, loc=true,
ver=2.8.1#20200521-sha1:86422096, isClient=true]

[06:32:46] Topology snapshot [ver=4, locNode=bcbaa5ad, servers=3, clients=1,
state=INACTIVE, CPUs=32, offheap=12.0GB, heap=28.0GB]
[06:32:46]   ^-- Baseline [id=0, size=3, online=3, offline=0]
2020-07-31 06:32:46,238
[disco-event-worker-#98%pricingClient-null-igniteclient-SINGLE%] INFO 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager [] -
Topology snapshot [ver=4, locNode=bcbaa5ad, servers=3, clients=1,
state=INACTIVE, CPUs=32, offheap=12.0GB, heap=28.0GB]
2020-07-31 06:32:46,238
[disco-event-worker-#98%pricingClient-null-igniteclient-SINGLE%] INFO 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager [] -  
^-- Baseline [id=0, size=3, online=3, offline=0]

2020-07-31 06:32:46,238
[disco-event-worker-#98%pricingClient-null-igniteclient-SINGLE%] DEBUG
SegmentationEventListener [] - /*Received Event 17 Event Node Id
bcbaa5ad-07aa-4118-802e-433dcfcb83b4 for Node :
bcbaa5ad-07aa-4118-802e-433dcfcb83b4*/

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

2.8.1 : EVT_NODE_RECONNECTED, EVT_NODE_SEGMENTED on the client side

[VeenaMithare]

Hello,

When do EVT_NODE_SEGMENTED event get generated on the client side in 2.8.1 ? 

I noticed that in 2.7.6 , when all the servers are restarted and the client
reconnects again with a new node id - it used to receive a
EVT_NODE_SEGMENTED . In 2.8.1 it receives EVT_NODE_RECONNECTED. 
Kindly confirm my understanding. 

regards,
Veena.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 - Loading Plugin Provider - Conflicting documentation

[VeenaMithare]

Okay.

>>   The recommended way is to use the service provider as detailed here:

How do we pass configuration for the provider in this approach.

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 - Loading Plugin Provider - Conflicting documentation

[VeenaMithare]

Thanks Alex,

There seems to be two conflicting documentation on loading the plugin
provider :  

1. The first way as shown in this documentation
https://apacheignite.readme.io/docs/plugins :
 uses the Java Service Loader to load the plugin provider . It loads the
plugin configurations through the ignite configuration and hence needs to
use  getPluginConfigurations to get the right configuration for the plugin.
/*Concern*/ : igniteconfiguration.getPluginConfigurations is deprecated in
2.8.1

2. The second way  : igniteconfiguration.getPluginConfigurations has been
deprecated in 2.8.1 .  The recommended way of sending configuration is by
calling igniteconfiguration.setPluginProvider . 

Please note there was no 'setPluginProvider' method in the versions prior to
2.8.1.
(
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/IgniteConfiguration.html#getPluginConfigurations--
)



/*Concern*/ : 2.8.1 plugin documentation talks about using service loader
framework. https://apacheignite.readme.io/docs/plugins. If we use
setPluginProvider, there is no need to load plugin provider through java
service loader. 



Kindly let me know if this is a documentation bug, or a miss in my
understanding,



regards,
Veena.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

2.8.1 : Ignite Security : Cache_Put event generated from a remote_client user action has subject uuid of Node that executes the request

[VeenaMithare]

Hi Team,

1. I noticed that this issue (
https://issues.apache.org/jira/browse/IGNITE-12781) is not resolved in
2.8.1.

Could you guide how can we get audit information if a cache record
modification is done on dbeaver and the cache_put event contains the node id
instead of the remote_client subject id ?

Please note this is a blocker issue for us to use Apache Ignite , since we
use dbeaver to update records sometimes. 
If this is not resolved, could we kindly ask this to be included in the next
release. 

2. Even if the cache_put event did contain the remote_client user id , how
are we supposed to fetch it from the auditstoragespi ?

The below link mentions 
http://apache-ignite-users.70518.x6.nabble.com/JDBC-thin-client-incorrect-security-context-td31354.html

public class EventStorageSpi extends IgniteSpiAdapter implements
EventStorageSpi {
@LoggerResource
private IgniteLogger log;

@Override
public  Collection localEvents(IgnitePredicate p)
{
return null;
}

@Override
public void record(Event evt) throws IgniteSpiException {
if (evt.type() == EVT_MANAGEMENT_TASK_STARTED) {
TaskEvent taskEvent = (TaskEvent) evt;

SecuritySubject subj = taskEvent.subjectId() != null
?
getSpiContext().authenticatedSubject(taskEvent.subjectId())
: null;

log.info("Management task started: [" +
"name=" + taskEvent.taskName() + ", " +
"eventNode=" + taskEvent.node() + ", " +
"timestamp=" + taskEvent.timestamp() + ", " +
"info=" + taskEvent.message() + ", " +
"subjectId=" + taskEvent.subjectId() + ", " +
"secureSubject=" + subj +
"]");
}
}

@Override
public void spiStart(@Nullable String igniteInstanceName) throws
IgniteSpiException {
/* No-op. */
}

@Override
public void spiStop() throws IgniteSpiException {
/* No-op. */
}
}

IgniteSpiContext exposes authenticatedSubject which according to some
discussions gets the subject *only for node* . (
http://apache-ignite-developers.2346864.n4.nabble.com/Security-Subject-of-thin-client-on-remote-nodes-td46029.html#a46412
)

/*securityContext(uuid ) was added to the GridSecurityProcessor to get the
securitycontext of the thin client. However this is not exposed via the
IgniteSpiContext.* /


3. The workaround I did was as follows. Please let me know if you see any
concerns on this approach - 
a. Add the remoteclientsubject into the authorizationcontext of the
authenticationcontext in the authenticate method of the securityprocessor.

b. This authorizationcontext is now put in a threadlocal variable ( Check
the class AuthorizationContext )
private static ThreadLocal actx = new ThreadLocal<>();

c. The following has been done in the storagespi when a change is made in
the dbeaver, 
c1. capture the EVT_TX_STARTED in the storage spi. The thread that generates
this event contains the subject in its threadlocal authorizationcontext.
Store this in a cache that holds the mapping transaction id to security
subject.

c2. capture the cache_put event and link the transaction id in the cache_put
event to the transaction id in the EVT_TX_STARTED and get the subject by
this mapping. 

c3. The transactionid in cache_put and the transactionid in EVT_TX_STARTED
could be same, in which case it is a direct mapping

c4. The transactionid in cache_put and the transactionid in EVT_TX_STARTED
could be different, in which case it is a case of finding the nearxid of the
transactionid in the cacheput event. And then find the security subject of
the nearxid


regards,
Veena.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: 2.8.1 - Loading Plugin Provider

[VeenaMithare]

Hi Alex,

Thanks for the reply, 

1. I noticed that the ml plugin uses getPluginConfigurations which seems to
have been deprecated..

https://github.com/apache/ignite/blob/master/modules/ml/src/main/java/org/apache/ignite/ml/util/plugin/MLPluginProvider.java

Would that mean that the preferred way  to load a PluginProvider is through
setPluginProvider in the latest ignite version  ?

2. Why did the earlier versions of ignite have service loader framework to
load plugins?

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

2.8.1 - Loading Plugin Provider

[VeenaMithare]

Hi , 

I saw that 'SetPluginConfiguration' in IgniteConfiguration has been
deprecated in 2.8.1: 
( IgniteConfiguration javadoc :
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/IgniteConfiguration.html
)
*IgniteConfigurationsetPluginConfigurations(PluginConfiguration...
pluginCfgs)*
Deprecated. 
Since PluginProviders can be set explicitly via
setPluginProviders(PluginProvider[]) it's preferable to store
PluginConfiguration as a part of PluginProvider.

*IgniteConfigurationsetPluginProviders(PluginProvider... pluginProvs)*
Sets plugin providers.


I guess we need to set Plugin Providers instead of setting Plugin
Configuration. The question is 
 do we also need to load the plugin provider through the service loader of
the JDK ?
(
https://apacheignite.readme.io/docs/plugins ( Section : Load Plugin Provider
)
)

What is the purpose of doing this ?

regards,
Veena.






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Event for an update that should have been filtered is received in Local Listener of Continuous Query when a 1000 row insert is triggered

[VeenaMithare]

Hello, 

Looks like this issue is not related to load, but related to remote filter
not getting redeployed when a server node is restarted on version 2.7.6. 

Related discussion at 
http://apache-ignite-users.70518.x6.nabble.com/Remote-Filter-Execution-td33274.html

regards,
Veena.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Remote Filter Execution

[VeenaMithare]

I can also confirm that both these issues happen consistently on 2.7.6. All
the nodes are in baseline when both these issues happen. 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Remote Filter Execution

[VeenaMithare]

Hi Ilya,

I can confirm that both issues do not occur in 2.8.1. ( The remote filter
evaluate executes only on 2 nodes as expected ..) Also the remote filter
gets deployed whenever there is a server restart.

The other thing for me to check is if we can use 2.8.1 ( We had issues on
2.8.0  jdbc thin client library , because of which we couldnt use it )

Thank you so much for the help ,

regards,
Veena.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Remote Filter Execution

[VeenaMithare]

HI Ilya,

Did you use the reproducer project ?

The CACHE configuration is PARTITIONED with ONE BACKUP. So should the remote
filter evaluate method run on all the 3 nodes ? I would expect the evaluate
method to run only 2 nodes

I have not tried 2.8.1 yet. Will try and let you know.
regards,
Veena.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/