Re: Native persistence - No space left on device error

[Denis Magda]

Keep an eye on the storage usage metrics to avoid such incidents
proactively in production:
https://www.gridgain.com/docs/tutorials/management-monitoring/ignite-storage-monitoring#track-disk-storage-size

-
Denis


On Wed, Nov 18, 2020 at 7:41 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> When you run out of disk space, the Ignite node will stop functioning
> normally, and your underlying PDS storage may become corrupted as well.
>
> Thus it is recommended to avoid running out of disk space. This is not in
> any means a normal mode of operation of Apache Ignite.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 18 нояб. 2020 г. в 18:29, facundo.maldonado <
> maldonadofacu...@gmail.com>:
>
>> What is the expected behavior of native persistence when there is no more
>> space on disk?
>>
>> I'm getting this error when I reach the max size of the mounted volume
>> (storage, not wal):
>> *class org.apache.ignite.IgniteCheckedException: No space left on device*
>>
>> I assumed, that when there is no more space for allocating pages, it
>> removes
>> the older ones. I may be wrong, confusing this with expiry.
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: Nodes failed to join the cluster after restarting

[Cong Guo]

Hi,

I attach the log from the only working node while two others are restarted.
There is no error message other than the "failed to join" message. I do not
see any clue in the log. I cannot reproduce this issue either. That's why I
am asking about the code. Maybe you know certain suspicious places. Thank
you.

On Wed, Nov 18, 2020 at 2:45 AM Ivan Bessonov  wrote:

> Sorry, I see that you use TcpDiscoverySpi.
>
> ср, 18 нояб. 2020 г. в 10:44, Ivan Bessonov :
>
>> Hello,
>>
>> these parameters are configured automatically, I know that you don't
>> configure them. And with the fact that all "automatic" configuration is
>> completed, chances of seeing the same bug are low.
>>
>> Understanding the reason is tricky, we would need to debug the starting
>> node or at least add more logs. Is this possible? I see that you're asking
>> me about the code.
>>
>> Knowing the content of "ver" and "histCache.toArray()" in
>> "org.apache.ignite.internal.processors.metastorage.persistence.DistributedMetaStorageImpl#collectJoiningNodeData"
>> would certainly help.
>> More specifically - *ver.id ()* and 
>> *Arrays.stream(histCache.toArray()).map(item
>> -> Arrays.toString(item.keys())).collect(Collectors.joining(","))*
>>
>> Honestly, I have no idea how your situation is even possible, otherwise
>> we would find the solution rather quickly. Needless to say, I can't
>> reproduce it. Error message that you see was created for the case when you
>> join your node to the wrong cluster.
>>
>> Do you have any custom code during the node start? And one more question
>> - what discovery SPI are you using? TCP or Zookeeper?
>>
>>
>> ср, 18 нояб. 2020 г. в 02:29, Cong Guo :
>>
>>> Hi,
>>>
>>> The parameters values on two other nodes are the same. Actually I do not
>>> configure these values. When you enable the native persistence, you will
>>> see these logs by default. Nothing is special. When this error occurs on
>>> the restarting node, nothing happens on two other nodes. When I restart the
>>> second node, it also fails due to the same error.
>>>
>>> I will still need to restart the nodes in the future,  one by one
>>> without stopping the service. This issue may happen again. The workaround
>>> has to deactivate the cluster and stop the service, which does not work in
>>> a production environment.
>>>
>>> I think we need to fix this bug or at least understand the reason to
>>> avoid it. Could you please tell me where this version value could be
>>> modified when a node just starts? Do you have any guess about this bug now?
>>> I can help analyze the code. Thank you.
>>>
>>> On Tue, Nov 17, 2020 at 4:09 AM Ivan Bessonov 
>>> wrote:
>>>
 Thank you for the reply!

 Right now the only existing distributed properties I see are these:
 - Baseline parameter 'baselineAutoAdjustEnabled' was changed from
 'null' to 'false'
 - Baseline parameter 'baselineAutoAdjustTimeout' was changed from
 'null' to '30'
 - SQL parameter 'sql.disabledFunctions' was changed from 'null' to
 '[FILE_WRITE, CANCEL_SESSION, MEMORY_USED, CSVREAD, LINK_SCHEMA,
 MEMORY_FREE, FILE_READ, CSVWRITE, SESSION_ID, LOCK_MODE]'

 I wonder what values they have on nodes that rejected the new node. I
 suggest sending logs of those nodes as well.
 Right now I believe that this bug won't happen again on your
 installation, but it only makes it more elusive...

 The most probable reason is that node (somehow) initialized some
 properties with defaults before joining the cluster, while cluster didn't
 have those values at all.
 The rule is that activated cluster can't accept changed properties from
 joining node. So, the workaround would be deactivating the cluster, joining
 the node and activating it again. But as I said, I don't think that you'll
 see this bug ever again.

 вт, 17 нояб. 2020 г. в 07:34, Cong Guo :

> Hi,
>
> Please find the attached log for a complete but failed reboot. You can
> see the exceptions.
>
> On Mon, Nov 16, 2020 at 4:00 AM Ivan Bessonov 
> wrote:
>
>> Hello,
>>
>> there must be a bug somewhere during node start, it updates its
>> distributed metastorage content and tries to join an already activated
>> cluster, thus creating a conflict. It's hard to tell the exact data that
>> caused conflict, especially without any logs.
>>
>> Topic that you mentioned (
>> http://apache-ignite-users.70518.x6.nabble.com/Question-about-baseline-topology-and-cluster-activation-td34336.html)
>> seems to be about the same problem, but the issue
>> https://issues.apache.org/jira/browse/IGNITE-12850 is not related to
>> it.
>>
>> If you have logs from those unsuccessful restart attempts, it would
>> be very helpful.
>>
>> Sadly, distributed metastorage is an internal component to store
>> settings and has no public documentation. Developers 

Apache Ignite Clientside NearCache and Serverside Eviction blocking cluster

[pvprsd]

Hi All,

We have recently enabled Eviction on Ignite server nodes. We were using
ignite nearcache for ignite client nodes.

Due to nearcache, the server side cache-entries are not getting evicted.
Eventually this leads to off-cache full, and "Too many attempts to evict
data page" error and ignite cluster stopped responding. We have given enough
memory to ignite server nodes i.e. 4+4gb for heap and off-heap.

>From debugging the server code, it looks like the near-cache enforces the
entries to maintain the client-node id; and due to this those entries are
not getting evicted from server.

When the clientside the entries got evicted (with clientside LRU eviction),
it seems the client-node reference in serverside entry is not getting
cleared. Hence after sometime it leads to ignite getting blocked. This is
consistently happening.

Either I am doing some thing wrong with the ignite configurations, OR there
is a bug in the ignite with this combination.

Can you please guide me, on how to find resolution to this problem.

Thanks,
Prasad





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Javamelody starting an Apache Ignite server node!

[Ilya Kasnacheev]

Hello!

Have you seen https://github.com/javamelody/javamelody/issues/858 ?

Regards,
-- 
Ilya Kasnacheev


ср, 18 нояб. 2020 г. в 22:14, David Tinker :

> https://github.com/javamelody/javamelody/issues/962
>
> "I am using the Apache Ignite thin client (
> https://ignite.apache.org/docs/latest/thin-clients/java-thin-client).
> Somehow after my app has been running for a few minutes an Ignite server
> node suddenly starts up! This happens on a "javamelody" thread."
>
> Any ideas?
>
> This is one of the more WTF things I have seen lately! :)
>

Javamelody starting an Apache Ignite server node!

[David Tinker]

https://github.com/javamelody/javamelody/issues/962

"I am using the Apache Ignite thin client (
https://ignite.apache.org/docs/latest/thin-clients/java-thin-client).
Somehow after my app has been running for a few minutes an Ignite server
node suddenly starts up! This happens on a "javamelody" thread."

Any ideas?

This is one of the more WTF things I have seen lately! :)

RE: Failed to prepare update plan error while inserting data, when select is used aong with insert

[harinath]

Hi Alexandar,

Sorry my bad. I just realized that the primary key is missed while
inserting.

Thanks for your response and help

Thanks.
Harinath



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Failed to prepare update plan error while inserting data, when select is used aong with insert

[Alexandr Shapkin]

Hi, That’s odd, could you please share the logs with  the error?   From: harinathSent: Wednesday, November 18, 2020 6:52 PMTo: user@ignite.apache.orgSubject: RE: Failed to prepare update plan error while inserting data, when select is used aong with insert Hi Alexandar, I have tried without parentheses, still i see the same error Thanks,Harinath   --Sent from: http://apache-ignite-users.70518.x6.nabble.com/ 

Re: C# SQL API on key-value entries in cache error

[Pavel Tupitsyn]

Hello,

The correct query would be "SELECT AssociateID FROM Associate",
because Ignite infers table name from the class name, and column names from
property names.

You can override table name with QueryEntity.TableName property,
and column names with QuerySqlFieldAttribute.Name property.

On Wed, Nov 18, 2020 at 6:38 PM ABDumalagan 
wrote:

> Hello,
>
> I ran into some errors trying to use Apache Ignite's SQL API, specifically
> the @QueryAnnotation
> 
> and executing a query
> .
>
> Right now, I start a cluster remotely with an XML file. Then, I start
> another node locally and programmatically with C# that connects to an
> underlying Oracle database in order to load the cache with data. The
> entries in the cache are key-value pairs, where the key is an Associate ID
> and the value is the entire Associate object.
>
> My objective is to be able to query the data in cache. The steps I've done
> so far was to annotate the fields in the value class definition with the
> @QuerySqlField annotation in Associate.cs , instantiate the QueryEntity in
> CacheConfiguration in Program.cs, and perform a query using SqlFieldsQuery
> in Program.cs
>
> The error that I'm running into is:
>
> 'Failed to parse query. Table "DEFAULT" not found; SQL statement: SELECT
> Associate_ID FROM default [42102-197]'
>
> Am I running into this error because my entries are in key-value pairs? Do
> I need to organize the entries in my cache into a SQL table?
>
> The code is attached as a file here:
>
> Program.cs
> 
>
> Associate.cs
> 
>
> Thank you!
>
>
> --
> Sent from the Apache Ignite Users mailing list archive
>  at Nabble.com.
>

RE: Failed to prepare update plan error while inserting data, when select is used aong with insert

[harinath]

Hi Alexandar,

I have tried without parentheses, still i see the same error

Thanks,
Harinath



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Native persistence - No space left on device error

[Ilya Kasnacheev]

Hello!

When you run out of disk space, the Ignite node will stop functioning
normally, and your underlying PDS storage may become corrupted as well.

Thus it is recommended to avoid running out of disk space. This is not in
any means a normal mode of operation of Apache Ignite.

Regards,
-- 
Ilya Kasnacheev


ср, 18 нояб. 2020 г. в 18:29, facundo.maldonado :

> What is the expected behavior of native persistence when there is no more
> space on disk?
>
> I'm getting this error when I reach the max size of the mounted volume
> (storage, not wal):
> *class org.apache.ignite.IgniteCheckedException: No space left on device*
>
> I assumed, that when there is no more space for allocating pages, it
> removes
> the older ones. I may be wrong, confusing this with expiry.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

C# SQL API on key-value entries in cache error

[ABDumalagan]

Hello, 
I ran into some errors trying to use Apache Ignite's SQL API, specifically
the  @QueryAnnotation
  
and  executing a query
  . 
Right now, I start a cluster remotely with an XML file. Then, I start
another node locally and programmatically with C# that connects to an
underlying Oracle database in order to load the cache with data. The entries
in the cache are key-value pairs, where the key is an Associate ID and the
value is the entire Associate object.
My objective is to be able to query the data in cache. The steps I've done
so far was to annotate the fields in the value class definition with the
@QuerySqlField annotation in Associate.cs , instantiate the QueryEntity in
CacheConfiguration in Program.cs, and perform a query using SqlFieldsQuery
in Program.cs
The error that I'm running into is: 
'Failed to parse query. Table "DEFAULT" not found; SQL statement:SELECT
Associate_ID FROM default [42102-197]'
Am I running into this error because my entries are in key-value pairs? Do I
need to organize the entries in my cache into a SQL table?
The code is attached as a file here:
Program.cs
  
Associate.cs
  
 Thank you!




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Native persistence - No space left on device error

[facundo.maldonado]

What is the expected behavior of native persistence when there is no more
space on disk?

I'm getting this error when I reach the max size of the mounted volume
(storage, not wal): 
*class org.apache.ignite.IgniteCheckedException: No space left on device*

I assumed, that when there is no more space for allocating pages, it removes
the older ones. I may be wrong, confusing this with expiry.




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: not in, not exists sql result is not correct

[Ilya Kasnacheev]

Hello!

This looks like a query where the result is only correct if the data in
orders and accounts are collocated.

If they are not, you may check it by setting distributedJoins=true
connection property and re-running the query.

Please see
https://ignite.apache.org/docs/latest/data-modeling/affinity-collocation

Regards,
-- 
Ilya Kasnacheev


ср, 18 нояб. 2020 г. в 06:03, marble.zh...@coinflex.com <
marble.zh...@coinflex.com>:

> Hi,
>
> We are using ignite 2.8.1, with this sql statement, SELECT * FROM orders p
> WHERE  NOT exists (SELECT accountid FROM account b WHERE b.accountid =
> p.accountid);
>
> Retured result accountid actually exists in table account, change to NOT IN
> also is not correct. But I have another table result is correct.
>
> Is there any suggestions? thanks.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

RE: Failed to prepare update plan error while inserting data, when select is used aong with insert

[Alexandr Shapkin]

Hi,  Seems like there is a syntax error and Apache Ignite failed to produce an execution plan Could you try to run it without the parentheses? Like: INSERT INTO desttable (name, lastname, age) SELECT name, lastname, age from srctable  From: Alexandr ShapkinSent: Wednesday, November 18, 2020 5:10 PMTo: user@ignite.apache.orgSubject: RE: Failed to prepare update plan error while inserting data, when select is used aong with insert Hi, Seems like there is a syntax error. From: harinathSent: Wednesday, November 18, 2020 10:28 AMTo: user@ignite.apache.orgSubject: Failed to prepare update plan error while inserting data, when select is used aong with insert Hi, Using ignite thick client, i'm trying to copy data from one table intoanother table using following sql statement "INSERT INTO desttable (name, lastname, age) (SELECT name, lastname, agefrom srctable)" I'm getting the following error while i run sql query."Failed to prepare update plan." Not sure of the reason for the error.Any help here will be much appreciated. Thanks in advance Thanks & Reagrds,Harinath   --Sent from: http://apache-ignite-users.70518.x6.nabble.com/  

RE: Failed to prepare update plan error while inserting data, when select is used aong with insert

[Alexandr Shapkin]

Hi, Seems like there is a syntax error. From: harinathSent: Wednesday, November 18, 2020 10:28 AMTo: user@ignite.apache.orgSubject: Failed to prepare update plan error while inserting data, when select is used aong with insert Hi, Using ignite thick client, i'm trying to copy data from one table intoanother table using following sql statement "INSERT INTO desttable (name, lastname, age) (SELECT name, lastname, agefrom srctable)" I'm getting the following error while i run sql query."Failed to prepare update plan." Not sure of the reason for the error.Any help here will be much appreciated. Thanks in advance Thanks & Reagrds,Harinath   --Sent from: http://apache-ignite-users.70518.x6.nabble.com/ 

Re: Java VM Parameters setting through cpp api

[Igor Sapego]

Yes, it is possible. See [1]

[1] -
https://ignite.apache.org/releases/latest/cppdoc/structignite_1_1IgniteConfiguration.html#ad7f632214a786dfdf8dc6f5824749e8a

Best Regards,
Igor


On Wed, Nov 18, 2020 at 3:18 PM Wolfgang Meyerle <
wolfgang.meye...@googlemail.com> wrote:

> Hi,
>
> Is it possible somehow to define specific Java VM Parameters through the
> cpp interfacte or the xml configuration files.
>
> I'm facing serious sys-stripe timeouts and one recommendation of Ignite
> is to switch to IPv4 only.
>
> Unfortunately this is an Java VM Parameter which I do not seem to have
> influence as soon as I start the Ignite Cpp Server Process via
> Ignite::Ignition::Start
>
> Regards,
>
> Wolfgang
>
>

Java VM Parameters setting through cpp api

[Wolfgang Meyerle]

Hi,

Is it possible somehow to define specific Java VM Parameters through the 
cpp interfacte or the xml configuration files.


I'm facing serious sys-stripe timeouts and one recommendation of Ignite 
is to switch to IPv4 only.


Unfortunately this is an Java VM Parameter which I do not seem to have 
influence as soon as I start the Ignite Cpp Server Process via 
Ignite::Ignition::Start


Regards,

Wolfgang

Re: What is considered high IO wait and partition exchange fiallure?

[Maxim Muzafarov]

Hello John,

Can you share the full log and cluster configuration?
For instance, the cluster may fail the node due to unresponsive socket
request if this node had long GC pauses due to high load.

> Does this mean this node was also the master?

Not sure I've got your question. Do you mean the coordinator node?

On Tue, 17 Nov 2020 at 20:52, John Smith  wrote:
>
> So if I understand correctly the logs below...
>
> The node that shut off was timing out trying to get partition exchange from 
> the indicated nodes and it shut itself off correct? Does this mean this node 
> was also the master?
>
> 1- The time indicated in the log is that UTC?
> 2- I'm trying to see if it was high IO, but the node that stopped had no high 
> IO.
> 3- the other nodes had an average peek of about 0.3%-0.5% And that was around 
> 16:30 EST So not sure if those times match with the log.
> 4- On super high load days we get up to 7%-9% IO Wait but we never lost a 
> node on those occasions.
> 5- The nodes are persistent nodes so I'm guessing on write there's high IO 
> can we reduce the commit time? but even then I'm not sure it's related. As 
> even on higher load we don't lose the node.
>
> [04:36:07,771][INFO][node-stopper][GridDhtPartitionsExchangeFuture] Completed 
> partition exchange [localNode=8d84b4e9-8c11-4166-a75a-9b3d959ff1fe, 
> exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion 
> [topVer=132, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode 
> [id=e5645874-77e9-4455-8db5-07fa63984276, addrs=[127.0.0.1, xxx.xxx.xxx.5], 
> sockAddrs=[/xxx.xxx.xxx.5:0, /127.0.0.1:0], discPort=0, order=108, 
> intOrder=60, lastExchangeTime=1600798045116, loc=false, 
> ver=2.7.0#20181130-sha1:256ae401, isClient=true], done=true], topVer=null, 
> durationFromInit=1605328567765]
> [04:36:07,771][INFO][node-stopper][GridDhtPartitionsExchangeFuture] Finish 
> exchange future [startVer=AffinityTopologyVersion [topVer=133, 
> minorTopVer=0], resVer=null, err=class 
> org.apache.ignite.internal.IgniteInterruptedCheckedException: Node is 
> stopping: xx]
> [04:36:07,772][INFO][node-stopper][GridDhtPartitionsExchangeFuture] Completed 
> partition exchange [localNode=8d84b4e9-8c11-4166-a75a-9b3d959ff1fe, 
> exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion 
> [topVer=133, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode 
> [id=13f473cb-5441-460b-9b9f-a23c1bcf3b0b, addrs=[127.0.0.1, xxx.xxx.xxx.3], 
> sockAddrs=[/127.0.0.1:0, /xxx.xxx.xxx.3:0], discPort=0, order=110, 
> intOrder=61, lastExchangeTime=1600798045127, loc=false, 
> ver=2.7.0#20181130-sha1:256ae401, isClient=true], done=true], topVer=null, 
> durationFromInit=1605328567765]
> [04:36:07,772][INFO][node-stopper][GridDhtPartitionsExchangeFuture] Finish 
> exchange future [startVer=AffinityTopologyVersion [topVer=134, 
> minorTopVer=0], resVer=null, err=class 
> org.apache.ignite.internal.IgniteInterruptedCheckedException: Node is 
> stopping: xx]
> [04:36:07,772][INFO][node-stopper][GridDhtPartitionsExchangeFuture] Completed 
> partition exchange [localNode=8d84b4e9-8c11-4166-a75a-9b3d959ff1fe, 
> exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion 
> [topVer=134, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode 
> [id=ef9f3750-edfa-474e-8e68-6f0abee5095a, addrs=[127.0.0.1, xxx.xxx.xxx.10], 
> sockAddrs=[/127.0.0.1:0, /xxx.xxx.xxx.10:0], discPort=0, order=112, 
> intOrder=62, lastExchangeTime=1600798045137, loc=false, 
> ver=2.7.0#20181130-sha1:256ae401, isClient=true], done=true], topVer=null, 
> durationFromInit=1605328567765]
> [04:36:07,772][INFO][node-stopper][GridDhtPartitionsExchangeFuture] Finish 
> exchange future [startVer=AffinityTopologyVersion [topVer=135, 
> minorTopVer=0], resVer=null, err=class 
> org.apache.ignite.internal.IgniteInterruptedCheckedException: Node is 
> stopping: xx]
> [04:36:07,772][INFO][node-stopper][GridDhtPartitionsExchangeFuture] Completed 
> partition exchange [localNode=8d84b4e9-8c11-4166-a75a-9b3d959ff1fe, 
> exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion 
> [topVer=135, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode 
> [id=e14e5bea-2784-4a6f-af2f-e98a4bf0ab52, addrs=[127.0.0.1, xxx.xxx.xxx.63], 
> sockAddrs=[xx-0001/xxx.xxx.xxx.63:47500, /127.0.0.1:47500], 
> discPort=47500, order=118, intOrder=65, lastExchangeTime=1600798763474, 
> loc=false, ver=2.7.0#20181130-sha1:256ae401, isClient=false], done=true], 
> topVer=null, durationFromInit=1605328567765]
> [04:36:07,772][INFO][node-stopper][GridDhtPartitionsExchangeFuture] Finish 
> exchange future [startVer=AffinityTopologyVersion [topVer=136, 
> minorTopVer=0], resVer=null, err=class 
> org.apache.ignite.internal.IgniteInterruptedCheckedException: Node is 
> stopping: xx]
> [04:36:07,772][INFO][node-stopper][GridDhtPartitionsExchangeFuture] Completed 
> partition exchange [localNode=8d84b4e9-8c11-4166-a75a-9b3d959ff1fe, 
>