Re: Issue in Distributed joins

[DS]

Hello Ilya,
Sure I'll do that


Regards 
Deepika 



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Using custom thread pool with listener

[Raymond Wilson]

I'm using the C# client on Ignite 2.8.0.

I have had an issue with thread deadlocking where the public thread pool is
saturated and then causes a deadlock when a message arrives from another
node using IMessageListener, eg

  public class MyListener : IMessageListener
  {
public bool Invoke(Guid nodeId, T message)
{
  // Process it

  return true;
}
 }

Ignite 2.8.0 now supports custom thread pools in the C# client :)

I can see how to create a custom thread pool in the Ignite configuration,
like this:

ExecutorConfiguration = new List
{
  new ExecutorConfiguration
  {
Name = "MyPool"
Size = 8
  }
}

And use it like this:

   ignite.Compute().WithExecutor("MyPool").Run() etc

However, I am not sure how to ask Ignite to run the Invoke() from the
MessageListener on that custom thread pool, rather than the public thread
pool.

Suggestions?

Thanks,
Raymond.

Re: read-though tutorial for a big table

[Evgenii Zhuravlev]

When you're saying that the result was poor, do you mean that data
preloading took too much time, or it's just about get operations?

Evgenii

вт, 10 мар. 2020 г. в 03:29, aealexsandrov :

> Hi,
>
> You can read the documentation articles:
>
> https://apacheignite.readme.io/docs/3rd-party-store
>
> In case if you are going to load the cache from 3-rd party store (RDBMS)
> then the default implementation of CacheJdbcPojoStore can take a lot of
> time
> for loading the data because it used JDBC connection inside (not pull of
> these connections).
>
> Probably you should implement your own version of CacheStore that will read
> data from RDBMS in several threads, e.g using the JDBC connection pull
> there. Sources are open for you, so you can copy the existed implementation
> and modify it:
>
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/cache/store/jdbc/CacheJdbcPojoStore.java
>
> Otherwise, you can do the initial data loading using some streaming tools:
>
> 1)Spark integration with Ignite -
> https://apacheignite-fs.readme.io/docs/ignite-data-frame
> 2)Kafka integration with Ignite -
> https://apacheignite-mix.readme.io/docs/kafka-streamer
>
> BR,
> Andrei
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: how to achieve this topology ?

[Evgenii Zhuravlev]

>Therefore, I think JVM1 and JVM2 should be client node with near caches
and also have node filter.
Not really, you can choose only one solution here - NodeFilter or client
nodes with Near cache. If you want to store data on nodes JVM1 and JVM2,
then you should go with NodeFilter approach. In this case they will be
server nodes and they will have all needed data locally.
In case of using Near cache, entry will be added to it only after reading
this entry from the remote cache first time. Also, this data will be
removed after node restart.

Evgenii



вт, 10 мар. 2020 г. в 06:20, Edward Chen :

> Here is a background. In enterprise wide, each application, like JVM1,
> will create its own cache and this data is useful for another application,
> so this data should push to server node , like JVM3. but each client node,
> like JVM1, should have freedom to contain other caches based on some rules,
> but definitely not every cache in JVM3. in this case, looks like Node
> filter will work.
>
> Therefore, I think JVM1 and JVM2 should be client node with near caches
> and also have node filter.
>
> My understanding is correct ?
>
> Thanks Stephen Darlington for the near cache idea.
>
>
> On 3/9/2020 7:04 PM, Evgenii Zhuravlev wrote:
>
> Hi,
>
> You can use NodeFilter for caches. Please use this JavaDoc for
> information:
> https://www.javadoc.io/doc/org.apache.ignite/ignite-core/latest/org/apache/ignite/util/AttributeNodeFilter.html
>
> Example can be found here:
> https://github.com/ezhuravl/ignite-code-examples/blob/master/src/main/java/examples/nodefilter/cache/CacheNodeFilterExample.java
>
> Evgenii
>
> пт, 6 мар. 2020 г. в 18:34, Edward Chen :
>
>> Hello,
>>
>> I want to achieve this topology, do you know how to configure ?
>>
>> The critical parts are, cache2 in JVM2 should not be replicated or copied
>> to JVM 1 . cache1 in JVM1 should not be replicated or copied to JVM 2 .
>> JVM3 and JVM 4 are each other failed-over backup.
>>
>> Thanks. Ed
>>
>

Re: SQL Error [08006]: Failed to communicate with Ignite cluster <-- BinaryObjectException: Not enough data to read the value

[Devin Bost]

It turns out that I still get the SQL SELECT error even when using Ignite
2.8.0.

Devin G. Bost


On Tue, Mar 10, 2020 at 3:19 PM Devin Bost  wrote:

> Hi Denis,
>
> Thanks for the information! I think it was just my IntelliJ that was
> complaining for no reason. I ran `mvn clean install`, and it pulled the
> package for me.
>
> Devin G. Bost
>
>
> On Tue, Mar 10, 2020 at 2:59 PM Denis Magda  wrote:
>
>> Hi Devin,
>>
>> I can see the latest Ignite artifacts in Maven:
>> https://search.maven.org/search?q=a:ignite-core
>>
>> Are you missing any particular package?
>>
>> -
>> Denis
>>
>>
>> On Tue, Mar 10, 2020 at 11:48 AM Devin Bost  wrote:
>>
>>> Hi Andrei,
>>>
>>> Thanks for the information. Do you know anything about when Ignite 2.8
>>> will be released? It looks like the Maven packages are unavailable.
>>>
>>> Devin G. Bost
>>>
>>>
>>> On Tue, Mar 10, 2020 at 4:17 AM aealexsandrov 
>>> wrote:
>>>
 Hi,

 Can you please test the Apache Ignite 2.8. It should contain the
 following
 fix:

 https://issues.apache.org/jira/browse/IGNITE-11953

 BR,
 Andrei



 --
 Sent from: http://apache-ignite-users.70518.x6.nabble.com/

>>>

Re: SQL Error [08006]: Failed to communicate with Ignite cluster <-- BinaryObjectException: Not enough data to read the value

[Devin Bost]

Hi Denis,

Thanks for the information! I think it was just my IntelliJ that was
complaining for no reason. I ran `mvn clean install`, and it pulled the
package for me.

Devin G. Bost


On Tue, Mar 10, 2020 at 2:59 PM Denis Magda  wrote:

> Hi Devin,
>
> I can see the latest Ignite artifacts in Maven:
> https://search.maven.org/search?q=a:ignite-core
>
> Are you missing any particular package?
>
> -
> Denis
>
>
> On Tue, Mar 10, 2020 at 11:48 AM Devin Bost  wrote:
>
>> Hi Andrei,
>>
>> Thanks for the information. Do you know anything about when Ignite 2.8
>> will be released? It looks like the Maven packages are unavailable.
>>
>> Devin G. Bost
>>
>>
>> On Tue, Mar 10, 2020 at 4:17 AM aealexsandrov 
>> wrote:
>>
>>> Hi,
>>>
>>> Can you please test the Apache Ignite 2.8. It should contain the
>>> following
>>> fix:
>>>
>>> https://issues.apache.org/jira/browse/IGNITE-11953
>>>
>>> BR,
>>> Andrei
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>

Re: Cache metrics on server nodes does not update correctly

[Denis Magda]

@Nikolay Izhikov , @Andrey Gura ,
could you folks check out this thread?

I have a feeling that what Dominik is describing was talked out before and
rather some sort of a limitation than an issue with the current
implementation.

-
Denis


On Tue, Mar 3, 2020 at 11:41 PM Dominik Przybysz 
wrote:

> Hi,
> I am trying to use partitioned cache on server nodes to which I connect
> with client node. Statistics of cache in the cluster are updated, but only
> for hits metric - misses metric is always 0.
>
> To reproduce this problem I created cluster of two nodes:
>
> Server node 1 adds 100 random test cases and prints cache statistics
> continuously:
>
> public class IgniteClusterNode1 {
> public static void main(String[] args) throws InterruptedException {
> IgniteConfiguration igniteConfiguration = new
> IgniteConfiguration();
>
> CacheConfiguration cacheConfiguration = new CacheConfiguration();
> cacheConfiguration.setName("test");
> cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> cacheConfiguration.setStatisticsEnabled(true);
> igniteConfiguration.setCacheConfiguration(cacheConfiguration);
>
> TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
> communicationSpi.setLocalPort(47500);
> igniteConfiguration.setCommunicationSpi(communicationSpi);
>
> TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
> discoverySpi.setLocalPort(47100);
> discoverySpi.setLocalPortRange(100);
> TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> "127.0.0.1:48100..48200"));
> igniteConfiguration.setDiscoverySpi(discoverySpi);
>
> try (Ignite ignite = Ignition.start(igniteConfiguration)) {
> try (IgniteCache cache =
> ignite.getOrCreateCache("test")) {
> new Random().ints(1000).map(i -> Math.abs(i %
> 1000)).distinct().limit(100).forEach(i -> {
> String key = "data_" + i;
> String value = UUID.randomUUID().toString();
> cache.put(key, value);
> }
> );
> }
> while (true) {
> System.out.println(ignite.cache("test").metrics());
> Thread.sleep(5000);
> }
> }
> }
> }
>
> Server node 2 only prints cache statistics continuously:
>
> public class IgniteClusterNode2 {
> public static void main(String[] args) throws InterruptedException {
> IgniteConfiguration igniteConfiguration = new
> IgniteConfiguration();
>
> CacheConfiguration cacheConfiguration = new CacheConfiguration();
> cacheConfiguration.setName("test");
> cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> cacheConfiguration.setStatisticsEnabled(true);
> igniteConfiguration.setCacheConfiguration(cacheConfiguration);
>
> TcpCommunicationSpi communicationSpi = new TcpCommunicationSpi();
> communicationSpi.setLocalPort(48500);
> igniteConfiguration.setCommunicationSpi(communicationSpi);
>
> TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
> discoverySpi.setLocalPort(48100);
> discoverySpi.setLocalPortRange(100);
> TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> ipFinder.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> "127.0.0.1:48100..48200"));
> igniteConfiguration.setDiscoverySpi(discoverySpi);
>
> try (Ignite ignite = Ignition.start(igniteConfiguration)) {
> while (true) {
> System.out.println(ignite.cache("test").metrics());
> Thread.sleep(5000);
> }
> }
> }
> }
>
> Next I start a client node which continuously read data from the cluster:
>
> public class CacheClusterReader {
> public static void main(String[] args) throws InterruptedException {
> IgniteConfiguration cfg = new IgniteConfiguration();
> cfg.setClientMode(true);
>
> TcpDiscoverySpi spi = new TcpDiscoverySpi();
> TcpDiscoveryVmIpFinder tcMp = new TcpDiscoveryVmIpFinder();
> tcMp.setAddresses(Arrays.asList("127.0.0.1:47100..47200",
> "127.0.0.1:48100..48200"));
> spi.setIpFinder(tcMp);
> cfg.setDiscoverySpi(spi);
>
> CacheConfiguration cacheConfig = new
> CacheConfiguration<>("test");
> cacheConfig.setStatisticsEnabled(true);
> cacheConfig.setCacheMode(CacheMode.PARTITIONED);
> cfg.setCacheConfiguration(cacheConfig);
>
> try (Ignite ignite = Ignition.start(cfg)) {
> System.out.println(ignite.cacheNames());
>
> while (true) {
> try (IgniteCache cache =
> ignite.getOrCreateCache(cacheConfig)) {
> 

Re: page is broken,cannot restore it from wal (2.8.0)

[Denis Magda]

Does this issue arise under special circumstances? Have you experienced
this before switching to Ignite 2.8?

-
Denis


On Sat, Mar 7, 2020 at 5:21 PM 18624049226 <18624049...@163.com> wrote:

> hi community,
>
> What are the reasons for the following problems? How to avoid it?
>
>

Re: SQL Error [08006]: Failed to communicate with Ignite cluster <-- BinaryObjectException: Not enough data to read the value

[Denis Magda]

Hi Devin,

I can see the latest Ignite artifacts in Maven:
https://search.maven.org/search?q=a:ignite-core

Are you missing any particular package?

-
Denis


On Tue, Mar 10, 2020 at 11:48 AM Devin Bost  wrote:

> Hi Andrei,
>
> Thanks for the information. Do you know anything about when Ignite 2.8
> will be released? It looks like the Maven packages are unavailable.
>
> Devin G. Bost
>
>
> On Tue, Mar 10, 2020 at 4:17 AM aealexsandrov 
> wrote:
>
>> Hi,
>>
>> Can you please test the Apache Ignite 2.8. It should contain the following
>> fix:
>>
>> https://issues.apache.org/jira/browse/IGNITE-11953
>>
>> BR,
>> Andrei
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: SQL Error [08006]: Failed to communicate with Ignite cluster <-- BinaryObjectException: Not enough data to read the value

[Devin Bost]

Hi Andrei,

Thanks for the information. Do you know anything about when Ignite 2.8 will
be released? It looks like the Maven packages are unavailable.

Devin G. Bost


On Tue, Mar 10, 2020 at 4:17 AM aealexsandrov 
wrote:

> Hi,
>
> Can you please test the Apache Ignite 2.8. It should contain the following
> fix:
>
> https://issues.apache.org/jira/browse/IGNITE-11953
>
> BR,
> Andrei
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Ignite 2.8 get exception while use batch insert in streaming mode

[Ilya Kasnacheev]

Hello!

Unfortunately this looks like a regression in 2.8 :(

I have filed an issue: https://issues.apache.org/jira/browse/IGNITE-12764

Regards,
-- 
Ilya Kasnacheev


чт, 5 мар. 2020 г. в 10:45, yangjiajun <1371549...@qq.com>:

> Hello.
>
> The following test code also throws the same exception:
> ps = conn.prepareStatement("SET STREAMING ON ALLOW_OVERWRITE ON");
>  ps.execute();
>  ps.close();
>
> String sql = "insert INTO  city1(id,name,name1)
> VALUES(?,?,RANDOM_UUID())";
> ps = conn.prepareStatement(sql);
> for (int i = 0; i < 1600; i++) {
> String s1 = String.valueOf(Math.random());
> ps.setInt(1, i);
> ps.setString(2, s1);
> ps.execute();
> }
>
> ps.close();
>
> ps = conn.prepareStatement("set streaming off");
> ps.execute();
> ps.close();
>
> conn.close();
>
> We can't use execute batch method and RANDOM_UUID() within streaming mode
> in
> ignite 2.8?
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: how to achieve this topology ?

[Edward Chen]

Here is a background. In enterprise wide, each application, like JVM1,  
will create its own cache and this data is useful for another 
application, so this data should push to server node , like JVM3. but 
each client node, like JVM1, should have freedom to contain other caches 
based on some rules, but definitely not every cache in JVM3. in this 
case, looks like Node filter will work.


Therefore, I think JVM1 and JVM2 should be client node with near caches 
and also have node filter.


My understanding is correct ?

Thanks Stephen Darlington for the near cache idea.


On 3/9/2020 7:04 PM, Evgenii Zhuravlev wrote:

Hi,

You can use NodeFilter for caches. Please use this JavaDoc for 
information: 
https://www.javadoc.io/doc/org.apache.ignite/ignite-core/latest/org/apache/ignite/util/AttributeNodeFilter.html


Example can be found here: 
https://github.com/ezhuravl/ignite-code-examples/blob/master/src/main/java/examples/nodefilter/cache/CacheNodeFilterExample.java


Evgenii

пт, 6 мар. 2020 г. в 18:34, Edward Chen >:


Hello,

I want to achieve this topology, do you know how to configure ?

The critical parts are, cache2 in JVM2 should not be replicated or
copied to JVM 1 . cache1 in JVM1 should not be replicated or
copied to JVM 2 .  JVM3 and JVM 4 are each other failed-over backup.

Thanks. Ed

Re: tcp-comm system-critical thread blocked

[Ilya Kasnacheev]

Hello!

Please make sure to read
https://apacheignite.readme.io/docs/critical-failures-handling about
configurability, etc.

Unfortunately there is too few context to say mor.e

Regards,
-- 
Ilya Kasnacheev


пн, 9 мар. 2020 г. в 18:33, Mitchell Rathbun (BLOOMBERG/ 731 LEX) <
mrathb...@bloomberg.net>:

> We have seen the following happen a couple of time recently during periods
> of high load/gc pauses in our system:
>
> 2020-03-02 11:38:56,803 ERROR STDIO
> [tcp-disco-msg-worker-#2%ignite_wingman_2931%] {} Mar 02, 2020 11:38:56 AM
> org.apache.ignite.logger.java.JavaLogger error
> SEVERE: Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=grid-nio-worker-tcp-comm-3,
> blockedFor=12s]
>
> I know it says this leads to undefined behavior, but I am wondering what
> this thread is for/what the effect of it being blocked is. Also, is this
> timeout something that is configurable?
>

Re: Unable to connect to Ignite Visor Console in Ignite 2.8.0

[Ilya Kasnacheev]

Hello!

Actually, this screen shot does actually contain relevant error message:
rebalance thread pool size should be < system thread pool size.

It is enforced and node will not start (even visor's Daemon node).

Please fix your cfg to enforce this constraint:
https://apacheignite.readme.io/docs/thread-pools

Regards,
-- 
Ilya Kasnacheev


пт, 6 мар. 2020 г. в 17:37, Николай Кулагин :

> Joshi,
>
> A ticket has been started for this issue. You can track his status here
> [1].
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12757
>
> чт, 5 мар. 2020 г. в 14:48, Kamlesh Joshi :
>
>> Hi Team,
>>
>>
>>
>> I have updated Ignite cluster to latest version 2.8.0 and update was
>> successful. However, I am unable connect to IgniteVisor console. Below is
>> the command I used (the same command was used for earlier versions and it
>> worked fine). Not sure am I missing something or is there any defect around
>> it? Ignite 2.8.0 docs are not available on the site !
>>
>>
>>
>> *$IGNITE_HOME/bin/ignitevisorcmd.sh -cfg="/app/Ignite/visorconfig.xml"*
>>
>>
>>
>>
>>
>>
>>
>> *Thanks and Regards,*
>>
>> *Kamlesh Joshi*
>>
>>
>>
>>
>> "*Confidentiality Warning*: This message and any attachments are
>> intended only for the use of the intended recipient(s), are confidential
>> and may be privileged. If you are not the intended recipient, you are
>> hereby notified that any review, re-transmission, conversion to hard copy,
>> copying, circulation or other use of this message and any attachments is
>> strictly prohibited. If you are not the intended recipient, please notify
>> the sender immediately by return email and delete this message and any
>> attachments from your system.
>>
>> *Virus Warning:* Although the company has taken reasonable precautions
>> to ensure no viruses are present in this email. The company cannot accept
>> responsibility for any loss or damage arising from the use of this email or
>> attachment."
>>
>

Re: page is broken,cannot restore it from wal (2.8.0)

[Ilya Kasnacheev]

Hello!

Maybe PDS is corrupted, if you have sufficient backups, I recommend
deleting persistence data from this node and re-add it to cluster.

Regards,
-- 
Ilya Kasnacheev


вс, 8 мар. 2020 г. в 04:20, 18624049226 <18624049...@163.com>:

> hi community,
>
> What are the reasons for the following problems? How to avoid it?
>
>

Re: Issue in Distributed joins

[Ilya Kasnacheev]

Hello again!

You can also frame it in the form of SQL script (DDL - INSERTs - SELECTs)
if you wish.

Regards,
-- 
Ilya Kasnacheev


вт, 10 мар. 2020 г. в 14:30, Ilya Kasnacheev :

> Hello!
>
> Can you please produce a small reproducer project which will, when
> started, bring up a node, populate it with some data and then run those
> queries?
>
> We will surely check.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 9 мар. 2020 г. в 09:01, DS :
>
>> Hello,
>> I'll appreciate, if you can find time to look into the issue.
>>
>> Regards
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: Issue in Distributed joins

[Ilya Kasnacheev]

Hello!

Can you please produce a small reproducer project which will, when started,
bring up a node, populate it with some data and then run those queries?

We will surely check.

Regards,
-- 
Ilya Kasnacheev


пн, 9 мар. 2020 г. в 09:01, DS :

> Hello,
> I'll appreciate, if you can find time to look into the issue.
>
> Regards
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: read-though tutorial for a big table

[aealexsandrov]

Hi,

You can read the documentation articles:

https://apacheignite.readme.io/docs/3rd-party-store

In case if you are going to load the cache from 3-rd party store (RDBMS)
then the default implementation of CacheJdbcPojoStore can take a lot of time
for loading the data because it used JDBC connection inside (not pull of
these connections).

Probably you should implement your own version of CacheStore that will read
data from RDBMS in several threads, e.g using the JDBC connection pull
there. Sources are open for you, so you can copy the existed implementation
and modify it:

https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/cache/store/jdbc/CacheJdbcPojoStore.java

Otherwise, you can do the initial data loading using some streaming tools:

1)Spark integration with Ignite -
https://apacheignite-fs.readme.io/docs/ignite-data-frame
2)Kafka integration with Ignite -
https://apacheignite-mix.readme.io/docs/kafka-streamer

BR,
Andrei



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: SQL Error [08006]: Failed to communicate with Ignite cluster <-- BinaryObjectException: Not enough data to read the value

[aealexsandrov]

Hi,

Can you please test the Apache Ignite 2.8. It should contain the following
fix:

https://issues.apache.org/jira/browse/IGNITE-11953

BR,
Andrei



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: how to achieve this topology ?

[Stephen Darlington]

Or JVM3 and JVM4 would be your Ignite cluster (server nodes) and JMV1 and JVM2 
would be client nodes, possibly with near caches.

> On 9 Mar 2020, at 23:04, Evgenii Zhuravlev  wrote:
> 
> Hi,
> 
> You can use NodeFilter for caches. Please use this JavaDoc for information: 
> https://www.javadoc.io/doc/org.apache.ignite/ignite-core/latest/org/apache/ignite/util/AttributeNodeFilter.html
>  
> 
> 
> Example can be found here: 
> https://github.com/ezhuravl/ignite-code-examples/blob/master/src/main/java/examples/nodefilter/cache/CacheNodeFilterExample.java
>  
> 
> 
> Evgenii
> 
> пт, 6 мар. 2020 г. в 18:34, Edward Chen  >:
> Hello,
> 
> I want to achieve this topology, do you know how to configure ? 
> 
> The critical parts are, cache2 in JVM2 should not be replicated or copied to 
> JVM 1 . cache1 in JVM1 should not be replicated or copied to JVM 2 .  JVM3 
> and JVM 4 are each other failed-over backup. 
> 
> 
> 
> Thanks. Ed
>