Re: Request for contributors permissions

[Ivan Pavlukhin]

Hello Steve,

I added your JIRA account to a contributors list. Now you can assign
tickets to yourself.

Looking forward for your contributions!

2020-05-21 9:52 GMT+03:00, steve.hostett...@gmail.com
:
> Hello,
>
> as discussed in  this thread
> 
>
> I would like to contribute to IGNITE-6499.
>
> Would you mind giving me the right to contribute.
>
> Thanks in advance
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>


-- 

Best regards,
Ivan Pavlukhin

[DISCUSSION] Ignite integration testing framework.

[Nikolay Izhikov]

Hello, Igniters.

I created a PoC [1] for the integration tests of Ignite.

Let me briefly explain the gap I want to cover:

1. For now, we don’t have a solution for automated testing of Ignite on «real 
cluster».
By «real cluster» I mean cluster «like a production»:
* client and server nodes deployed on different hosts.
* thin clients perform queries from some other hosts
* etc.

2. We don’t have a solution for automated benchmarks of some internal Ignite 
process
* PME
* rebalance.
This means we don’t know - Do we perform rebalance(or PME) in 2.7.0 faster or 
slower than in 2.8.0 for the same cluster?

3. We don’t have a solution for automated testing of Ignite integration in a 
real-world environment:
Ignite-Spark integration can be taken as an example.
I think some ML solutions also should be tested in real-world deployments.

Solution:

I propose to use duck tape library from confluent (apache 2.0 license)
I tested it both on the real cluster(Yandex Cloud) and on the local 
environment(docker) and it works just fine.

PoC contains following services:

* Simple rebalance test: 
Start 2 server nodes, 
Create some data with Ignite client, 
Start one more server node, 
Wait for rebalance finish
* Simple Ignite-Spark integration test: 
Start 1 Spark master, start 1 Spark worker, 
Start 1 Ignite server node
Create some data with Ignite client, 
Check data in application that queries it from Spark.

All tests are fully automated.
Logs collection works just fine.
You can see an example of the tests report - [4].

Pros:

* Ability to test local changes(no need to public changes to some remote 
repository or similar).
* Ability to parametrize test environment(run the same tests on different JDK, 
JVM params, config, etc.)
* Isolation by default so system tests are as reliable as possible.
* Utilities for pulling up and tearing down services easily in clusters in 
different environments (e.g. local, custom cluster, Vagrant, K8s, Mesos, 
Docker, cloud providers, etc.)
* Easy to write unit tests for distributed systems
* Adopted and successfully used by other distributed open source project - 
Apache Kafka.
* Collect results (e.g. logs, console output)
* Report results (e.g. expected conditions met, performance results, etc.)

WDYT?

[1] https://github.com/nizhikov/ignite/pull/15
[2] https://github.com/confluentinc/ducktape 
[3] https://ducktape-docs.readthedocs.io/en/latest/run_tests.html
[4] https://yadi.sk/d/JC8ciJZjrkdndg

Re: [DISCUSSION] Ignite integration testing framework.

[Anton Vinogradov]

Nikolay,

Great proposal!

Could you please explain how to perform testing on the different
environments?
For Example, you provided a rebalance test.

Is it possible to perform this test with TDE enabled?
Is it possible to perform it on custom OS (eg. Windows)?
Is it possible to perform in on bare metal?

I hope you'll answer yes to all, and the question mostly about details.

On Thu, May 21, 2020 at 10:27 AM Nikolay Izhikov 
wrote:

> Hello, Igniters.
>
> I created a PoC [1] for the integration tests of Ignite.
>
> Let me briefly explain the gap I want to cover:
>
> 1. For now, we don’t have a solution for automated testing of Ignite on
> «real cluster».
> By «real cluster» I mean cluster «like a production»:
> * client and server nodes deployed on different hosts.
> * thin clients perform queries from some other hosts
> * etc.
>
> 2. We don’t have a solution for automated benchmarks of some internal
> Ignite process
> * PME
> * rebalance.
> This means we don’t know - Do we perform rebalance(or PME) in 2.7.0 faster
> or slower than in 2.8.0 for the same cluster?
>
> 3. We don’t have a solution for automated testing of Ignite integration in
> a real-world environment:
> Ignite-Spark integration can be taken as an example.
> I think some ML solutions also should be tested in real-world deployments.
>
> Solution:
>
> I propose to use duck tape library from confluent (apache 2.0 license)
> I tested it both on the real cluster(Yandex Cloud) and on the local
> environment(docker) and it works just fine.
>
> PoC contains following services:
>
> * Simple rebalance test:
> Start 2 server nodes,
> Create some data with Ignite client,
> Start one more server node,
> Wait for rebalance finish
> * Simple Ignite-Spark integration test:
> Start 1 Spark master, start 1 Spark worker,
> Start 1 Ignite server node
> Create some data with Ignite client,
> Check data in application that queries it from Spark.
>
> All tests are fully automated.
> Logs collection works just fine.
> You can see an example of the tests report - [4].
>
> Pros:
>
> * Ability to test local changes(no need to public changes to some remote
> repository or similar).
> * Ability to parametrize test environment(run the same tests on different
> JDK, JVM params, config, etc.)
> * Isolation by default so system tests are as reliable as possible.
> * Utilities for pulling up and tearing down services easily in clusters in
> different environments (e.g. local, custom cluster, Vagrant, K8s, Mesos,
> Docker, cloud providers, etc.)
> * Easy to write unit tests for distributed systems
> * Adopted and successfully used by other distributed open source project -
> Apache Kafka.
> * Collect results (e.g. logs, console output)
> * Report results (e.g. expected conditions met, performance results, etc.)
>
> WDYT?
>
> [1] https://github.com/nizhikov/ignite/pull/15
> [2] https://github.com/confluentinc/ducktape
> [3] https://ducktape-docs.readthedocs.io/en/latest/run_tests.html
> [4] https://yadi.sk/d/JC8ciJZjrkdndg

Re: [DISCUSSION] Ignite integration testing framework.

[Nikolay Izhikov]

Hello, Anton.

> Could you please explain how to perform testing on the different environments?

Different JDK:

1. We should prepare our environment and put the desired JDK in the known 
locations.  [/opt/jdk1.8, /opt/jdk11], for example.
2. We should parametrize test and just setup java environment variables as 
usual(see ignite.py [1], for example):
export JAVA_HOME,
export PATH

Different JVM parameters - We need to support it into the desired `service` or 
perform command from the test itself.
A different version of Ignite(or any other `service`) - We can run the same 
test on the different Ignite versions.
All we need, just change the root path for the commands

Makes sense?

Is it possible to perform this test with TDE enabled?

Yes. All you need is parametrize Ignite config and enable some features when 
required.

Is it possible to perform it on custom OS (eg. Windows)?

No, This not supported out of the box.
I guess we can investigate the features of PowerShell to make it possible.

Is it possible to perform in on bare metal?

Yes.
All you need is an ssh connection.
As I said before, you can run the same tests on the dev-environment(local 
source + docker) with on command and right after development runs it on the 
bare metal.
The only change you need is one config file (cluster.json to be clear)

[1] 
https://github.com/nizhikov/ignite/pull/15/files#diff-463ad0ed102bd795ac78a50a868b5425R69

> 21 мая 2020 г., в 10:46, Anton Vinogradov  написал(а):
> 
> Nikolay,
> 
> Great proposal!
> 
> Could you please explain how to perform testing on the different
> environments?
> For Example, you provided a rebalance test.
> 
> Is it possible to perform this test with TDE enabled?
> Is it possible to perform it on custom OS (eg. Windows)?
> Is it possible to perform in on bare metal?
> 
> I hope you'll answer yes to all, and the question mostly about details.
> 
> On Thu, May 21, 2020 at 10:27 AM Nikolay Izhikov 
> wrote:
> 
>> Hello, Igniters.
>> 
>> I created a PoC [1] for the integration tests of Ignite.
>> 
>> Let me briefly explain the gap I want to cover:
>> 
>> 1. For now, we don’t have a solution for automated testing of Ignite on
>> «real cluster».
>> By «real cluster» I mean cluster «like a production»:
>>* client and server nodes deployed on different hosts.
>>* thin clients perform queries from some other hosts
>>* etc.
>> 
>> 2. We don’t have a solution for automated benchmarks of some internal
>> Ignite process
>>* PME
>>* rebalance.
>> This means we don’t know - Do we perform rebalance(or PME) in 2.7.0 faster
>> or slower than in 2.8.0 for the same cluster?
>> 
>> 3. We don’t have a solution for automated testing of Ignite integration in
>> a real-world environment:
>> Ignite-Spark integration can be taken as an example.
>> I think some ML solutions also should be tested in real-world deployments.
>> 
>> Solution:
>> 
>> I propose to use duck tape library from confluent (apache 2.0 license)
>> I tested it both on the real cluster(Yandex Cloud) and on the local
>> environment(docker) and it works just fine.
>> 
>> PoC contains following services:
>> 
>>* Simple rebalance test:
>>Start 2 server nodes,
>>Create some data with Ignite client,
>>Start one more server node,
>>Wait for rebalance finish
>>* Simple Ignite-Spark integration test:
>>Start 1 Spark master, start 1 Spark worker,
>>Start 1 Ignite server node
>>Create some data with Ignite client,
>>Check data in application that queries it from Spark.
>> 
>> All tests are fully automated.
>> Logs collection works just fine.
>> You can see an example of the tests report - [4].
>> 
>> Pros:
>> 
>> * Ability to test local changes(no need to public changes to some remote
>> repository or similar).
>> * Ability to parametrize test environment(run the same tests on different
>> JDK, JVM params, config, etc.)
>> * Isolation by default so system tests are as reliable as possible.
>> * Utilities for pulling up and tearing down services easily in clusters in
>> different environments (e.g. local, custom cluster, Vagrant, K8s, Mesos,
>> Docker, cloud providers, etc.)
>> * Easy to write unit tests for distributed systems
>> * Adopted and successfully used by other distributed open source project -
>> Apache Kafka.
>> * Collect results (e.g. logs, console output)
>> * Report results (e.g. expected conditions met, performance results, etc.)
>> 
>> WDYT?
>> 
>> [1] https://github.com/nizhikov/ignite/pull/15
>> [2] https://github.com/confluentinc/ducktape
>> [3] https://ducktape-docs.readthedocs.io/en/latest/run_tests.html
>> [4] https://yadi.sk/d/JC8ciJZjrkdndg

Re: [DISCUSSION] Separate code sanity check and build task

[Konstantin Orlov]

Hi Ivan,

Thanks for your reply! It’s better to get an answer late than never :)

-- 
Regards,
Konstantin Orlov


> On 18 May 2020, at 09:04, Ivan Pavlukhin  wrote:
> 
> Hi Konstantin,
> 
> Surprisingly, I found your message in a Spam folder (gmail).
> 
> We had discussions about the subject before. The most recent one and
> reflecting a current state is [1]. You can find many thoughts and
> arguments in another discussion [2] (it might be better to start
> reading from a bottom).
> 
> [1] 
> https://lists.apache.org/thread.html/6995a4e789117ba3f5577651866cfa99a6ffcc208cf60330d17d5a48%40%3Cdev.ignite.apache.org%3E
> [2] 
> http://apache-ignite-developers.2346864.n4.nabble.com/Code-inspection-td27709i80.html#a41297
> 
> 2020-04-20 11:00 GMT+03:00, Konstantin Orlov :
>> Igniters,
>> 
>> Currently we have code sanity checks [1][2] integrated within a build task
>> [3]. Do we really need to fail the build (and therefore the other tasks) if
>> there is a minor flaw like a missing line at the end of a file or an unused
>> import? As for me it could be separated from the build task.
>> 
>> What do you think?
>> 
>> [1]
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_CheckCodeStyle
>> 
>> [2]
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_LicensesHeaders
>> 
>> [3]
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite
>> 
>> 
>> --
>> Regards,
>> Konstantin Orlov
>> 
>> 
>> 
> 
> 
> -- 
> 
> Best regards,
> Ivan Pavlukhin

[jira] [Created] (IGNITE-13048) WAL FSYNC mode doesn't work with disabled archiver

[Ivan Bessonov (Jira)]

Ivan Bessonov created IGNITE-13048:
--

 Summary: WAL FSYNC mode doesn't work with disabled archiver
 Key: IGNITE-13048
 URL: https://issues.apache.org/jira/browse/IGNITE-13048
 Project: Ignite
  Issue Type: Bug
Reporter: Ivan Bessonov
Assignee: Ivan Bessonov


{noformat}
Caused by: 
org.apache.ignite.internal.processors.cache.persistence.StorageException: 
Failed to initialize WAL log segment (WAL segment size change is not supported 
in 'DEFAULT' WAL mode) 
[filePath=/home/vsisko/gridgain/backend-work/work/db/wal/web_console_data/0001.wal,
 fileSize=24313258, configSize=10]
at 
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.checkFiles(FileWriteAheadLogManager.java:2427)
 ~[ignite-core-8.7.5.jar:8.7.5]
at 
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.checkOrPrepareFiles(FileWriteAheadLogManager.java:1404)
 ~[ignite-core-8.7.5.jar:8.7.5]
at 
org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.start0(FileWriteAheadLogManager.java:435)
 ~[ignite-core-8.7.5.jar:8.7.5]
at 
org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.start(GridCacheSharedManagerAdapter.java:60)
 ~[ignite-core-8.7.5.jar:8.7.5]
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:841)
 ~[ignite-core-8.7.5.jar:8.7.5]
at 
org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1717) 
~[ignite-core-8.7.5.jar:8.7.5]
at 
org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1020) 
~[ignite-core-8.7.5.jar:8.7.5]
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2039)
 ~[ignite-core-8.7.5.jar:8.7.5]
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1731)
 ~[ignite-core-8.7.5.jar:8.7.5]
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1157) 
~[ignite-core-8.7.5.jar:8.7.5]
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:677) 
~[ignite-core-8.7.5.jar:8.7.5]
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:602) 
~[ignite-core-8.7.5.jar:8.7.5]
at org.apache.ignite.Ignition.start(Ignition.java:322) 
~[ignite-core-8.7.5.jar:8.7.5]
at 
org.apache.ignite.console.config.GridConfiguration.igniteInstance(GridConfiguration.java:38)
 ~[classes/:?]
at 
org.apache.ignite.console.config.GridConfiguration$$EnhancerBySpringCGLIB$$b50da981.CGLIB$igniteInstance$0()
 ~[classes/:?]
at 
org.apache.ignite.console.config.GridConfiguration$$EnhancerBySpringCGLIB$$b50da981$$FastClassBySpringCGLIB$$d486ae88.invoke()
 ~[classes/:?]
at 
org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) 
~[spring-core-4.3.23.RELEASE.jar:4.3.23.RELEASE]
at 
org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358)
 ~[spring-context-4.3.23.RELEASE.jar:4.3.23.RELEASE]
at 
org.apache.ignite.console.config.GridConfiguration$$EnhancerBySpringCGLIB$$b50da981.igniteInstance()
 ~[classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:1.8.0_222]
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
~[?:1.8.0_222]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at 
org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
 ~[spring-beans-4.3.23.RELEASE.jar:4.3.23.RELEASE]
at 
org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
 ~[spring-beans-4.3.23.RELEASE.jar:4.3.23.RELEASE]
... 83 more{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13049) AssertionError with JmxExporterSpi on cache stop. Test testCacheConfigSingleLineOutputFormatTwoNodeManyCaches

[Maxim Muzafarov (Jira)]

Maxim Muzafarov created IGNITE-13049:


 Summary: AssertionError with JmxExporterSpi on cache stop. Test 
testCacheConfigSingleLineOutputFormatTwoNodeManyCaches
 Key: IGNITE-13049
 URL: https://issues.apache.org/jira/browse/IGNITE-13049
 Project: Ignite
  Issue Type: Test
Reporter: Maxim Muzafarov


The test throws AssertionError if {{JmxExporterSpi}} configured 
{{GridCommandHandlerClusterByClassWithSSLTest#testCacheConfigSingleLineOutputFormatTwoNodeManyCaches}}.
 Reproduced locally time to time.

1. Configure JmxExporterSpi for default {{getConfiguration}} method (see 
{{org.apache.ignite.util.GridCommandHandlerAbstractTest#getConfiguration}})
2. Run with SSL testCacheConfigSingleLineOutputFormatTwoNodeManyCaches

{code}
[12:17:28]W: [org.apache.ignite:ignite-core] [2020-05-21 
12:17:28,500][ERROR][exchange-worker-#16994%gridCommandHandlerTest0%][IgniteTestResources]
 Critical system error detected. Will be handled accordingly to configured 
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler 
[ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, 
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext 
[type=SYSTEM_WORKER_TERMINATION, err=java.lang.AssertionError]]
[12:17:28]W: [org.apache.ignite:ignite-core] 
java.lang.AssertionError
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.spi.metric.jmx.JmxMetricExporterSpi.unregister(JmxMetricExporterSpi.java:110)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.util.IgniteUtils.notifyListeners(IgniteUtils.java:11694)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.processors.metric.GridMetricManager.remove(GridMetricManager.java:393)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.processors.cache.CacheGroupContext.removeIOStatistic(CacheGroupContext.java:1321)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.cleanup(GridCacheProcessor.java:513)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.stopCacheGroup(GridCacheProcessor.java:2900)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.stopCacheGroup(GridCacheProcessor.java:2888)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.processCacheStopRequestOnExchangeDone(GridCacheProcessor.java:2780)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.onExchangeDone(GridCacheProcessor.java:2877)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:2404)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:3822)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:3590)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processSingleMessage(GridDhtPartitionsExchangeFuture.java:3180)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$200(GridDhtPartitionsExchangeFuture.java:155)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply(GridDhtPartitionsExchangeFuture.java:2967)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply(GridDhtPartitionsExchangeFuture.java:2955)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:399)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:354)
[12:17:28]W: [org.apache.ignite:ignite-core]at 
org.apache.ignite.internal.processors.

[jira] [Created] (IGNITE-13050) ClusterGroup that is recomputed on topology change

[Ivan Bessonov (Jira)]

Ivan Bessonov created IGNITE-13050:
--

 Summary: ClusterGroup that is recomputed on topology change
 Key: IGNITE-13050
 URL: https://issues.apache.org/jira/browse/IGNITE-13050
 Project: Ignite
  Issue Type: Improvement
Reporter: Ivan Bessonov
Assignee: Ivan Bessonov


Currently, ClusterGroup comes in two favors:
One is a static set of UUIDs which will not change, second is predicate that is 
recomputed over ALL nodes on EVERY operation. This has bitten our client 
because recomputing of ClusterGroup happens in tcp-communication thread 
clogging it and delaying every operation in cluster. This is a major problem.

It would be nice if there was a ClusterGroup with predicate which would 
recompute once per topology affinity change. Bonus points if it precisely 
tracks current topology with zero delay or overrun.

Would be nice to upgrade firstNode/lastNode predicates to that mechanism since 
now they are static - topology changes but firstNode/lastNode projections 
don't, they may point to absent node.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSSION] New Ignite settings for IGNITE-12438 and IGNITE-13013

[Ivan Bessonov]

Hello Igniters,

I'd like to discuss with you changes related to [1] and [2]. Both issues
are mostly the same so
let's discuss the core idea.

*Motivation.*

There are certain environments that don't allow Ignite server nodes to open
TCP connections to
thick clients, e.g. K8S, AWS Lambda or Azure Functions. To operate in such
environments, the
server needs a way to request a client to open an "inverse" communication
connection to it.

I've prepared a PR (still in progress) that introduces new mechanism of
opening connection and
related configuration.

*Main idea*

This mechanism is called "communication via discovery" or "inverse
connection", it works as
follows:
 - server that needs to connect to "unreachable" thick client sends a
specific Discovery message
   (inverse communication request) to that client;
 - client node upon receiving the request opens communication connection to
that server;
 - server sees connection opened by client and proceeds with its task (that
required opening
   connection to the client).

Working name for new configuration parameter for this feature is
environmentType, it is an
enum with two values (again, working names): STANDALONE (default) and
VIRTUALIZED.
It is used as a hint to server to speed-up establishing of connections:
when server sees a client
with VIRTUALIZED environmentType it doesn't try to open connection to it
and sends inverse
communication request right away.
If environmentType is STANDALONE then server tries to open a connection in
a regular way
(iterating over all client addresses) and sends request only if all
attempts failed.

There is a concern about naming of the configuration as it catches only one
use-case: when we
deal with some kind of virtualized environment like K8S.
There are other options I've encountered in private discussion:
- connectionMode - ALWAYS_INITIATE / INITIATE_OR_ACCEPT
- networkConnectivity - REACHABLE_ALWAYS / REACHABLE_ONE_WAY
- communicationViaDiscovery - ALWAYS / FALLBACK
- isReachableFromAllNodes (true/false flag)
- initiateConnectionsOnThisNode (true/false flag).

*Limitations*

The feature cannot be used along with pairedConnection setting as this
setting implies
establishing connections in both directions. Also current implementation
supports opening only
client-to-server connections. Other types of connections like
client-to-client or server-to-server
will be implemented in separate tickets.

[1] https://issues.apache.org/jira/browse/IGNITE-12438
[2] https://issues.apache.org/jira/browse/IGNITE-13013

-- 
Sincerely yours,
Ivan Bessonov

Re: Apache Ignite 2.8.1 RELEASE [Time, Scope, Manager]

[Ilya Kasnacheev]

Hello!

Do you mind if I push additional change to fix English in some of these
entries? A lot of these has times not concording with "fixed".

Regards,
-- 
Ilya Kasnacheev


ср, 20 мая 2020 г. в 15:47, Nikolay Izhikov :

> Hello, Igniters.
>
> I prepared a PR with release notes.
> Please, take a look [1]
>
> [1] https://github.com/apache/ignite/pull/7821
>
> > 20 мая 2020 г., в 13:06, Nikolay Izhikov 
> написал(а):
> >
> > Hello, Jury.
> >
> > It seems we reached code freeze step for 2.8.1 release.
> > I will prepare rc for it shortly.
> >
> >> 14 мая 2020 г., в 11:08, Юрий  написал(а):
> >>
> >> Nikolay,
> >>
> >> Release 2.8.1 are delayed and announced dates [1] at release page is not
> >> actual. Could you update it to reflect current vision of release date?
> >>
> >> [1].
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8.1
> >>
> >> чт, 7 мая 2020 г. в 17:23, Nikolay Izhikov :
> >>
> >>> Done.
> >>>
> >>>
>  7 мая 2020 г., в 13:20, Denis Garus  написал(а):
> 
>  Nikolay,
>  could we add the simple improvement [1] to 2.8.1 scope?
> 
>  1. https://issues.apache.org/jira/browse/IGNITE-12983
> 
>  чт, 30 апр. 2020 г. в 11:26, Alex Plehanov :
> 
> > Nikolay,
> >
> > TC results: [1], [2]
> > I've cherry-picked IGNITE-12933 and IGNITE-12855 to 2.8.1
> >
> > [1]:
> >
> >
> >>>
> https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull%2F7754%2Fhead&action=Latest&baseBranchForTc=ignite-2.8.1
> > [2]:
> >
> >
> >>>
> https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull%2F7755%2Fhead&action=Latest&baseBranchForTc=ignite-2.8.1
> >
> > вт, 28 апр. 2020 г. в 15:19, Nikolay Izhikov :
> >
> >> Hello, Alex.
> >>
> >> +1 from me.
> >>
> >>> 28 апр. 2020 г., в 15:03, Alex Plehanov 
> >> написал(а):
> >>>
> >>> Hello guys,
> >>>
> >>> While we are still waiting for some tickets to resolve I propose to
> >>> cherry-pick to 2.8.1 two more bugfixes:
> >>> IGNITE-12933 Fixed node failure after put incorrect key class for
> >>> cache
> >>> with indexed types
> >>> IGNITE-12855 Fixed node failure with concurrent get operation and
> >>> entry
> >>> expiration when persistent is enabled
> >>> Both fixes prevent node failure in some circumstances, both fixed
> > already
> >>> merged to master.
> >>>
> >>> WDYT?
> >>>
> >>> пн, 27 апр. 2020 г. в 11:53, Nikolay Izhikov  >:
> >>>
>  Taras.
> 
>  Thank you, very much!
>  You changes merged to 2.8.1 branch.
> 
>  Igniters,
> 
>  We have 10 tickets scheduled for 2.8.1 release:
> 
>  OPEN:
> 
>  IGNITE-11687Concurrent WAL replay & log may fail with CRC
> error
> >>> on
>  read - Dmitriy Govorukhin
>  IGNITE-12346.NET: Platform
> error:System.NullReferenceException -
>  Pavel Tupitsyn
> 
>  IN PROGRESS:
> 
>  IGNITE-12637IgniteSparkSession doesn't start the clients on
> >>> really
>  distributed cluster - Yaroslav Molochkov
>  IGNITE-12788Cluster achieved fully rebalanced (PME-free ready)
> > state
>  metric - Nikolay Izhikov
> 
>  PATCH AVAILABLE:
> 
>  IGNITE-10417notifyDiscoveryListener() call can be lost. -
> Pavel
>  Voronkin
>  IGNITE-12852Comma in field is not supported by COPY command -
> > YuJue
> >> Li
>  IGNITE-12252Unchecked exceptions during rebalancing should be
> >> handled
>  - Nikolai Kulagin
>  IGNITE-12905QueryKeyValueIterable missing custom spliterator()
>  implementation - Johnny Galatikitis
>  IGNITE-12801Possible extra page release when throttling and
> >> checkpoint
>  thread store its concurrently - Anton Kalashnikov
>  IGNITE-12794Scan query fails with an assertion error:
> Unexpected
> > row
>  key - Denis Mekhanikov
> 
> 
> > 27 апр. 2020 г., в 11:08, Taras Ledkov 
>  написал(а):
> >
> > Hi,
> >
> > Nikolay, i've created PR [1] that contains the SQL-related
> tickets
> >>> to
>  port into 2.8.1:
> >
> > IGNITE-12790 Introduce distributed SQL configuration and ability
> to
>  disable SQL functions.
> > IGNITE-12887 Fix handle type mismatch exception on compare values
> > while
>  traversing index tree.
> > IGNITE-12848 fix H2Connection leaks on INSERT
> > IGNITE-12800  SQL: local queries cursors must be closed or full
> read
> > to
>  unlock the GridH2Table.
> >
> > TC test are OK. Please take a look at the TC bot report [2].
> > Please review & merge the pa

Re: Apache Ignite 2.8.1 RELEASE [Time, Scope, Manager]

[Nikolay Izhikov]

Hello, Ilya.

Actually, I’m uploading RC0 right now.
Let’s include these fixes in RC1, if you don’t mind?

> 21 мая 2020 г., в 16:00, Ilya Kasnacheev  
> написал(а):
> 
> Hello!
> 
> Do you mind if I push additional change to fix English in some of these
> entries? A lot of these has times not concording with "fixed".
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> ср, 20 мая 2020 г. в 15:47, Nikolay Izhikov :
> 
>> Hello, Igniters.
>> 
>> I prepared a PR with release notes.
>> Please, take a look [1]
>> 
>> [1] https://github.com/apache/ignite/pull/7821
>> 
>>> 20 мая 2020 г., в 13:06, Nikolay Izhikov 
>> написал(а):
>>> 
>>> Hello, Jury.
>>> 
>>> It seems we reached code freeze step for 2.8.1 release.
>>> I will prepare rc for it shortly.
>>> 
 14 мая 2020 г., в 11:08, Юрий  написал(а):
 
 Nikolay,
 
 Release 2.8.1 are delayed and announced dates [1] at release page is not
 actual. Could you update it to reflect current vision of release date?
 
 [1].
>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8.1
 
 чт, 7 мая 2020 г. в 17:23, Nikolay Izhikov :
 
> Done.
> 
> 
>> 7 мая 2020 г., в 13:20, Denis Garus  написал(а):
>> 
>> Nikolay,
>> could we add the simple improvement [1] to 2.8.1 scope?
>> 
>> 1. https://issues.apache.org/jira/browse/IGNITE-12983
>> 
>> чт, 30 апр. 2020 г. в 11:26, Alex Plehanov :
>> 
>>> Nikolay,
>>> 
>>> TC results: [1], [2]
>>> I've cherry-picked IGNITE-12933 and IGNITE-12855 to 2.8.1
>>> 
>>> [1]:
>>> 
>>> 
> 
>> https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull%2F7754%2Fhead&action=Latest&baseBranchForTc=ignite-2.8.1
>>> [2]:
>>> 
>>> 
> 
>> https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull%2F7755%2Fhead&action=Latest&baseBranchForTc=ignite-2.8.1
>>> 
>>> вт, 28 апр. 2020 г. в 15:19, Nikolay Izhikov :
>>> 
 Hello, Alex.
 
 +1 from me.
 
> 28 апр. 2020 г., в 15:03, Alex Plehanov 
 написал(а):
> 
> Hello guys,
> 
> While we are still waiting for some tickets to resolve I propose to
> cherry-pick to 2.8.1 two more bugfixes:
> IGNITE-12933 Fixed node failure after put incorrect key class for
> cache
> with indexed types
> IGNITE-12855 Fixed node failure with concurrent get operation and
> entry
> expiration when persistent is enabled
> Both fixes prevent node failure in some circumstances, both fixed
>>> already
> merged to master.
> 
> WDYT?
> 
> пн, 27 апр. 2020 г. в 11:53, Nikolay Izhikov >> :
> 
>> Taras.
>> 
>> Thank you, very much!
>> You changes merged to 2.8.1 branch.
>> 
>> Igniters,
>> 
>> We have 10 tickets scheduled for 2.8.1 release:
>> 
>> OPEN:
>> 
>> IGNITE-11687Concurrent WAL replay & log may fail with CRC
>> error
> on
>> read - Dmitriy Govorukhin
>> IGNITE-12346.NET: Platform
>> error:System.NullReferenceException -
>> Pavel Tupitsyn
>> 
>> IN PROGRESS:
>> 
>> IGNITE-12637IgniteSparkSession doesn't start the clients on
> really
>> distributed cluster - Yaroslav Molochkov
>> IGNITE-12788Cluster achieved fully rebalanced (PME-free ready)
>>> state
>> metric - Nikolay Izhikov
>> 
>> PATCH AVAILABLE:
>> 
>> IGNITE-10417notifyDiscoveryListener() call can be lost. -
>> Pavel
>> Voronkin
>> IGNITE-12852Comma in field is not supported by COPY command -
>>> YuJue
 Li
>> IGNITE-12252Unchecked exceptions during rebalancing should be
 handled
>> - Nikolai Kulagin
>> IGNITE-12905QueryKeyValueIterable missing custom spliterator()
>> implementation - Johnny Galatikitis
>> IGNITE-12801Possible extra page release when throttling and
 checkpoint
>> thread store its concurrently - Anton Kalashnikov
>> IGNITE-12794Scan query fails with an assertion error:
>> Unexpected
>>> row
>> key - Denis Mekhanikov
>> 
>> 
>>> 27 апр. 2020 г., в 11:08, Taras Ledkov 
>> написал(а):
>>> 
>>> Hi,
>>> 
>>> Nikolay, i've created PR [1] that contains the SQL-related
>> tickets
> to
>> port into 2.8.1:
>>> 
>>> IGNITE-12790 Introduce distributed SQL configuration and ability
>> to
>> disable SQL functions.
>>> IGNITE-12887 Fix handle type mismatch exception on compare values
>>> while
>> traversing index tree.
>>> IGNITE-12848 fix H2Connection leaks on IN

Re: Apache Ignite 2.8.1 RELEASE [Time, Scope, Manager]

[Ilya Kasnacheev]

Hello!

Please apply https://github.com/apache/ignite/pull/7832

Regards,
-- 
Ilya Kasnacheev


чт, 21 мая 2020 г. в 16:02, Nikolay Izhikov :

> Hello, Ilya.
>
> Actually, I’m uploading RC0 right now.
> Let’s include these fixes in RC1, if you don’t mind?
>
> > 21 мая 2020 г., в 16:00, Ilya Kasnacheev 
> написал(а):
> >
> > Hello!
> >
> > Do you mind if I push additional change to fix English in some of these
> > entries? A lot of these has times not concording with "fixed".
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 20 мая 2020 г. в 15:47, Nikolay Izhikov :
> >
> >> Hello, Igniters.
> >>
> >> I prepared a PR with release notes.
> >> Please, take a look [1]
> >>
> >> [1] https://github.com/apache/ignite/pull/7821
> >>
> >>> 20 мая 2020 г., в 13:06, Nikolay Izhikov 
> >> написал(а):
> >>>
> >>> Hello, Jury.
> >>>
> >>> It seems we reached code freeze step for 2.8.1 release.
> >>> I will prepare rc for it shortly.
> >>>
>  14 мая 2020 г., в 11:08, Юрий 
> написал(а):
> 
>  Nikolay,
> 
>  Release 2.8.1 are delayed and announced dates [1] at release page is
> not
>  actual. Could you update it to reflect current vision of release date?
> 
>  [1].
> >> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8.1
> 
>  чт, 7 мая 2020 г. в 17:23, Nikolay Izhikov :
> 
> > Done.
> >
> >
> >> 7 мая 2020 г., в 13:20, Denis Garus 
> написал(а):
> >>
> >> Nikolay,
> >> could we add the simple improvement [1] to 2.8.1 scope?
> >>
> >> 1. https://issues.apache.org/jira/browse/IGNITE-12983
> >>
> >> чт, 30 апр. 2020 г. в 11:26, Alex Plehanov  >:
> >>
> >>> Nikolay,
> >>>
> >>> TC results: [1], [2]
> >>> I've cherry-picked IGNITE-12933 and IGNITE-12855 to 2.8.1
> >>>
> >>> [1]:
> >>>
> >>>
> >
> >>
> https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull%2F7754%2Fhead&action=Latest&baseBranchForTc=ignite-2.8.1
> >>> [2]:
> >>>
> >>>
> >
> >>
> https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull%2F7755%2Fhead&action=Latest&baseBranchForTc=ignite-2.8.1
> >>>
> >>> вт, 28 апр. 2020 г. в 15:19, Nikolay Izhikov  >:
> >>>
>  Hello, Alex.
> 
>  +1 from me.
> 
> > 28 апр. 2020 г., в 15:03, Alex Plehanov  >
>  написал(а):
> >
> > Hello guys,
> >
> > While we are still waiting for some tickets to resolve I propose
> to
> > cherry-pick to 2.8.1 two more bugfixes:
> > IGNITE-12933 Fixed node failure after put incorrect key class for
> > cache
> > with indexed types
> > IGNITE-12855 Fixed node failure with concurrent get operation and
> > entry
> > expiration when persistent is enabled
> > Both fixes prevent node failure in some circumstances, both fixed
> >>> already
> > merged to master.
> >
> > WDYT?
> >
> > пн, 27 апр. 2020 г. в 11:53, Nikolay Izhikov <
> nizhi...@apache.org
> >>> :
> >
> >> Taras.
> >>
> >> Thank you, very much!
> >> You changes merged to 2.8.1 branch.
> >>
> >> Igniters,
> >>
> >> We have 10 tickets scheduled for 2.8.1 release:
> >>
> >> OPEN:
> >>
> >> IGNITE-11687Concurrent WAL replay & log may fail with CRC
> >> error
> > on
> >> read - Dmitriy Govorukhin
> >> IGNITE-12346.NET: Platform
> >> error:System.NullReferenceException -
> >> Pavel Tupitsyn
> >>
> >> IN PROGRESS:
> >>
> >> IGNITE-12637IgniteSparkSession doesn't start the clients on
> > really
> >> distributed cluster - Yaroslav Molochkov
> >> IGNITE-12788Cluster achieved fully rebalanced (PME-free
> ready)
> >>> state
> >> metric - Nikolay Izhikov
> >>
> >> PATCH AVAILABLE:
> >>
> >> IGNITE-10417notifyDiscoveryListener() call can be lost. -
> >> Pavel
> >> Voronkin
> >> IGNITE-12852Comma in field is not supported by COPY command
> -
> >>> YuJue
>  Li
> >> IGNITE-12252Unchecked exceptions during rebalancing should
> be
>  handled
> >> - Nikolai Kulagin
> >> IGNITE-12905QueryKeyValueIterable missing custom
> spliterator()
> >> implementation - Johnny Galatikitis
> >> IGNITE-12801Possible extra page release when throttling and
>  checkpoint
> >> thread store its concurrently - Anton Kalashnikov
> >> IGNITE-12794Scan query fails with an assertion error:
> >> Unexpected
> >>> row
> >> key - Denis Mekhanikov
> >>
> >>
> >>> 27 апр. 2020 г., в 11:08, Taras Ledkov 
> >> написал(а):
> >>>
> >>>

Re: Apache Ignite 2.8.1 RELEASE [Time, Scope, Manager]

[Nikolay Izhikov]

Done, thanks.

> 21 мая 2020 г., в 16:34, Ilya Kasnacheev  
> написал(а):
> 
> Hello!
> 
> Please apply https://github.com/apache/ignite/pull/7832
> 
> Regards,
> -- 
> Ilya Kasnacheev
> 
> 
> чт, 21 мая 2020 г. в 16:02, Nikolay Izhikov :
> 
>> Hello, Ilya.
>> 
>> Actually, I’m uploading RC0 right now.
>> Let’s include these fixes in RC1, if you don’t mind?
>> 
>>> 21 мая 2020 г., в 16:00, Ilya Kasnacheev 
>> написал(а):
>>> 
>>> Hello!
>>> 
>>> Do you mind if I push additional change to fix English in some of these
>>> entries? A lot of these has times not concording with "fixed".
>>> 
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>> 
>>> 
>>> ср, 20 мая 2020 г. в 15:47, Nikolay Izhikov :
>>> 
 Hello, Igniters.
 
 I prepared a PR with release notes.
 Please, take a look [1]
 
 [1] https://github.com/apache/ignite/pull/7821
 
> 20 мая 2020 г., в 13:06, Nikolay Izhikov 
 написал(а):
> 
> Hello, Jury.
> 
> It seems we reached code freeze step for 2.8.1 release.
> I will prepare rc for it shortly.
> 
>> 14 мая 2020 г., в 11:08, Юрий 
>> написал(а):
>> 
>> Nikolay,
>> 
>> Release 2.8.1 are delayed and announced dates [1] at release page is
>> not
>> actual. Could you update it to reflect current vision of release date?
>> 
>> [1].
 https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8.1
>> 
>> чт, 7 мая 2020 г. в 17:23, Nikolay Izhikov :
>> 
>>> Done.
>>> 
>>> 
 7 мая 2020 г., в 13:20, Denis Garus 
>> написал(а):
 
 Nikolay,
 could we add the simple improvement [1] to 2.8.1 scope?
 
 1. https://issues.apache.org/jira/browse/IGNITE-12983
 
 чт, 30 апр. 2020 г. в 11:26, Alex Plehanov >> :
 
> Nikolay,
> 
> TC results: [1], [2]
> I've cherry-picked IGNITE-12933 and IGNITE-12855 to 2.8.1
> 
> [1]:
> 
> 
>>> 
 
>> https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull%2F7754%2Fhead&action=Latest&baseBranchForTc=ignite-2.8.1
> [2]:
> 
> 
>>> 
 
>> https://mtcga.gridgain.com/pr.html?serverId=apache&suiteId=IgniteTests24Java8_RunAll&branchForTc=pull%2F7755%2Fhead&action=Latest&baseBranchForTc=ignite-2.8.1
> 
> вт, 28 апр. 2020 г. в 15:19, Nikolay Izhikov >> :
> 
>> Hello, Alex.
>> 
>> +1 from me.
>> 
>>> 28 апр. 2020 г., в 15:03, Alex Plehanov >> 
>> написал(а):
>>> 
>>> Hello guys,
>>> 
>>> While we are still waiting for some tickets to resolve I propose
>> to
>>> cherry-pick to 2.8.1 two more bugfixes:
>>> IGNITE-12933 Fixed node failure after put incorrect key class for
>>> cache
>>> with indexed types
>>> IGNITE-12855 Fixed node failure with concurrent get operation and
>>> entry
>>> expiration when persistent is enabled
>>> Both fixes prevent node failure in some circumstances, both fixed
> already
>>> merged to master.
>>> 
>>> WDYT?
>>> 
>>> пн, 27 апр. 2020 г. в 11:53, Nikolay Izhikov <
>> nizhi...@apache.org
> :
>>> 
 Taras.
 
 Thank you, very much!
 You changes merged to 2.8.1 branch.
 
 Igniters,
 
 We have 10 tickets scheduled for 2.8.1 release:
 
 OPEN:
 
 IGNITE-11687Concurrent WAL replay & log may fail with CRC
 error
>>> on
 read - Dmitriy Govorukhin
 IGNITE-12346.NET: Platform
 error:System.NullReferenceException -
 Pavel Tupitsyn
 
 IN PROGRESS:
 
 IGNITE-12637IgniteSparkSession doesn't start the clients on
>>> really
 distributed cluster - Yaroslav Molochkov
 IGNITE-12788Cluster achieved fully rebalanced (PME-free
>> ready)
> state
 metric - Nikolay Izhikov
 
 PATCH AVAILABLE:
 
 IGNITE-10417notifyDiscoveryListener() call can be lost. -
 Pavel
 Voronkin
 IGNITE-12852Comma in field is not supported by COPY command
>> -
> YuJue
>> Li
 IGNITE-12252Unchecked exceptions during rebalancing should
>> be
>> handled
 - Nikolai Kulagin
 IGNITE-12905QueryKeyValueIterable missing custom
>> spliterator()
 implementation - Johnny Galatikitis
 IGNITE-12801Possible extra page release when throttling and
>> checkpoint
 thread store its concurrently - Anton Kalashnikov
 IGNITE-12794Scan query fails with an assertion error:
 Unexpected
>>

Re: [DISCUSSION] Key and Value fields with same name and SQL DML

[Alexey Sasov]

Hello, Denis. This is the analogy of DB records. We've got keys in the table.
But querying the record with the key we also have this key in the returned
record. The question of having all data in the value is a domain driven one.
If i have a model (ex. Person) containing PassportNo, I don't want to change
the model just to store it in cache. A lot of outer code uses these classes,
so i have to either create an independent model layer to store it in cache,
or copy data from keys to values before returning object to callers.

Thanks. 



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: [DISCUSSION] Key and Value fields with same name and SQL DML

[Alexey Kukushkin]

I vote for implementing this enhancement: having key information separated
from value means either cluttering domain model with
KeyValue/CacheEntry-like structures or developing an Ignite data access
layer to insert key info into domain entity after getting it from Ignite. 

Although both the options solve the issue, I think the right approach is to
address it in Ignite rather than make all the Ignite users repeatedly deal
with the issue. It is the Ignite which is a re-usable platform and not
vise-versa.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[jira] [Created] (IGNITE-13051) Optimized affinity switch on baseline node left is broken for client topology

[Alexey Scherbakov (Jira)]

Alexey Scherbakov created IGNITE-13051:
--

 Summary: Optimized affinity switch on baseline node left is broken 
for client topology
 Key: IGNITE-13051
 URL: https://issues.apache.org/jira/browse/IGNITE-13051
 Project: Ignite
  Issue Type: Bug
Reporter: Alexey Scherbakov
 Fix For: 2.9






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13052) Calculate result of reserveHistoryForExchange in advance

[Ivan Rakov (Jira)]

Ivan Rakov created IGNITE-13052:
---

 Summary: Calculate result of reserveHistoryForExchange in advance
 Key: IGNITE-13052
 URL: https://issues.apache.org/jira/browse/IGNITE-13052
 Project: Ignite
  Issue Type: Improvement
Reporter: Ivan Rakov


Method reserveHistoryForExchange() is called on every partition map exchange. 
It's an expensive call: it requires iteration over the whole checkpoint history 
with possible retrieve of GroupState from WAL (it's stored on heap with 
SoftReference). On some deployments this operation can take several minutes.

The idea of optimization is to calculate it's result only on first PME 
(ideally, even before first PME, on recovery stage), keep resulting map {grpId, 
partId -> earlisetCheckpoint} on heap and update it if necessary. From the 
first glance, map should be updated:
1) On checkpoint. If a new partition appears on local node, it should be 
registered in the map with current checkpoint. If a partition is evicted from 
local node, or changed its state to non-OWNING, it should removed from the map. 
If checkpoint is marked as inapplicable for a certain group, the whole group 
should be removed from the map.
2) On checkpoint history cleanup. For every (grpId, partId), previous earliest 
checkpoint should be changed with setIfGreater to new earliest checkpoint.

Memory overhead of storing described map on heap in significant. It's size 
isn't greater than size of map returned from reserveHistoryForExchange().

Described fix should be much simpler than IGNITE-12429.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13053) resetLostPartitions is not working if invoked from a node where a cache not started

[Alexey Scherbakov (Jira)]

Alexey Scherbakov created IGNITE-13053:
--

 Summary: resetLostPartitions is not working if invoked from a node 
where a cache not started
 Key: IGNITE-13053
 URL: https://issues.apache.org/jira/browse/IGNITE-13053
 Project: Ignite
  Issue Type: Bug
Reporter: Alexey Scherbakov
 Fix For: 2.9






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Apache Ignite mascot

[Denis Magda]

Folks, thanks for sharing ideas.

Let me find a designer who can turn the ideas into pictures. Then we can
compare and adjust.

-
Denis


On Sun, May 17, 2020 at 5:50 AM Manuel Núñez  wrote:

> What about a hellboy’s style animal?, with Ignite’s flame between the horns
>
> El 16 may 2020, a las 23:22, Saikat Maitra 
> escribió:
>
> 
> Hi Denis,
>
> Awesome ideas !!! yes, I am aligned on a friendly dragon character as a
> mascot, like Spyro. I also like the idea of peacock as its feathers colors
> shows a nice clusters.
>
> Here are few proposals I wanted to share:
>
> 1. Dragon (fire, spark, ignite)
> 2. Peacock (feathers, node clusters)
> 3. Cheetah (speed, fast)
> 4. Beluga Whales (echolocation, multicasting)
>
> Let me know if you have more suggestions.
>
> Regards,
> Saikat
>
>
>
>
> On Sun, May 10, 2020 at 3:16 AM Denis Magda  wrote:
>
>> Hi Saikat,
>>
>> I love this idea and, frankly, have been contemplating for a while about
>> the same. Such characters (aka. mascots) can convey the spirit of our
>> project and community as well as make life more entertaining.
>>
>> I bet that every Java developer instantly associates Java (as a project
>> and
>> community) once he/she comes accross Java Duke [1]. Those of us who live
>> in
>> the U.S. know who Geigo Gecko [1] is. What to say about Ronald McDonald
>> [3]
>> is famous worldwide.
>>
>> How do you see the Ignite's mascot? I imagine it as a dragon who is kind
>> and friendly rather than furious and hostile. Our logo is a spark of fire
>> and dragons are those whimsical creatures who are usually associated with
>> fire and flames.
>>
>> [1] 568px-Duke_(Java_mascot)_waving.svg.png
>> <
>> https://upload.wikimedia.org/wikipedia/commons/thumb/5/5d/Duke_%28Java_mascot%29_waving.svg/568px-Duke_%28Java_mascot%29_waving.svg.png
>> >
>> [2] maxresdefault.jpg <
>> https://i.ytimg.com/vi/8lZeTTXyIk4/maxresdefault.jpg>
>> [3] ronald-mcdonald-052011.jpg
>> <
>> https://s3-prod.adage.com/s3fs-public/styles/800x600/public/ronald-mcdonald-052011.jpg
>> >
>>
>> -
>> Denis
>>
>>
>> On Sat, May 9, 2020 at 1:07 PM Saikat Maitra 
>> wrote:
>>
>> > Hi,
>> >
>> > I wanted to discuss the idea of creating Apache Ignite mascot and if it
>> is
>> > something we would be interested. I was recently following the process
>> of
>> > creating a mascot  Firefly[1] in Apache Beam project and absolutely
>> loved
>> > the concept.
>> >
>> >
>> https://blogs.apache.org/foundation/entry/success-at-apache-bringing-the
>> >
>> > Regards,
>> > Saikat
>> >
>>
>

[jira] [Created] (IGNITE-13054) Prevent nodes with stale data joining the active topology.

[Alexey Scherbakov (Jira)]

Alexey Scherbakov created IGNITE-13054:
--

 Summary: Prevent nodes with stale data joining the active topology.
 Key: IGNITE-13054
 URL: https://issues.apache.org/jira/browse/IGNITE-13054
 Project: Ignite
  Issue Type: Bug
Reporter: Alexey Scherbakov
 Fix For: 2.9


After IGNITE-13003 LOST state is preserved then nodes with lost data are 
retuned to topology after failure.

If resetting is performed on lesser topology, it's possible to get into 
sutiation where a node with most recent data is returned to active topology 
where some key already could be modified.

This should not be allowed because brings conflicting data into grid. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSSION] Key and Value fields with same name and SQL DML

[Denis Magda]

Folks, thanks for the details. I do acknowledge that this awkward usability
issue is quite annoying and complicates applications' architecture. Problem
with Spring Data

At the moment, I use the workaround mentioned by Alexey Kukushkin. Guess
many of us already use it in productions:

   - PersonKey and PersonValue POJOs are used by Ignite repository
   abstractions that put or read data to/from Ignite.  Those POJOs don't have
   repetitive fields.
   - Person POJO class is a high-level model that has all the fields of the
   table (key + value). PersonKey and PersonValue are converted to a Person
   instance by a repository abstraction and that instance is used by other
   high-level abstractions of my applications (services, controller, etc.).

Personally, the proposed solution for the problem still sounds complicated
to me. It still requires me, as an application developer, to do some extra
settings in the CacheConfiguration and CREATE TABLE statements.

Let me know what you think about an alternative approach, I've been
contemplating for a while. Those of us who use JPA, Spring Data or
Hibernate frameworks are aware of the Entity abstraction. Which is
basically a POJO class of your business domain model that goes with all the
fields defined in an associated relational/NoSQL table. You use special
annotations to define indexes, primary keys and other relations in that
Entity class. My thinking was to upgrade Ignite cache APIs to the state
when we no longer require to create key POJOs. Instead, you create a single
POJO/Entity, annotate its primary key and pass that object to the cache API
or get it back for read operations.

As an example, let's assume we have this POJO:

@IgnitePrimaryKey(fields = {"id", "passportNo"}, affinityKey="id")
> class Person {
> int id;
> String passportNo;
> String name;
> }


(the primary key fields must always be annotated so that Ignite could
extract those fields internally).

Next, if you want to write an object instance into the cache you do this:

Person personInstance = new Person(5, "23324234", "Thomas");
> ignite.cache("Person").put(personInstance);


To read it back, you just do this

Person personInstance = ignite.cache("Person").get(5, "23324234"); //the
> method can accept an arbitrary number of fields that define a primary key).



The idea is rough and not finalized but I would like us to explore the APIs
evolution in this direction.

-
Denis


On Thu, May 21, 2020 at 7:19 AM Alexey Kukushkin 
wrote:

> I vote for implementing this enhancement: having key information separated
> from value means either cluttering domain model with
> KeyValue/CacheEntry-like structures or developing an Ignite data access
> layer to insert key info into domain entity after getting it from Ignite.
>
> Although both the options solve the issue, I think the right approach is to
> address it in Ignite rather than make all the Ignite users repeatedly deal
> with the issue. It is the Ignite which is a re-usable platform and not
> vise-versa.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

[jira] [Created] (IGNITE-13055) ExecutorServices must handle AssertionErrors and log uncaught exceptions

[Maxim Muzafarov (Jira)]

Maxim Muzafarov created IGNITE-13055:


 Summary: ExecutorServices must handle AssertionErrors and log 
uncaught exceptions
 Key: IGNITE-13055
 URL: https://issues.apache.org/jira/browse/IGNITE-13055
 Project: Ignite
  Issue Type: Improvement
Reporter: Maxim Muzafarov


Currently, most of ExecuterServices (e.g. {{sysExecSvc}}) handles only 
OutOfMemoryException. Services must also handle AssertionErrors and log any 
uncaught exceptions.

The most important list of executor services configured with OOM handler only:
- execSvc
- svcExecSvc
- sysExecSvc
- p2pExecSvc
- restExecSvc
- utilityCacheExecSvc
- affExecSvc
- qryExecSvc

See the dev-list discussion for details.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: ExecutorServices hide assertions without logging and node stop

[Maxim Muzafarov]

Folks,

I've created an issue.
https://issues.apache.org/jira/browse/IGNITE-13055

On Wed, 6 May 2020 at 10:28, Nikolay Izhikov  wrote:
>
> Hello, Maxim.
>
> I can confirm this itching issue.
> It also happens when some custom Security plugin throws an exception while 
> processing a communication message.
>
> ```
> UUID newSecSubjId = secSubjId != null ? secSubjId : nodeId;
>
> try (OperationSecurityContext s = ctx.security().withContext(newSecSubjId)) {
> lsnr.onMessage(nodeId, msg, plc);
> }
> finally {
> if (change)
> CUR_PLC.set(oldPlc);
> }
> ```
>
> If an exception thrown from `withContext` it is hidden from the user.
>
> > 4 мая 2020 г., в 19:15, Maxim Muzafarov  написал(а):
> >
> > Igniters,
> >
> >
> > I've spent a couple of days in debug mode and found that some of the
> > configured ExecutorServices of an Ignite instance completely hide
> > assertion errors without any logging and node killing. This may
> > violate some internal guarantees and hide serious errors.
> >
> > Let's consider, for instance, GridDhtPartitionsExchangeFuture [1]. It
> > has three places of submitting some Runnable on system executor
> > service. If an assertion error (or even any uncached exception) occurs
> > in the code block below it will be swallowed without logging, exchange
> > future completion or node stoping.
> >
> > cctx.kernalContext().getSystemExecutorService().submit(new Runnable() {
> >@Override public void run() {
> >sendPartitions(newCrd);
> >}
> > });
> >
> > I've checked that these executor services and most of them configured
> > to catch only OutOfMemoryError.
> >
> > Should we consider catching AssertionErrors as well and treat them as
> > CRITICAL_ERRORS for the Failure Handler?
> > Should we log uncached errors on each of them?
> >
> >
> > The most important list of executor services configured with OOM handler 
> > only:
> > execSvc
> > svcExecSvc
> > sysExecSvc
> > p2pExecSvc
> > restExecSvc
> > utilityCacheExecSvc
> > affExecSvc
> > qryExecSvc
> >
> > [1] 
> > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPartitionsExchangeFuture.java#L4848
>

Re: [DISCUSSION] Ignite integration testing framework.

[Denis Magda]

Hi Nikolay,

Thanks for kicking off this conversation and sharing your findings with the
results. That's the right initiative. I do agree that Ignite needs to have
an integration testing framework with capabilities listed by you.

As we discussed privately, I would only check if instead of
Confluent's Ducktape library, we can use an integration testing framework
developed by GridGain for testing of Ignite/GridGain clusters. That
framework has been battle-tested and might be more convenient for
Ignite-specific workloads. Let's wait for @Maksim Shonichev
 who promised to join this thread once he finishes
preparing the usage examples of the framework. To my knowledge, Max has
already been working on that for several days.

-
Denis


On Thu, May 21, 2020 at 12:27 AM Nikolay Izhikov 
wrote:

> Hello, Igniters.
>
> I created a PoC [1] for the integration tests of Ignite.
>
> Let me briefly explain the gap I want to cover:
>
> 1. For now, we don’t have a solution for automated testing of Ignite on
> «real cluster».
> By «real cluster» I mean cluster «like a production»:
> * client and server nodes deployed on different hosts.
> * thin clients perform queries from some other hosts
> * etc.
>
> 2. We don’t have a solution for automated benchmarks of some internal
> Ignite process
> * PME
> * rebalance.
> This means we don’t know - Do we perform rebalance(or PME) in 2.7.0 faster
> or slower than in 2.8.0 for the same cluster?
>
> 3. We don’t have a solution for automated testing of Ignite integration in
> a real-world environment:
> Ignite-Spark integration can be taken as an example.
> I think some ML solutions also should be tested in real-world deployments.
>
> Solution:
>
> I propose to use duck tape library from confluent (apache 2.0 license)
> I tested it both on the real cluster(Yandex Cloud) and on the local
> environment(docker) and it works just fine.
>
> PoC contains following services:
>
> * Simple rebalance test:
> Start 2 server nodes,
> Create some data with Ignite client,
> Start one more server node,
> Wait for rebalance finish
> * Simple Ignite-Spark integration test:
> Start 1 Spark master, start 1 Spark worker,
> Start 1 Ignite server node
> Create some data with Ignite client,
> Check data in application that queries it from Spark.
>
> All tests are fully automated.
> Logs collection works just fine.
> You can see an example of the tests report - [4].
>
> Pros:
>
> * Ability to test local changes(no need to public changes to some remote
> repository or similar).
> * Ability to parametrize test environment(run the same tests on different
> JDK, JVM params, config, etc.)
> * Isolation by default so system tests are as reliable as possible.
> * Utilities for pulling up and tearing down services easily in clusters in
> different environments (e.g. local, custom cluster, Vagrant, K8s, Mesos,
> Docker, cloud providers, etc.)
> * Easy to write unit tests for distributed systems
> * Adopted and successfully used by other distributed open source project -
> Apache Kafka.
> * Collect results (e.g. logs, console output)
> * Report results (e.g. expected conditions met, performance results, etc.)
>
> WDYT?
>
> [1] https://github.com/nizhikov/ignite/pull/15
> [2] https://github.com/confluentinc/ducktape
> [3] https://ducktape-docs.readthedocs.io/en/latest/run_tests.html
> [4] https://yadi.sk/d/JC8ciJZjrkdndg

Re: [DISCUSSION] Ignite integration testing framework.

[Nikolay Izhikov]

Hello, Denis.

There is no rush with these improvements.
We can wait for Maxim proposal and compare two solutions :)

> 21 мая 2020 г., в 22:24, Denis Magda  написал(а):
> 
> Hi Nikolay,
> 
> Thanks for kicking off this conversation and sharing your findings with the
> results. That's the right initiative. I do agree that Ignite needs to have
> an integration testing framework with capabilities listed by you.
> 
> As we discussed privately, I would only check if instead of
> Confluent's Ducktape library, we can use an integration testing framework
> developed by GridGain for testing of Ignite/GridGain clusters. That
> framework has been battle-tested and might be more convenient for
> Ignite-specific workloads. Let's wait for @Maksim Shonichev
>  who promised to join this thread once he finishes
> preparing the usage examples of the framework. To my knowledge, Max has
> already been working on that for several days.
> 
> -
> Denis
> 
> 
> On Thu, May 21, 2020 at 12:27 AM Nikolay Izhikov 
> wrote:
> 
>> Hello, Igniters.
>> 
>> I created a PoC [1] for the integration tests of Ignite.
>> 
>> Let me briefly explain the gap I want to cover:
>> 
>> 1. For now, we don’t have a solution for automated testing of Ignite on
>> «real cluster».
>> By «real cluster» I mean cluster «like a production»:
>>* client and server nodes deployed on different hosts.
>>* thin clients perform queries from some other hosts
>>* etc.
>> 
>> 2. We don’t have a solution for automated benchmarks of some internal
>> Ignite process
>>* PME
>>* rebalance.
>> This means we don’t know - Do we perform rebalance(or PME) in 2.7.0 faster
>> or slower than in 2.8.0 for the same cluster?
>> 
>> 3. We don’t have a solution for automated testing of Ignite integration in
>> a real-world environment:
>> Ignite-Spark integration can be taken as an example.
>> I think some ML solutions also should be tested in real-world deployments.
>> 
>> Solution:
>> 
>> I propose to use duck tape library from confluent (apache 2.0 license)
>> I tested it both on the real cluster(Yandex Cloud) and on the local
>> environment(docker) and it works just fine.
>> 
>> PoC contains following services:
>> 
>>* Simple rebalance test:
>>Start 2 server nodes,
>>Create some data with Ignite client,
>>Start one more server node,
>>Wait for rebalance finish
>>* Simple Ignite-Spark integration test:
>>Start 1 Spark master, start 1 Spark worker,
>>Start 1 Ignite server node
>>Create some data with Ignite client,
>>Check data in application that queries it from Spark.
>> 
>> All tests are fully automated.
>> Logs collection works just fine.
>> You can see an example of the tests report - [4].
>> 
>> Pros:
>> 
>> * Ability to test local changes(no need to public changes to some remote
>> repository or similar).
>> * Ability to parametrize test environment(run the same tests on different
>> JDK, JVM params, config, etc.)
>> * Isolation by default so system tests are as reliable as possible.
>> * Utilities for pulling up and tearing down services easily in clusters in
>> different environments (e.g. local, custom cluster, Vagrant, K8s, Mesos,
>> Docker, cloud providers, etc.)
>> * Easy to write unit tests for distributed systems
>> * Adopted and successfully used by other distributed open source project -
>> Apache Kafka.
>> * Collect results (e.g. logs, console output)
>> * Report results (e.g. expected conditions met, performance results, etc.)
>> 
>> WDYT?
>> 
>> [1] https://github.com/nizhikov/ignite/pull/15
>> [2] https://github.com/confluentinc/ducktape
>> [3] https://ducktape-docs.readthedocs.io/en/latest/run_tests.html
>> [4] https://yadi.sk/d/JC8ciJZjrkdndg

Re: [DISCUSSION] Key and Value fields with same name and SQL DML

[Denis Magda]

>
> I do acknowledge that this awkward usability issue is quite annoying and
> complicates applications' architecture. Problem with Spring Data


Sorry, sent the message with the first paragraph left unfinished. "Problem
with Spring Data" - just wanted to highlight that the current separation on
key and value POJOs leads to other clumsy usability issues:
https://issues.apache.org/jira/browse/IGNITE-13028

-
Denis


On Thu, May 21, 2020 at 12:03 PM Denis Magda  wrote:

> Folks, thanks for the details. I do acknowledge that this awkward
> usability issue is quite annoying and complicates applications'
> architecture. Problem with Spring Data
>
> At the moment, I use the workaround mentioned by Alexey Kukushkin. Guess
> many of us already use it in productions:
>
>- PersonKey and PersonValue POJOs are used by Ignite repository
>abstractions that put or read data to/from Ignite.  Those POJOs don't have
>repetitive fields.
>- Person POJO class is a high-level model that has all the fields of
>the table (key + value). PersonKey and PersonValue are converted to a
>Person instance by a repository abstraction and that instance is used by
>other high-level abstractions of my applications (services, controller,
>etc.).
>
> Personally, the proposed solution for the problem still sounds complicated
> to me. It still requires me, as an application developer, to do some extra
> settings in the CacheConfiguration and CREATE TABLE statements.
>
> Let me know what you think about an alternative approach, I've been
> contemplating for a while. Those of us who use JPA, Spring Data or
> Hibernate frameworks are aware of the Entity abstraction. Which is
> basically a POJO class of your business domain model that goes with all the
> fields defined in an associated relational/NoSQL table. You use special
> annotations to define indexes, primary keys and other relations in that
> Entity class. My thinking was to upgrade Ignite cache APIs to the state
> when we no longer require to create key POJOs. Instead, you create a single
> POJO/Entity, annotate its primary key and pass that object to the cache API
> or get it back for read operations.
>
> As an example, let's assume we have this POJO:
>
> @IgnitePrimaryKey(fields = {"id", "passportNo"}, affinityKey="id")
>> class Person {
>> int id;
>> String passportNo;
>> String name;
>> }
>
>
> (the primary key fields must always be annotated so that Ignite could
> extract those fields internally).
>
> Next, if you want to write an object instance into the cache you do this:
>
> Person personInstance = new Person(5, "23324234", "Thomas");
>> ignite.cache("Person").put(personInstance);
>
>
> To read it back, you just do this
>
> Person personInstance = ignite.cache("Person").get(5, "23324234"); //the
>> method can accept an arbitrary number of fields that define a primary key).
>
>
>
> The idea is rough and not finalized but I would like us to explore the
> APIs evolution in this direction.
>
> -
> Denis
>
>
> On Thu, May 21, 2020 at 7:19 AM Alexey Kukushkin <
> kukushkinale...@gmail.com> wrote:
>
>> I vote for implementing this enhancement: having key information separated
>> from value means either cluttering domain model with
>> KeyValue/CacheEntry-like structures or developing an Ignite data access
>> layer to insert key info into domain entity after getting it from Ignite.
>>
>> Although both the options solve the issue, I think the right approach is
>> to
>> address it in Ignite rather than make all the Ignite users repeatedly deal
>> with the issue. It is the Ignite which is a re-usable platform and not
>> vise-versa.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>>
>

[jira] [Created] (IGNITE-13056) SchemaManager refactoring

[Nikolay Izhikov (Jira)]

Nikolay Izhikov created IGNITE-13056:


 Summary: SchemaManager refactoring
 Key: IGNITE-13056
 URL: https://issues.apache.org/jira/browse/IGNITE-13056
 Project: Ignite
  Issue Type: New Feature
  Components: sql
Affects Versions: 2.8
Reporter: Nikolay Izhikov
Assignee: Nikolay Izhikov


Since Ignite wants to leverage from several SQL engines we need to make 
SchemaManager independent from H2.

We also should consider moving it to the core module to make it available from 
any module that wants to use it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Request for contributors permissions

[steve.hostett...@gmail.com]

Thanks a lot



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: [DISCUSSION] New Ignite settings for IGNITE-12438 and IGNITE-13013

[Denis Magda]

Ivan,

Considering that the setting controls the way a communication SPI
connection is open I would add the new parameter to CommunicationSpi
interface naming it as follows:

>
> CommunicationSpi.connectionInitiationMode
> {
> BIDIRECTIONAL, //both clients and servers initiate a connection
> initiation procedure
> CLIENTS_TO_SERVERS //servers cannot open a connection to clients, only
> clients can do that
> }


The problem with the environment type approach is that private networks of
bare-metal environments usually impose restrictions similar to virtual
environments powered by Kubernetes. Thus, environmentType.VIRTUALIZED
doesn't cover all the cases and I'm struggling to come up with a universal
alternative.

-
Denis


On Thu, May 21, 2020 at 5:38 AM Ivan Bessonov  wrote:

> Hello Igniters,
>
> I'd like to discuss with you changes related to [1] and [2]. Both issues
> are mostly the same so
> let's discuss the core idea.
>
> *Motivation.*
>
> There are certain environments that don't allow Ignite server nodes to open
> TCP connections to
> thick clients, e.g. K8S, AWS Lambda or Azure Functions. To operate in such
> environments, the
> server needs a way to request a client to open an "inverse" communication
> connection to it.
>
> I've prepared a PR (still in progress) that introduces new mechanism of
> opening connection and
> related configuration.
>
> *Main idea*
>
> This mechanism is called "communication via discovery" or "inverse
> connection", it works as
> follows:
>  - server that needs to connect to "unreachable" thick client sends a
> specific Discovery message
>(inverse communication request) to that client;
>  - client node upon receiving the request opens communication connection to
> that server;
>  - server sees connection opened by client and proceeds with its task (that
> required opening
>connection to the client).
>
> Working name for new configuration parameter for this feature is
> environmentType, it is an
> enum with two values (again, working names): STANDALONE (default) and
> VIRTUALIZED.
> It is used as a hint to server to speed-up establishing of connections:
> when server sees a client
> with VIRTUALIZED environmentType it doesn't try to open connection to it
> and sends inverse
> communication request right away.
> If environmentType is STANDALONE then server tries to open a connection in
> a regular way
> (iterating over all client addresses) and sends request only if all
> attempts failed.
>
> There is a concern about naming of the configuration as it catches only one
> use-case: when we
> deal with some kind of virtualized environment like K8S.
> There are other options I've encountered in private discussion:
> - connectionMode - ALWAYS_INITIATE / INITIATE_OR_ACCEPT
> - networkConnectivity - REACHABLE_ALWAYS / REACHABLE_ONE_WAY
> - communicationViaDiscovery - ALWAYS / FALLBACK
> - isReachableFromAllNodes (true/false flag)
> - initiateConnectionsOnThisNode (true/false flag).
>
> *Limitations*
>
> The feature cannot be used along with pairedConnection setting as this
> setting implies
> establishing connections in both directions. Also current implementation
> supports opening only
> client-to-server connections. Other types of connections like
> client-to-client or server-to-server
> will be implemented in separate tickets.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12438
> [2] https://issues.apache.org/jira/browse/IGNITE-13013
>
> --
> Sincerely yours,
> Ivan Bessonov
>

Re: [DISCUSSION] Key and Value fields with same name and SQL DML

[Alexey Kukushkin]

Denis,I like your idea. It would be like native "OKVM" (Object-KeyValue
Mapper) allowing Ignite to natively operate on domain entities. It also
means no key and value data duplication and thus addressing the original SQL
DML problem when only key field is modified.We could add a
configuration-based alternative to the AOP-style API you suggested. Of all
languages that Ignite supports I think only Java and C# have AOP. We need to
think about all languages when we add a new feature. I think your solution
does not completely overlap with the solution suggested in  IGNITE-12807
  : - You suggested a
new API that does not have a "Key" entity by definition. A huge advantage of
the existing Key/Value is it is a widely adopted JCache standard supported
by all Ignite competitors I am aware of. That means low cost of switching to
Ignite for users of other product and more confidence for existing Ignite
users. IGNITE-12807 suggests solving the problem with the Key/Value API. -
Implementing your solution is probably a significantly bigger effort than
what IGNITE-12807 suggests. IGNITE-12807 already has a patch with a
proof-of-concept implemented.Still, I think the urge for addressing it in
K/V API would be less if we have your kind of API in Java and C#.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[MTCGA]: new failures in builds [5324265] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *New test failure in master 
IgniteProjectionStartStopRestartSelfTest.testStartFiveWithTwoSpecs 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=-6854063062309244670&branch=%3Cdefault%3E&tab=testDetails
 Changes may lead to failure were done by 
 - pavel tupitsyn  
https://ci.ignite.apache.org/viewModification.html?modId=902045
 - nikolay izhikov  
https://ci.ignite.apache.org/viewModification.html?modId=902079
 - nikolay  
https://ci.ignite.apache.org/viewModification.html?modId=902072

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 01:16:33 22-05-2020 

Re: Using GraalVM instead of standard JVM

[Denis Magda]

Folks,

I've looked into GraalVM and would like to share some observations and
suggestions:

1. GridGain nodes running on Graal can execute native Node.JS, Python, LLVM
(C/C++) code. This feature will let Ignite developers who are not using
Java as their primary programming language, deploy their custom non-Java
compute tasks to the server nodes and execute them via the Ignite compute
APIs. I see a little value of adding the compute APIs to Node/Python thin
clients if developers need to code compute tasks in Java (the scope of
IEP-42
).
With Graal, this inconvenience goes away if our internal compute
implementation can detect that a node is running on Graal and load/execute
the code of a guest language:
https://www.graalvm.org/docs/reference-manual/embed/

2. Speaking of the compute tasks (re)deployment briefly touched here, we
can enhance our command-line tool to take care of this task. See how custom
tasks of an arbitrary language can be deployed on Oracle Database and
execute: https://www.graalvm.org/docs/examples/mle-oracle/

3. Presently, Ignite compute APIs are the only interface for the tasks'
execution. At the same time, we can treat a deployed compute task as an
advanced stored procedure and call it from SQL. Scroll to the bottom of the
same Oracle DB example: https://www.graalvm.org/docs/examples/mle-oracle/

Should we move forward creating an IEP for Graal which goal is to enable
the capabilities listed above?

@Pavel Tupitsyn , @Igor Sapego ,
@plehanov.a...@gmail.com , it would be nice to
hear your perspective on 1 and 2. @Igor Seliverstov
, @Taras Ledkov , @Roman
Kondakov , @Ivan Pavlukhin
, @Yuriy
Gerzhedovich  what are your thoughts on #3?


-
Denis


On Wed, May 6, 2020 at 1:07 PM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> I just switched out OpenJDK and used GraalVM instead. Everything seemed to
> work but I wasn’t looking terribly hard. We’d need to do some more QA but I
> think chances are good that it’ll work just fine.
>
> Regards,
> Stephen
>
> >> On 6 May 2020, at 20:31, Denis Magda  wrote:
> > Stephen, that's terrific! To Ivan's first question, did you just swap
> > HotSpot with GraalVM and got the thing working? Or did it require some
> > extra work?
> >
> > -
> > Denis
> >
> >
> >> On Wed, May 6, 2020 at 10:10 AM Stephen Darlington <
> >> stephen.darling...@gridgain.com> wrote:
> >>
> >> I’ve been playing around with it. I’ve was really impressed that I could
> >> run JavaScript on Ignite with comparatively little code:
> >>
> >> https://github.com/sdarlington/ignite-graalvm <
> >> https://github.com/sdarlington/ignite-graalvm>
> >>
> >> I’ve not been looking at performance, though.
> >>
> >> Regards,
> >> Stephen
> >>
> > On 6 May 2020, at 17:52, Denis Magda  wrote:
> >>> I'll leave this reference here so that we have a better understanding
> of
> >>> why it's worthwhile to support GraalVM:
> >>> https://blogs.oracle.com/graalvm/apache-spark
> >>> —lightning-fast-on-graalvm-enterprise
> >>> Spark benefits from running on GraalVM, so should we. Apart from memory
> >>> usage and performance advantages, this JVM can execute Python code.
> With
> >>> that, we can enable compute APIs support for Python.
> >>> -
> >>> Denis
> >>> On Sun, May 13, 2018 at 12:23 PM Sven Beauprez <
> >> sven.beaup...@theglue.com>
> >>> wrote:
>  Thnx all for the feedback.
>  Looking forward to the results of such a test run.
>  Regards,
>  Sven
>  SVEN BEAUPREZ
>  L e a d   A r c h i t e c t
>  De Kleetlaan 5, B-1831 Diegem
>  www.theglue.com 
>  On 10/05/2018, 17:44, "Petr Ivanov"  wrote:
>    File the ticket and specify priority — and I will start researching.
>    For test runs — we can have a copy of current test project and run
>  some tests in different VMs (as you rightly remarked — right after
> JDK9
>  task is complete).
> > On 10 May 2018, at 18:34, Dmitry Pavlov 
>  wrote:
> > Hi Peter,
> > It seems it is one more argument to implement selectable VM for
>  existing run-all chain instead of creating one more.
> > Would it be easy to add one more option once JDK 9 run is ready?
> > Sincerely,
> > Dmitriy Pavlov
> > чт, 10 мая 2018 г. в 15:58, Dmitriy Setrakyan   >:
> > Would be nice to have a TC run on Graal, just to have an
>  understanding
> > whether we support it or not.
> > D.
> > On Wed, May 9, 2018 at 4:28 PM, Denis Magda   > wrote:
> >> The performance might become better just by replacing HotSpot with
>  Graal,
> >> but something suggests me that Ignite has to be adopted for this
>  JVM (as
> >> well as for Azul VM) to get more benefits. Probably, someone will
>  get
> >> interested and pick this task up.
> >> What stands out is that the Graal folks

[jira] [Created] (IGNITE-13057) Do not print /tmp path if pds file is not stored in tmp folder.

[Semyon Danilov (Jira)]

Semyon Danilov created IGNITE-13057:
---

 Summary: Do not print /tmp path if pds file is not stored in tmp 
folder.
 Key: IGNITE-13057
 URL: https://issues.apache.org/jira/browse/IGNITE-13057
 Project: Ignite
  Issue Type: Improvement
Reporter: Semyon Danilov
Assignee: Semyon Danilov


Temporary folder for pds warning can be printed even if pds is not located there



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSSION] New Ignite settings for IGNITE-12438 and IGNITE-13013

[Valentin Kulichenko]

Ivan,

Have you considered eliminating server to client connections altogether?
Or, at the very least making the "client to server only" mode the default
one?

All the suggested names are confusing and not intuitive, and I doubt we
will be able to find a good one. A server initiating a TCP connection with
a client is confusing in the first place and creates a usability issue. We
now want to solve it by introducing an additional configuration
parameter, and therefore additional complexity. I don't think this is the
right approach.

What are the drawbacks of permanently switching to client-to-server
connections? Is there any value provided by the server-to-client option?

As for pair connections, I'm not sure I understand why there is a
limitation. As far as I know, the idea behind this feature is that we
maintain two connections between two nodes instead of one, so that every
connection is used for communication in a single direction only. Why does
it matter which node initiates the connection? Why can't one of the nodes
(e.g., a client) initiate both connections, and then use them accordingly?
Correct me if I'm wrong, but I don't see why we can't do this.

-Val

On Thu, May 21, 2020 at 1:58 PM Denis Magda  wrote:

> Ivan,
>
> Considering that the setting controls the way a communication SPI
> connection is open I would add the new parameter to CommunicationSpi
> interface naming it as follows:
>
> >
> > CommunicationSpi.connectionInitiationMode
> > {
> > BIDIRECTIONAL, //both clients and servers initiate a connection
> > initiation procedure
> > CLIENTS_TO_SERVERS //servers cannot open a connection to clients,
> only
> > clients can do that
> > }
>
>
> The problem with the environment type approach is that private networks of
> bare-metal environments usually impose restrictions similar to virtual
> environments powered by Kubernetes. Thus, environmentType.VIRTUALIZED
> doesn't cover all the cases and I'm struggling to come up with a universal
> alternative.
>
> -
> Denis
>
>
> On Thu, May 21, 2020 at 5:38 AM Ivan Bessonov 
> wrote:
>
> > Hello Igniters,
> >
> > I'd like to discuss with you changes related to [1] and [2]. Both issues
> > are mostly the same so
> > let's discuss the core idea.
> >
> > *Motivation.*
> >
> > There are certain environments that don't allow Ignite server nodes to
> open
> > TCP connections to
> > thick clients, e.g. K8S, AWS Lambda or Azure Functions. To operate in
> such
> > environments, the
> > server needs a way to request a client to open an "inverse" communication
> > connection to it.
> >
> > I've prepared a PR (still in progress) that introduces new mechanism of
> > opening connection and
> > related configuration.
> >
> > *Main idea*
> >
> > This mechanism is called "communication via discovery" or "inverse
> > connection", it works as
> > follows:
> >  - server that needs to connect to "unreachable" thick client sends a
> > specific Discovery message
> >(inverse communication request) to that client;
> >  - client node upon receiving the request opens communication connection
> to
> > that server;
> >  - server sees connection opened by client and proceeds with its task
> (that
> > required opening
> >connection to the client).
> >
> > Working name for new configuration parameter for this feature is
> > environmentType, it is an
> > enum with two values (again, working names): STANDALONE (default) and
> > VIRTUALIZED.
> > It is used as a hint to server to speed-up establishing of connections:
> > when server sees a client
> > with VIRTUALIZED environmentType it doesn't try to open connection to it
> > and sends inverse
> > communication request right away.
> > If environmentType is STANDALONE then server tries to open a connection
> in
> > a regular way
> > (iterating over all client addresses) and sends request only if all
> > attempts failed.
> >
> > There is a concern about naming of the configuration as it catches only
> one
> > use-case: when we
> > deal with some kind of virtualized environment like K8S.
> > There are other options I've encountered in private discussion:
> > - connectionMode - ALWAYS_INITIATE / INITIATE_OR_ACCEPT
> > - networkConnectivity - REACHABLE_ALWAYS / REACHABLE_ONE_WAY
> > - communicationViaDiscovery - ALWAYS / FALLBACK
> > - isReachableFromAllNodes (true/false flag)
> > - initiateConnectionsOnThisNode (true/false flag).
> >
> > *Limitations*
> >
> > The feature cannot be used along with pairedConnection setting as this
> > setting implies
> > establishing connections in both directions. Also current implementation
> > supports opening only
> > client-to-server connections. Other types of connections like
> > client-to-client or server-to-server
> > will be implemented in separate tickets.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-12438
> > [2] https://issues.apache.org/jira/browse/IGNITE-13013
> >
> > --
> > Sincerely yours,
> > Ivan Bessonov
> >
>

Re: [DISCUSSION] Key and Value fields with same name and SQL DML

[Nikolay Izhikov]

> Personally, I treat this as an anti-pattern that should have been influenced 
> by some of the old Ignite examples.

+1

I don’t see any reason to support fields with the same name both in key and in 
value from SQL.
Moreover, I think if we implement this feature then for some part of the users 
it will be confusing.

«Why SQL sets both fields? I just want to set the key object field!» 

It seems to me like one of the examples where

1. The issue can be easily fixed on the application side(just rename the fields)
2. We want to make Ignite(or any database) too smart.

Please, consider renaming of the key or value field.

> 22 мая 2020 г., в 00:14, Alexey Kukushkin  
> написал(а):
> 
> Denis,I like your idea. It would be like native "OKVM" (Object-KeyValue
> Mapper) allowing Ignite to natively operate on domain entities. It also
> means no key and value data duplication and thus addressing the original SQL
> DML problem when only key field is modified.We could add a
> configuration-based alternative to the AOP-style API you suggested. Of all
> languages that Ignite supports I think only Java and C# have AOP. We need to
> think about all languages when we add a new feature. I think your solution
> does not completely overlap with the solution suggested in  IGNITE-12807
>   : - You suggested a
> new API that does not have a "Key" entity by definition. A huge advantage of
> the existing Key/Value is it is a widely adopted JCache standard supported
> by all Ignite competitors I am aware of. That means low cost of switching to
> Ignite for users of other product and more confidence for existing Ignite
> users. IGNITE-12807 suggests solving the problem with the Key/Value API. -
> Implementing your solution is probably a significantly bigger effort than
> what IGNITE-12807 suggests. IGNITE-12807 already has a patch with a
> proof-of-concept implemented.Still, I think the urge for addressing it in
> K/V API would be less if we have your kind of API in Java and C#.
> 
> 
> 
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[jira] [Created] (IGNITE-13058) GridCommandHandlerTest.testKillHangingRemoteTransactions failed

[Ivan Bessonov (Jira)]

Ivan Bessonov created IGNITE-13058:
--

 Summary: GridCommandHandlerTest.testKillHangingRemoteTransactions 
failed
 Key: IGNITE-13058
 URL: https://issues.apache.org/jira/browse/IGNITE-13058
 Project: Ignite
  Issue Type: Bug
Reporter: Ivan Bessonov
Assignee: Ivan Bessonov


Test may fail if not all clients completed local cache start routine.
{code:java}
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertNotNull(Assert.java:712)
at org.junit.Assert.assertNotNull(Assert.java:722)
at 
org.apache.ignite.testframework.junits.JUnitAssertAware.assertNotNull(JUnitAssertAware.java:178)
at 
org.apache.ignite.util.GridCommandHandlerTest.testKillHangingRemoteTransactions(GridCommandHandlerTest.java:1044)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$6.run(GridAbstractTest.java:2127)
at java.lang.Thread.run(Thread.java:748)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)