Re: Ignite not friendly for Monitoring

[Alexey Kukushkin]

Hi Alexey,
A nice thing about delegating alerting to 3rd party enterprise systems is that 
those systems already deal with lots of things including distributed apps.
What is needed from Ignite is to consistently write to log files (again that 
means stable event IDs, proper event granularity, no repetition, 
documentation). This would be 3rd party monitoring system's responsibility to 
monitor log files on all nodes, filter, aggregate, process, visualize and 
notify on events.
How a monitoring tool would deal with an event like "node left":
The only thing needed from Ignite is to write an entry like below to log files 
on all Ignite servers. In this example 3300 identifies this "node left" event 
and will never change in the future even if text description changes:
[2017-09-01 10:00:14] [WARN] 3300 Node DF2345F-XCVDS4-34ETJH left the cluster
Then we document somewhere on the web that Ignite has event 3300 and it means a 
node left the cluster. Maybe provide documentation how to deal with it. Some 
examples:Oracle Web Cache events: 
https://docs.oracle.com/cd/B14099_19/caching.1012/b14046/event.htm#sthref2393MS 
SQL Server events: 
https://msdn.microsoft.com/en-us/library/cc645603(v=sql.105).aspx 
That is all for Ignite! Everything else is handled by specific monitoring 
system configured by DevOps on the customer side. 
Basing on the Ignite documentation similar to above, DevOps of a company where 
Ignite is going to be used will configure their monitoring system to understand 
Ignite events. Consider the "node left" event as an example.
- This event is output on every node but DevOps do not want to be notified many 
times. To address this, they will build an "Ignite model" where there will be a 
parent-child dependency between components "Ignite Cluster" and "Ignite Node". 
For example, this is how you do it in Nagios: 
https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/4/en/dependencies.html
 and this is how you do it in Microsoft SCSM: 
https://docs.microsoft.com/en-us/system-center/scsm/auth-classes. Then DevOps 
will configure "node left" monitors in SCSM (or a "checks" in Nagios) for 
parent "Ignite Cluster" and child "Ignite Service" components. State change (OK 
-> WARNING) and notification (email, SMS, whatever) will be configured only for 
the "Ignite Cluster"'s "node left" monitor.- Now suppose a node left. The "node 
left" monitor (that uses log file monitoring plugin) on "Ignite Node" will 
detect the event and pass it to the parent. This will trigger "Ignite Cluster" 
state change from OK to WARNING and send a notification. No more notification 
will be sent unless the "Ignite Cluster" state is reset back to OK, which 
happens either manually or on timeout or automatically on "node joined". 
This was just FYI. We, Ignite developers, do not care about how monitoring 
works - this is responsibility of customer's DevOps. Our responsibility is 
consistent event logging.
Thank you!


Best regards, Alexey


On Tuesday, August 15, 2017, 6:16:25 PM GMT+3, Alexey Kuznetsov 
<akuznet...@apache.org> wrote:

Alexey,

How you are going to deal with distributed nature of Ignite cluster?
And how do you propose handle nodes restart / stop?

On Tue, Aug 15, 2017 at 9:12 PM, Alexey Kukushkin <
alexeykukush...@yahoo.com.invalid> wrote:

> Hi Denis,
> Monitoring tools simply watch event logs for patterns (regex in case of
> unstructured logs like text files). A stable (not changing in new releases)
> event ID identifying specific issue would be such a pattern.
> We need to introduce such event IDs according to the principles I
> described in my previous mail.
> Best regards, Alexey
>
>
> On Tuesday, August 15, 2017, 4:53:05 AM GMT+3, Denis Magda <
> dma...@apache.org> wrote:
>
> Hello Alexey,
>
> Thanks for the detailed input.
>
> Assuming that Ignite supported the suggested events based model, how can
> it be integrated with mentioned tools like DynaTrace or Nagios? Is this all
> we need?
>
> —
> Denis
>
> > On Aug 14, 2017, at 5:02 AM, Alexey Kukushkin <alexeykukush...@yahoo.com
> .INVALID> wrote:
> >
> > Igniters,
> > While preparing some Ignite materials for Administrators I found Ignite
> is not friendly for such a critical DevOps practice as monitoring.
> > TL;DRI think Ignite misses structured descriptions of abnormal events
> with references to event IDs in the logs not changing as new versions are
> released.
> > MORE DETAILS
> > I call an application “monitoring friendly” if it allows DevOps to:
> > 1. immediately receive a notification (email, SMS, etc.)
> > 2. understand what a problem is without involving developers
> > 3. provide automated recovery actio

Re: [jira] [Commented] (IGNITE-6026) init cluster for Ignite Persistence by xml

[Alexey Kukushkin]

Hi Alex,

Please specify "--port 11211" port when running "control.sh --activate". Also, 
add --host IGNITE_NODE if you run control.sh on a host where there are no 
ignite nodes. For example:

./bin/control.sh --host 127.0.0.1 --port 11211 --activate

It looks we have an issue that default Ignite command port is 11211 but 
control.sh uses 11212 by default.

Let me know if it did not help.

Best regards, Alexey


On Friday, August 11, 2017, 10:36:08 AM GMT+3, Alex Negashev (JIRA) 
 wrote:


    [ 
https://issues.apache.org/jira/browse/IGNITE-6026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122971#comment-16122971
 ] 

Alex Negashev commented on IGNITE-6026:
---

Hello Alexey, we add client to cluster with config:

        
        http://www.springframework.org/schema/beans; 
xmlns:cache="http://www.springframework.org/schema/cache; 
xmlns:context="http://www.springframework.org/schema/context; 
xmlns:mvc="http://www.springframework.org/schema/mvc; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="      
  http://www.springframework.org/schema/beans         
http://www.springframework.org/schema/beans/spring-beans.xsd         
http://www.springframework.org/schema/cache         
http://www.springframework.org/schema/cache/spring-cache-3.1.xsd         
http://www.springframework.org/schema/context         
http://www.springframework.org/schema/context/spring-context.xsd ">
          
              
              
              
              
                
                    
                
              

              
              
                  
              
              
                
                    
                      
                          
                          
                          
                          
                      
                    
                
              
          
        


some logs from instances:
ignite-cache-client-5 | [07:29:53] Topology snapshot [ver=30, servers=5, 
clients=5, CPUs=40, heap=10.0GB]
ignite-cache-client-3 | [07:29:53] Topology snapshot [ver=30, servers=5, 
clients=5, CPUs=40, heap=10.0GB]
ignite-cache-client-2 | [07:29:53] Topology snapshot [ver=30, servers=5, 
clients=5, CPUs=40, heap=10.0GB]

but if i start script `./bin/control.sh --activate` it output: 
root@36ff2f004f78:/opt/ignite/apache-ignite-fabric-2.1.0-bin# ./bin/control.sh 
--activate
Aug 11, 2017 7:33:59 AM org.apache.ignite.internal.client.impl.GridClientImpl 

WARNING: Failed to initialize topology on client start. Will retry in 
background.
Aug 11, 2017 7:33:59 AM org.apache.ignite.internal.client.impl.GridClientImpl 

INFO: Client started [id=6b6279f1-6d5c-46a4-b88a-83d1378d94d2, protocol=TCP]
Something fail during activation, exception message: Latest topology update 
failed.
Aug 11, 2017 7:33:59 AM org.apache.ignite.internal.client.impl.GridClientImpl 
stop
INFO: Client stopped [id=6b6279f1-6d5c-46a4-b88a-83d1378d94d2, 
waitCompletion=true]


> init cluster for Ignite Persistence by xml 
> ---
>
>                Key: IGNITE-6026
>                URL: https://issues.apache.org/jira/browse/IGNITE-6026
>            Project: Ignite
>          Issue Type: Wish
>          Components: cache, examples, persistence
>    Affects Versions: 2.1
>        Environment: ignite in docker with zk
>            Reporter: Alex Negashev
>              Labels: documentation
>
> Hello! We use Ignite 2.1 and would like to use Ignite Persistence, how i can 
> do this without java code? xml only.
> Example attached.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Ignite not friendly for Monitoring

[Alexey Kukushkin]

Hi Denis,
Monitoring tools simply watch event logs for patterns (regex in case of 
unstructured logs like text files). A stable (not changing in new releases) 
event ID identifying specific issue would be such a pattern. 
We need to introduce such event IDs according to the principles I described in 
my previous mail.
Best regards, Alexey


On Tuesday, August 15, 2017, 4:53:05 AM GMT+3, Denis Magda <dma...@apache.org> 
wrote:

Hello Alexey,

Thanks for the detailed input.

Assuming that Ignite supported the suggested events based model, how can it be 
integrated with mentioned tools like DynaTrace or Nagios? Is this all we need?

—
Denis
 
> On Aug 14, 2017, at 5:02 AM, Alexey Kukushkin 
> <alexeykukush...@yahoo.com.INVALID> wrote:
> 
> Igniters,
> While preparing some Ignite materials for Administrators I found Ignite is 
> not friendly for such a critical DevOps practice as monitoring. 
> TL;DRI think Ignite misses structured descriptions of abnormal events with 
> references to event IDs in the logs not changing as new versions are released.
> MORE DETAILS
> I call an application “monitoring friendly” if it allows DevOps to:
> 1. immediately receive a notification (email, SMS, etc.)
> 2. understand what a problem is without involving developers 
> 3. provide automated recovery action.
> 
> Large enterprises do not implement custom solutions. They usually use tools 
> like DynaTrace, Nagios, SCOM, etc. to monitor all apps in the enterprise 
> consistently. All such tools have similar architecture providing a dashboard 
> showing apps as “green/yellow/red”, and numerous “connectors” to look for 
> events in text logs, ESBs, database tables, etc.
> 
> For each app DevOps build a “health model” - a diagram displaying the app’s 
> “manageable” components and the app boundaries. A “manageable” component is 
> something that can be started/stopped/configured in isolation. “System 
> boundary” is a list of external apps that the monitored app interacts with.
> 
> The main attribute of a manageable component is a list of “operationally 
> significant events”. Those are the events that DevOps can do something with. 
> For example, “failed to connect to cache store” is significant, while “user 
> input validation failed” is not.
> 
> Events shall be as specific as possible so that DevOps do not spend time for 
> further analysis. For example, a “database failure” event is not good. There 
> should be “database connection failure”, “invalid database schema”, “database 
> authentication failure”, etc. events.  
> 
> “Event” is NOT the same as exception occurred in the code. Events identify 
> specific problem from the DevOps point of view. For example, even if 
> “connection to cache store failed” exception might be thrown from several 
> places in the code, that is still the same event. On the other side, even if 
> a SqlServerConnectionTimeout and OracleConnectionTimeout exceptions might be 
> caught in the same place, those are different events since MS SQL Server and 
> Oracle are usually different DevOps groups in large enterprises!
> 
> The operationally significant event IDs must be stable: they must not change 
> from one release to another. This is like a contract between developers and 
> DevOps.
> 
> This should be the developer’s responsibility to publish and maintain a table 
> with attributes:
> 
> - Event ID
> - Severity: Critical (Red) - the system is not operational; Warning (Yellow) 
> - the system is operational but health is degraded; None - just an info.
> - Description: concise but enough for DevOps to act without developer’s help
> - Recovery actions: what DevOps shall do to fix the issue without developer’s 
> help. DevOps might create automated recovery scripts based on this 
> information.
> 
> For example:
> 10100 - Critical - Could not connect to Zookeeper to discovery nodes - 1) 
> Open ignite configuration and find zookeeper connection string 2) Make sure 
> the Zookeeper is running
> 10200 - Warning - Ignite node left the cluster.
> 
> Back to Ignite: it looks to me we do not design for operations as described 
> above. We have no event IDs: our logging is subject to change in new version 
> so that any patterns DevOps might use to detect significant events would stop 
> working after upgrade.
> 
> If I am not the only one how have such concerns then we might open a ticket 
> to address this.
> 
> 
> Best regards, Alexey

Ignite not friendly for Monitoring

[Alexey Kukushkin]

Igniters,
While preparing some Ignite materials for Administrators I found Ignite is not 
friendly for such a critical DevOps practice as monitoring. 
TL;DRI think Ignite misses structured descriptions of abnormal events with 
references to event IDs in the logs not changing as new versions are released.
MORE DETAILS
I call an application “monitoring friendly” if it allows DevOps to:
1. immediately receive a notification (email, SMS, etc.)
2. understand what a problem is without involving developers 
3. provide automated recovery action.

Large enterprises do not implement custom solutions. They usually use tools 
like DynaTrace, Nagios, SCOM, etc. to monitor all apps in the enterprise 
consistently. All such tools have similar architecture providing a dashboard 
showing apps as “green/yellow/red”, and numerous “connectors” to look for 
events in text logs, ESBs, database tables, etc.

For each app DevOps build a “health model” - a diagram displaying the app’s 
“manageable” components and the app boundaries. A “manageable” component is 
something that can be started/stopped/configured in isolation. “System 
boundary” is a list of external apps that the monitored app interacts with.

The main attribute of a manageable component is a list of “operationally 
significant events”. Those are the events that DevOps can do something with. 
For example, “failed to connect to cache store” is significant, while “user 
input validation failed” is not.

Events shall be as specific as possible so that DevOps do not spend time for 
further analysis. For example, a “database failure” event is not good. There 
should be “database connection failure”, “invalid database schema”, “database 
authentication failure”, etc. events.  

“Event” is NOT the same as exception occurred in the code. Events identify 
specific problem from the DevOps point of view. For example, even if 
“connection to cache store failed” exception might be thrown from several 
places in the code, that is still the same event. On the other side, even if a 
SqlServerConnectionTimeout and OracleConnectionTimeout exceptions might be 
caught in the same place, those are different events since MS SQL Server and 
Oracle are usually different DevOps groups in large enterprises!

The operationally significant event IDs must be stable: they must not change 
from one release to another. This is like a contract between developers and 
DevOps.

This should be the developer’s responsibility to publish and maintain a table 
with attributes:
 
- Event ID
- Severity: Critical (Red) - the system is not operational; Warning (Yellow) - 
the system is operational but health is degraded; None - just an info.
- Description: concise but enough for DevOps to act without developer’s help
- Recovery actions: what DevOps shall do to fix the issue without developer’s 
help. DevOps might create automated recovery scripts based on this information.

For example:
10100 - Critical - Could not connect to Zookeeper to discovery nodes - 1) Open 
ignite configuration and find zookeeper connection string 2) Make sure the 
Zookeeper is running
10200 - Warning - Ignite node left the cluster.

Back to Ignite: it looks to me we do not design for operations as described 
above. We have no event IDs: our logging is subject to change in new version so 
that any patterns DevOps might use to detect significant events would stop 
working after upgrade.

If I am not the only one how have such concerns then we might open a ticket to 
address this.


Best regards, Alexey

Re: [jira] [Created] (IGNITE-6026) init cluster for Ignite Ignite Persistence by xml

[Alexey Kukushkin]

Hi Alex,
The main idea from the API perspective is that Ignite Persistence is 
transparent for the API. After you enable persistence all your APIs including 
get/put, SQL, etc would work as before. 
Enable Persistence with 1 line:




And then activate cluster explicitly - With script:
${IGNITE_HOME}/bin/control.sh --activate

OR in the code:
ignite.active(true);


Best regards, Alexey


On Thursday, August 10, 2017, 3:50:35 PM GMT+3, Alex Negashev (JIRA) 
 wrote:

Alex Negashev created IGNITE-6026:
-

            Summary: init cluster for Ignite  Ignite Persistence by xml 
                Key: IGNITE-6026
                URL: https://issues.apache.org/jira/browse/IGNITE-6026
            Project: Ignite
          Issue Type: Wish
          Components: cache, examples, persistence
    Affects Versions: 2.1
        Environment: ignite in docker with zk
            Reporter: Alex Negashev


Hello! We use Ignite 2.1 and would like to use Ignite Persistence, how i can do 
this with out java code? xml only.
Example attached.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

IGNITE-4901 (Decrease logging level for DataStremer retry)

[Alexey Kukushkin]

Hi,
Presently when DataStreamer is flushing data and cluster topology changes, the 
DataStreamer would output an ERROR message to the console and retry flushing 
the data with the latest topology. 
We have this IGNITE-4901 issue asking to decrease logging level since a 
topology change is normal and DataStreamer reliably handles it.
In addition to setting the log level to INFO, I suggest we change Ignite to 
fail to update the cache only if MAJOR topology version changed (another node 
joined/left) since minor version change (registering/unregistering another 
cache) does not impact updating the cache unless we delete the cache-in-use but 
the latter error is handled differently anyway.
Please let me know if anyone has objections or comments. Otherwise I will 
submit this solution.
Thank you!

Best regards, Alexey

I am a new Ignite Development Community member

[Alexey Kukushkin]

Dear Ignite Development Community,
I am a software engineer and joined the community to contribute to the product. 
Could you please help with granting me proper Ignite Jira permissions? My Jira 
user ID is "kukushal". Thank you!

Best regards, Alexey

Re: Ignite cluster with persistent store enabled did not load from wal after restarting.

[Alexey Kukushkin]

Ray,

Yakov just pointed me to this thread

where
user had a problem with IP address changing between the cluster restarts.
The persistent data directories inside work/db have names built as a
concatenation of local IP addresses and ports. Thus, if your node receives
a different IP address between the restarts, you will not have your
persistent data since the node will initialise from a different persistent
data file. You would need to bind your discovery SPI to specific address to
work around that. There is an issue opened to properly fix it.

Please also consider that as a potential cause.

Re: integrate with prestodb

[Alexey Kukushkin]

Cross-sending to the DEV community.

On Mon, Oct 16, 2017 at 12:14 PM, shawn.du  wrote:

> Hi community,
>
> I am trying to implement a connector for presto to connect ignite.
> I think it will be a very interest thing to connect ignite and presto.
>
> In fact, currently we use ignite and it works very well.  but in order to
> save memory, we build compressed binary data.
> thus we cannot query them using SQL. We use ignite map-reduce to query the
> data.
>
> Using presto, we may use SQL again. If it is fast enough, ignite will be
> our in memory storage and not responsible for computing or only for simple
> query.
> The only thing I concern about is presto is fast enough or not like
> Ignite. For now all ignite query cost less than 5 seconds and most are
> hundreds of milliseconds.
> Also presto provides a connector for redis.  I don't know community has
> interest to contribute to presto-ignite?
>
> Thanks
> Shawn
>
>


-- 
Best regards,
Alexey

Re: Some questions regard Cache Query and performance

[Alexey Kukushkin]

Aaron, let me forward this to DEV:

Ignite DEV community,

I tried to confirm setLocal(true) semantics for ContinuousQuery and I have
got not what I expected. I found that executing a ContinuousQuery with
setLocal(true):

   - Still executes initial query remotely
   - Listens for modifications locally

I expected that initial query shall also be local.

Is this by design or a bug?



On Fri, Oct 13, 2017 at 3:45 PM, aa...@tophold.com 
wrote:

> hi All,
>
> Regards ContinuousQuery, I notice there is a #setLocal  will this meaning
> CQ only query against local cache?  while this query only have 
> #setRemoteFilterFactory
> method.
>
> So the question is will my CQ's local listener only receive event updates
> from local cache or entire cluster?
>
> Also the SqlFieldsQuery, we use this to batch update entries in cache, the
> frequent may be once several seconds.
>
> The entry may be updated real time, sometime the data may inconsistent
> seem, will there any lock/transaction control of this?
>
> We use the JDBC as back-end storage;  and notice if we run the instance
> for several days, the memory usage keep increase, the performance downgrade
> very badly,
>
> And eventually can not be used, from log we receive the ignite node stop
> event, but the memory not be reclaimed until we kill the JVM finally.
>
> Our cache defined as:
>
> 
> 
> 
> 
>
> 
> 
> 
> 
> 
>
>
> 
> 
> 
>
> 
> 
> 
> 
> 
>
> 
> 
> 
>  class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
> 
> 
> 
> 
> 
> 
>
> 
>
> 
> 
> 
> 
> 
> 
>
>
> Thanks for your time!
>
> Regards
> Aaron
> --
> aa...@tophold.com
>



-- 
Best regards,
Alexey

Re: Massive commit sizes for processes with local Ignite Grid servers

[Alexey Kukushkin]

Raymond,

So you see 3 "extra" GB that your server app takes. Is it possible your app
really loads additional 3GB of referenced libraries and data besides
Ignite? Did you try temporarily changing the code to NOT start Ignite and
see how much memory such an app takes?

On Thu, Sep 7, 2017 at 1:49 PM, Raymond Wilson 
wrote:

> Hi Dmitry,
>
>
>
> Thanks for the pointer to the MemoryPolicy.
>
>
>
> I added the following:
>
>
>
> cfg.MemoryConfiguration = new MemoryConfiguration()
>
> {
>
> SystemCacheMaxSize = (long)1 * 1024 * 1024 * 1024,
>
> DefaultMemoryPolicyName = "defaultPolicy",
>
> MemoryPolicies = new[]
>
> {
>
> new MemoryPolicyConfiguration
>
> {
>
> Name = "defaultPolicy",
>
> InitialSize = 128 * 1024 * 1024,  // 128 MB
>
> MaxSize = 1L * 1024 * 1024 * 1024  // 1 GB
>
> }
>
> }
>
> };
>
>
>
> After running both servers the commit size peaked at 4Gb for both
> processes (with ~430Mb actual allocated memory) which is s significant
> improvement, though still seems higher than might be expected.
>
>
>
> Thanks,
>
> Raymond.
>
>
>
>
>
>
>
> *From:* Dmitry Pavlov [mailto:dpavlov@gmail.com]
> *Sent:* Thursday, September 7, 2017 10:22 PM
> *To:* u...@ignite.apache.org; dev@ignite.apache.org
> *Subject:* Re: Massive commit sizes for processes with local Ignite Grid
> servers
>
>
>
> Hi Raymond,
>
>
>
> Total memory usage since 2.0 version is determined as sum of heap size and
> memory policies MaxSizes (overall segment sizes). If it is not configured
> there is 80% of physical RAM is used for each node (before 2.2). In 2.2
> this behaviour will be changed.
>
>
>
> To run several nodes at one PC it may be required to manually setup Memory
> Configuration and Memory Policy(ies).
>
>
>
>
>
> Hi Igniters, esp. Pavel T.
>
>
>
> please share your thoughts. To which Java property value of
> SystemCacheMaxSize is now mapped?
>
>
>
> Sincerely,
>
> Dmitriy Pavlov
>
>
>
> P.S. Please see example of configuration
>
> https://apacheignite-net.readme.io/docs/durable-memory
>
>
>
> MemoryPolicies = new[]
>
> {
>
>   new MemoryPolicyConfiguration
>
>   {
>
> Name = "defaultPolicy",
>
> MaxSize = 4L * 1024 * 1024 * 1025  // 4 GB
>
>   }
>
> }
>
>
>
> чт, 7 сент. 2017 г. в 12:44, Raymond Wilson :
>
> I tried an experiment where I ran only two instances of the server
> locally, this is the result in the Task Manager:
>
>
>
>
>
> *From:* Raymond Wilson [mailto:raymond_wil...@trimble.com]
> *Sent:* Thursday, September 7, 2017 9:21 PM
> *To:* u...@ignite.apache.org; 'dev@ignite.apache.org' <
> dev@ignite.apache.org>
> *Subject:* Massive commit sizes for processes with local Ignite Grid
> servers
>
>
>
> I’m running a set of four server applications on a local system to
> simulate a cluster.
>
>
>
> Each of the servers has the following memory configurations set:
>
>
>
> public override void ConfigureRaptorGrid(IgniteConfiguration cfg)
>
> {
>
> cfg.JvmInitialMemoryMb = 512; // Set to minimum advised
> memory for Ignite grid JVM of 512Mb
>
> cfg.JvmMaxMemoryMb = 1 * 1024; // Set max to 1Gb
>
>
>
> // Don't permit the Ignite node to use more than 1Gb RAM
> (handy when running locally...)
>
> cfg.MemoryConfiguration = new MemoryConfiguration()
>
> {
>
> SystemCacheMaxSize = (long)1 * 1024 * 1024 * 1024
>
> };
>
> }
>
>
>
> The snap below is from the Windows 10 Task Manager where I have included
> the Commit Size value. As can be seen, the four identical servers are using
> very large and wildly varying commit sizes. Some Googling suggests this is
> due to the JVM allocating the largest contiguous block of virtual memory it
> can, but I would not have expected this size to be larger than the
> configured memory for the JVM (1Gb plus memory from the wider process it is
> running in, though this is only a few hundred Mb at most)
>
>
>
>
>
> The result is that my local system reports ~50-60Gb committed memory on a
> system with 16Gb of physical RAM, and I don’t think it likes it!
>
>
>
> Is there are way to configure the Ignite JVM to be a better citizen with
> respect to the commited size it requests from the host operating system?
>
>
>
> Thanks,
>
> Raymond.
>
>
>
>


-- 
Best regards,
Alexey

Re: Massive commit sizes for processes with local Ignite Grid servers

[Alexey Kukushkin]

OK, please ignore - I see you might have found a cause in the other thread.

On Thu, Sep 7, 2017 at 2:45 PM, Alexey Kukushkin <kukushkinale...@gmail.com>
wrote:

> Raymond,
>
> So you see 3 "extra" GB that your server app takes. Is it possible your
> app really loads additional 3GB of referenced libraries and data besides
> Ignite? Did you try temporarily changing the code to NOT start Ignite and
> see how much memory such an app takes?
>
> On Thu, Sep 7, 2017 at 1:49 PM, Raymond Wilson <raymond_wil...@trimble.com
> > wrote:
>
>> Hi Dmitry,
>>
>>
>>
>> Thanks for the pointer to the MemoryPolicy.
>>
>>
>>
>> I added the following:
>>
>>
>>
>> cfg.MemoryConfiguration = new MemoryConfiguration()
>>
>> {
>>
>> SystemCacheMaxSize = (long)1 * 1024 * 1024 * 1024,
>>
>> DefaultMemoryPolicyName = "defaultPolicy",
>>
>> MemoryPolicies = new[]
>>
>> {
>>
>> new MemoryPolicyConfiguration
>>
>> {
>>
>> Name = "defaultPolicy",
>>
>> InitialSize = 128 * 1024 * 1024,  // 128 MB
>>
>> MaxSize = 1L * 1024 * 1024 * 1024  // 1 GB
>>
>> }
>>
>> }
>>
>> };
>>
>>
>>
>> After running both servers the commit size peaked at 4Gb for both
>> processes (with ~430Mb actual allocated memory) which is s significant
>> improvement, though still seems higher than might be expected.
>>
>>
>>
>> Thanks,
>>
>> Raymond.
>>
>>
>>
>>
>>
>>
>>
>> *From:* Dmitry Pavlov [mailto:dpavlov@gmail.com]
>> *Sent:* Thursday, September 7, 2017 10:22 PM
>> *To:* u...@ignite.apache.org; dev@ignite.apache.org
>> *Subject:* Re: Massive commit sizes for processes with local Ignite Grid
>> servers
>>
>>
>>
>> Hi Raymond,
>>
>>
>>
>> Total memory usage since 2.0 version is determined as sum of heap size
>> and memory policies MaxSizes (overall segment sizes). If it is not
>> configured there is 80% of physical RAM is used for each node (before 2.2).
>> In 2.2 this behaviour will be changed.
>>
>>
>>
>> To run several nodes at one PC it may be required to manually setup
>> Memory Configuration and Memory Policy(ies).
>>
>>
>>
>>
>>
>> Hi Igniters, esp. Pavel T.
>>
>>
>>
>> please share your thoughts. To which Java property value of
>> SystemCacheMaxSize is now mapped?
>>
>>
>>
>> Sincerely,
>>
>> Dmitriy Pavlov
>>
>>
>>
>> P.S. Please see example of configuration
>>
>> https://apacheignite-net.readme.io/docs/durable-memory
>>
>>
>>
>> MemoryPolicies = new[]
>>
>> {
>>
>>   new MemoryPolicyConfiguration
>>
>>   {
>>
>> Name = "defaultPolicy",
>>
>> MaxSize = 4L * 1024 * 1024 * 1025  // 4 GB
>>
>>   }
>>
>> }
>>
>>
>>
>> чт, 7 сент. 2017 г. в 12:44, Raymond Wilson <raymond_wil...@trimble.com>:
>>
>> I tried an experiment where I ran only two instances of the server
>> locally, this is the result in the Task Manager:
>>
>>
>>
>>
>>
>> *From:* Raymond Wilson [mailto:raymond_wil...@trimble.com]
>> *Sent:* Thursday, September 7, 2017 9:21 PM
>> *To:* u...@ignite.apache.org; 'dev@ignite.apache.org' <
>> dev@ignite.apache.org>
>> *Subject:* Massive commit sizes for processes with local Ignite Grid
>> servers
>>
>>
>>
>> I’m running a set of four server applications on a local system to
>> simulate a cluster.
>>
>>
>>
>> Each of the servers has the following memory configurations set:
>>
>>
>>
>> public override void ConfigureRaptorGrid(IgniteConfiguration cfg)
>>
>> {
>>
>> cfg.JvmInitialMemoryMb = 512; // Set to minimum advised
>> memory for Ignite grid JVM of 512Mb
>>
>> cfg.JvmMaxMemoryMb = 1 * 1024; // Set max to 1Gb
>>
>>
>>
>> // Don't permit the Ignite node to use more than 1Gb RAM
>> (handy when running locally...)
>>
>> cfg.MemoryConfiguration = new

Re: Ignite: configuration changes at runtime

[Alexey Kukushkin]

I would vote for not automatically persisting runtime configuration
changes. Still it makes sense to expose a "save(fileName)" method to allow
explicitly saving changes to a different or same config file.

On Mon, Aug 21, 2017 at 5:22 PM, Dmitriy Setrakyan 
wrote:

> On Mon, Aug 21, 2017 at 7:15 AM, Alexey Dmitriev 
> wrote:
>
> > I would say it should as soon as we moving to "persistence".
> > It will require to look at the things a bit different than before, but I
> > would say that's an evolution for the product.
> > We probably also should think how our configuration system should be
> > changed to make it more obvious.
> >
>
> But in that case, what to do with XML configuration? Should it be updated
> as well? Or even worse, what if user added configuration from code?
>
> On Mon, Aug 21, 2017 at 5:11 PM, Dmitriy Setrakyan 
> > wrote:
> >
> > > I think the main question I have is whether this configuration change
> > will
> > > survive restarts. Part of me says it should, and the other part says it
> > > shouldn't.
> > >
> > > Does anyone have a strong opinion about this?
> > >
> > > D.
> > >
> > > On Mon, Aug 21, 2017 at 7:07 AM, Alexey Dmitriev <
> admitr...@gridgain.com
> > >
> > > wrote:
> > >
> > > > It looks like very useful and natural thing having the parameters
> > change
> > > on
> > > > the fly.
> > > > Maybe we should design something like OLCC (on-line configuration
> > change)
> > > > module, which will request different procedures for different bunch
> of
> > > > parameters.
> > > > All the parameters, this way, will be splitted into several groups.
> > > > For example in the worst case, when all the grid has to be
> synchronized
> > > for
> > > > particular parameter change, some preparation on the grid has to be
> > > > performed, than when all the nodes reports readiness, the parameter
> > > change
> > > > could be initiated.
> > > > On the opposite if it is parameter configured per node, OLCC state
> > > machine
> > > > will work simpler without inter-node communication.
> > > >
> > > > On Mon, Aug 21, 2017 at 4:21 PM, Yakov Zhdanov 
> > > > wrote:
> > > >
> > > > > >I see your point. In this case, we should have a special package
> > > > > containing
> > > > > >all the runtime config properties.
> > > > >
> > > > > Dmitry, I think this will be a mess.
> > > > >
> > > > > Igniters, any more opinions?
> > > > >
> > > > > --Yakov
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Alexey Dmitriev, VP Engineering
> > > > *GridGain Systems*
> > > > www.gridgain.com
> > > >
> > >
> >
> >
> >
> > --
> > Alexey Dmitriev, VP Engineering
> > *GridGain Systems*
> > www.gridgain.com
> >
>



-- 
Best regards,
Alexey

Re: Ignite node crashes after one query fetches many entries from cache

[Alexey Kukushkin]

Ignite Developers,

I know community is developing an "Internal Problems Detection" feature
.
Do you know if it addresses the problem Ray described below? May be we
already have a setting to prevent this from happening?

On Tue, Nov 28, 2017 at 5:13 PM, Ray  wrote:

> I try to fetch all the results of a table with billions of entries using
> sql
> like this "select * from table_name".
> As far as my understanding, Ignite will prepare all the data on the node
> running this query then return the results to the client.
> The problem is that after a while, the node crashes(probably because of
> long
> GC pause or running out of memory).
> Is node crashing the expected behavior?
> I mean it's unreasonable that Ignite node crashes after this kind of query.
>
> From my experience with other databases,  running this kind of full table
> scan will not crash the node.
>
> The optimal way for handling this kind of situation is Ignite node stays
> alive, the query will be stopped by Ignite node when the node find out it
> will run out of memory soon.
> Then an error response shall be returned to the client.
>
> Please advice me if this mechanism already exists and there is hidden
> switch
> to turn it on.
> Thanks
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>



-- 
Best regards,
Alexey

Re: Transport compression (not store compression)

[Alexey Kukushkin]

Forwarding to DEV list: Ignite developers, could you please share your
thoughts on how hard it is to extend Ignite to compress data on the network.

On Wed, Nov 22, 2017 at 10:04 AM, Gordon Reid (Nine Mile) <
gordon.r...@ninemilefinancial.com> wrote:

> Hi Igniters,
>
>
>
> I see there is a lot of discussion in certain threads about compression.
> This seems to have diverged into conversations about object versus field
> compression, and even throwing encryption into the mix. For my use case, I
> am not interested in compressing the cache stored in memory, I have plenty
> of memory for my application. What I don’t have is a good network. I have a
> high latency, low bandwidth network between my C# ignite client and my Java
> ignite server. I only want to compress data when it is sent over the
> network to remote nodes. It should be stored in the local memory
> uncompressed. How can we achive this? Can the TcpCommunicationSpi support
> compression?
>
>
>
> Thanks,
>
> Gordon.
>
>
>
>
>
>
>
>
>
> This email and any attachments are proprietary & confidential and are
> intended solely for the use of the individuals to whom it is addressed. Any
> views or opinions expressed are solely for those of the author and do not
> necessarily reflect those of Nine Mile Financial Pty. Limited. If you have
> received this email in error, please let us know immediately by reply email
> and delete from your system. Nine Mile Financial Pty. Limited. ABN: 346
> 1349 0252
>



-- 
Best regards,
Alexey

Re: Ignite Events Remote Filter

[Alexey Kukushkin]

Also, I ran our CacheEventsExample and I confirm the remote filter is NOT
unsubscribed when it returns false. So it looks like a documentation bug
only.

On Fri, Oct 27, 2017 at 1:10 PM, Alexey Kukushkin <kukushkinale...@gmail.com
> wrote:

> Hi,
>
> I think it is a documentation bug. I do not think remote listener will be
> unsubscribed if it returns false. Let's confirm with the developers
> community.
>
> *Ignite Developers*,
>
> We have this comment for the remoteListener argument of the
> IgniteEvents#remoteListen(...):
>
> rmtFilter - Filter callback that is called on remote node. Only events
> that pass the remote filter will be sent to local node. If null, all events
> of specified types will be sent to local node. This remote filter can be
> used to pre-handle events remotely, before they are passed in to local
> callback. *It will be auto-unsubsribed on the node where event occurred
> in case if it returns false*.
>
> The documentation in bold looks like a bug to me. Why we would do it that
> way? To get only the first event? I think the idea is to listen until
> either master node leaves or you call remoteUnsubscribe(). I feel we just
> need to remove that sentence from the comments. Could you please confirm?
>
>
> On Fri, Oct 27, 2017 at 4:25 AM, rajivgandhi <rajiv.gan...@gmail.com>
> wrote:
>
>> Hi,
>> We wish to listen to remote events with a remote filter and local
>> listener:
>> https://apacheignite.readme.io/docs/events#section-remote-events
>>
>> Our requirement is to entertain only those events on local listener which
>> are allowed by remote filter. This is the API we are trying to use:
>> https://ignite.apache.org/releases/latest/javadoc/org/apache
>> /ignite/IgniteEvents.html#remoteListen(org.apache.ignite
>> .lang.IgniteBiPredicate,%20org.apache.ignite.lang.Ignit
>> ePredicate,%20int...)
>>
>> As per this API:
>> "rmtFilter - Filter callback that is called on remote node. Only events
>> that
>> pass the remote filter will be sent to local node. If null, all events of
>> specified types will be sent to local node. This remote filter can be used
>> to pre-handle events remotely, before they are passed in to local
>> callback.
>> *It will be auto-unsubsribed on the node where event occurred in case if
>> it
>> returns false.*"
>>
>> As per the bolded part, it seems the remote listener will be unsubscribed
>> as
>> soon as the remote filter returns false. Is that right? We want to be able
>> to continue listening even after the remote filter has once
>> filtered/blocked
>> the event on remote nodes. If that is indeed the case (remote filter stops
>> listening), are there any other work arounds?
>>
>> thanks!
>> Rajeev Gandhi
>>
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>
>
> --
> Best regards,
> Alexey
>



-- 
Best regards,
Alexey

Re: Ignite Events Remote Filter

[Alexey Kukushkin]

Hi,

I think it is a documentation bug. I do not think remote listener will be
unsubscribed if it returns false. Let's confirm with the developers
community.

*Ignite Developers*,

We have this comment for the remoteListener argument of the
IgniteEvents#remoteListen(...):

rmtFilter - Filter callback that is called on remote node. Only events that
pass the remote filter will be sent to local node. If null, all events of
specified types will be sent to local node. This remote filter can be used
to pre-handle events remotely, before they are passed in to local callback. *It
will be auto-unsubsribed on the node where event occurred in case if it
returns false*.

The documentation in bold looks like a bug to me. Why we would do it that
way? To get only the first event? I think the idea is to listen until
either master node leaves or you call remoteUnsubscribe(). I feel we just
need to remove that sentence from the comments. Could you please confirm?


On Fri, Oct 27, 2017 at 4:25 AM, rajivgandhi  wrote:

> Hi,
> We wish to listen to remote events with a remote filter and local listener:
> https://apacheignite.readme.io/docs/events#section-remote-events
>
> Our requirement is to entertain only those events on local listener which
> are allowed by remote filter. This is the API we are trying to use:
> https://ignite.apache.org/releases/latest/javadoc/org/
> apache/ignite/IgniteEvents.html#remoteListen(org.apache.
> ignite.lang.IgniteBiPredicate,%20org.apache.ignite.lang.
> IgnitePredicate,%20int...)
>
> As per this API:
> "rmtFilter - Filter callback that is called on remote node. Only events
> that
> pass the remote filter will be sent to local node. If null, all events of
> specified types will be sent to local node. This remote filter can be used
> to pre-handle events remotely, before they are passed in to local callback.
> *It will be auto-unsubsribed on the node where event occurred in case if it
> returns false.*"
>
> As per the bolded part, it seems the remote listener will be unsubscribed
> as
> soon as the remote filter returns false. Is that right? We want to be able
> to continue listening even after the remote filter has once
> filtered/blocked
> the event on remote nodes. If that is indeed the case (remote filter stops
> listening), are there any other work arounds?
>
> thanks!
> Rajeev Gandhi
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>



-- 
Best regards,
Alexey

Thin client: Java bindings

[Alexey Kukushkin]

Igniters,

I am working on a project where users will use Ignite via a thin client
protocol. The API will be Java and the client will have to support failover
and encryption.

I know community already developed the thin client protocol and .NET
bindings and is going to develop failover and encryption (I found backlog
tickets addressing failover and encryption). Thus, the only missing part is
Java bindings.

Unless someone is already working on Java bindings, let us develop and
contribute the thin client Java API to the Apache Ignite project. I believe
many can benifit from it.

There are two options to design the API: 1) implement existing interface
like IgniteCache and throw UnsupportedOperationExcception if something is
not supported 2) implement new interface and define only supported methods. 
The Community  already discussed the options

  
decided to use the latter option 2 for .NET bindings so we will follow the
same approach for the Java API.

On the first phase we will implement the below data-only API. Let us know if
you have comments. Thank you!

Cache
JCache (limited)
getName(): String
put(key, val)
get(key): V
getAll(keys: Set): Map
containsKey(key): boolean
getAndPut(key, val): V
getAndReplace(key, val): V
getAndRemove(key): V
putIfAbsent
replace(key, val)
replace(key, oldVal, newVal)
putAll
clear
remove(key)
remove(key, val)
removeAll()
removeAll(keys: Set)
getConfiguration(clazz): Configuration
close()
size(modes: CachePeekMode...)
query(qry: Query): QueryCursor
query(qry: SqlFieldsQuery): FieldsQueryCursor
withKeepBinary(): IgniteCache
Ignite
cache(name: String)
cacheNames(): Collection
binary(): IgniteBinary
createCache(name): Cache
getOrCreateCache(name): Cache
destroyCache(name)
Ignition
startClient(:ClientConfiguration): Ignite
ClientConfiguration(port, host, binaryConfiguration, sslConfiguration,
etc...)







--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Adopting mock framework (ex. Mockito) for unit tests

[Alexey Kukushkin]

Hi Jason,

I heartily support unit testing practices and introducing a mocking
framework into ignite development environment. Today I can hardly find a
single unit test in Apache Ignite, which does not allow me to use the best
TDD and CI/CD practices like running tests on every commit (or even on
every "save file"!).

I recently started developing an isolated component based on Apache Ignite
and, since I use TDD and write lots of unit tests, I had to add a mocking
framework to my project. I started from Mockito (version 1.something) and
found I could not do some things with Mockito due to Ignite currently not
designed for unit testing. I believe I could not find a way to mock some
private initialisation block with Mockito.

Thus, I switched to JMockit - it allowed me to mock what I wanted and it
seems you can mock virtually everything with JMockit.

I know that a situation when you have to mock something private or static
indicates not very modular and extendable design but you do not have much
of a choice with Ignite since you already have huge amount of code and it
would be really hard to refactor everything to make it testable since
Ignite development process is heavy and your project could be stuck waiting
for Ignite changes.

Did you consider JMockit over Mockito? It seems JMockit supports both
record-replay-verity and setup-run-verify models as well as BDD semantics
API.

Re: Thin client: Java bindings

[Alexey Kukushkin]

Dmitriy,

Although having invoke() was really important for us, implementing invoke()
does not look trivial (need to think about real multi-lingual and
cross-platform design) and we do not have time to implement it on the first
phase. Currently thin client protocol does not support it so we would have
to design it ourselves. Still we will need invoke and transactions on the
next phases and really appreciate if community would design something to
support it in the binary protocol.

Re: Thin client: Java bindings

[Alexey Kukushkin]

We also need asynchronous versions of all the thin client APIs. I created 
JIRA-7623    to address
async APIs. Proposed design:
Async methods are named by appending "Async" to the corresponding
syncMethod, e.g. putAsync, clearAsync.
Async methods return java.util.concurrent.Future
Async methods are not cancellable (presently cancellation is not supported
by the thin client protocol)
Use thin client protocol's Request ID to map async requests and responses.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Ignite Thin clients for Node.js, Python, PHP

[Alexey Kukushkin]

Alexey, Ekaterina, Pavel,

These resources will be useful for you:

   - Thin client protocol spec
   
   - .NET thin client code
   

   - Java thin client code
   

(this
   is a work-in-progress in a development branch)

Please post your questions and ideas - the community will help.

Re: Ignite Thin clients for Node.js, Python, PHP

[Alexey Kukushkin]

Also, we could have a Hangouts call or something to share thin client
development experience. You can reach me at kukushkinale...@gmail.com.

Re: Thin client: Java bindings

[Alexey Kukushkin]

Igniters!

Could you please help with the Java thin client code review:

Documentation <https://apacheignite.readme.io/v2.3/docs/java-thin-client>
Code <https://reviews.ignite.apache.org/apache-ignite/review/IG-CR-23>

Thank you for the help.

On Mon, Feb 5, 2018 at 5:45 PM, Alexey Kukushkin <kukushkinale...@gmail.com>
wrote:

> We also need asynchronous versions of all the thin client APIs. I created
> JIRA-7623 <https://issues.apache.org/jira/browse/IGNITE-7623>   to address
> async APIs. Proposed design:
> Async methods are named by appending "Async" to the corresponding
> syncMethod, e.g. putAsync, clearAsync.
> Async methods return java.util.concurrent.Future
> Async methods are not cancellable (presently cancellation is not supported
> by the thin client protocol)
> Use thin client protocol's Request ID to map async requests and responses.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>



-- 
Best regards,
Alexey

Re: Thin client / Binary protocol: questions

[Alexey Kukushkin]

Denis and All,

The links you asked for:

   - Java thin client development branch
   <https://github.com/gridgain/apache-ignite/tree/ignite-7421>
   - Code review in Upsource
   <https://reviews.ignite.apache.org/apache-ignite/review/IG-CR-23>
   - Thin client documentation
   <https://apacheignite.readme.io/v2.3/docs/java-thin-client>
  - Quick Start
  <https://apacheignite.readme.io/v2.3/docs/java-thin-client-quick-start>
  - API <https://apacheignite.readme.io/v2.3/docs/java-thin-client-api>
  - Security
  <https://apacheignite.readme.io/v2.3/docs/java-thin-client-security>
  - High Availability
  
<https://apacheignite.readme.io/v2.3/docs/java-thin-client-high-availability>
  - Monitoring
  <https://apacheignite.readme.io/v2.3/docs/java-thin-client-monitoring>


On Wed, Feb 28, 2018 at 10:44 PM, Denis Magda <dma...@apache.org> wrote:

> Guys, please consider my five cents
>
> >>> - Name/password authentication.
>> >>> When will it be available in the spec?
>> >>> Is there any draft we can look at?
>> It is still under development. Will be available in several weeks. It is
>> better to skip this part for now.
>
>
> Let's give a reference to the branch where the guys can see those changes
> at the protocol level. My guess they want to know what to expect and how it
> can affect the design they've been working on. *Alexey Kukushkin*, please
> point out to your Java thin client branch were you already have this
> functionality embedded.
>
> >>> - SSL/TLS communication.
>> >>> When will it be available in the spec?
>
>
> This doc explains how SSL is enabled on the server side:
> https://apacheignite.readme.io/docs/ssltls
>
> As always, the thin client needs to establish the SSL tunneling first and
> then starts sending the protocol messages. Hope this is good for the
> beginning. Will be happy to answer more specific questions.
>
> >>>  - May we easily use 3rd-party components with the following licenses?
>> As a general rule of thumb it is not recommended to use any external
>> libraries. The whole Ignite core is built with almost no dependencies, not
>> to say about much less complex thin clients ans JDBC/ODBC drivers. IMO the
>> only possible place where external dependency might be required is SSL
>> support (e.g. we use OpenSSL for ODBC).
>
>
> I wouldn't discourage Alexey from using 3rd party libs taking Ignite core
> as an example. Ignite optional libs use 3rd party libs a lot. Take our REST
> protocol for instance that uses Jetty and not based on its own HTTP server
> implementation. So, I would alter the rule of thumb a little bit - if it
> takes weeks to develop a functionality already available in a 3rd party lib
> then let's go for the 3rd party.
>
> Here is a list of licenses that can be included in Ignite without any
> extra permissions or licensing concerns: https://www.apache.
> org/legal/resolved.html
>
> According to the doc you're free to use Apache 2.0 and MIT, but LGPL code
> can't be delivered within Ignite.
>
> --
> Denis
>
> On Wed, Feb 28, 2018 at 2:32 AM, Vladimir Ozerov <voze...@gridgain.com>
> wrote:
>
>> Hi Alex,
>>
>> >>> Just to double-check - we should support the types from the spec only.
>> Right?
>> Please provide the list of types you are in doubt of
>>
>> >>> - Key-Value and SQL and Scan Queries.
>> >>>  The most of operations has "Flag" field in Request: "Pass 0 for
>> default, or 1 to keep the value in binary form."
>> >>>  What is it for?
>> Please see IgniteCache.withKeepBinary method. For SQL this flag has no
>> effect. For complex SCAN queries with filters this defines whether with
>> pass real objects or binary objects to the filter.
>>
>> >>>  Java and .net libs always pass 0. Why?
>> There is no Java client at the moment. As far as .NET, it doesn't support
>> binary mode at the moment.
>>
>> >>>  Why an app working remotely via the binary protocol should know the
>> exact platform where the node is running?
>> Because if node is started from .NET, you may execute both Java filters
>> and
>> .NET fitlers on it. This flag defines what kind of filter is passed.
>>
>> >>> - Binary Type operations (register/get type name, put/get type).
>> >>>  What are they for?
>> >>>  What is a use case?
>> Please get familiar with binary type concepts, especially binary metadata:
>> https://apacheignite.readme.io/docs#binaryobject-type-metadata
>>
>> >>

Re: IGNITE-6879

[Alexey Kukushkin]

Dmitry, Roman, I can review the fix - send me a code review link when it is
ready.

On Wed, Mar 14, 2018 at 5:14 PM, Dmitry Pavlov <dpavlov@gmail.com>
wrote:

> Igniters,
>
> it is about spring-data integration support.
>
> Who has intereset about styding this technology?
>
> Alexey Kukushkin would you like to be reviewer of this issue?
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 14 мар. 2018 г. в 1:08, Denis Magda <dma...@apache.org>:
>
>> Hello Roman,
>>
>> It's not a big deal, I've added you to the contributors' list in JIRA. Go
>> ahead and assign the ticket to yourself.
>>
>> Folks, anyway, who can review Roman's contribution that brings Spring 2.0
>> support to Ignite?
>>
>> --
>> Denis
>>
>>
>>
>> On Tue, Mar 13, 2018 at 2:21 PM, Роман Меерсон <homich1...@gmail.com>
>> wrote:
>>
>> > Hello!
>> >
>> > I want to work on https://issues.apache.org/jira/browse/IGNITE-6879
>> issue.
>> > Following the rules here
>> > https://ignite.apache.org/community/contribute.html#contribute my Jira
>> > username is "homich" so assign me to this ticket please.
>> >
>> > P.S. I found this rules page after I made PR, so sorry for this
>> > inconvenience.
>> >
>> > Regards.
>> >
>>
>


-- 
Best regards,
Alexey

Re: IGNITE-6879

[Alexey Kukushkin]

Roman,

Just two small comments from me:

   1. I suggest renaming IgniteSpringDataTestSuite to
   IgniteSpringData2TestSuite: we must be able to test both spring-data and
   spring-data-2 JARs with single "mvn test"
   2. Make sure "Ignite Spring Data" test job in TeamCity is extended to
   run the new test suite after the code is committed.

Other than that the new module looks OK to me.

Re: IGNITE-6879

[Alexey Kukushkin]

Roman,

Just pay commiter's (Dmitry Pavlov will most likely commit your code)
attention to include the new test suite to TeamCity configuration.

Re: IGNITE-6879

[Alexey Kukushkin]

Roman, Dmitry,

I also reviewed the fix and the code looks OK to me. But the fix has
significant implication - Ignite no longer can be used with spring-data 1.0
due to no backward compatibility between spring 2.0 and 1.0 APIs. With this
approach we must remember to add corresponding spring-data migration
instructions to the future ignite 2.5 migration guide.

We could keep spring 1 support and backward compatibility by creating a new
module "ignite-spring-2-data" and keeping existing ignite-spring-data as
is. I do not like this option since to me increased complexity and
maintainability costs overweight the benefits of protecting
"Ignite-spring-1" users.

I suggest you find a committer (see this list
), communicate
the implication I mentioned above and say that two people already approved
the code providing we are OK with the chosen approach.

Re: What's about releasing Ignite 2.5 a bit earlier?

[Alexey Kukushkin]

Denis,

Java thin client is in the final review stage - I expect it will go to
master sometime next week. April 30 release date seems to have enough
contingency for this component.

Thanks!

Re: Lightweight profiling of messages processing

[Alexey Kukushkin]

Hi Alexei,

Did you consider general-purpose APM
 tools
like free InspectIT  or commercial
DynaTrace  or AppDynamics
?

Java APM tools do not require writing code or even instrumenting Ignite
binaries: you just attach them as javaagent
 to
Ignite JVM and then you can configure "sensors" to track whatever API you
want. APM tools collect metrics from the sensors and provide sophisticated
analysis tools.

DynaTrace claims they have 2%-7% overhead (depending on application) but
you can always detach the tool if you do not always need it in production.

I did not try APM for Ignite myself but it might work well.

On Tue, Nov 27, 2018 at 4:37 PM Alexei Scherbakov <
alexey.scherbak...@gmail.com> wrote:

> Igniters,
>
> At work I often have to solve performance issues with Ignite cluster
> without having access to source code of running user application.
>
> Looks like Ignite have limited capabilities to identify bottlenecks without
> extensive profiling on server and client side (JFR recording , sampling
> profilers, regular thread dumps,  etc), which is not always possible.
>
> Even having profiling data not always helpful for determining several types
> of bottlenecks, on example, if where is a contention on single
> key/partition.
>
> I propose to implement new feature, like lightweight profiling of message
> processing.
>
> The feature will allow to have view on message processing statistics for
> each node and for all grid nodes.
>
> In short, it's necessary to track each message execution in executors and
> record execution statistics like synchronous execution time in executor
> thread and waiting time in queue.
>
> Full description:
>
> 1. Implement detailed tracking of message waiting in queue and actual
> processing by executors with splitting to several time bins. Example of
> detailed statistics for each processed message:
>
>
> Processing time(%)
>  Message   Total Average(ms)
>  < 1 ms   < 10 ms< 30ms< 50ms   < 100ms   < 250ms   <
> 500ms   < 750ms  < 1000ms  > 1000ms
>
> 
> GridNearSingleGetRequest   9043113720.023000
>   904240521 57394  7242  3961  1932   229
> 6124 4 4
>GridNearSingleGetResponse   3401344160.041000
>   340118791 11660  1167   729   901  1001
> 158 8 1 0
>  GridNearLockRequest770886890.079000
>77073458 11945  2299   643   31131
> 2 0 0 0
>  GridNearAtomicSingleUpdateInvokeRequest396457520.298000
>39580914 28222  6469  4638  9870 13414
> 2087   137 1 0
> GridDhtAtomicSingleUpdateRequest376368290.277000
>37579375 23247  5915  4210  8954 12917
> 2048   162 1 0
>  GridDhtAtomicDeferredUpdateResponse335801980.002000
>33579805   33751 3 1 1
> 0 0 0 0
> GridNearTxPrepareRequest216677900.238000
>21078069580126  1622  1261  2531  3631
> 49640 014
> GridDhtTxPrepareResponse209498730.316000
>17130161   3803105  4615  3162  4489  3721
> 57734 1 8
>  GridDhtTxPrepareRequest209498380.501000
>16158732   4750217 16183  5735  8472  8994
> 1353881153
>  GridDhtTxFinishResponse138350650.007000
>13832519  2476272814 1
> 0 0 0 0
>   GridDhtTxFinishRequest138350280.547000
>12084106   1736789  8971  2340  1792   807
> 11841 460
>  GridNearTxFinishRequest137621970.725000
>11811828   1942499  4441  1400  1201   524
> 893419   162
>GridDhtAtomicNearResponse 27844220.122000
> 2783393  1022 5 2 0 0
> 0 0 0 0
>   GridNearGetRequest 23604830.484000
> 2345937 14129   244   10164 8
> 0 0 0 0
>  GridNearGetResponse 19842430.054000
> 1981905  2327 8 1 1 1
> 

Re: {DISCUSSION] Cluster read-only mode.

[Alexey Kukushkin]

I agree with Ivan's concern - do we really need the "activation" concept in
Ignite?

Activation was introduced with Ignite persistence: we must prevent both the
read and write operations on a cluster with persistence on until full data
set is loaded (all the nodes are started). Cluster "activation" was a hint
to the cluster to know that enough nodes had started for the cluster to have
all the data.

Then we introduced the concept of "baseline topology". It looks like the
"cluster is active" is similar to "cluster has baseline topology defined". 

Can we remove the concept of "activation" now and leave only "set baseline
topology" ? Having duplicate concepts negatively impacts Ignite's usability,
making it unnecessary more complex. 



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Replacing NodeFilter functionality with label approach

[Alexey Kukushkin]

Pavel,

Just a real example you asked for: we have a user attribute "ROLE_DC",
which is a comma separated list like "wfe_a, as_a, db_a" (server role and
data center designator) and we have node filters to deploy services and
start caches on servers with specific role (like WFE) and sometimes
specific role and DC (like WFE_A). The node filter splits the list and uses
a regular expression to match each segment.

If you replace generic node filter with a user attribute filter then we
still can achieve what we need  by creating 3 user attributes (ROLE_DC,
ROLE and DC) but we lose flexibility since now adding a new data center
requires updating all cache and service configurations. With regex matching
we do not need to update the configurations since we still match all the
roles in the new DC.

So we would have a solution with user attributes filter but I we lose some
flexibility.

-- 
Best regards,
Alexey

Re: Java thin client errors handling

[Alexey Kukushkin]

Hi Guys,

Most likely I just forgot to remove the "internal" ClientError and
ClientProtocolError when I was implementing a review comment to merge the
Java thin client into ignite-core (originally I developed the Java thin
client as a separate module). I see that the ClientProtocolError is not
used in the code at all and the ClientError is used to report some SSL
initialisation failures.

We cannot throw "internal" exceptions from public API. I would open a
"newbie" ticket to get rid of the "internal" ClientError and
ClientProtocolError.

The public ClientException and its subclasses are used to report the Java
thin client errors now.

As I understand we are reviewing some new feature that utilises the
internal "ClientProtocolError". I would create a new public
org.apache.ignite.client.ClientProtocolException extending ClientException
instead.



On Tue, Mar 3, 2020 at 2:50 PM Igor Sapego  wrote:

> Hi,
>
> I believe, we definitely should not ignore this.
>
> Alexey, you are the author of this code. What do you think?
>
> Best Regards,
> Igor
>
>
> On Thu, Feb 27, 2020 at 3:56 PM Aleksandr Shapkin 
> wrote:
>
> > Hello!
> >
> >
> >
> > I just noticed that the Java thin client throws the following internal
> > exceptions:
> >
> > ClientProtocolError
> >
> > ClientError
> >
> >
> >
> > Since the classes are not public, there is no way to catch them properly
> in
> > user code.
> >
> > Consider the recent changes, introduced by IGNITE-9410:
> >
> >
> >
> > throw new ClientProtocolError(String.format("Transactions have not
> > supported by the server's " +
> >
> > "protocol version %s, required version %s",
> > req.clientChannel().serverVersion(), V1_5_0));
> >
> >
> >
> > The code above correctly verifies the server version against the current
> > client and throws an exception
> >
> > In case of an outdated server. The only way to catch it in user code is
> by
> > RuntimeException
> >
> > that feels too broad to use.
> >
> >
> >
> > I’d like to discuss what’d be the best option to handle this scenario:
> >
> > -Should we make the ClientError public?
> >
> > -Should we introduce a new public error for every particular
> > exception?
> >
> > -Just ignore this?
> >
> >
> >
> > Thoughts?
> >
> > --
> > Alex.
> >
>


-- 
Best regards,
Alexey

Re: [DISCUSSION] Key and Value fields with same name and SQL DML

[Alexey Kukushkin]

Denis,I like your idea. It would be like native "OKVM" (Object-KeyValue
Mapper) allowing Ignite to natively operate on domain entities. It also
means no key and value data duplication and thus addressing the original SQL
DML problem when only key field is modified.We could add a
configuration-based alternative to the AOP-style API you suggested. Of all
languages that Ignite supports I think only Java and C# have AOP. We need to
think about all languages when we add a new feature. I think your solution
does not completely overlap with the solution suggested in  IGNITE-12807
  : - You suggested a
new API that does not have a "Key" entity by definition. A huge advantage of
the existing Key/Value is it is a widely adopted JCache standard supported
by all Ignite competitors I am aware of. That means low cost of switching to
Ignite for users of other product and more confidence for existing Ignite
users. IGNITE-12807 suggests solving the problem with the Key/Value API. -
Implementing your solution is probably a significantly bigger effort than
what IGNITE-12807 suggests. IGNITE-12807 already has a patch with a
proof-of-concept implemented.Still, I think the urge for addressing it in
K/V API would be less if we have your kind of API in Java and C#.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: [DISCUSSION] Key and Value fields with same name and SQL DML

[Alexey Kukushkin]

> «Why SQL sets both fields? I just want to set the key object field!»

The solution from IGNITE-12807 has backward compatible API: you have to
explicitly specify fields having same name inside the Key and Value.
Otherwise the semantics is as is.

Denis's solution (that I like) is a new API not changing the K/V API at all.

> We want to make Ignite(or any database) too smart.

I agree that introducing a new feature or even a new small configuration
setting often makes a product more complex: more expensive to learn and
maintain. One should think hard whether benefits of some new function
overweight added complexity to the product.

To me in this case complexity and costs of the new configuration setting
(KeyValue field list in QueryEntity and CREATE TABLE params) justifies the
value for the users. 

In my experience "just rename the fields" is a huge inconvenience for real
applications:

 - Many are the existing (legacy) apps adding Ignite as one of the pieces.
Many are large enterprise apps with many thousands lines of code deployed in
mission critical PROD environments. Such apps already have a domain model
used by other enterprise components. "Renaming a field" in the domain model
entities is just not a practical solution for them. They will choose
developing a data access layer to map domain model to K/Vs.

 - As for having the user to always develop a data access layer - I already
mentioned popular problems should be solved in re-usable parts, which is
Ignite in this case. Besides, additional layers have performance and memory
impact and the suggested solution has none. 






--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: [DISCUSSION] Key and Value fields with same name and SQL DML

[Alexey Kukushkin]

Nikolay,

I think you and I agree here: there is no problem at all if the user uses
K/Vs in domain model OR developed a domain-to-K/V mapper OR does not use SQL
DML.

I think we just have different experience as to how "popular" is the user's
unwillingness to use one of the three existing approaches and desire to have
that addressed in Ignite.

Let's see what other community members think.




--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: [DISCUSSION] Key and Value fields with same name and SQL DML

[Alexey Kukushkin]

Denis, I like your proposal and will be glad to participate in developing a
solution in that way.

In additional to Nikolay's opinion about your ideas I think it is important
to get Dmitry Frolov's (who started the discussion) and Alexey Sasov's (who
also voted for the having this enhancement) opinions.

Dmitry F, Alexey S, suppose we solve this like Denis suggested - will you
then be happy to use the new API instead of fixing the existing K/V API? Or
you still think the K/V API has to be fixed?



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: [DISCUSSION] Consistency across java thin/thick APIs

[Alexey Kukushkin]

I am wrong Alex but let me provide some history: when I originally
developed the Java thin client 3 years ago I suggested to take all public
API out of ignite-core.jar into a separate ignite-api.jar and develop Java
thin client implementing the common API from the ignite-api.jar. So the
ignite-core and ignite-client would both depend on ignite-api. I was told
on the design review it was not an appropriate moment to make such a
decision. (I still think a separate ignite-api.jar would be a good idea).
Then I implemented the thin client as a separate JAR having its own API. On
the code review I was told to merge the client into ignite-core but keep
the API separate. The motivation was there was too much unsupported
functionality in the thin client to implement the thick client API at that
moment.

вт, 25 авг. 2020 г. в 00:13, Igor Sapego :

> Yes, it was an attempt to separate thick and thin clients as much as
> possible
> to move them in separate libs in future.
>
> Alex, what do you think? What is the right path here from the Java
> developer viewpoint?
>
> Best Regards,
> Igor
>
>
> On Mon, Aug 24, 2020 at 9:40 AM Alex Plehanov 
> wrote:
>
> > Hi Val,
> >
> > No, I don't know why ClientException was introduced. Perhaps it was an
> > attempt to strictly separate thin and thick implementations. But there
> > still some API and thick implementation reused in thin client
> > (BinaryObjects, Queries).
> >
> > пт, 21 авг. 2020 г. в 23:28, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com>:
> >
> > > Hi Alex,
> > >
> > > Do you know if there is any particular reason why we introduced
> > > the ClientException in the first place? Generally, I believe that
> > > consistency is a good thing, but there might be something that we're
> > > missing.
> > >
> > > -Val
> > >
> > > On Fri, Aug 21, 2020 at 2:33 AM Alex Plehanov  >
> > > wrote:
> > >
> > > > Hello, Igniters!
> > > >
> > > > Since we touched on the subject of consistency across thin/thick APIs
> > in
> > > > thread [1] I would like to continue the discussion here.
> > > >
> > > > Currently, thick and thin APIs look similar but have no common
> > parents. A
> > > > set of ClientCache methods is almost a subset of IgniteCache methods,
> > but
> > > > these interfaces are fully independent. So we can't build common
> > > > abstraction which can work with ClientCache and IgniteCache without
> > code
> > > > duplication. For example, we have ignite-spring-data module, to make
> it
> > > > thin-client compliant we should copy-paste all cache methods
> > invocations.
> > > >
> > > > It's impossible to make thin and thick interfaces absolutely
> identical
> > > and
> > > > we can't extend one from another since there thick and thin specific
> > > > methods needed, but we can move common methods to the parent
> interface.
> > > But
> > > > here is another problem: exceptions. Thick methods declared as
> throwing
> > > > IgniteException and IgniteCheckedException, but all thin-client
> > > interfaces
> > > > use ClientException. Without common exceptions for thin and thick
> > > clients,
> > > > we can't build common parent interface.
> > > >
> > > > Currently, only a small amount of thick interfaces can be fully
> reused
> > by
> > > > thin-client (Queries, IgniteBinary, ClusterNode)
> > > >
> > > > Also, I would like to discuss JCache support by thin-client. We
> already
> > > use
> > > > some of JCache specification for thin-client, but not fully support
> > > JCache.
> > > > For example, we use Cache.Entry for queries and JCache expiration
> > > policies
> > > > in public thin-client API (also, some of JCache interfaces are used
> by
> > > the
> > > > internal implementation).
> > > >
> > > > I've tried to implement POC for continuous queries for java thin
> > client,
> > > > but without JCache support API looks weird. On the one hand, we
> should
> > > use
> > > > CacheEntryEventFilter (JCache interface) since it's required by the
> > > > server-side, on the other hand, we can't use
> CacheEntryUpdatedListener
> > > > since it requires CacheEntryEvent which requires an instance of Cache
> > > > (JCache interface), which doesn't exist on the thin-client side.
> > > >
> > > > We have plans to change public API in Ignite 3.0, perhaps it makes
> > sense
> > > to
> > > > make thick and thin API consistent. WDYT?
> > > > What about JCache support by thin-client in Ignite 3.0?
> > > > Please, share your thoughts.
> > > >
> > > > [1]:
> > > >
> > > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/IEP-51-Java-Thin-Client-Async-API-td48900.html
> > > >
> > >
> >
>


-- 
Best regards,
Alexey

Re: [DISCUSSION] Ignite.C++ and CMake

[Alexey Kukushkin]

+1. I recently completed a cross-IDE (MS Visual Studio & GCC/GDB) Ignite
C++ project and CMake in Ignite C++ would save me a day of effort.

вт, 26 мая 2020 г. в 12:09, Pavel Tupitsyn :

> +1
>
> On Tue, May 26, 2020 at 12:02 PM Zhenya Stanilovsky
>  wrote:
>
> >
> > Ivan huge +1 with your proposal.
> > I had some problems with odbc tests building too, looks like cmake will
> > make it more easy.
> > >Hello Igniters.
> > >
> > >I’d like to discuss porting build process of Ignite.C++. I think that
> > there is time to change it.
> > >
> > >*Motivation*
> > >Currently, it is hard to build Ignite.C++. Different build process for
> > windows and linux, lack of building support on Mac OS X (quite popular OS
> > among developers), absolutely not IDE support, except windows and only
> > Visual Studio is supported.
> > >
> > >*Suggestion*
> > >I’d suggest to migrate to CMake build system. It is very popular among
> > open source projects, and in Apache Software Foundation too. Notable
> user:
> > Apache Mesos, Apache Zookeeper (C client offers CMake as an alternative
> to
> > autoconf and only option on windows), Apache Kafka (librdkafka - C/C++
> > client), Apache Thrift. Popular column-oriented database ClickHouse also
> > uses CMake.
> > >
> > >CMake is widely supported in many IDE’s on various platforms, notably
> > Visual Studio, CLion, Xcode, QtCreator, KDevelop.
> > >
> > >*Current status*
> > >
> > >Currently, the most of work is done (see [1]) and tested on Mac OS X
> > 10.15 (some C++ porting). All tests are run without any flaws, including
> > odbc (unixodbc), ssl, thin and thick client, installation, IDE
> integration
> > (CLion). Next steps is to test linux and windows.
> > >
> > >But full migration isn’t possible without agreement and help of
> > community. Even if most of all you agree, migration requires additional
> > efforts in TC agents tuning and so on (event though test running fully
> > automated by CMake CTest).
> > >
> > >Lets discuss my proposition and idea.
> > >
> > >[1] -  https://github.com/apache/ignite/pull/7845
> > >
> > >
> > >
> >
> >
> >
> >
>


-- 
Best regards,
Alexey

Re: [DISCUSSION] Key and Value fields with same name and SQL DML

[Alexey Kukushkin]

I vote for implementing this enhancement: having key information separated
from value means either cluttering domain model with
KeyValue/CacheEntry-like structures or developing an Ignite data access
layer to insert key info into domain entity after getting it from Ignite. 

Although both the options solve the issue, I think the right approach is to
address it in Ignite rather than make all the Ignite users repeatedly deal
with the issue. It is the Ignite which is a re-usable platform and not
vise-versa.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Interoperable Ignite.NET Dates

[Alexey Kukushkin]

Igniters,

What do you think about changing .NET API to read/write portable dates by
default and making that really portable?

*The Problem*
Presently .NET API writes dates as composite Ignite objects. Only .NET
clients can read such dates: any other client (JDBC, Java, etc) does not
understand it without custom deserialization.

It is still possible to configure .NET serialization to write dates as
Ignite dates - see DateTime Serialization note
.
But then Ignite accepts only UTC dates, requiring the application
developers to convert local dates to UTC dates and back. This task is not
trivial: DateTime.ToUniversalTime

uses
calendars different from Java (and the .NET calendars are the invalid ones
- especially for pre-1990 dates).

The motivation for the current default behavior was probably the desire to
keep the Time Zone information: Ignite dates do not store time zones.

In our experience interoperability is more important than storing time zone
info.

*The Proposal*

   1. Always write .NET dates as portable Ignite dates: get rid of the
   BinaryReflectiveSerializer.ForceTimestamp flag that currently triggers this
   behavior.
  - We could still keep the ForceTimestamp flag if saving .NET dates as
  transparent objects seems a useful case. We do not think it is useful.
   2. Automatically convert Local dates to UTC and back *inside*
   Ignite.NET.
  - In this case we lose the DateTime.Kind of UTC dates: we write a UTC
  date but we would read a Local date since Ignite would always convert UTC
  to Local when reading.
  We could add a UtcDate date flag to QuerySqlFieldAttribute
  and BinaryReflectiveSerializer to control the deserialization behavior if
  keeping dates in UTC format use case seems important.
   3. Use NodaTime  for UTC<->Local conversions.
   Noda time uses Java calendars making the conversion truely portable.

*The Benefits*

   1. We think portable dates are much more important than storing time
   zone info. Why do we store time zones for every date on the server anyway?
   Time zone is client-side info.
   2. Simpler application code: no more manual configuration to trigger the
   portable behavior.
   3. Non-trivial code to make the truly portable UTC<->Local conversion is
   implemented once inside Ignite instead of having every Ignite.NET
   application implementing it.

Thank you!

-- 
Best regards,
Alexey

Re: Interoperable Ignite.NET Dates

[Alexey Kukushkin]

Val, absolutely - our proposal affects .NET API only.

вт, 3 нояб. 2020 г. в 15:00, Valentin Kulichenko <
valentin.kuliche...@gmail.com>:

> Got it. So it only affects the .NET side, correct?
>
> -Val
>
> On Tue, Nov 3, 2020 at 3:52 AM Alexey Kukushkin  >
> wrote:
>
> > Hi Val,
> >
> > Our proposal does not overlap with IEP-54
> > <
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach
> > >,
> > which proposes changing Ignite date storage format. Our proposal is
> > enhancing Ignite to handle .NET dates in a truly portable way instead of
> > requiring the application developers to repeat this task every time:
> >
> >- Write .NET dates as Ignite dates (today .NET dates are written as
> >generic Ignite objects)
> >- Convert Local .NET dates to UTC inside Ignite and to it using Java
> >calendars.
> >
> >
> > вт, 3 нояб. 2020 г. в 11:10, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com>:
> >
> > > Hi Alexey,
> > >
> > > The IEP-54 [1] describes the data layout proposed for Ignite 3.0, it
> > > includes various date/time types. Can you please take a look and check
> if
> > > this addresses your concerns? We can then discuss further if needed.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach
> > >
> > > -Val
> > >
> > > On Tue, Nov 3, 2020 at 12:05 AM Alexey Kukushkin <
> > > kukushkinale...@gmail.com>
> > > wrote:
> > >
> > > > Pavel,
> > > >
> > > > We propose changing public API so this is for Ignite 3.0.
> > > >
> > > > Thanks!
> > > >
> > > > вт, 3 нояб. 2020 г. в 02:17, Pavel Tupitsyn :
> > > >
> > > > > Alexey,
> > > > >
> > > > > Just to clarify before we start the discussion:
> > > > > this proposal seems to introduce some breaking changes, so we are
> > > talking
> > > > > about Ignite 3.0, correct?
> > > > >
> > > > > Pavel
> > > > >
> > > > > On Tue, Nov 3, 2020 at 12:13 AM Alexey Kukushkin <
> > > > > kukushkinale...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > What do you think about changing .NET API to read/write portable
> > > dates
> > > > by
> > > > > > default and making that really portable?
> > > > > >
> > > > > > *The Problem*
> > > > > > Presently .NET API writes dates as composite Ignite objects. Only
> > > .NET
> > > > > > clients can read such dates: any other client (JDBC, Java, etc)
> > does
> > > > not
> > > > > > understand it without custom deserialization.
> > > > > >
> > > > > > It is still possible to configure .NET serialization to write
> dates
> > > as
> > > > > > Ignite dates - see DateTime Serialization note
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ignite.apache.org/docs/latest/net-specific/net-platform-interoperability#types-compatibility
> > > > > > >.
> > > > > > But then Ignite accepts only UTC dates, requiring the application
> > > > > > developers to convert local dates to UTC dates and back. This
> task
> > is
> > > > not
> > > > > > trivial: DateTime.ToUniversalTime
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.microsoft.com/en-us/dotnet/api/system.datetime.touniversaltime?view=netcore-3.1
> > > > > > >
> > > > > > uses
> > > > > > calendars different from Java (and the .NET calendars are the
> > invalid
> > > > > ones
> > > > > > - especially for pre-1990 dates).
> > > > > >
> > > > > > The motivation for the current default behavior was probably the
> > > desire
> > > > > to
> > > > > > keep the Time Zone information: Ignite dates do not store time
> > zones.
> > > > > >
> > > > > > In our experience interoperabili

Re: Interoperable Ignite.NET Dates

[Alexey Kukushkin]

Pavel,

We propose changing public API so this is for Ignite 3.0.

Thanks!

вт, 3 нояб. 2020 г. в 02:17, Pavel Tupitsyn :

> Alexey,
>
> Just to clarify before we start the discussion:
> this proposal seems to introduce some breaking changes, so we are talking
> about Ignite 3.0, correct?
>
> Pavel
>
> On Tue, Nov 3, 2020 at 12:13 AM Alexey Kukushkin <
> kukushkinale...@gmail.com>
> wrote:
>
> > Igniters,
> >
> > What do you think about changing .NET API to read/write portable dates by
> > default and making that really portable?
> >
> > *The Problem*
> > Presently .NET API writes dates as composite Ignite objects. Only .NET
> > clients can read such dates: any other client (JDBC, Java, etc) does not
> > understand it without custom deserialization.
> >
> > It is still possible to configure .NET serialization to write dates as
> > Ignite dates - see DateTime Serialization note
> > <
> >
> https://ignite.apache.org/docs/latest/net-specific/net-platform-interoperability#types-compatibility
> > >.
> > But then Ignite accepts only UTC dates, requiring the application
> > developers to convert local dates to UTC dates and back. This task is not
> > trivial: DateTime.ToUniversalTime
> > <
> >
> https://docs.microsoft.com/en-us/dotnet/api/system.datetime.touniversaltime?view=netcore-3.1
> > >
> > uses
> > calendars different from Java (and the .NET calendars are the invalid
> ones
> > - especially for pre-1990 dates).
> >
> > The motivation for the current default behavior was probably the desire
> to
> > keep the Time Zone information: Ignite dates do not store time zones.
> >
> > In our experience interoperability is more important than storing time
> zone
> > info.
> >
> > *The Proposal*
> >
> >1. Always write .NET dates as portable Ignite dates: get rid of the
> >BinaryReflectiveSerializer.ForceTimestamp flag that currently triggers
> > this
> >behavior.
> >   - We could still keep the ForceTimestamp flag if saving .NET dates
> as
> >   transparent objects seems a useful case. We do not think it is
> > useful.
> >2. Automatically convert Local dates to UTC and back *inside*
> >Ignite.NET.
> >   - In this case we lose the DateTime.Kind of UTC dates: we write a
> UTC
> >   date but we would read a Local date since Ignite would always
> > convert UTC
> >   to Local when reading.
> >   We could add a UtcDate date flag to QuerySqlFieldAttribute
> >   and BinaryReflectiveSerializer to control the deserialization
> > behavior if
> >   keeping dates in UTC format use case seems important.
> >3. Use NodaTime <https://nodatime.org/> for UTC<->Local conversions.
> >Noda time uses Java calendars making the conversion truely portable.
> >
> > *The Benefits*
> >
> >1. We think portable dates are much more important than storing time
> >zone info. Why do we store time zones for every date on the server
> > anyway?
> >Time zone is client-side info.
> >2. Simpler application code: no more manual configuration to trigger
> the
> >portable behavior.
> >3. Non-trivial code to make the truly portable UTC<->Local conversion
> is
> >implemented once inside Ignite instead of having every Ignite.NET
> >application implementing it.
> >
> > Thank you!
> >
> > --
> > Best regards,
> > Alexey
> >
>


-- 
Best regards,
Alexey

Re: Interoperable Ignite.NET Dates

[Alexey Kukushkin]

Hi Val,

Our proposal does not overlap with IEP-54
<https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach>,
which proposes changing Ignite date storage format. Our proposal is
enhancing Ignite to handle .NET dates in a truly portable way instead of
requiring the application developers to repeat this task every time:

   - Write .NET dates as Ignite dates (today .NET dates are written as
   generic Ignite objects)
   - Convert Local .NET dates to UTC inside Ignite and to it using Java
   calendars.


вт, 3 нояб. 2020 г. в 11:10, Valentin Kulichenko <
valentin.kuliche...@gmail.com>:

> Hi Alexey,
>
> The IEP-54 [1] describes the data layout proposed for Ignite 3.0, it
> includes various date/time types. Can you please take a look and check if
> this addresses your concerns? We can then discuss further if needed.
>
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach
>
> -Val
>
> On Tue, Nov 3, 2020 at 12:05 AM Alexey Kukushkin <
> kukushkinale...@gmail.com>
> wrote:
>
> > Pavel,
> >
> > We propose changing public API so this is for Ignite 3.0.
> >
> > Thanks!
> >
> > вт, 3 нояб. 2020 г. в 02:17, Pavel Tupitsyn :
> >
> > > Alexey,
> > >
> > > Just to clarify before we start the discussion:
> > > this proposal seems to introduce some breaking changes, so we are
> talking
> > > about Ignite 3.0, correct?
> > >
> > > Pavel
> > >
> > > On Tue, Nov 3, 2020 at 12:13 AM Alexey Kukushkin <
> > > kukushkinale...@gmail.com>
> > > wrote:
> > >
> > > > Igniters,
> > > >
> > > > What do you think about changing .NET API to read/write portable
> dates
> > by
> > > > default and making that really portable?
> > > >
> > > > *The Problem*
> > > > Presently .NET API writes dates as composite Ignite objects. Only
> .NET
> > > > clients can read such dates: any other client (JDBC, Java, etc) does
> > not
> > > > understand it without custom deserialization.
> > > >
> > > > It is still possible to configure .NET serialization to write dates
> as
> > > > Ignite dates - see DateTime Serialization note
> > > > <
> > > >
> > >
> >
> https://ignite.apache.org/docs/latest/net-specific/net-platform-interoperability#types-compatibility
> > > > >.
> > > > But then Ignite accepts only UTC dates, requiring the application
> > > > developers to convert local dates to UTC dates and back. This task is
> > not
> > > > trivial: DateTime.ToUniversalTime
> > > > <
> > > >
> > >
> >
> https://docs.microsoft.com/en-us/dotnet/api/system.datetime.touniversaltime?view=netcore-3.1
> > > > >
> > > > uses
> > > > calendars different from Java (and the .NET calendars are the invalid
> > > ones
> > > > - especially for pre-1990 dates).
> > > >
> > > > The motivation for the current default behavior was probably the
> desire
> > > to
> > > > keep the Time Zone information: Ignite dates do not store time zones.
> > > >
> > > > In our experience interoperability is more important than storing
> time
> > > zone
> > > > info.
> > > >
> > > > *The Proposal*
> > > >
> > > >1. Always write .NET dates as portable Ignite dates: get rid of
> the
> > > >BinaryReflectiveSerializer.ForceTimestamp flag that currently
> > triggers
> > > > this
> > > >behavior.
> > > >   - We could still keep the ForceTimestamp flag if saving .NET
> > dates
> > > as
> > > >   transparent objects seems a useful case. We do not think it is
> > > > useful.
> > > >2. Automatically convert Local dates to UTC and back *inside*
> > > >Ignite.NET.
> > > >   - In this case we lose the DateTime.Kind of UTC dates: we
> write a
> > > UTC
> > > >   date but we would read a Local date since Ignite would always
> > > > convert UTC
> > > >   to Local when reading.
> > > >   We could add a UtcDate date flag to QuerySqlFieldAttribute
> > > >   and BinaryReflectiveSerializer to control the deserialization
> > > > behavior if
> > > >   keeping dates in UTC format use case seems important.
> > > >3. Use NodaTime <https://nodatime.org/> for UTC<->Local
> > conversions.
> > > >Noda time uses Java calendars making the conversion truely
> portable.
> > > >
> > > > *The Benefits*
> > > >
> > > >1. We think portable dates are much more important than storing
> time
> > > >zone info. Why do we store time zones for every date on the server
> > > > anyway?
> > > >Time zone is client-side info.
> > > >2. Simpler application code: no more manual configuration to
> trigger
> > > the
> > > >portable behavior.
> > > >3. Non-trivial code to make the truly portable UTC<->Local
> > conversion
> > > is
> > > >implemented once inside Ignite instead of having every Ignite.NET
> > > >application implementing it.
> > > >
> > > > Thank you!
> > > >
> > > > --
> > > > Best regards,
> > > > Alexey
> > > >
> > >
> >
> >
> > --
> > Best regards,
> > Alexey
> >
>


-- 
Best regards,
Alexey

Re: Interoperable Ignite.NET Dates

[Alexey Kukushkin]

We already created tickets and tested the proposed solution with our
applications (already in PROD) and an internal fork of Ignite. This
discussion is a proposal to donate this change to open source Apache Ignite
is the community accepts it:

   - IGNITE-12824 .NET: Write DateTime in interoperable format by default
   <https://issues.apache.org/jira/browse/IGNITE-12824>
   - IGNITE-12825 Serialize Java and .NET dates using same calendars
   <https://issues.apache.org/jira/browse/IGNITE-12825>


вт, 3 нояб. 2020 г. в 15:08, Valentin Kulichenko <
valentin.kuliche...@gmail.com>:

> Cool, makes sense to me then. Please feel free to create a ticket and mark
> it with the "ignite-3" label.
>
> -Val
>
> On Tue, Nov 3, 2020 at 4:03 AM Alexey Kukushkin  >
> wrote:
>
> > Val, absolutely - our proposal affects .NET API only.
> >
> > вт, 3 нояб. 2020 г. в 15:00, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com>:
> >
> > > Got it. So it only affects the .NET side, correct?
> > >
> > > -Val
> > >
> > > On Tue, Nov 3, 2020 at 3:52 AM Alexey Kukushkin <
> > kukushkinale...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi Val,
> > > >
> > > > Our proposal does not overlap with IEP-54
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach
> > > > >,
> > > > which proposes changing Ignite date storage format. Our proposal is
> > > > enhancing Ignite to handle .NET dates in a truly portable way instead
> > of
> > > > requiring the application developers to repeat this task every time:
> > > >
> > > >- Write .NET dates as Ignite dates (today .NET dates are written
> as
> > > >generic Ignite objects)
> > > >- Convert Local .NET dates to UTC inside Ignite and to it using
> Java
> > > >calendars.
> > > >
> > > >
> > > > вт, 3 нояб. 2020 г. в 11:10, Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com>:
> > > >
> > > > > Hi Alexey,
> > > > >
> > > > > The IEP-54 [1] describes the data layout proposed for Ignite 3.0,
> it
> > > > > includes various date/time types. Can you please take a look and
> > check
> > > if
> > > > > this addresses your concerns? We can then discuss further if
> needed.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach
> > > > >
> > > > > -Val
> > > > >
> > > > > On Tue, Nov 3, 2020 at 12:05 AM Alexey Kukushkin <
> > > > > kukushkinale...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Pavel,
> > > > > >
> > > > > > We propose changing public API so this is for Ignite 3.0.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > вт, 3 нояб. 2020 г. в 02:17, Pavel Tupitsyn <
> ptupit...@apache.org
> > >:
> > > > > >
> > > > > > > Alexey,
> > > > > > >
> > > > > > > Just to clarify before we start the discussion:
> > > > > > > this proposal seems to introduce some breaking changes, so we
> are
> > > > > talking
> > > > > > > about Ignite 3.0, correct?
> > > > > > >
> > > > > > > Pavel
> > > > > > >
> > > > > > > On Tue, Nov 3, 2020 at 12:13 AM Alexey Kukushkin <
> > > > > > > kukushkinale...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Igniters,
> > > > > > > >
> > > > > > > > What do you think about changing .NET API to read/write
> > portable
> > > > > dates
> > > > > > by
> > > > > > > > default and making that really portable?
> > > > > > > >
> > > > > > > > *The Problem*
> > > > > > > > Presently .NET API writes dates as composite Ignite objects.
> > Only
> > > > > .NET
> > > > > > > > clients can read such dates: any other cl

Re: Interoperable Ignite.NET Dates

[Alexey Kukushkin]

Pavel,

Thank you for your positive feedback! I will create an IEP.

вт, 3 нояб. 2020 г. в 15:48, Pavel Tupitsyn :

> Alexey,
>
> The proposal looks good to me.
>
> Usually I would try to avoid extra dependencies in the core assembly,
> but NodaTime seems to be worth it.
>
> Would you like to prepare an IEP?
>
> Thanks,
> Pavel
>
>
>
> On Tue, Nov 3, 2020 at 3:15 PM Alexey Kukushkin  >
> wrote:
>
> > We already created tickets and tested the proposed solution with our
> > applications (already in PROD) and an internal fork of Ignite. This
> > discussion is a proposal to donate this change to open source Apache
> Ignite
> > is the community accepts it:
> >
> >- IGNITE-12824 .NET: Write DateTime in interoperable format by default
> ><https://issues.apache.org/jira/browse/IGNITE-12824>
> >- IGNITE-12825 Serialize Java and .NET dates using same calendars
> ><https://issues.apache.org/jira/browse/IGNITE-12825>
> >
> >
> > вт, 3 нояб. 2020 г. в 15:08, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com>:
> >
> > > Cool, makes sense to me then. Please feel free to create a ticket and
> > mark
> > > it with the "ignite-3" label.
> > >
> > > -Val
> > >
> > > On Tue, Nov 3, 2020 at 4:03 AM Alexey Kukushkin <
> > kukushkinale...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Val, absolutely - our proposal affects .NET API only.
> > > >
> > > > вт, 3 нояб. 2020 г. в 15:00, Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com>:
> > > >
> > > > > Got it. So it only affects the .NET side, correct?
> > > > >
> > > > > -Val
> > > > >
> > > > > On Tue, Nov 3, 2020 at 3:52 AM Alexey Kukushkin <
> > > > kukushkinale...@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Val,
> > > > > >
> > > > > > Our proposal does not overlap with IEP-54
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach
> > > > > > >,
> > > > > > which proposes changing Ignite date storage format. Our proposal
> is
> > > > > > enhancing Ignite to handle .NET dates in a truly portable way
> > instead
> > > > of
> > > > > > requiring the application developers to repeat this task every
> > time:
> > > > > >
> > > > > >- Write .NET dates as Ignite dates (today .NET dates are
> written
> > > as
> > > > > >generic Ignite objects)
> > > > > >- Convert Local .NET dates to UTC inside Ignite and to it
> using
> > > Java
> > > > > >calendars.
> > > > > >
> > > > > >
> > > > > > вт, 3 нояб. 2020 г. в 11:10, Valentin Kulichenko <
> > > > > > valentin.kuliche...@gmail.com>:
> > > > > >
> > > > > > > Hi Alexey,
> > > > > > >
> > > > > > > The IEP-54 [1] describes the data layout proposed for Ignite
> 3.0,
> > > it
> > > > > > > includes various date/time types. Can you please take a look
> and
> > > > check
> > > > > if
> > > > > > > this addresses your concerns? We can then discuss further if
> > > needed.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach
> > > > > > >
> > > > > > > -Val
> > > > > > >
> > > > > > > On Tue, Nov 3, 2020 at 12:05 AM Alexey Kukushkin <
> > > > > > > kukushkinale...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Pavel,
> > > > > > > >
> > > > > > > > We propose changing public API so this is for Ignite 3.0.
> > > > > > > >
>

Re: [DISCUSSION] IgniteFuture class future in Ignite-3.0.

[Alexey Kukushkin]

I do not like Java's CompletableFuture and prefer our own Future (revised
IgniteFuture).

My understanding of the Future (or Promise) pattern in general is having
two separate APIs:

   1. Server-side: create, set result, raise error, cancel from server.
   2. Client-side: get result, handle error, cancel from client

Java's CompletableFuture looks like both the client-side and
server-side API. The "Completeable" prefix in the name is already confusing
for a client since it cannot "complete" an operation, only a server can.

I would create our own IgniteFuture adding client-side functionality we
currently miss (like client-side cancellation).


пт, 26 мар. 2021 г. в 01:08, Valentin Kulichenko <
valentin.kuliche...@gmail.com>:

> Andrey,
>
> Can you compile a full list of these risky methods, and elaborate on what
> the risks are?
>
> Generally, CompletableFuture is a much better option, because it's
> standard. But we need to make sure it actually fits our needs and doesn't
> do more harm than good.
>
> -Val
>
> On Thu, Mar 25, 2021 at 12:23 PM Alexei Scherbakov <
> alexey.scherbak...@gmail.com> wrote:
>
> > I think both options are fine, but personally lean toward
> > CompletableFuture.
> >
> > чт, 25 мар. 2021 г. в 17:56, Atri Sharma :
> >
> > > I would suggest using CompletableFuture -- I don't see a need for a
> > custom
> > > interface that is unique to us.
> > >
> > > It also allows a lower barrier for new contributors for understanding
> > > existing code
> > >
> > > On Thu, 25 Mar 2021, 20:18 Andrey Mashenkov, <
> andrey.mashen...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Igniters,
> > > >
> > > > I'd like to start a discussion about replacing our custom
> IgniteFuture
> > > > class with CompletableFuture - existed JDK class
> > > > or rework it's implementation (like some other products done) to a
> > > > composition of CompletionStage and Future interfaces.
> > > > or maybe other option if you have any ideas. Do you?
> > > >
> > > > 1. The first approach pros and cons are
> > > > + Well-known JDK class
> > > > + Already implemented
> > > > - It is a class, not an interface.
> > > > - Expose some potentially harmful methods like "complete()".
> > > >
> > > > On the other side, it has copy() method to create defensive copy and
> > > > minimalCompletionStage() to restrict harmful method usage.
> > > > Thus, this look like an applicable solution, but we should be careful
> > > > exposing internal future to the outside.
> > > >
> > > > 2. The second approach is to implement our own interface like the
> next
> > > one:
> > > >
> > > > interface IgniteFuture extends CompletableStage, Future {
> > > > }
> > > >
> > > > Pros and cons are
> > > > + Our interfaces/classes contracts will expose an interface rather
> than
> > > > concrete implementation.
> > > > + All methods are safe.
> > > > - Some implementation is required.
> > > > - CompletableStage has a method toCompletableFuture() and can be
> > > converted
> > > > to CompletableFuture. This should be supported.
> > > >
> > > > However, we still could wrap CompletableFuture and don't bother about
> > > > creating a defensive copy.
> > > >
> > > >
> > > > Other project experience:
> > > > * Spotify uses CompletableFuture directly [1].
> > > > * Redis goes the second approach [2]
> > > > * Vertx explicitly extends CompletableFuture [3]. However, they have
> > > custom
> > > > future classes and a number of helpers that could be replaced with
> > > > CompletableStage. Maybe it is just a legacy.'
> > > >
> > > > Any thoughts?
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://spotify.github.io/completable-futures/apidocs/com/spotify/futures/ConcurrencyReducer.html
> > > > [2]
> > > >
> > > >
> > >
> >
> https://lettuce.io/lettuce-4/release/api/com/lambdaworks/redis/RedisFuture.html
> > > > [3]
> > > >
> > > >
> > >
> >
> https://javadoc.io/static/org.jspare.vertx/vertx-jspare/1.1.0-M03/org/jspare/vertx/concurrent/VertxCompletableFuture.html
> > > > --
> > > > Best regards,
> > > > Andrey V. Mashenkov
> > > >
> > >
> >
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
> >
>


-- 
Best regards,
Alexey

Re: IEP-70: Async Continuation Executor

[Alexey Kukushkin]

Hi Pavel,

Extending Java async API with additional Executor parameters looks OK to me.

It is not clear from the IEP how you are going to do that for .NET async
API. My understanding is in .NET we do not add any Executors. Instead, the
Ignite Async API should use (SynchronizationContext.Current ??
TaskScheduler.Current) by default and it should have exciting behavior (use
Ignite striped pool) if ConfigureAwait(false) was specified for the Task
result.

Is my understanding correct?


пн, 15 мар. 2021 г. в 19:24, Pavel Tupitsyn :

> Igniters,
>
> Please review the IEP [1] and let me know your thoughts.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-70%3A+Async+Continuation+Executor
>


-- 
Best regards,
Alexey

Re: IEP-70: Async Continuation Executor

[Alexey Kukushkin]

Raymond,

The article you referenced
<https://devblogs.microsoft.com/dotnet/configureawait-faq/> is great! It
seems to me the article rather proved what I suggested: an async operation
implementation:

   - Uses the async operation thread (Ignite's thread) if ConfigureAwait is
   false (by default it is true)
   - Uses caller's current SynchornizationContext if it is specified
   - Uses caller's TaskScheduler.Current if current
   SynchornizationContext is null

In the application code (outside framework callbacks) the
SynchornizationContext.Current will be null and TaskScheduler.Current is
the .NET's fork-join thread pool. Thus, normally the .NET thread pool would
be used for continuations as Pavel suggested in the IEP.

Executing Async operation in a context where
SynchornizationContext.Current is not null means some framework explicitly
configured the context and that means it is important to execute the
continuation in that context (like UI or xUnit main thread)

I agree that routing back to original context might result in waiting,
which is very dangerous for an Ignite thread. We can create a worker thread
to route continuation to original context.


вт, 16 мар. 2021 г. в 21:40, Raymond Wilson :

> There is a (long) discussion here (
> https://devblogs.microsoft.com/dotnet/configureawait-faq/) regarding use
> of
> ConfigureAwait().
>
> Putting edge cases aside, there is a general use case for
> .ConfigureAwait(true) in application UI contexts to ensure the continuation
> joins a UI thread (for example), where failing to do so results in an
> error.
>
> The other general use case relates more to library code where you are
> typically running your tasks on the managed thread pool (assuming no custom
> task schedulers etc). In this case the .library is agnostic about which
> thread pool thread picks up the continuation, so forcing the continuation
> to join to the original thread may be both a performance penalty from the
> join overhead, but also may add latency waiting for that thread to become
> available again.
>
> I don't have good insight into how the striped thread pool is used in
> Ignite, but in highly concurrent environments letting those threads escape
> into the calling client context seems like a bad idea in general.
>
> Stepping back a little, the Cache Async operations are good for when there
> will be an IO operation incurred because the requested element is not
> present in memory. If it is present in memory, then a Sync operation will
> be more performant. Is it possible to do a two step operation like this
> 'Cache.GetIfInMemory() ?? await Cache.GetAsync()' to allow the calling
> context to optimise the calling model dynamically?
>
> Thanks,
> Raymond.
>
>
> On Wed, Mar 17, 2021 at 6:14 AM Alexey Kukushkin <
> kukushkinale...@gmail.com>
> wrote:
>
> > Pavel,
> >
> > My understanding might be wrong but I think the best practice (or even
> > strongly recommended way) to implement async methods in .NET is to
> execute
> > continuation on the caller's thread if ConfigureAwait(false) was not
> > specified. Pseudo-code might look like:
> >
> > async Task PutAsync(K k, V v)
> > {
> > var continuationExecutor = configureAwait
> > ? (SynchronizationContext.Current ?? TaskScheduler.Current)
> > : null;
> >
> > await <>
> >
> > continuationExecutor.Post(continuation);
> > }
> >
> > I got this understanding from reading some blog
> > about SynchronizationContext lots of time ago. They were saying they
> > created SynchronizationContext specifically to allow posting
> continuations
> > to the caller's thread.
> >
> > The reason for that is to simplify the user's code to avoid routing in
> the
> > code. Suppose you have a UI (like WPF or WinForms) event handler that
> must
> > be processed on the U thread:
> >
> > async Task Button1_Click(EventArgs args)
> > {
> > ignite.PutAsync(args.Key, args.Value);
> > Button1.Disabled = true;
> > }
> >
> > Executing the "Button1.Disabled = true" on a ForkJoinPool pool would
> cause
> > a "Trying to modify UI on a non-UI thread" exception. But if you
> > capture SynchronizationContext.Current in PutAsync and then route
> > continuation to the captured context then the code would work.
> >
> > I think the users really expect the continuations to be executed on the
> > caller's thread.
> >
> > Sometimes you know that your continuation is really fast and safe and you
> > want to avoid switching threads to improve performance. In this case you
> > use ConfigureAwait(false) like
> >
> > ignite.PutA

Re: IEP-70: Async Continuation Executor

[Alexey Kukushkin]

Pavel,

So what you are saying is submitting a continuation to .NET Thread Pool
already respects current  SynchronizationContext and ConfigureAwait. In
this case the IEP looks complete (for some reason I thought that we should
take care about the SynchronizationContext inside the Ignite.NET
implementation).

-- 
Best regards,
Alexey

Re: IEP-70: Async Continuation Executor

[Alexey Kukushkin]

My initial confusion was due to I did not know the
TaskCompletionSource.SetResult() internally handles
SynchronizationContext captured before the async operation. I thought we
would have to do it in Ignite. Now I know that and have no problems with
the IEP.

ср, 17 мар. 2021 г. в 11:18, Pavel Tupitsyn :

> Denis,
>
> > we don't try to solve the problem mentioned in the IEP
>
> The problem: user code executes on striped pool (which causes issues)
> The solution: don't execute user code on striped pool
>
> I believe this solves the problem and removes any and all restrictions
> for async listeners/continuations. Am I missing something else?
>
> On Wed, Mar 17, 2021 at 10:16 AM Denis Garus  wrote:
>
> > Hello!
> >
> > As I understood, we don't try to solve the problem mentioned in the IEP:
> > there is unexpected behavior,
> > users should carefully read the docs to know about this, and so on.
> > A thread that executes an incorrect listener hung in any case,
> > and the suggested solution is to change the pool which owns this thread.
> > Did you consider an option to execute user-defined listeners with a
> > timeout?
> > I think that implicit using java pools to run a code that can be hung is
> a
> > questionable solution.
> >
> > вт, 16 мар. 2021 г. в 23:30, Alexey Kukushkin  >:
> >
> > > Pavel,
> > >
> > > So what you are saying is submitting a continuation to .NET Thread Pool
> > > already respects current  SynchronizationContext and ConfigureAwait. In
> > > this case the IEP looks complete (for some reason I thought that we
> > should
> > > take care about the SynchronizationContext inside the Ignite.NET
> > > implementation).
> > >
> > > --
> > > Best regards,
> > > Alexey
> > >
> >
>


-- 
Best regards,
Alexey

Re: IEP-70: Async Continuation Executor

[Alexey Kukushkin]

Pavel,

My understanding might be wrong but I think the best practice (or even
strongly recommended way) to implement async methods in .NET is to execute
continuation on the caller's thread if ConfigureAwait(false) was not
specified. Pseudo-code might look like:

async Task PutAsync(K k, V v)
{
var continuationExecutor = configureAwait
? (SynchronizationContext.Current ?? TaskScheduler.Current)
: null;

await <>

continuationExecutor.Post(continuation);
}

I got this understanding from reading some blog
about SynchronizationContext lots of time ago. They were saying they
created SynchronizationContext specifically to allow posting continuations
to the caller's thread.

The reason for that is to simplify the user's code to avoid routing in the
code. Suppose you have a UI (like WPF or WinForms) event handler that must
be processed on the U thread:

async Task Button1_Click(EventArgs args)
{
ignite.PutAsync(args.Key, args.Value);
Button1.Disabled = true;
}

Executing the "Button1.Disabled = true" on a ForkJoinPool pool would cause
a "Trying to modify UI on a non-UI thread" exception. But if you
capture SynchronizationContext.Current in PutAsync and then route
continuation to the captured context then the code would work.

I think the users really expect the continuations to be executed on the
caller's thread.

Sometimes you know that your continuation is really fast and safe and you
want to avoid switching threads to improve performance. In this case you
use ConfigureAwait(false) like

ignite.PutAsync(args.Key, args.Value).ConfigureAwat(false);

In this case the continuation executes on the Ignite thread without routing
to the caller's thread.

вт, 16 мар. 2021 г. в 18:49, Pavel Tupitsyn :

> Alexey,
>
> .NET thick API delegates to Java directly.
>
> When you do ICache.PutAsync():
> * Future is created on Java side, .listen() is called
> * TaskCompletionSource is created on .NET side, its Task is returned to the
> user
> * Operation completes, Future listener is called on the Java side
> * Listener invokes JNI callback to .NET, where
> TaskCompletionSource.SetResult is called
>
> Therefore, .NET user code (in ContinueWith or after await) will be executed
> on the Java
> thread that invokes the future listener.
>
> After the proposed fix, future listeners will be invoked on
> ForkJoinPool#commonPool (instead of striped pool).
> So .NET continuations will end up in commonPool as well, which solves the
> problem for .NET automatically, no changes required.
>
> Does that make sense?
>
> On Tue, Mar 16, 2021 at 1:52 PM Alexey Kukushkin <
> kukushkinale...@gmail.com>
> wrote:
>
> > Hi Pavel,
> >
> > Extending Java async API with additional Executor parameters looks OK to
> > me.
> >
> > It is not clear from the IEP how you are going to do that for .NET async
> > API. My understanding is in .NET we do not add any Executors. Instead,
> the
> > Ignite Async API should use (SynchronizationContext.Current ??
> > TaskScheduler.Current) by default and it should have exciting behavior
> (use
> > Ignite striped pool) if ConfigureAwait(false) was specified for the Task
> > result.
> >
> > Is my understanding correct?
> >
> >
> > пн, 15 мар. 2021 г. в 19:24, Pavel Tupitsyn :
> >
> > > Igniters,
> > >
> > > Please review the IEP [1] and let me know your thoughts.
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-70%3A+Async+Continuation+Executor
> > >
> >
> >
> > --
> > Best regards,
> > Alexey
> >
>


-- 
Best regards,
Alexey

Re: [DISCUSSION] Javadoc style requirements and checks in Ignite-3.

[Alexey Kukushkin]

I personally do not understand why we need mandatory javadoc even in public
modules. I think javadoc is needed only when the code is not
self-descriptive. What value does a javadoc like this bring

to a developer:

/** Default metrics history size (value is {@code 1}). */
public static final int DFLT_METRICS_HISTORY_SIZE = 1;

/** Metrics history time. */
private int metricsHistSize = DFLT_METRICS_HISTORY_SIZE;

BTW, I picked a random example and it already has a copy-paste mistake in
the javadoc, which is there for years. I think that indicates it has no
real value and developers just do it to comply with the rule.

Some say mandatory javadoc is for the discipline but I think Apache Ignite
developers are mature enough to understand when additional documentation is
really required.



пн, 19 апр. 2021 г. в 16:37, Ivan Pavlukhin :

> Hi Andrey and Igniters,
>
> Sorry if I misread something but I have totally different opinion in
> one aspect. In my mind it is much better in practice to follow strict
> rules for public API javadocs but not for internals. I would use
> static checks for API packages and disable them for internal ones and
> refine code readability during code review. Main motivation here is
> that ubiquitous javadocs did not work well in ignite-2 and I believe
> it would not in ignite-3.
>
> 2021-04-19 13:30 GMT+03:00, Andrey Mashenkov :
> > Hi Igniters,
> >
> > We use JDK Javadoc tool to validate javadocs on TC (Javadoc suite) in
> > Ignite 2 and now in Ignite 3.
> > A javadoc tool is designed for javadoc site generation that also performs
> > some basic checks and markup validation,
> > but has nothing to do with javadoc styles.
> >
> > I've found maven-checkstyle-plugin has modules for javadoc style
> checking,
> > which looks more suited for the issue.
> > I've tried to turn on javadoc checks in checkstyle plugin, then run it
> > against Ignite-3 main branch and got 1200+ errors.
> >
> > For Ignite-2 thing may goes worse and numbers can be much higher,
> > but let please, start a separate discussion for this if one feels it make
> > sense.
> >
> > Javadoc is part of documentation which a face of product and we MUST keep
> > high standards for javadocs as well.
> >
> > Let's improve this within the ticket [1] regarding the next suggestions:
> > 1. Add Javadoc checks to mavan-checkstyle-plugin. I've made a PR for
> > Ignite-3 [1] to turn on style checks for javadocs.
> >
> > 2. Keep the current Javadoc TC suite as is. because it allow detecting
> > markup errors regarding current javadoc tool version capabilities.
> >
> > 3. Fix the Codestyle guidelines to follow higher standards.
> > 3.1. Disallow empty javadocs (or with missed description) for member that
> > can be used outside the class/package/module by users or other
> developers.
> > I believe all methods/classes/fields that can be used by developers
> > (public, protected, package visible) MUST have meaningful description,
> > excepts may be tests, well-known constants (e.g. serialVersionUid), and
> > private members.
> > Actually, it not a big deal to put few words into javadoc even if the
> > method looks simple,
> > if one feels difficulties expressing a class/method purpose, then most
> > likely refactoring is needed.
> >
> > 3.2. Check all params/throws/returns/generics/deprecates MUST be
> > well-documented and goes in order.
> >
> > 3.4. Suggest to allow optional tags @apiNote, @implSpec, @implNote as
> > described in [3],
> > to put e.g. expectations/requirements of implementation for developers
> that
> > may be non-obvious.
> > The tags values are rendered in separate blocks on javadoc site.
> >
> > 3.5 However, one-liner javadoc like "{@inheritDoc}" does nothing and can
> be
> > safely omitted. I'd keep it.
> >
> > 3.6 Add javadoc checks for 'package-info'. Do we want an additional
> > requirement to every package has package-info?
> >
> > Working on the patch I've found it is impossible to have different
> policies
> > in the same config for different scopes: source and test code.
> > Thus, we can either exclude tests from style checks at all, which looks
> > like not a good idea,
> > or have different configs with different policies for source and test
> code.
> >
> > Any thoughts?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-14591
> > [2] https://github.com/apache/ignite-3/pull/98
> > [3] http://openjdk.java.net/jeps/8068562
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
> >
>
>
> --
>
> Best regards,
> Ivan Pavlukhin
>


-- 
Best regards,
Alexey

Re: [DISCUSSION] Error handling in Ignite 3

[Alexey Kukushkin]

Just some points looking questionable to me, although I realize the error
handling style may be very opinionated:

   - Would it make sense splitting the proposed composite error code
   (TBL-0001) into separate numeric code (0001) and scope/category ("TBL") to
   avoid parsing the code when an error handler needs to analyze only the
   category or the code?
   - "*The cause - short string description of an issue, readable by user.*".
   This terminology sounds confusing to me since that "cause" sounds like Java
   Throwable's Message to me and the "Cause" is a lower level exception.
   - "*The action - steps for a user to resolve error...*". The action is
   very important but do we want to make it part of the IgniteException? I do
   not think the recovery action text should be part of the exception.
   IgniteException may include a URL pointing to the corresponding
   documentation - this is discussable.
   - "*All public methods throw only unchecked IgniteException*" - I assume
   we still respect JCache specification and prefer using standard Java
   exception to signal about invalid parameters.
   - Why we do not follow the "classic" structured exception handling
   practices in Ignite:
  - Why do we not allow using checked exceptions? It seems to me
  sometimes forcing the user to handle an error serves as a hint and thus
  improves usability. For example, handling an optimistic/pessimistic
  transaction conflict/deadlock. Or handling a timeout for operations with
  timeouts.
  - Why single public IgniteException and no exception hierarchy? Java
  is optimized for structured exception handling instead of using
IF-ELSE to
  analyze the codes.
  - Why no nested exceptions? Sometimes an error handler is interested
  only in high level exceptions (like Invalid Configuration) and sometimes
  details are needed (like specific configuration parser exceptions).
   - For async methods returning a Future we may have a universal rule on
   how to handle exceptions. For example, we may specify that any async method
   can throw only invalid argument exceptions. All other errors are reported
   via the exceptionally(IgniteException -> {}) callback even if the async
   method was executed synchronously.


вт, 13 апр. 2021 г. в 12:08, Alexei Scherbakov :

> Igniters,
>
> I would like to start the discussion about error handling in Ignite 3 and
> how we can improve it compared to Ignite 2.
>
> The error handling in Ignite 2 was not very good because of generic
> CacheException thrown on almost any occasion, having deeply nested root
> cause and often containing no useful information on further steps to fix
> the issue.
>
> I aim to fix it by introducing some rules on error handling.
>
> *Public exception structure.*
>
> A public exception must have an error code, a cause, and an action.
>
> * The code - the combination of 2 byte scope id and 2 byte error number
> within the module. This allows up to 2^16 errors for each scope, which
> should be enough. The error code string representation can look like
> RFT-0001 or TBL-0001
> * The cause - short string description of an issue, readable by user. This
> can have dynamic parameters depending on the error type for better user
> experience, like "Can't write a snapshot, no space left on device {0}"
> * The action - steps for a user to resolve error situation described in the
> documentation in the corresponding error section, for example "Clean up
> disk space and retry the operation".
>
> Common errors should have their own scope, for example IGN-0001
>
> All public methods throw only unchecked
> org.apache.ignite.lang.IgniteException containing aforementioned fields.
> Each public method must have a section in the javadoc with a list of all
> possible error codes for this method.
>
> A good example with similar structure can be found here [1]
>
> *Async timeouts.*
>
> Because almost all API methods in Ignite 3 are async, they all will have a
> configurable default timeout and can complete with timeout error if a
> computation is not finished in time, for example if a response has not been
> yet received.
> I suggest to complete the async op future with TimeoutException in this
> case to make it on par with synchronous execution using future.get, which
> will throw java.util.concurrent.TimeoutException on timeout.
> For reference, see java.util.concurrent.CompletableFuture#orTimeout
> No special error code should be used for this scenario.
>
> *Internal exceptions hierarchy.*
>
> All internal exceptions should extend
> org.apache.ignite.internal.lang.IgniteInternalException for checked
> exceptions and
> org.apache.ignite.internal.lang.IgniteInternalCheckedException for
> unchecked exceptions.
>
> Thoughts ?
>
> [1] https://docs.oracle.com/cd/B10501_01/server.920/a96525/preface.htm
>
> --
>
> Best regards,
> Alexei Scherbakov
>


-- 
Best regards,
Alexey

[DISCUSSION] IgniteOutOfMemoryException may not be a critical failure

[Alexey Kukushkin]

Hi Igniters,

Currently Ignite treats the "not enough data region capacity" case as a
critical failure and does not allow configuring any of the default critical
failure handlers to ignore that error.

In our company we have different teams using Apache Ignite and none of them
wants to apply a default "stop server" or "restart server" handler when
encountering the problem. We rather want to report this problem to DevOps
and the end users.

We developed a custom failure handler to deal with the problem but the
solution is really clumsy. And the most important thing is we think
treating this problem as a critical failure is not what most users would
want.

What do you think about enhancing Ignite not to treat the "not enough data
region capacity" case as a critical failure?

We opened IGNITE-16272  for
this discussion with the description below:

The Problem
Ignite raises the IgniteOutOfMemoryException
if
a data region size is exceeded when trying to add more data to a cache.
Ignite considers the IgniteOutOfMemoryException as a critical failure. This
causes shutting down the Ignite server with the default failure handler.

However, reaching the data region capacity does not seem to be such a
critical problem requiring the server shutdown or restart. For example, in
our application we just want to report this problem back to the users and
notify the DevOps without applying the critical failure handler. To achieve
that, we had to define a custom FailureHandler that detects and ignores the
IgniteOutOfMemoryException and all the caused by the
IgniteOutOfMemoryException, allowing the final exception to reach the
application. This solution is clumsy and unreliable since it uses the
internal IgniteOutOfMemoryException definition and relies on a complex
secondary exception structure trying to find the IgniteOutOfMemoryException
among the suppressed exception and causes.

Ignite out-of-the-box failure handlers have the ignoredFailure property
that allows filtering out some kinds of failures. However, the
IgniteOutOfMemoryException is not among the FailureType
that
can be ignored.

The Proposal

   1. Does anyone really want to treat the "data region capacity exceeded"
   problem as a critical failure and stop or restart the server?
  - Consider never treating this condition as a critical failure. This
  change is not backward compatible.
  - Or add another item to the FailureType enumeration to optionally
  allow the users not to have that treated as a critical failure. This is
  backward-compatible.
   2. Make the IgniteOutOfMemoryException a public API (now it is in the
   internal package)
   3. Consider renaming IgniteOutOfMemoryException (for example, to
   something like NotEnoughStorageException) since the current name is similar
   to a really critical and usually unrecoverable Java's OutOfMemoryError
   although the IgniteOutOfMemoryException is not that critical.

--
Best regards,
Alexey

[jira] [Created] (IGNITE-5966) IgniteCache#get() fails with "Requesting mapping from grid failed" when deserialising binary object loaded from CacheJdbcPojoStoreFactory

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-5966:


 Summary: IgniteCache#get() fails with "Requesting mapping from 
grid failed" when deserialising binary object loaded from 
CacheJdbcPojoStoreFactory
 Key: IGNITE-5966
 URL: https://issues.apache.org/jira/browse/IGNITE-5966
 Project: Ignite
  Issue Type: Bug
 Environment: Ignite 2.1.4
Reporter: Alexey Kukushkin
Assignee: Alexey Kukushkin


STEPS TO REPRODUCE
1. A running MySQL database with at least one table with an Integer key and 
some data
2. Use WebConsole to automatically generate an Ignite project from the RDBMS. 
In the WebConsole add a cache for the table containing data
3. Build the project
4. Start the cluster (run ServerNodeSpringStartup)
5. Load the data (run LoadCaches)
6. Run simple "get data" code against the running cluster with the data loaded. 
Make sure you do NOT keep binary and do NOT put anything to the cache except 
loading data on step #5. For example, if the cache is "AircraftCache", the type 
is "Aircraft" and a row with ID 1 exists in the DB, then:
IgniteCache<Integer, Aircraft> aircraftCache = 
ignite.getOrCreateCache("AircraftCache");
System.out.format("1->%s\n", aircraftCache.get(1));
EXPECTED:
1...5: Project is generated, cluster runs, data is loaded
6: The entry with ID 1 is output to the console
ACTUAL:"
1..5: As expected
6: javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: 
Requesting mapping from grid failed for [platformId=0, typeId=-1267085398]
ANALYSIS
The “typeId -> MappedName” mappings are stored in the 
MarshallerContextImpl$allCaches[platformId] map.
My understanding is according to the existing implementations it is expected 
the mapping will always be registered when BinaryContext#descriptorForClass() 
-> MarshallerContextImpl#registerClassName(typeId) is called either from 
BinaryWriterExImpl or BinaryReaderExImpl.
However, that mechanism is never called when 
CacheJdbcPojoStore@buildBinaryObject builds the object, calling 
BinaryObjectBuilderImpl#build(). The latter method still requests 
BinaryContext#updateMetadata, which updates 
CacheObjectBinaryProcessorImpl#metadataFileStore on all server nodes. But the 
metadataFileStore is not the place where MarshallerContextImpl get mappings 
from.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6879) Support Spring 2.0

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-6879:


 Summary: Support Spring 2.0
 Key: IGNITE-6879
 URL: https://issues.apache.org/jira/browse/IGNITE-6879
 Project: Ignite
  Issue Type: Improvement
  Security Level: Public (Viewable by anyone)
  Components: spring
Affects Versions: 2.3
Reporter: Alexey Kukushkin


Ignite-spring and ignite-spring-data now use Spring 1.1 and cannot be rebuilt 
with Spring 2.0. Trying to change the Spring dependency version to 
2.0.0.release results in compile errors like below and requires regression in 
general. 
This improvement was created to either migrate from Spring 1.1 to 2.0 or create 
another set of modules ignite-spring-xxx-2 to have backward compatibility with 
Spring 1.1.

[ERROR] 
/Users/kukushal/Dev/incubator-ignite/modules/spring-data/src/main/java/org/apache/ignite/springdata/repository/IgniteRepository.java:[57,10]
 name clash: deleteAll(java.lang.Iterable) in 
org.apache.ignite.springdata.repository.IgniteRepository and 
deleteAll(java.lang.Iterable) in 
org.springframework.data.repository.CrudRepository have the same erasure, yet 
neither overrides the other




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7065) Cannot use multiple Ignite Kafka Sink Connectors

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7065:


 Summary: Cannot use multiple Ignite Kafka Sink Connectors 
 Key: IGNITE-7065
 URL: https://issues.apache.org/jira/browse/IGNITE-7065
 Project: Ignite
  Issue Type: Bug
  Components: 2.3
Affects Versions: 2.3
Reporter: Alexey Kukushkin
 Fix For: 2.4


Stopping an Ignite Kafka sink connector makes other connectors deployed in the 
same worker unusable. As a result we cannot stream different topics in 
different caches since connector support streaming to single cache only.

See 
http://apache-ignite-users.70518.x6.nabble.com/Issues-with-multiple-kafka-connectors-questions-regarding-ignite-caches-tc18297.html
 for more details. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7248) "Schema not found" error when setting streaming mode for JDBC driver

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7248:


 Summary: "Schema not found" error when setting streaming mode for 
JDBC driver
 Key: IGNITE-7248
 URL: https://issues.apache.org/jira/browse/IGNITE-7248
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.3
Reporter: Alexey Kukushkin


Using JDBC "thick" driver in streaming mode fails with a "Schema not found" 
erorr:

Connection connection = 
DriverManager.getConnection("jdbc:ignite:cfg://streaming=true:cache=CACHE@file:/path-to-ignite-config.xml");

// using connection to create a table works fine but this exception is 
generated when using the connection to execute INSER INTO...

class org.apache.ignite.internal.processors.query.IgniteSQLException: Failed
to set schema for DB connection for thread [schema=]

org.h2.jdbc.JdbcSQLException: Schema  not found; SQL statement:
SET SCHEMA "" [90079-195]

See [link 
UserList|https://mail.google.com/mail/u/0/#label/Apache+Ignite+Users/15f25f4d70ade793]
 for more details




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7215) Documentation bug: "Cross-cache Query" page is dead

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7215:


 Summary: Documentation bug: "Cross-cache Query" page is dead
 Key: IGNITE-7215
 URL: https://issues.apache.org/jira/browse/IGNITE-7215
 Project: Ignite
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.3
Reporter: Alexey Kukushkin


Could not find "Cross-cache Queries" section in the latest Ignite 2.3 docs. The 
only page that references "cross-cache queries" is [JDBC Client 
Driver|https://apacheignite-sql.readme.io/docs#section-jdbc-client-driver] and 
it points to a [dead 
link|https://apacheignite-sql.readme.io/docs/cache-queries#cross-cache-queries]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6889) ignite.queue() returns null when queue configuration is null

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-6889:


 Summary: ignite.queue() returns null when queue configuration is 
null
 Key: IGNITE-6889
 URL: https://issues.apache.org/jira/browse/IGNITE-6889
 Project: Ignite
  Issue Type: Bug
  Security Level: Public (Viewable by anyone)
Affects Versions: 2.3, 2.2
Reporter: Alexey Kukushkin


ignite.queue() returns null when queue configuration is null.

Reproducer: assertTrue(ignite.queue("a_queue", 1000, null) != null)

We should either allow specifying null as a queue configuration (meaning 
default configuration) or raise an IllegalArgumentException if collection 
configuration is null



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6802) NullPointerException when WAL store and WAL archive paths are same

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-6802:


 Summary: NullPointerException when WAL store and WAL archive paths 
are same 
 Key: IGNITE-6802
 URL: https://issues.apache.org/jira/browse/IGNITE-6802
 Project: Ignite
  Issue Type: Bug
  Security Level: Public (Viewable by anyone)
Affects Versions: 2.2
Reporter: Alexey Kukushkin
 Fix For: 2.4


Configuring WAL store and WAL archive paths to be the same results in 
NullPointerException. We should display a meaningful error about the 
misconfiguration instead.

See 
http://apache-ignite-users.70518.x6.nabble.com/NullPointerException-in-FileWriteAheadLogManager-java-1313-tc17860.html
The thread includes the reproducer code and stack trace. I was able to 
reproduce the issue with today's (31-Oct-2017) Apache master.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6801) Ignite Kafka Connector certification with Confluent

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-6801:


 Summary: Ignite Kafka Connector certification with Confluent
 Key: IGNITE-6801
 URL: https://issues.apache.org/jira/browse/IGNITE-6801
 Project: Ignite
  Issue Type: Improvement
  Security Level: Public (Viewable by anyone)
Affects Versions: 2.2
 Environment: 

Reporter: Alexey Kukushkin
Assignee: Alexey Kukushkin
 Fix For: 2.4


Certifying Ignite Kafka connector with Confluent will give Ignite Community 
benefits like:
* TBD (I will put the benefits here after I confirm them)

A certified connector must be coded according to [these 
requirements](https://www.confluent.io/wp-content/uploads/Partner-Dev-Guide-for-Kafka-Connect.pdf),
 reviewed and approved by Confluent.
The existing ignite-kafka module does not comply with the requirements.

The purpose of this task to develop a certified version of the connector.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7422) Thin client Java API - startClient

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7422:


 Summary: Thin client Java API - startClient
 Key: IGNITE-7422
 URL: https://issues.apache.org/jira/browse/IGNITE-7422
 Project: Ignite
  Issue Type: Sub-task
  Components: thin client
Reporter: Alexey Kukushkin


Implement below thin client Java API including unit and system tests and 
samples. BinaryObject, failover and encryption is not part of this task.

Ignition
    startClient(:ClientConfiguration): Ignite
ClientConfiguration(port, host, binaryConfiguration, sslConfiguration,
etc...)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7421) Thin client Java API - data grid API

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7421:


 Summary: Thin client Java API - data grid API
 Key: IGNITE-7421
 URL: https://issues.apache.org/jira/browse/IGNITE-7421
 Project: Ignite
  Issue Type: New Feature
  Components: thin client
 Environment: Implement below Java bindings for the thin client 
protocol. The client configuration must support failover and encryption.

Cache
    JCache (limited)
        getName(): String
        put(key, val)
        get(key): V
        getAll(keys: Set): Map
        containsKey(key): boolean
        getAndPut(key, val): V
        getAndReplace(key, val): V
        getAndRemove(key): V
        putIfAbsent
        replace(key, val)
        replace(key, oldVal, newVal)
        putAll
        clear
        remove(key)
        remove(key, val)
        removeAll()
        removeAll(keys: Set)
        getConfiguration(clazz): Configuration
        close()
    size(modes: CachePeekMode...)
    query(qry: Query): QueryCursor
    query(qry: SqlFieldsQuery): FieldsQueryCursor
    withKeepBinary(): IgniteCache
Ignite
    cache(name: String)
    cacheNames(): Collection
    binary(): IgniteBinary
    createCache(name): Cache
    getOrCreateCache(name): Cache
    destroyCache(name)
Ignition
    startClient(:ClientConfiguration): Ignite
ClientConfiguration(port, host, binaryConfiguration, sslConfiguration,
etc...)
Reporter: Alexey Kukushkin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7423) Thin client Java API - cache management

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7423:


 Summary: Thin client Java API - cache management
 Key: IGNITE-7423
 URL: https://issues.apache.org/jira/browse/IGNITE-7423
 Project: Ignite
  Issue Type: Sub-task
 Environment: Implement get/create/destroy cache thin client Java API 
including unit and system tests and samples.

Ignite
    cache(name: String)
    createCache(name): Cache
    getOrCreateCache(name): Cache
    destroyCache(name)
Reporter: Alexey Kukushkin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7424) Thin client Java API - cache enumeration

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7424:


 Summary: Thin client Java API - cache enumeration
 Key: IGNITE-7424
 URL: https://issues.apache.org/jira/browse/IGNITE-7424
 Project: Ignite
  Issue Type: Sub-task
 Environment: Implement list caches thin client Java API including unit 
and system tests and samples.

Ignite
    cacheNames(): Collection
Reporter: Alexey Kukushkin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7426) Thin client Java API - encryption

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7426:


 Summary: Thin client Java API - encryption
 Key: IGNITE-7426
 URL: https://issues.apache.org/jira/browse/IGNITE-7426
 Project: Ignite
  Issue Type: Sub-task
 Environment: Implement SSL/TLS configuration client Java API including 
unit and system tests and samples.

ClientConfiguration(ssl configuration)
Reporter: Alexey Kukushkin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7427) Thin client Java API - failover

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7427:


 Summary: Thin client Java API - failover
 Key: IGNITE-7427
 URL: https://issues.apache.org/jira/browse/IGNITE-7427
 Project: Ignite
  Issue Type: Sub-task
 Environment: Implement failover in thin client Java API including unit 
and system tests and samples.

ClientConfiguration(failover configuration)
Reporter: Alexey Kukushkin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7430) Thin client Java API - cache putAll/getAll

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7430:


 Summary: Thin client Java API - cache putAll/getAll
 Key: IGNITE-7430
 URL: https://issues.apache.org/jira/browse/IGNITE-7430
 Project: Ignite
  Issue Type: Sub-task
 Environment: Implement cache putAll/getAll thin client Java API 
including unit and system tests and samples.

Cache
        getAll(keys: Set): Map
         getAll
 
Reporter: Alexey Kukushkin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7429) Thin client Java API - cache put/get/contains

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7429:


 Summary: Thin client Java API - cache put/get/contains
 Key: IGNITE-7429
 URL: https://issues.apache.org/jira/browse/IGNITE-7429
 Project: Ignite
  Issue Type: Sub-task
 Environment: Implement cache put/get/contains thin client Java API 
including unit and system tests and samples.

Cache
       put(key, val)
        get(key): V
        containsKey(key): boolean
 
Reporter: Alexey Kukushkin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7433) Thin client Java API - cache clear

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7433:


 Summary: Thin client Java API - cache clear
 Key: IGNITE-7433
 URL: https://issues.apache.org/jira/browse/IGNITE-7433
 Project: Ignite
  Issue Type: Sub-task
 Environment: Implement cache clear thin client Java API including unit 
and system tests and samples.

Cache
        clear()
Reporter: Alexey Kukushkin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7432) Thin client Java API - cache atomic get/put/replace/remove

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7432:


 Summary: Thin client Java API - cache atomic 
get/put/replace/remove 
 Key: IGNITE-7432
 URL: https://issues.apache.org/jira/browse/IGNITE-7432
 Project: Ignite
  Issue Type: Sub-task
 Environment: Implement cache atomic get/put/replace/remove  thin 
client Java API including unit and system tests and samples.

Cache
        getAndPut(key, val): V
        getAndReplace(key, val): V
        getAndRemove(key): V
        putIfAbsent
Reporter: Alexey Kukushkin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7434) Thin client Java API - cache query API

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7434:


 Summary: Thin client Java API - cache query API
 Key: IGNITE-7434
 URL: https://issues.apache.org/jira/browse/IGNITE-7434
 Project: Ignite
  Issue Type: Sub-task
 Environment: Implement cache query thin client Java API including unit 
and system tests and samples.

Cache
        query(qry: Query): QueryCursor
Reporter: Alexey Kukushkin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7435) Thin client Java API - fields query API

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7435:


 Summary: Thin client Java API - fields query API
 Key: IGNITE-7435
 URL: https://issues.apache.org/jira/browse/IGNITE-7435
 Project: Ignite
  Issue Type: Sub-task
 Environment: Implement cache fields query thin client Java API 
including unit and system tests and samples.

Cache
         query(qry: SqlFieldsQuery): FieldsQueryCursor
Reporter: Alexey Kukushkin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7425) Thin client Java API - binary objects

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7425:


 Summary: Thin client Java API - binary objects
 Key: IGNITE-7425
 URL: https://issues.apache.org/jira/browse/IGNITE-7425
 Project: Ignite
  Issue Type: Sub-task
 Environment: Implement binary object thin client Java API including 
unit and system tests and samples.

Ignite
    binary(): IgniteBinary
Reporter: Alexey Kukushkin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7431) Thin client Java API - cache replace/remove API

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7431:


 Summary: Thin client Java API - cache replace/remove API
 Key: IGNITE-7431
 URL: https://issues.apache.org/jira/browse/IGNITE-7431
 Project: Ignite
  Issue Type: Sub-task
 Environment: Implement cache replace/remove thin client Java API 
including unit and system tests and samples.

Cache
        replace(key, val)
        replace(key, oldVal, newVal)
        remove(key)
        remove(key, val)
        removeAll()
        removeAll(keys: Set)
Reporter: Alexey Kukushkin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7411) JDBC thin client selects NULL cache ID values of entries added with Java API

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7411:


 Summary: JDBC thin client selects NULL cache ID values of entries 
added with Java API
 Key: IGNITE-7411
 URL: https://issues.apache.org/jira/browse/IGNITE-7411
 Project: Ignite
  Issue Type: Bug
  Components: thin client
Affects Versions: 2.3
Reporter: Alexey Kukushkin


I am using SQL to create caches and run queries and DataStreamer to load lots 
of data. I must use MD5 digests as entity IDs (BINARY(16) SQL type). The 
problem is when I select data loaded with DataStreamer the fields participating 
in PRIMARY KEY are null.
 * Note: binary fields not part of primary keys are OK (not null)
 * Note: if I use JAVA API even the binary primary key fields are OK (not null).

Reproducer below fails on the last assertion since SSN is NULL.

See full reproducer code in branch ignite-TBD.

@Test
 public void javaPutIntoSqlCacheWithBinaryAffinityKey() throws SQLException {
 try (Ignite srv = Ignition.start(getServerConfig());
 Ignite cln = Ignition.start(getClientConfig());
 Connection conn = DriverManager.getConnection("jdbc:ignite:thin://127.0.0.1/")
 ) {
 conn.prepareStatement(
 "CREATE TABLE " + CACHE_NAME + "(" +
 "ssn BINARY(16), orgId BINARY(16), name BINARY(16), PRIMARY KEY(ssn, orgId)" +
 ") WITH \"affinitykey=orgId,value_type=org.apache.ignite.Reproducer$Person\""
 ).execute();

IgniteCache<AffinityKey<byte[]>, Person> cache = cln.cache("SQL_PUBLIC_" + 
CACHE_NAME);

AffinityKey<byte[]> key = new AffinityKey<>(new byte[] \{1, 2}, new byte[] \{3, 
4});

cache.put(key, new Person(key.key(), key.affinityKey(), new byte[] \{5, 6}));

List entries = convert(conn.prepareStatement("SELECT * from " + 
CACHE_NAME).executeQuery());

assertEquals("1 person must be in the cache", 1, entries.size());

assertArrayEquals("Person SSN must be same as affinity key's key", key.key(), 
entries.get(0).getSsn());
 }
 }

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7623) Thin client Java API - async API

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7623:


 Summary: Thin client Java API - async API
 Key: IGNITE-7623
 URL: https://issues.apache.org/jira/browse/IGNITE-7623
 Project: Ignite
  Issue Type: Sub-task
  Components: clients
Reporter: Alexey Kukushkin


Implement Async version of all the client APIs. This is a big task - consider 
splitting it into multiple tasks. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9197) Java thin client querying empty table results in NoSuchElementException

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-9197:


 Summary: Java thin client querying empty table results in 
NoSuchElementException
 Key: IGNITE-9197
 URL: https://issues.apache.org/jira/browse/IGNITE-9197
 Project: Ignite
  Issue Type: Bug
  Components: thin client
Affects Versions: 2.6
 Environment: +*Reproducer*+
{code:java}
@Test
public void testGettingEmptyResultWhenQueryingEmptyTable() throws Exception {
try (Ignite ignored = Ignition.start(Config.getServerConfiguration());
 IgniteClient client = Ignition.startClient(new 
ClientConfiguration().setAddresses(Config.SERVER))
) {
final String TBL = "Person";

client.query(
new SqlFieldsQuery("CREATE TABLE IF NOT EXISTS " + TBL + " (id INT 
PRIMARY KEY, name VARCHAR)")
).getAll();

List> res = client.query(new SqlFieldsQuery("SELECT * FROM " + 
TBL)).getAll();

assertNotNull(res);
assertEquals(0, res.size());
}
}

{code}
*Expected*

The test above should pass

*Actual*

java.util.NoSuchElementException
 at java.util.ArrayList$Itr.next(ArrayList.java:862)
 at 
org.apache.ignite.internal.client.thin.ClientQueryCursor$1.next(ClientQueryCursor.java:87)
 at 
org.apache.ignite.internal.client.thin.ClientQueryCursor.getAll(ClientQueryCursor.java:45)
 at 
org.apache.ignite.client.FunctionalQueryTest.testGettingEmptyResultWhenQueryingEmptyTable(FunctionalQueryTest.java:171)
Reporter: Alexey Kukushkin
Assignee: Alexey Kukushkin
 Fix For: 2.7


+*Reproducer*+

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9204) Test org.apache.ignite.client.ReliabilityTest@testFailover fails intermittently

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-9204:


 Summary: Test 
org.apache.ignite.client.ReliabilityTest@testFailover fails intermittently
 Key: IGNITE-9204
 URL: https://issues.apache.org/jira/browse/IGNITE-9204
 Project: Ignite
  Issue Type: Test
  Components: thin client
Affects Versions: 2.6
Reporter: Alexey Kukushkin
Assignee: Alexey Kukushkin
 Fix For: 2.7


According to [Apache Ignite Team 
City|https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=1817609924700192054=testDetails],
 test org.apache.ignite.client.ReliabilityTest@testFailover fails 
intermittently with this message:

 
{noformat}
java.lang.AssertionError: Ignite cluster is unavailable expected null, but 
was: at 
org.apache.ignite.client.ReliabilityTest.assertOnUnstableCluster(ReliabilityTest.java:184)
 at 
org.apache.ignite.client.ReliabilityTest.testFailover(ReliabilityTest.java:88) 
* Text was not loaded fully because its' size exceeds 2 MB, see full log 
for the whole text *
{noformat}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9357) Spark Structured Streaming with Ignite as data source and sink

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-9357:


 Summary: Spark Structured Streaming with Ignite as data source and 
sink
 Key: IGNITE-9357
 URL: https://issues.apache.org/jira/browse/IGNITE-9357
 Project: Ignite
  Issue Type: New Feature
  Components: spark
Affects Versions: 2.7
Reporter: Alexey Kukushkin
Assignee: Alexey Kukushkin


We are working on a PoC where we want to use Ignite as a data storage and Spark 
as a computation engine. We found that Ignite is not supported neither as a 
source nor as a Sink when using Spark Structured Streaming, which is a must for 
us.

We are enhancing Ignite to support Spark streaming with Ignite. We will send 
docs and code for review for the Ignite Community to consider if the Community 
want to accept this feature. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7858) "Cannot find cache" error for existing cache on unstable topology

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-7858:


 Summary: "Cannot find cache" error for existing cache on unstable 
topology 
 Key: IGNITE-7858
 URL: https://issues.apache.org/jira/browse/IGNITE-7858
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.5
Reporter: Alexey Kukushkin


Java thin client high availability test fails with "Cannot find cache" error 
for existing cache.

Reproducer:

git fetch professional
git checkout ignite-7421
Open org.apache.ignite.thinclient.system.ReliabilityTest and search for "TODO: 
fix CACHE_DOES_NOT_EXIST server error". Then remove the try/catch block leaving 
only assertion call.
Run test testFailover



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8135) Missing SQL-DDL Authorization

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-8135:


 Summary: Missing SQL-DDL Authorization
 Key: IGNITE-8135
 URL: https://issues.apache.org/jira/browse/IGNITE-8135
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.5
Reporter: Alexey Kukushkin


Ignite has infrastructure to support 3-rd party security plugins. To support 
authorization, Ignite has security checks spread all over the code delegating 
actual authorization to a 3rd party security plugins if configured.

In addition to existing checks, Ignite 2.5 will authorise "create" and 
"destroy" cache operations.

The problem is authorization is not implemented for SQL at all - even if 
authorization is enabled, it is currently possible to run any SQL to 
create/drop/alter caches and read/modify/remove the cache data thus bypassing 
security. The problem exists for both DDL (create/drop/alter table) and DML 
(select/merge/insert/delete).

This ticket addresses DDL only: DML will be addressed by a different ticket.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8221) Cache management and server node authorisation

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-8221:


 Summary: Cache management and server node authorisation
 Key: IGNITE-8221
 URL: https://issues.apache.org/jira/browse/IGNITE-8221
 Project: Ignite
  Issue Type: Task
Reporter: Alexey Kukushkin
Assignee: Vladimir Ozerov


Add new authorisation checks requested by multiple Apache Ignite users:

CACHE_CREATE

CACHE_DESTROY

JOIN_AS_SERVER

Also, create an Ignite system property to allow disabling "on-heap" cache 
feature. 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8237) Ignite blocks on SecurityException in exchange-worker due to unauthorised off-heap cache configuration

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-8237:


 Summary: Ignite blocks on SecurityException in exchange-worker due 
to unauthorised off-heap cache configuration 
 Key: IGNITE-8237
 URL: https://issues.apache.org/jira/browse/IGNITE-8237
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.5
Reporter: Alexey Kukushkin
Assignee: Alexey Kukushkin


Ignite blocks on SecurityException in exchange-worker due to unauthorised 
off-heap cache configuration. Consider moving IGNITE_DISABLE_OFFHEAP_CACHE 
system property check to a more appropriate place to avoid blocking Ignite.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8076) Java Thin Client Authentication

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-8076:


 Summary: Java Thin Client Authentication
 Key: IGNITE-8076
 URL: https://issues.apache.org/jira/browse/IGNITE-8076
 Project: Ignite
  Issue Type: Task
Reporter: Alexey Kukushkin
Assignee: Alexey Kukushkin


Java thin client was merged into branch 2.4.4. Java thin client supports 
authentication by sending user credentials to server. Implement handling 
authentication on the server side.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8067) Java thin client code review: move parsing from TcpClientCache to ClientMessageParser

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-8067:


 Summary: Java thin client code review: move parsing from 
TcpClientCache to ClientMessageParser
 Key: IGNITE-8067
 URL: https://issues.apache.org/jira/browse/IGNITE-8067
 Project: Ignite
  Issue Type: Task
  Components: thin client
Affects Versions: 2.5
Reporter: Alexey Kukushkin


Code review comments 9 and 10 from 
https://issues.apache.org/jira/browse/IGNITE-7421?focusedCommentId=16398310=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16398310
 were left as technical debt due to Java thin client delivery urgency.
This task is to address:

9) {{SerDes}} - configuration serialization/deserialization is already 
implemented({{ClientCacheConfigurationSerializer}}). I think all we need is to 
implement converter from CacheConfiguration to ClientCacheConfiguration and 
vice versa.
10) {{ClientOperation}} and {{TcpClientCache}} - duplicates logic of 
{{ClientMessageParser}}. Let's simply add two new methods to decode response 
and encode request to this class, and remove all lambdas from 
{{TcpClientCache}} - we should not define the same format twice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10100) No public Java API to call Ignite.NET service

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-10100:
-

 Summary: No public Java API to call Ignite.NET service
 Key: IGNITE-10100
 URL: https://issues.apache.org/jira/browse/IGNITE-10100
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Affects Versions: 2.6
Reporter: Alexey Kukushkin
Assignee: Alexey Kukushkin


Ignite wraps .NET services in PlatformDotNetServiceImpl implementing 
PlatformService interface.

To call a .NET service from a Java client I have to use invokeMethod API 
defined in PlatformService interface. 

The problem is for some reason PlatformService is defined in internal Ignite 
package apache.ignite.internal.processors.platform.services. This API must be 
public.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10072) Apache.Ignite NuGet package removes existing post-build actions

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-10072:
-

 Summary: Apache.Ignite NuGet package removes existing post-build 
actions
 Key: IGNITE-10072
 URL: https://issues.apache.org/jira/browse/IGNITE-10072
 Project: Ignite
  Issue Type: Bug
  Components: platforms
Affects Versions: 2.6
Reporter: Alexey Kukushkin
Assignee: Alexey Kukushkin


Setup:
 # A *.csproj file with any one ore more post-build actions specified opened in 
Visual Studio

Steps to reproduce:
 # In Visual Studio -> Package Manager Console run "Install-Package 
Apache.Ignite"
 # Open the csproj file in a test editor and check the post-build actions 

Expected:
 # Package Apache.Ignite installed OK
 # The initial post-build actions are still in the csproj file

Actual:
 # As expected
 # The initial post-build actions are gone



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10074) Structured Exception information is lost when Ignite .NET client calls Ignite Java service

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-10074:
-

 Summary: Structured Exception information is lost when Ignite .NET 
client calls Ignite Java service
 Key: IGNITE-10074
 URL: https://issues.apache.org/jira/browse/IGNITE-10074
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Affects Versions: 2.6
Reporter: Alexey Kukushkin


Suppose an Ignite service in Java throws *new 
ModelVerificationException(“ERROR!”)* to signal about a failure.

An Ignite.NET client of such a Java service would receive this exception:

Apache.Ignite.Core.Services.ServiceInvocationException
 * *Message*: Proxy method invocation failed with an exception. Examine 
InnerException for details.
 * *InnerException*: Apache.Ignite.Core.Common.IgniteException

o   *Message*: ERROR!

o   *InnerException*: Apache.Ignite.Core.Common.JavaException
 * *JavaClassName*: class org.apache.ignite.IgniteCheckedException
 * *JavaMessage*: ERROR!
 * *InnerException*: null
 * *Message*: class org.apache.ignite.IgniteCheckedException: ERROR!
   at 
org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7332)
   at 
org.apache.ignite.internal.processors.platform.services.PlatformServices$ServiceProxyHolder.invoke(PlatformServices.java:589)
   at 
org.apache.ignite.internal.processors.platform.services.PlatformServices.processInObjectStreamOutObjectStream(PlatformServices.java:289)
   at 
org.apache.ignite.internal.processors.platform.PlatformTargetProxyImpl.inObjectStreamOutObjectStream(PlatformTargetProxyImpl.java:172)
Caused by: ModelVerificationException: ERROR!
   at ... 

Ignite service wraps the custom ModelVerificationException in a 
IgniteCheckedException, which we get information for on the client side. There 
is no structured information about the custom exception in the client.

We need to have information about the custom exception on the client.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10075) Avoid binary configurations of Ignite Java service params and result when calling it from Ignite.NET

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-10075:
-

 Summary: Avoid binary configurations of Ignite Java service params 
and result when calling it from Ignite.NET
 Key: IGNITE-10075
 URL: https://issues.apache.org/jira/browse/IGNITE-10075
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Affects Versions: 2.6
Reporter: Alexey Kukushkin
Assignee: Alexey Kukushkin


Presently if there is an Ignite Java service taking parameters of complex 
(composite) types and returning a result of complex type then all the complex 
types must be explicitly specified in the binary configuration.

We need to enhance Ignite not to require binary configuration assuming that we 
use the same type, package/namespace and field/property names on both the .NET 
and Java sides (or provide a custom name mapper for relaxed naming conventions).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10073) Ignite NuGet package without embedded Ignite JARs

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-10073:
-

 Summary: Ignite NuGet package without embedded Ignite JARs
 Key: IGNITE-10073
 URL: https://issues.apache.org/jira/browse/IGNITE-10073
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Affects Versions: 2.6
Reporter: Alexey Kukushkin


The existing Apache.Ignite NuGet package includes Ignite JARs deployed into the 
"libs" directory in the .NET project output directory upon the package 
installation.

We prefer using external Ignite JARs from $IGNITE_HOME/libs instead of the JARs 
in the local libs directory.

Right now we have to manually remove local "libs" directory after every 
Apache.Ignite package installation or upgrade.

It would help us having another Ignite NuGet package without the embedded 
Ignite JARs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9454) SecurityPermissionSetBuilder fails to append system permission CACHE_CREATE

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-9454:


 Summary: SecurityPermissionSetBuilder fails to append system 
permission CACHE_CREATE
 Key: IGNITE-9454
 URL: https://issues.apache.org/jira/browse/IGNITE-9454
 Project: Ignite
  Issue Type: Bug
  Components: security
Affects Versions: 2.5
Reporter: Alexey Kukushkin
Assignee: Alexey Kukushkin


SecurityPermissionSetBuilder fails to append system permission CACHE_CREATE, 
CACHE_DESTROY and JOIN_AS_SERVER

+*Reproducer*+:

SecurityPermissionSetBuilder.create().appendSystemPermission(SecurityPermission.CACHE_CREATE)

The code above throw IgniteException saying that a security permission name 
should begin with either EVENTS_ or ADMIN_

*+Solution+:*

Update SecurityPermissionSetBuilder to:
 # allow system permissions CACHE_CREATE, CACHE_DESTROY and JOIN_AS_SERVER
 # restrict cache presmissions CACHE_CREATE, CACHE_DESTROY
 # add automated tests

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10430) Ignite.NET Data Region Metrics: PagesRead, PagesWritten, PagesReplaced, OffHeapSize, OffheapUsedSize

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-10430:
-

 Summary: Ignite.NET Data Region Metrics: PagesRead, PagesWritten, 
PagesReplaced, OffHeapSize, OffheapUsedSize
 Key: IGNITE-10430
 URL: https://issues.apache.org/jira/browse/IGNITE-10430
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 2.6
Reporter: Alexey Kukushkin
Assignee: Alexey Kukushkin


Add Ignite.NET Data Region Metrics presently existing in Java but missing in 
.NET: PagesRead, PagesWritten, PagesReplaced, OffHeapSize, OffheapUsedSize



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10874) JDBC thin driver metadata misses caches with queryEntities and names containing underscores

[Alexey Kukushkin (JIRA)]

Alexey Kukushkin created IGNITE-10874:
-

 Summary: JDBC thin driver metadata misses caches with 
queryEntities and names containing underscores
 Key: IGNITE-10874
 URL: https://issues.apache.org/jira/browse/IGNITE-10874
 Project: Ignite
  Issue Type: Improvement
  Components: jdbc
Affects Versions: 2.7
Reporter: Alexey Kukushkin


+*Steps to reproduce*+

1) Build and run this app:

 
{code:java}
public class App {
public static void main(String[] ags) {
try (Ignite ignite = Ignition.start(
new IgniteConfiguration()
.setDiscoverySpi(
new TcpDiscoverySpi()
.setIpFinder(
new 
TcpDiscoveryVmIpFinder().setAddresses(Collections.singleton("127.0.0.1:47500"))
)
)
)) {
final String NAME = "V_MODELS_SHORT";

ignite.getOrCreateCache(
new CacheConfiguration<>()
.setName(NAME)
.setCacheMode(CacheMode.PARTITIONED)
.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL)
.setQueryEntities(Collections.singleton(
new QueryEntity()
.setKeyType(Integer.class.getTypeName())
.setValueType(NAME)
.setKeyFieldName("VMS_ID")
.setFields(
Stream.of(
new AbstractMap.SimpleEntry<>("VMS_ID", 
Integer.class.getTypeName()),
new 
AbstractMap.SimpleEntry<>("VMS_HASVERSION", Boolean.class.getTypeName())
).collect(Collectors.toMap(
AbstractMap.SimpleEntry::getKey,
AbstractMap.SimpleEntry::getValue,
(u, v) -> {
throw new 
IllegalStateException(String.format("Duplicate key %s", u));
},
LinkedHashMap::new
))
)
))
);

System.in.read();
}
}
}
{code}
2) While the app is running use a JDBC database management tool like DBeaver to 
open a JDBC thin driver connection to Ignite.

3) Check table "V_MODELS_SHORT" under schema "V_MODELS_SHORT"

+*Expected*+

3) Table "V_MODELS_SHORT" is visible under  schema "V_MODELS_SHORT"

+*Actual*+

 

3) There is no table "V_MODELS_SHORT" under  schema "V_MODELS_SHORT"

+*Notes*+

You still can run queries on table V_MODELS_SHORT in DBeaver. For example, 
these queries work fine:
{code:java}
INSERT INTO V_MODELS_SHORT.V_MODELS_SHORT VALUES (1, 'false');

SELECT * FROM V_MODELS_SHORT.V_MODELS_SHORT;
{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-12898) Server node with CacheStore fails to re-join the cluster: IgniteCheckedException: Cannot enable read-through (loader or store is not provided) for cache

[Alexey Kukushkin (Jira)]

Alexey Kukushkin created IGNITE-12898:
-

 Summary: Server node with CacheStore fails to re-join the cluster: 
IgniteCheckedException: Cannot enable read-through (loader or store is not 
provided) for cache
 Key: IGNITE-12898
 URL: https://issues.apache.org/jira/browse/IGNITE-12898
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.8
Reporter: Alexey Kukushkin


If a cache with external persistence is dynamically created on a non-affinity 
node then the cache affinity node cannot join the cluster after restart.
h2. Repro Steps
 # Run an "empty" Ignite node where no cache is going to be started
 # Run a cache affinity node having the "ROLE" attribute set to "DATA"
 # Create the cache from the "empty" node and use a Node Filter to limit the 
cache to the "data" node. External persistence is configured for the cache.
 # Restart the "data" node

h3. Actual Result
{{IgniteCheckedException: Cannot enable read-through (loader or store is not 
provided) for cache}}

h2. Reproducer
h3. Reproducer.java
{code:java}
public class Reproducer {
@Test
public void test() throws Exception {
final String DB_URL = "jdbc:h2:mem:test";
final String ENTITY_NAME = "Person";

Function igniteCfgFactory = instanceName ->
new IgniteConfiguration()
.setIgniteInstanceName(instanceName)
.setDiscoverySpi(new TcpDiscoverySpi()
.setIpFinder(new 
TcpDiscoveryVmIpFinder().setAddresses(Collections.singleton("127.0.0.1:47500")))
);

// 1. Run an "empty" Ignite node where no cache is going to be started
try (Connection dbConn = DriverManager.getConnection(DB_URL, "sa", "");
 Statement dbStmt = dbConn.createStatement();
 Ignite emptyNode = 
Ignition.start(igniteCfgFactory.apply("emptyyNode"))) {
// 2. Run a "Person" cache affinity node having the "ROLE" 
attribute set to "DATA"
Map dataNodeAttrs = new HashMap<>(1);
dataNodeAttrs.put(DataNodeFilter.ATTR_NAME, 
DataNodeFilter.ATTR_VAL);

Ignite dataNode = 
Ignition.start(igniteCfgFactory.apply("dataNode").setUserAttributes(dataNodeAttrs));

// 3. Create the "Person" cache from the "empty" node and use a 
Node Filter to limit the cache to the
// "data" node. External persistence to the "Person" table in H2 DB 
is configured for the cache.
dbStmt.execute("CREATE TABLE " + ENTITY_NAME + " (id int PRIMARY 
KEY, name varchar)");

CacheJdbcPojoStoreFactory igniteStoreFactory 
= new CacheJdbcPojoStoreFactory<>();
igniteStoreFactory.setDataSourceFactory(() -> 
JdbcConnectionPool.create(DB_URL, "sa", ""))
.setTypes(
new JdbcType()
.setCacheName(ENTITY_NAME)
.setDatabaseTable(ENTITY_NAME)
.setKeyType(Integer.class)
.setValueType(ENTITY_NAME)
.setKeyFields(new JdbcTypeField(java.sql.Types.INTEGER, 
"id", Integer.class, "id"))
.setValueFields(
new JdbcTypeField(java.sql.Types.INTEGER, "id", 
Integer.class, "id"),
new JdbcTypeField(java.sql.Types.VARCHAR, "name", 
String.class, "name")
)
);

CacheConfiguration cacheCfg =
new CacheConfiguration(ENTITY_NAME)
.setCacheMode(CacheMode.REPLICATED)
.setCacheStoreFactory(igniteStoreFactory)
.setWriteThrough(true)
.setReadThrough(true)
.setNodeFilter(new DataNodeFilter());

emptyNode.createCache(cacheCfg).withKeepBinary();

// 4. Restart the "data" node
dataNode.close();
dataNode = 
Ignition.start(igniteCfgFactory.apply("node2").setUserAttributes(dataNodeAttrs));

dataNode.close();
}
}

private static class DataNodeFilter implements IgnitePredicate 
{
public static final String ATTR_NAME = "ROLE";
public static final String ATTR_VAL = "DATA";

@Override public boolean apply(ClusterNode node) {
Object role = node.attributes().get(ATTR_NAME);
return role != null && ATTR_VAL.equalsIgnoreCase(role.toString());
}
}
}
{code}

h3. build.gradle
{code:groovy}
dependencies {

[jira] [Created] (IGNITE-12894) Cannot use IgniteAtomicSequence in Ignite services

[Alexey Kukushkin (Jira)]

Alexey Kukushkin created IGNITE-12894:
-

 Summary: Cannot use IgniteAtomicSequence in Ignite services
 Key: IGNITE-12894
 URL: https://issues.apache.org/jira/browse/IGNITE-12894
 Project: Ignite
  Issue Type: Bug
  Components: compute
Affects Versions: 2.8
Reporter: Alexey Kukushkin
Assignee: Alexey Kukushkin


h2. Repro Steps
Execute the below steps in default service deployment mode and in 
discovery-based service deployment mode. 
Use the {{ -DIGNITE_EVENT_DRIVEN_SERVICE_PROCESSOR_ENABLED=true}} JVM option to 
switch to the discovery-based service deployment mode.
 * Create a service initializing an IgniteAtomicService in method 
Service#init() and using the IgniteAtomicService in a business method.
 * Start an Ignite node with the service specified in the IgniteConfiguration
 * Invoke the service's business method on the Ignite node

h3. Actual Result
h4. In Default Service Deployment Mode
Deadlock on the business method invocation
h4. In Discovery-Based Service Deployment Mode
The method invocation fails with {{IgniteException: Failed to find deployed 
service: IgniteTestService}}

h2. Reproducer
h3. Test.java
{code:java}
public interface Test {
String sayHello(String name);
}
{code}

h3. IgniteTestService.java
{code:java}
public class IgniteTestService implements Test, Service {
private @IgniteInstanceResource Ignite ignite;
private IgniteAtomicSequence seq;

@Override public void cancel(ServiceContext ctx) {
}

@Override public void init(ServiceContext ctx) throws InterruptedException {
seq = ignite.atomicSequence("TestSeq", 0, true);
}

@Override public void execute(ServiceContext ctx) {
}

@Override public String sayHello(String name) {
return "Hello, " + name + "! #" + seq.getAndIncrement();
}
}
{code}

h3. Reproducer.java
{code:java}
public class Reproducer {
public static void main(String[] args) {
IgniteConfiguration igniteCfg = new IgniteConfiguration()
.setServiceConfiguration(
new ServiceConfiguration()
.setName(IgniteTestService.class.getSimpleName())
.setMaxPerNodeCount(1)
.setTotalCount(0)
.setService(new IgniteTestService())
)
.setDiscoverySpi(
new TcpDiscoverySpi()
.setIpFinder(new 
TcpDiscoveryVmIpFinder().setAddresses(Collections.singleton("127.0.0.1:47500")))
);

try (Ignite ignite = Ignition.start(igniteCfg)) {

ignite.services().serviceProxy(IgniteTestService.class.getSimpleName(), 
Test.class, false)
.sayHello("World");
}
}
}
{code}

h2. Workaround
Specifying a service wait timeout solves the problem in the discovery-based 
service deployment mode (but not in the default deployment mode):
{code:java}

ignite.services().serviceProxy(IgniteTestService.class.getSimpleName(), 
Test.class, false, 1_000)
.sayHello("World");
{code}

This workaround cannot be used in Ignite.NET clients since .NET 
{{GetServiceProxy}} API does not support the service wait timeout, which is 
hard-coded to 0 on the server side.

h2. Full Exception in Discovery-Based Service Deployment Mode
{noformat}
[01:08:54,653][SEVERE][services-deployment-worker-#52][IgniteServiceProcessor] 
Failed to initialize service (service will not be deployed): IgniteTestService
class org.apache.ignite.IgniteInterruptedException: Got interrupted while 
waiting for future to complete.
at 
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:888)
at 
org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:886)
at 
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:1062)
at 
org.apache.ignite.internal.IgniteKernal.atomicSequence(IgniteKernal.java:3999)
at 
org.apache.ignite.internal.IgniteKernal.atomicSequence(IgniteKernal.java:3985)
at Sandbox.Net.IgniteTestService.init(IgniteTestService.java:17)
at 
org.apache.ignite.internal.processors.service.IgniteServiceProcessor.redeploy(IgniteServiceProcessor.java:1188)
at 
org.apache.ignite.internal.processors.service.ServiceDeploymentTask.lambda$processDeploymentActions$5(ServiceDeploymentTask.java:318)
at java.base/java.util.HashMap.forEach(HashMap.java:1336)
at 
org.apache.ignite.internal.processors.service.ServiceDeploymentTask.processDeploymentActions(ServiceDeploymentTask.java:302)
at 
org.apache.ignite.internal.processors.service.ServiceDeploymentTask.init(ServiceDeploymentTask.java:262)
at 
org.apache.ignite.internal.processors.service.ServiceDeploymentManager$Ser

[jira] [Created] (IGNITE-12901) SQL: Uncorrelated subquery should run only once.

[Alexey Kukushkin (Jira)]

Alexey Kukushkin created IGNITE-12901:
-

 Summary: SQL: Uncorrelated subquery should run only once.
 Key: IGNITE-12901
 URL: https://issues.apache.org/jira/browse/IGNITE-12901
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.8
Reporter: Alexey Kukushkin


Currently uncorrelated subqueries (where subquery is not depends on the outer 
query) are executed on each nested loop iteration in the 
org.h2.command.dml.Select#isConditionMet method. 
We may avoid this, for example, using results caching.

h2. Reproducer
{code:java}
public class SubQueryTest extends AbstractIndexingCommonTest {
/** Keys counts at the RIGHT table. */
private static final int RIGHT_CNT = 10;

/** Keys counts at the LEFT table. */
private static final int LEFT_CNT = 50;

/** {@inheritDoc} */
@SuppressWarnings("unchecked")
@Override protected void beforeTest() throws Exception {
super.beforeTest();

startGrids(1);

IgniteCache cacheA = grid(0).createCache(new CacheConfiguration()
.setName("A")
.setSqlSchema("TEST")
.setQueryEntities(Collections.singleton(new 
QueryEntity(Long.class.getTypeName(), "A_VAL")
.setTableName("A")
.addQueryField("ID", Long.class.getName(), null)
.addQueryField("JID", Long.class.getName(), null)
.addQueryField("VAL", Long.class.getName(), null)
.setKeyFieldName("ID")
)));

IgniteCache cacheB = grid(0).createCache(new CacheConfiguration()
.setCacheMode(CacheMode.REPLICATED)
.setName("B")
.setSqlSchema("TEST")
.setQueryEntities(Collections.singleton(new 
QueryEntity(Long.class.getName(), "B_VAL")
.setTableName("B")
.addQueryField("ID", Long.class.getName(), null)
.addQueryField("A_JID", Long.class.getName(), null)
.addQueryField("VAL0", String.class.getName(), null)
.setKeyFieldName("ID")
)));

Map batch = new HashMap<>();
for (long i = 0; i < LEFT_CNT; ++i) {
batch.put(i, grid(0).binary().builder("A_VAL")
.setField("JID", i % RIGHT_CNT)
.setField("VAL", i)
.build());

if (batch.size() > 1000) {
cacheA.putAll(batch);

batch.clear();
}
}
if (batch.size() > 0) {
cacheA.putAll(batch);

batch.clear();
}

for (long i = 0; i < RIGHT_CNT; ++i)
cacheB.put(i, grid(0).binary().builder("B_VAL")
.setField("A_JID", i)
.setField("VAL0", String.format("val%03d", i))
.build());
}

/** {@inheritDoc} */
@Override protected void afterTest() throws Exception {
stopAllGrids();

super.afterTest();
}

/**
 * Test local query execution.
 */
@Test
public void test() {
sql(true, "SELECT * FROM A WHERE A.JID IN (SELECT A_JID FROM 
B)").getAll();
}


/**
 * @param enforceJoinOrder Enforce join order mode.
 * @param sql SQL query.
 * @param args Query parameters.
 * @return Results cursor.
 */
private FieldsQueryCursor> sql(boolean enforceJoinOrder, String 
sql, Object... args) {
return grid(0).context().query().querySqlFields(new SqlFieldsQuery(sql)
.setSchema("TEST")
.setLazy(true)
.setEnforceJoinOrder(enforceJoinOrder)
.setArgs(args), false);
}
}
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)