[jira] [Created] (IGNITE-8215) Web console: some fields are not generated

[Pavel Konstantinov (JIRA)]

Pavel Konstantinov created IGNITE-8215:
--

 Summary: Web console: some fields are not generated
 Key: IGNITE-8215
 URL: https://issues.apache.org/jira/browse/IGNITE-8215
 Project: Ignite
  Issue Type: Bug
Reporter: Pavel Konstantinov
Assignee: Vasiliy Sisko


Data Region Configuration 
- Metrics sub interval count
- Metrics rate time interval



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8214) Web console: add validation for Persistent + Swap file for data region configuration

[Pavel Konstantinov (JIRA)]

Pavel Konstantinov created IGNITE-8214:
--

 Summary: Web console: add validation for Persistent + Swap file 
for data region configuration
 Key: IGNITE-8214
 URL: https://issues.apache.org/jira/browse/IGNITE-8214
 Project: Ignite
  Issue Type: Bug
Reporter: Pavel Konstantinov
Assignee: Vasiliy Sisko


The 'Swap file' option can be set only if 'Persistent Enable' is OFF.
Please add corresponding validation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8213) control.sh: make a correct error message in case of adding non-existen node to baseline

[Pavel Konstantinov (JIRA)]

Pavel Konstantinov created IGNITE-8213:
--

 Summary: control.sh: make a correct error message in case of 
adding non-existen node to baseline
 Key: IGNITE-8213
 URL: https://issues.apache.org/jira/browse/IGNITE-8213
 Project: Ignite
  Issue Type: Improvement
Reporter: Pavel Konstantinov


We need to print out a readable error message with a corresponding error code 
instead of this
{code}
PS C:\work\master\bin> .\control.bat --baseline add 121212
Warning: the command will perform changes in baseline.
Press 'y' to continue...y
Failed to add nodes to baseline.
Error: Failed to reduce job results due to undeclared user exception 
[task=org.apache.ignite.internal.visor.baseline.VisorBaselineTask@71e96f24, 
err=class org.apache.ignite.compute.ComputeUserUndeclaredException: Failed to 
execute job due to unexpected runtime exception 
[jobId=3064672b261-df38f2fd-ca81-423c-b4c3-dd002a77f0af, ses=GridJobSessionImpl 
[ses=GridTaskSessionImpl 
[taskName=org.apache.ignite.internal.visor.baseline.VisorBaselineTask, 
dep=LocalDeployment [super=GridDeployment [ts=1523412519601, depMode=SHARED, 
clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, 
clsLdrId=ba54672b261-df38f2fd-ca81-423c-b4c3-dd002a77f0af, userVer=0, loc=true, 
sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, 
usage=0]], 
taskClsName=org.apache.ignite.internal.visor.baseline.VisorBaselineTask, 
sesId=2064672b261-df38f2fd-ca81-423c-b4c3-dd002a77f0af, 
startTime=1523412552937, endTime=9223372036854775807, 
taskNodeId=df38f2fd-ca81-423c-b4c3-dd002a77f0af, 
clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, closed=false, cpSpi=null, 
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=true, 
topPred=null, subjId=df38f2fd-ca81-423c-b4c3-dd002a77f0af, mapFut=IgniteFuture 
[orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, 
hash=1531276047]], execName=null], 
jobId=3064672b261-df38f2fd-ca81-423c-b4c3-dd002a77f0af], err=Node not found for 
consistent ID: 121212]]
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: iep-6 metrics ticket review

[Valentin Kulichenko]

This is on my plate, will try to take a look this week.

-Val

On Mon, Apr 9, 2018 at 10:28 AM, Denis Magda  wrote:

> Val,
>
> As an initial reviewer and reporter, could you have a look and sign the
> contribution off?
>
> --
> Denis
>
> On Mon, Apr 9, 2018 at 12:56 AM, Aleksey Kuznetsov <
> alkuznetsov...@gmail.com
> > wrote:
>
> > Hi ,Igniters!
> >
> > Do we still need this ticket, about invoke metrics : [1] ?
> >
> > If yes, than could somebody review it ?
> >
> > If no, should we close this ticket ?
> >
> > [1] : https://issues.apache.org/jira/browse/IGNITE-6846
> > --
> >
> > *Best Regards,*
> >
> > *Kuznetsov Aleksey*
> >
>

[GitHub] ignite pull request #3773: IGNITE-8153 Nodes fail to connect each other when...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3773


---

Re: Move documentation from readme.io to GitHub pages

[Denis Magda]

Look into both Docusaraus and Jekyll from the usage perspective. Here is my
summary:

*Documentation Sources *

Will be stored on GitHub. My preference is to store them in "ignite/docs"
folder as many other ASF projects do (Spark [1], Flink [2] and Storm [3]).
If we need to update the sources of an already released version, then we
can create ignite-{version}-docs branch, edit it directly and generate HTML
pages from it.

*Versioning*

Since the docs are stored in the main repo, a doc version will correspond
to an Ignite version. If changes incorporated in the master version of the
docs have to be merged to a previous version and redeployed on the site, we
will use standard 'git' facilities to propagate the changes whenever is
needed.

*Documentation Deployment and Automation*

Documentation engines usually go with a set of scripts that produce an HTML
version of the docs out of the sources. In our scenario, we will use the
scripts to convert the sources stored in GitHub to HTML pages stored in SVN
repo of Ignite site. The docs' HTML pages will be hosted on
ignite.apache.org.

By default, the one has to run the scripts on a local machine to produce
the HTML pages. However, nothing prevents us from tweaking the scripts and
using them in a way that would do this on a fly - "a page has changed in
sources"->"update site button is pressed"->"HTML is generated and
automatically deployed to the site".


Btw, *Prachi*, have you checked up Jekyll [4]? It's used by Spark, Flink,
Storm, and even Github Pages. It's simpler than Docusarus and still gives a
way to generate customized sites with navigation menus and table of
contents: https://ci.apache.org/projects/flink/flink-docs-release-1.4/


Does anyone else have any open questions we need to solve before starting a
migration process?



[1] https://github.com/apache/spark/tree/master/docs
[2] https://github.com/apache/flink/tree/master/docs
[3] https://github.com/apache/storm/tree/master/docs
[4] https://github.com/jekyll/jekyll

On Wed, Mar 21, 2018 at 6:15 PM, Dmitriy Setrakyan 
wrote:

> On Wed, Mar 21, 2018 at 9:27 PM, Prachi Garg  wrote:
>
> > We can store the project (Markdown & Docusaurus config files) in Git, use
> > Docusaurus to build html, and upload them to Ignite website.
> >
>
> Sounds good!
>

Re: affinityRun/Call in C++

[Valentin Kulichenko]

Is there a ticket? Let's create one if not.

-Val

On Tue, Apr 10, 2018 at 6:17 AM, Vladimir Ozerov 
wrote:

> Val,
>
> They are simply not implemented yet. I am not aware of concrete plans to
> support them.
>
> On Mon, Apr 9, 2018 at 11:33 PM, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Guys,
> >
> > Is there a way to run collocated compute job in C++? I can't find
> > affinityRun and affinityCall method in C++ compute API, am I missing
> > something? If we really don't have them, is there any particular reason
> for
> > this and/or plans to add them?
> >
> > -Val
> >
>

Re: Atomic caches

[Valentin Kulichenko]

Guys,

I also not sure I understand the purpose of methods like [1] that accept
instance of AtomicConfiguration to create new atomic structure. Per my
knowledge, all atomics are stored in a single cache which is configured by
AtomicConfiguration provided on startup as part of IgniteConfiguration. If
that's the case, providing another configuration for a particular atomic
doesn't make sense because it will never be used.

Any thoughts on this? Unless I'm missing something, I think we can
deprecate these methods.

[1]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/Ignite.html#atomicLong-java.lang.String-org.apache.ignite.configuration.AtomicConfiguration-long-boolean-

-Val

On Tue, Apr 10, 2018 at 3:28 PM, Dmitriy Setrakyan 
wrote:

> On Tue, Apr 10, 2018 at 2:03 PM, akurbanov  wrote:
>
> > Dmitry,
> >
> > Sorry for confusing topic. I I'm pretty sure that configuration for
> atomic
> > caches is validated, will double-check this. I was referring only atomic
> > data structures cache.
> >
>
> Got it. We should definitely add validation for the atomic data structures
> configuration as well.
>

Re: Warning message

[Alew]

Hi, Dmitriy

Of course I understand what a local cache means and it is not a mistake. 
But why does Ignite warn me on every query?
My logs full with this warning. Maybe log level should be information or 
debug.


On 11/04/2018 01:33, Dmitriy Setrakyan wrote:

On Tue, Apr 10, 2018 at 2:22 PM, Alew  wrote:


Hi!

I use local cache and get a lot of warnings when execute ScanQueries.

[13:26:25 WRN]  Ignoring query projection because it's
executed over LOCAL cache (only local node will be queried):
GridCacheQueryAdapter [type=SCAN, clsName=null, clause=null,
filter=o.a.i.i.processors.platform.cache.PlatformCacheEntryF
ilterImpl@7629939a, transform=null, part=null, incMeta=false,
metrics=GridCacheQueryMetricsAdapter [minTime=9223372036854775807,
maxTime=0, sumTime=0, avgTime=0.0, execs=0, completed=0, fails=0],
pageSize=1024, timeout=0, keepAll=true, incBackups=false, dedup=false,
prj=o.a.i.i.cluster.ClusterGroupAdapter@708472f7, keepBinary=true,
subjId=null, taskHash=0]

What does it warn me about? Is it really warning?


Your cache is LOCAL, i.e. not distributed. This means that all the data is
located only on the local node. More on cache modes here:
https://apacheignite.readme.io/docs/cache-modes

D.

Re: Warning message

[Dmitriy Setrakyan]

On Tue, Apr 10, 2018 at 2:22 PM, Alew  wrote:

> Hi!
>
> I use local cache and get a lot of warnings when execute ScanQueries.
>
> [13:26:25 WRN]  Ignoring query projection because it's
> executed over LOCAL cache (only local node will be queried):
> GridCacheQueryAdapter [type=SCAN, clsName=null, clause=null,
> filter=o.a.i.i.processors.platform.cache.PlatformCacheEntryF
> ilterImpl@7629939a, transform=null, part=null, incMeta=false,
> metrics=GridCacheQueryMetricsAdapter [minTime=9223372036854775807,
> maxTime=0, sumTime=0, avgTime=0.0, execs=0, completed=0, fails=0],
> pageSize=1024, timeout=0, keepAll=true, incBackups=false, dedup=false,
> prj=o.a.i.i.cluster.ClusterGroupAdapter@708472f7, keepBinary=true,
> subjId=null, taskHash=0]
>
> What does it warn me about? Is it really warning?
>

Your cache is LOCAL, i.e. not distributed. This means that all the data is
located only on the local node. More on cache modes here:
https://apacheignite.readme.io/docs/cache-modes

D.

Re: Atomic caches

[Dmitriy Setrakyan]

On Tue, Apr 10, 2018 at 2:03 PM, akurbanov  wrote:

> Dmitry,
>
> Sorry for confusing topic. I I'm pretty sure that configuration for atomic
> caches is validated, will double-check this. I was referring only atomic
> data structures cache.
>

Got it. We should definitely add validation for the atomic data structures
configuration as well.

[jira] [Created] (IGNITE-8212) Binary Client Protocol spec: Key-Value Queries clarifications

[Alexey Kosenchuk (JIRA)]

Alexey Kosenchuk created IGNITE-8212:


 Summary: Binary Client Protocol spec: Key-Value Queries 
clarifications
 Key: IGNITE-8212
 URL: https://issues.apache.org/jira/browse/IGNITE-8212
 Project: Ignite
  Issue Type: Bug
  Components: documentation, thin client
Affects Versions: 2.4
Reporter: Alexey Kosenchuk


https://apacheignite.readme.io/v2.4/docs/binary-client-protocol-key-value-operations

- all operations - "Flag. Pass 0 for default, or 1 to keep the value in binary 
form":
-- remove words about binary form and keep "Pass 0 for default" for all 
operations where this flag has no meaning.
-- clarify about binary form in operations where it has a meaning.

- OP_CACHE_GET: clarify that null is returned if the provided key does not 
exist in the cache.

- OP_CACHE_GET_AND_PUT: clarify that new entry is created, if the provided key 
does not exist in the cache, and null is returned in this case.

- OP_CACHE_GET_AND_REPLACE:
-- "if and only if there is a value currently mapped for that key" - is 
confusing. Is it possible that an existing key has no associated value? Suggest 
to rephrase as "if and only if the key exists in the cache"
-- clarify that null is returned if the key does not exist in the cache.

- OP_CACHE_GET_AND_REMOVE: clarify that null is returned if the key does not 
exist in the cache.

- OP_CACHE_PUT_IF_ABSENT: clarify what is "Success Flag" - true if the new 
entry is created, false if the specified key already exists in the cache.

- OP_CACHE_GET_AND_PUT_IF_ABSENT: "Previously contained value regardless of 
whether put happened or not" - is confusing. Suggest to rewrite "the current 
value associated with the key if it already exists in the cache, null if the 
new entry is created".

- OP_CACHE_GET_SIZE: clarify Peek modes

- OP_CACHE_CLEAR_XXX and OP_CACHE_REMOVE_XXX:
-- what is the difference between the corresponding operations? Only in 
notifying / not notifying listeners and cache writers? Or there are other 
differences not clarified by the spec?
-- (the operations could be combined in one set - with a flag required/not 
required notifications)
-- there is OP_CACHE_REMOVE_IF_EQUALS. But there is no 
OP_CACHE_CLEAR_IF_EQUALS. Is it intentional?
-- OP_CACHE_REMOVE_KEY and OP_CACHE_REMOVE_IF_EQUALS have "Value indicating 
whether remove happened" in the response. But OP_CACHE_CLEAR_KEY does not have 
such a flag in the response. Is it intentional?
-- OP_CACHE_CLEAR_KEYS, OP_CACHE_REMOVE_KEYS: clarify that even if some of the 
specified keys do not exist other entries are removed anyway.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: IGNITE-6879

[Denis Magda]

Roman,

Your suggestion sounds reasonable to me. Backing it up.

--
Denis

On Tue, Apr 10, 2018 at 2:50 PM, Роман Меерсон  wrote:

> Hi all!
>
> IMHO if we do so we'll produce big pain for everybody while migrating on
> new version, because ones should change method and others should change
> their poms. This change would be backward incompatible so it probably
> should follow with major version upgrade, but I'm not sure about it.
>
> Otherwise if we leave current state( spring-data for old and
> spring-data-2.0 for new) we could support old users who probably use spring
> data 1.0 (because spring data 2.0 release was not so long ago) and provide
> new functionality for users who want to use new spring data.
> In this case old users wouldn't change any in their code except ignite
> version, and new users would include ignite in their Pom anyway and could
> choose which module of spring data to bring.
>
> After some time (probably on 3.0 release) we could change naming as Denis
> suggested.
>
> Anyway I leave this decision up to you, just tell me what is the way to
> finish this PR.
>
> Regards, Roman.
>
> ср, 11 апр. 2018 г. в 1:35, Denis Magda :
>
> > In our Hibernate integration we define following two modules to
> > distinguish incompatible versions:
> >
> >- ignite-hiberbate_4.2
> >- ignite-hibernate_5.1
> >
> > In Spark we have:
> >
> >- ignite-spark
> >- ignite-spark_2.10
> >
> > After thinking this over, I would do the following with Spring Data:
> >
> >- ignite-spring-data for the latest Sprind Data 2.0
> >- ignite-spring-data_1.0
> >
> > What do you think?
> >
> > --
> > Denis
> >
> > On Tue, Apr 10, 2018 at 3:50 AM, Dmitry Pavlov 
> > wrote:
> >
> >> Thank you, Roman.
> >>
> >> Igniters,
> >>
> >> IMO we should consider one more alternative - renaming of old module and
> >> package names. Users, which prefer to stay on previous version will be
> >> requiered to update their pom's. In the same time users which are ready
> to
> >> migrate to spring data 2.0 will need to update methods naming.
> >>
> >> Denis M, what would you say?
> >>
> >> Sincerely,
> >> Dmitriy Pavlov
> >>
> >> вт, 10 апр. 2018 г. в 11:27, Роман Меерсон :
> >>
> >>> Hi Dmitry!
> >>>
> >>> I`ve just commited new fix. I renamed package of new module to
> >>> springdata20, it helps us to separate old implementation from new and
> also
> >>> should fix all compilation errors.
> >>>
> >>> пн, 9 апр. 2018 г. в 23:54, Роман Меерсон :
> >>>
>  Ok, I'll check it, but I haven't face this problem.
>  If I'll find same issue, what is the proper way? Renaming to something
>  like Ignite2QueryGenerator or module removing?
>  пн, 9 апр. 2018 г. в 23:40, Dmitry Pavlov :
> 
> > There are 2 classes IgniteQueryGenerator with same package name.
> > Ignite in Idea can't compile.
> >
> >
> > пн, 9 апр. 2018 г., 21:38 Роман Меерсон :
> >
> >> Hi Dmitry!
> >
> >
> >> Could you specify where you find conflict? Because I don’t have any.
> >> пн, 9 апр. 2018 г. в 21:09, Dmitry Pavlov :
> >>
> >>> Hi Denis,
> >>>
> >>> could we support just one version instead of leaving compatible
> >>> module?
> >>>
> >>> Sincerely,
> >>> Dmitriy Pavlov
> >>>
> >>> пн, 9 апр. 2018 г. в 20:08, Dmitry Pavlov :
> >>>
> 
> 
>  пн, 9 апр. 2018 г. в 20:07, Dmitry Pavlov  >:
> 
> > Hi Roman,
> >
> > I've applied PR locally and I have class name conflict at least
> > for
> > org.apache.ignite.springdata.repository.query.
> IgniteQueryGenerator
> >
> > How could we solve it? Is it better to rename class for new
> plugin
> > version?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пт, 6 апр. 2018 г. в 17:38, Dmitry Pavlov  >:
> >
> >> Excellend picture. I remember about this change.
> >>
> >> If Denis M. would be able to look througt the changes faster
> than
> >> me, I can merge without detailed review.
> >>
> >> пт, 6 апр. 2018 г. в 16:15, Роман Меерсон  >:
> >>
> >>> OK
> >>>
> >>> [image: 1486924635147168240.jpg]
> >>>
> >>>
> >>> пт, 6 апр. 2018 г. в 17:08, Igor Sapego :
> >>>
>  Hi,
>  Well, Dmitry has said he's going to merge it in 3-4 days 2
> days
>  ago,
>  so I guess, the merge is going to happen in 1-2 days or so.
> 
> 
>  Best Regards,
>  Igor
> 
>  On Fri, Apr 6, 2018 at 3:48 PM, Роман 

Re: IGNITE-6879

[Роман Меерсон]

Hi all!

IMHO if we do so we'll produce big pain for everybody while migrating on
new version, because ones should change method and others should change
their poms. This change would be backward incompatible so it probably
should follow with major version upgrade, but I'm not sure about it.

Otherwise if we leave current state( spring-data for old and
spring-data-2.0 for new) we could support old users who probably use spring
data 1.0 (because spring data 2.0 release was not so long ago) and provide
new functionality for users who want to use new spring data.
In this case old users wouldn't change any in their code except ignite
version, and new users would include ignite in their Pom anyway and could
choose which module of spring data to bring.

After some time (probably on 3.0 release) we could change naming as Denis
suggested.

Anyway I leave this decision up to you, just tell me what is the way to
finish this PR.

Regards, Roman.

ср, 11 апр. 2018 г. в 1:35, Denis Magda :

> In our Hibernate integration we define following two modules to
> distinguish incompatible versions:
>
>- ignite-hiberbate_4.2
>- ignite-hibernate_5.1
>
> In Spark we have:
>
>- ignite-spark
>- ignite-spark_2.10
>
> After thinking this over, I would do the following with Spring Data:
>
>- ignite-spring-data for the latest Sprind Data 2.0
>- ignite-spring-data_1.0
>
> What do you think?
>
> --
> Denis
>
> On Tue, Apr 10, 2018 at 3:50 AM, Dmitry Pavlov 
> wrote:
>
>> Thank you, Roman.
>>
>> Igniters,
>>
>> IMO we should consider one more alternative - renaming of old module and
>> package names. Users, which prefer to stay on previous version will be
>> requiered to update their pom's. In the same time users which are ready to
>> migrate to spring data 2.0 will need to update methods naming.
>>
>> Denis M, what would you say?
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>> вт, 10 апр. 2018 г. в 11:27, Роман Меерсон :
>>
>>> Hi Dmitry!
>>>
>>> I`ve just commited new fix. I renamed package of new module to
>>> springdata20, it helps us to separate old implementation from new and also
>>> should fix all compilation errors.
>>>
>>> пн, 9 апр. 2018 г. в 23:54, Роман Меерсон :
>>>
 Ok, I'll check it, but I haven't face this problem.
 If I'll find same issue, what is the proper way? Renaming to something
 like Ignite2QueryGenerator or module removing?
 пн, 9 апр. 2018 г. в 23:40, Dmitry Pavlov :

> There are 2 classes IgniteQueryGenerator with same package name.
> Ignite in Idea can't compile.
>
>
> пн, 9 апр. 2018 г., 21:38 Роман Меерсон :
>
>> Hi Dmitry!
>
>
>> Could you specify where you find conflict? Because I don’t have any.
>> пн, 9 апр. 2018 г. в 21:09, Dmitry Pavlov :
>>
>>> Hi Denis,
>>>
>>> could we support just one version instead of leaving compatible
>>> module?
>>>
>>> Sincerely,
>>> Dmitriy Pavlov
>>>
>>> пн, 9 апр. 2018 г. в 20:08, Dmitry Pavlov :
>>>


 пн, 9 апр. 2018 г. в 20:07, Dmitry Pavlov :

> Hi Roman,
>
> I've applied PR locally and I have class name conflict at least
> for
> org.apache.ignite.springdata.repository.query.IgniteQueryGenerator
>
> How could we solve it? Is it better to rename class for new plugin
> version?
>
> Sincerely,
> Dmitriy Pavlov
>
> пт, 6 апр. 2018 г. в 17:38, Dmitry Pavlov :
>
>> Excellend picture. I remember about this change.
>>
>> If Denis M. would be able to look througt the changes faster than
>> me, I can merge without detailed review.
>>
>> пт, 6 апр. 2018 г. в 16:15, Роман Меерсон :
>>
>>> OK
>>>
>>> [image: 1486924635147168240.jpg]
>>>
>>>
>>> пт, 6 апр. 2018 г. в 17:08, Igor Sapego :
>>>
 Hi,
 Well, Dmitry has said he's going to merge it in 3-4 days 2 days
 ago,
 so I guess, the merge is going to happen in 1-2 days or so.


 Best Regards,
 Igor

 On Fri, Apr 6, 2018 at 3:48 PM, Роман Меерсон <
 homich1...@gmail.com> wrote:

 > Hi all!
 >
 > As i see everything is awesome and there is no objections, so
 when my PR
 > would be merged?
 >
 > чт, 5 апр. 2018 г. в 18:58, Вячеслав Коптилин <
 slava.kopti...@gmail.com>:
 >
 > > Thank you, Roman!
 > >
 > > 2018-04-05 

[jira] [Created] (IGNITE-8211) .Net Invalid cast to CacheEvent

[Aleksandr Tceluiko (JIRA)]

Aleksandr Tceluiko created IGNITE-8211:
--

 Summary: .Net Invalid cast to CacheEvent
 Key: IGNITE-8211
 URL: https://issues.apache.org/jira/browse/IGNITE-8211
 Project: Ignite
  Issue Type: Bug
  Components: platforms
Affects Versions: 2.3
Reporter: Aleksandr Tceluiko


Subscribed to all events

config.IncludedEventTypes = EventType.All;
ignite = Ignition.Start(config);
 var events = ignite.GetEvents();
 var listener = new CacheEventListener();
 events.LocalListen(listener, EventType.All);

and got exceptions:

[23:56:22 ERR] Failure in Java callback
System.InvalidCastException: Unable to cast object of type 
'Apache.Ignite.Core.Events.CacheQueryExecutedEvent' to type 
'Apache.Ignite.Core.Events.CacheEvent'.
 at Apache.Ignite.Core.Events.EventReader.Read[T](BinaryReader reader)
 at Apache.Ignite.Core.Impl.Events.Events.InvokeLocalListener[T](IBinaryStream 
stream, IEventListener`1 listener)
 at 
Apache.Ignite.Core.Impl.Events.Events.LocalHandledEventFilter.Invoke(IBinaryStream
 stream)
 at Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.EventFilterApply(Int64 
ptr, Int64 memPtr, Int64 unused, Void* arg)
 at 
Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.InLongLongLongObjectOutLong(Void*
 target, Int32 type, Int64 val1, Int64 val2, Int64 val3, Void* arg)
[23:56:22 ERR] Failure in Java callback
System.InvalidCastException: Unable to cast object of type 
'Apache.Ignite.Core.Events.CacheQueryExecutedEvent' to type 
'Apache.Ignite.Core.Events.CacheEvent'.
 at Apache.Ignite.Core.Events.EventReader.Read[T](BinaryReader reader)
 at Apache.Ignite.Core.Impl.Events.Events.InvokeLocalListener[T](IBinaryStream 
stream, IEventListener`1 listener)
 at 
Apache.Ignite.Core.Impl.Events.Events.LocalHandledEventFilter.Invoke(IBinaryStream
 stream)
 at Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.EventFilterApply(Int64 
ptr, Int64 memPtr, Int64 unused, Void* arg)
 at 
Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.InLongLongLongObjectOutLong(Void*
 target, Int32 type, Int64 val1, Int64 val2, Int64 val3, Void* arg)
[23:56:22 ERR] Failure in Java callback
System.InvalidCastException: Unable to cast object of type 
'Apache.Ignite.Core.Events.DiscoveryEvent' to type 
'Apache.Ignite.Core.Events.CacheEvent'.
 at Apache.Ignite.Core.Events.EventReader.Read[T](BinaryReader reader)
 at Apache.Ignite.Core.Impl.Events.Events.InvokeLocalListener[T](IBinaryStream 
stream, IEventListener`1 listener)
 at 
Apache.Ignite.Core.Impl.Events.Events.LocalHandledEventFilter.Invoke(IBinaryStream
 stream)
 at Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.EventFilterApply(Int64 
ptr, Int64 memPtr, Int64 unused, Void* arg)
 at 
Apache.Ignite.Core.Impl.Unmanaged.UnmanagedCallbacks.InLongLongLongObjectOutLong(Void*
 target, Int32 type, Int64 val1, Int64 val2, Int64 val3, Void* arg)
[23:56:22 ERR] Unexpected exception in listener notification for event: 
CacheQueryExecutedEvent [qryType=SCAN, cacheName=ignite-sys-cache, 
clsName=null, clause=null, scanQryFilter=ServiceAssignmentsPredicate [], 
contQryFilter=null, args=null, subjId=782e4d31-f4c5-4bea-a9fb-0f2e4d6b602f, 
taskName=null, nodeId8=782e4d31, msg=Scan query executed., 
type=CACHE_QUERY_EXECUTED, tstamp=1523393782062]
[23:56:23 ERR] Unexpected exception in listener notification for event: 
CacheQueryExecutedEvent [qryType=CONTINUOUS, cacheName=Authorization, 
clsName=null, clause=null, scanQryFilter=null, 
contQryFilter=o.a.i.i.processors.platform.cache.query.PlatformContinuousQueryImpl@1b84f475,
 args=null, subjId=782e4d31-f4c5-4bea-a9fb-0f2e4d6b602f, taskName=null, 
nodeId8=782e4d31, msg=Continuous query executed., type=CACHE_QUERY_EXECUTED, 
tstamp=1523393782095]
[23:56:23 ERR] Unexpected exception in listener notification for event: 
DiscoveryEvent [evtNode=TcpDiscoveryNode 
[id=0ee0988a-8364-4560-8215-e1dbe673bcd5, addrs=[0:0:0:0:0:0:0:1, 10.77.6.203, 
127.0.0.1, 192.168.2.21], sockAddrs=[WA/192.168.2.21:47500, /10.77.6.203:47500, 
/0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, 
lastExchangeTime=1523393780217, loc=false, ver=2.3.0#20171028-sha1:8add7fd5, 
isClient=false], topVer=3, nodeId8=782e4d31, msg=Metrics were updated: 
TcpDiscoveryNode [id=0ee0988a-8364-4560-8215-e1dbe673bcd5, 
addrs=[0:0:0:0:0:0:0:1, 10.77.6.203, 127.0.0.1, 192.168.2.21], 
sockAddrs=[WA/192.168.2.21:47500, /10.77.6.203:47500, /0:0:0:0:0:0:0:1:47500, 
/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, 
lastExchangeTime=1523393780217, loc=false, ver=2.3.0#20171028-sha1:8add7fd5, 
isClient=false], type=NODE_METRICS_UPDATED, tstamp=1523393782728]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: IGNITE-6879

[Denis Magda]

In our Hibernate integration we define following two modules to distinguish
incompatible versions:

   - ignite-hiberbate_4.2
   - ignite-hibernate_5.1

In Spark we have:

   - ignite-spark
   - ignite-spark_2.10

After thinking this over, I would do the following with Spring Data:

   - ignite-spring-data for the latest Sprind Data 2.0
   - ignite-spring-data_1.0

What do you think?

--
Denis

On Tue, Apr 10, 2018 at 3:50 AM, Dmitry Pavlov 
wrote:

> Thank you, Roman.
>
> Igniters,
>
> IMO we should consider one more alternative - renaming of old module and
> package names. Users, which prefer to stay on previous version will be
> requiered to update their pom's. In the same time users which are ready to
> migrate to spring data 2.0 will need to update methods naming.
>
> Denis M, what would you say?
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 10 апр. 2018 г. в 11:27, Роман Меерсон :
>
>> Hi Dmitry!
>>
>> I`ve just commited new fix. I renamed package of new module to
>> springdata20, it helps us to separate old implementation from new and also
>> should fix all compilation errors.
>>
>> пн, 9 апр. 2018 г. в 23:54, Роман Меерсон :
>>
>>> Ok, I'll check it, but I haven't face this problem.
>>> If I'll find same issue, what is the proper way? Renaming to something
>>> like Ignite2QueryGenerator or module removing?
>>> пн, 9 апр. 2018 г. в 23:40, Dmitry Pavlov :
>>>
 There are 2 classes IgniteQueryGenerator with same package name. Ignite
 in Idea can't compile.


 пн, 9 апр. 2018 г., 21:38 Роман Меерсон :

> Hi Dmitry!


> Could you specify where you find conflict? Because I don’t have any.
> пн, 9 апр. 2018 г. в 21:09, Dmitry Pavlov :
>
>> Hi Denis,
>>
>> could we support just one version instead of leaving compatible
>> module?
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>> пн, 9 апр. 2018 г. в 20:08, Dmitry Pavlov :
>>
>>>
>>>
>>> пн, 9 апр. 2018 г. в 20:07, Dmitry Pavlov :
>>>
 Hi Roman,

 I've applied PR locally and I have class name conflict at least for
 org.apache.ignite.springdata.repository.query.IgniteQueryGenerator

 How could we solve it? Is it better to rename class for new plugin
 version?

 Sincerely,
 Dmitriy Pavlov

 пт, 6 апр. 2018 г. в 17:38, Dmitry Pavlov :

> Excellend picture. I remember about this change.
>
> If Denis M. would be able to look througt the changes faster than
> me, I can merge without detailed review.
>
> пт, 6 апр. 2018 г. в 16:15, Роман Меерсон :
>
>> OK
>>
>> [image: 1486924635147168240.jpg]
>>
>>
>> пт, 6 апр. 2018 г. в 17:08, Igor Sapego :
>>
>>> Hi,
>>> Well, Dmitry has said he's going to merge it in 3-4 days 2 days
>>> ago,
>>> so I guess, the merge is going to happen in 1-2 days or so.
>>>
>>>
>>> Best Regards,
>>> Igor
>>>
>>> On Fri, Apr 6, 2018 at 3:48 PM, Роман Меерсон <
>>> homich1...@gmail.com> wrote:
>>>
>>> > Hi all!
>>> >
>>> > As i see everything is awesome and there is no objections, so
>>> when my PR
>>> > would be merged?
>>> >
>>> > чт, 5 апр. 2018 г. в 18:58, Вячеслав Коптилин <
>>> slava.kopti...@gmail.com>:
>>> >
>>> > > Thank you, Roman!
>>> > >
>>> > > 2018-04-05 17:49 GMT+03:00 Роман Меерсон <
>>> homich1...@gmail.com>:
>>> > >
>>> > > > Hi Slava,
>>> > > >
>>> > > > Fixed
>>> > > >
>>> > > > чт, 5 апр. 2018 г. в 18:41, Вячеслав Коптилин <
>>> > slava.kopti...@gmail.com
>>> > > >:
>>> > > >
>>> > > > > Hi Roman,
>>> > > > >
>>> > > > > please take into account my comment
>>> IgniteQueryGenerator.java
>>> > > > > <
>>> > > > > https://reviews.ignite.apache.
>>> org/ignite/review/IGNT-CR-541?
>>> > > > commentId=de43c65f-9ac7-4080-9904-aec119138c94=/
>>> > > > modules/spring-data-2.0/src/main/java/org/apache/ignite/
>>> > > > springdata/repository/query/IgniteQueryGenerator.java
>>> > > > > >
>>> > > > >
>>> > > > > Best regards,
>>> > > > > Slava.
>>> > > > >
>>> > > > > 2018-04-05 14:59 GMT+03:00 Роман Меерсон <
>>> homich1...@gmail.com>:
>>> > > > >
>>> > > > > > Ok, so waiting for accept and commit
>>> > > > > >
>>> > > > > > чт, 5 апр. 2018 г. в 15:29, Alexey 

Re: Remove cache groups in AI 3.0

[Denis Magda]

Vladimir,

- Data size per-cache


Could you elaborate how the data size per-cache/table task will be
addressed with proposed architecture? Are you going to store data of a
specific cache in dedicated pages/segments? What's about index size?

--
Denis

On Tue, Apr 10, 2018 at 2:31 AM, Vladimir Ozerov 
wrote:

> Dima,
>
> 1) Easy to understand for users
> AI 2.x: cluster -> cache group -> cache -> table
> AI 3.x: cluster -> cache(==table)
>
> 2) Fine grained cache management
> - MVCC on/off per-cache
> - WAL mode on/off per-cache
> - Data size per-cache
>
> 3) Performance:
> - Efficient scans are not possible with cache groups
> - Efficient destroy/DROP - O(N) now, O(1) afterwards
>
> "Huge refactoring" is not precise estimate. Let's think on how to do that
> instead of how not to do :-)
>
> On Tue, Apr 10, 2018 at 11:41 AM, Dmitriy Setrakyan  >
> wrote:
>
> > Vladimir, sounds like a huge refactoring. Other than "cache groups are
> > confusing", are we solving any other big issues with the new proposed
> > approach?
> >
> > (every time we try to refactor rebalancing, I get goose bumps)
> >
> > D.
> >
> > On Tue, Apr 10, 2018 at 1:32 AM, Vladimir Ozerov 
> > wrote:
> >
> > > Igniters,
> > >
> > > Cache groups were implemented for a sole purpose - to hide internal
> > > inefficiencies. Namely (add more if I missed something):
> > > 1) Excessive heap usage for affinity/partition data
> > > 2) Too much data files as we employ file-per-partition approach.
> > >
> > > These problems were resolved, but now cache groups are a great source
> of
> > > confusion both for users and us - hard to understand, no way to
> configure
> > > it in deterministic way. Should we resolve mentioned performance issues
> > we
> > > would never had cache groups. I propose to think we would it take for
> us
> > to
> > > get rid of cache groups.
> > >
> > > Please provide your inputs to suggestions below.
> > >
> > > 1) "Merge" partition data from different caches
> > > Consider that we start a new cache with the same affinity configuration
> > > (cache mode, partition number, affinity function) as some of already
> > > existing caches, Is it possible to re-use partition distribution and
> > > history of existing cache for a new cache? Think of it as a kind of
> > > automatic cache grouping which is transparent to the user. This would
> > > remove heap pressure. Also it could resolve our long-standing issue
> with
> > > FairAffinityFunction when tow caches with the same affinity
> configuration
> > > are not co-located when started on different topology versions.
> > >
> > > 2) Employ segment-extent based approach instead of file-per-partition
> > > - Every object (cache, index) reside in dedicated segment
> > > - Segment consists of extents (minimal allocation units)
> > > - Extents are allocated and deallocated as needed
> > > - *Ignite specific*: particular extent can be used by only one
> partition
> > > - Segments may be located in any number of data files we find
> convenient
> > > With this approach "too many fsyncs" problem goes away automatically.
> At
> > > the same time it would be possible to implement efficient rebalance
> still
> > > as partition data will be split across moderate number of extents, not
> > > chaotically.
> > >
> > > Once we have p.1 and p.2 ready cache groups could be removed, couldn't
> > > they?
> > >
> > > Vladimir.
> > >
> >
>

Warning message

[Alew]

Hi!

I use local cache and get a lot of warnings when execute ScanQueries.

[13:26:25 WRN]  Ignoring query projection because 
it's executed over LOCAL cache (only local node will be queried): 
GridCacheQueryAdapter [type=SCAN, clsName=null, clause=null, 
filter=o.a.i.i.processors.platform.cache.PlatformCacheEntryFilterImpl@7629939a, 
transform=null, part=null, incMeta=false, 
metrics=GridCacheQueryMetricsAdapter [minTime=9223372036854775807, 
maxTime=0, sumTime=0, avgTime=0.0, execs=0, completed=0, fails=0], 
pageSize=1024, timeout=0, keepAll=true, incBackups=false, dedup=false, 
prj=o.a.i.i.cluster.ClusterGroupAdapter@708472f7, keepBinary=true, 
subjId=null, taskHash=0]


What does it warn me about? Is it really warning?

Re: Atomic caches

[akurbanov]

Dmitry,

Sorry for confusing topic. I I'm pretty sure that configuration for atomic
caches is validated, will double-check this. I was referring only atomic
data structures cache.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[GitHub] ignite pull request #3709: IGNITE-5979: for TC reproducing

[daradurvs]

Github user daradurvs closed the pull request at:

https://github.com/apache/ignite/pull/3709


---

[GitHub] ignite pull request #3792: -

[daradurvs]

GitHub user daradurvs opened a pull request:

https://github.com/apache/ignite/pull/3792

-



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/daradurvs/ignite ignite-8141

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3792.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3792


commit 5ee431ad7b8a5df5748bc69cc4ba5f239784b487
Author: Vyacheslav Daradur 
Date:   2018-04-10T18:50:14Z

ignite-4756: checks modified




---

[GitHub] ignite pull request #3791: IGNITE-8116 Historical rebalance fixes

[Jokser]

GitHub user Jokser opened a pull request:

https://github.com/apache/ignite/pull/3791

IGNITE-8116 Historical rebalance fixes



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8116-8122

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3791.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3791


commit 604ee719b304d0b4cf4caabaa6fa16b5a980e04e
Author: Pavel Kovalenko 
Date:   2018-04-04T09:33:10Z

IGNITE-8122 Restore partition state to OWNING if unable to read from page 
memory.

commit c43c598916bd9bda2f624a214cefcc28b0380a5c
Author: Pavel Kovalenko 
Date:   2018-04-05T14:26:53Z

IGNITE-8122 Restore partitions when persistence is enabled with OWNING 
default state.

commit a061cdba3a46f5384910fecf89366101f904ccdf
Author: Pavel Kovalenko 
Date:   2018-04-05T14:50:20Z

IGNITE-8122 Move OWN logic to GridDhtLocalPartition constructor.

commit 9259407a462633a68de6dcd7d3ae135c8c7c0b37
Author: Pavel Kovalenko 
Date:   2018-04-05T17:15:39Z

IGNITE-8122 Docs.

commit 20979dc4874cfeca649b29d8827d522f3e57ee67
Author: Pavel Kovalenko 
Date:   2018-04-05T17:16:54Z

Merge branch 'master' into ignite-8122

commit 791ef918335e66d274a10c2b5d21c9da1c212a0b
Author: Pavel Kovalenko 
Date:   2018-04-06T12:53:21Z

IGNITE-8122 Restore partition in OWNING state correctly.

commit 56bdb20513731f49bf3a2b51f672396efc16bfe1
Author: Pavel Kovalenko 
Date:   2018-04-09T11:42:59Z

IGNITE-8122 Restore partition states on before exchange in case of starting 
cache group.

commit dcea5b47a5a08f165930a2e0235ecf51385f4997
Author: Pavel Kovalenko 
Date:   2018-04-09T11:55:53Z

IGNITE-8122 Fixed test with auto-activation.

commit 915788c7ea25f04ccacfa06e1f19c06c92f7c141
Author: Pavel Kovalenko 
Date:   2018-04-09T15:32:59Z

IGNITE-8116 WIP

commit d224cd5663415dc8636f6ba01bfd5002fdb35a3e
Author: Pavel Kovalenko 
Date:   2018-04-10T10:59:20Z

Merge branch 'ignite-8122' into ignite-8116-8122

commit e988272e57760ee44753314ad8db2adab344a6c0
Author: Pavel Kovalenko 
Date:   2018-04-10T11:47:04Z

IGNITE-8116 WIP

commit bd53648935b24fb2c68a885db81656f2d960ba1e
Author: Pavel Kovalenko 
Date:   2018-04-10T18:09:32Z

IGNITE-8116 WAL rebalance issues.




---

Re: IGNITE-6827 - Review needed.

[Dmitry Pavlov]

Hi Pavel,

 thank you for bring up test questions. It seems my previous comments were
not taken into account.

Igniters,

 let me remind we should get passing TC suites before merge,
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers
(highlighted
note).

For disabling parity test checks please consider steps describled in
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Tests+How+To#IgniteTestsHowTo-Testof.NETAPIparitywithJavaAPI

Sincerely,
Dmitriy Pavlov


пн, 9 апр. 2018 г. в 21:18, Pavel Tupitsyn :

> > Pavel Tupitsyn, what about .NET stuff ?
>
> 1) Thank you for filing the ticket, personally I have no plans to work on
> it in the near future.
>
> 2) .NET tests fail, please make sure they are fixed before merging:
> https://ci.ignite.apache.org/viewLog.html?buildId=1175956
>
> TransactionsParityTest should be fixed by adding new properties to ignore
> list with a reference to IGNITE-8075, this is simple.
>
> But I have concerns about
> *CachePartitionedTest.TestTransactionScopeMultiCache, *
> seems like something is broken with multi-cache transactions. Please
> investigate this one.
>
> Thanks,
> Pavel
>
>
> On Mon, Apr 9, 2018 at 6:24 PM, Alexei Scherbakov <
> alexey.scherbak...@gmail.com> wrote:
>
> > Guys,
> >
> > I've slightly modified public API javadoc as Denis Magda has suggested in
> > PR review.
> >
> > Please take a look.
> >
> > Pavel Tupitsyn, what about .NET stuff ?
> >
> > I provided all necessary information in ticket [2]
> >
> > Upsource link [1]
> >
> > [1] https://reviews.ignite.apache.org/ignite/branch/PR%203624
> >
> > [2] https://issues.apache.org/jira/browse/IGNITE-8075
> >
> >
> >
> > пн, 9 апр. 2018 г. в 16:57, Alexey Goncharuk  >:
> >
> > > I am not aware of any additional timeouts that we are willing to add in
> > the
> > > nearest future.
> > >
> > > 2018-04-09 16:01 GMT+03:00 Dmitriy Setrakyan :
> > >
> > > > On Mon, Apr 9, 2018 at 5:42 AM, Alexey Goncharuk <
> > > > alexey.goncha...@gmail.com
> > > > > wrote:
> > > >
> > > > > Guys,
> > > > >
> > > > > After the review in Upsource the configuration parameter was
> renamed
> > > > > to txTimeoutOnPartMapSync, and it makes sense to me because PME is
> an
> > > > > implementation detail and it may change in future, partition map
> sync
> > > is
> > > > a
> > > > > more abstract term. For the same reason I like this parameter being
> > > > placed
> > > > > on transactions configuration - we do not have any parameters for
> > PME,
> > > so
> > > > > the configuration property goes to an object which affects a
> > > user-exposed
> > > > > API.
> > > > >
> > > >
> > > > AG, are we going to have any other timeouts on PME, like locks? If
> yes,
> > > > then I would still vote of adding PmeTimeout property.
> > > >
> > >
> >
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
> >
>

[jira] [Created] (IGNITE-8210) Rebalancing sometimes is not triggered when a node is added to the baseline topology

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-8210:
--

 Summary: Rebalancing sometimes is not triggered when a node is 
added to the baseline topology
 Key: IGNITE-8210
 URL: https://issues.apache.org/jira/browse/IGNITE-8210
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Reporter: Stanislav Lukyanov


When a cluster already has baseline topology set and activated adding a new 
node to the baseline should lead to rebalancing. However, in some cases the new 
node will not receive any data when joined and added to the baseline; if a 
cluster is later deactivated and activated again, the rebalancing happens 
properly and partitions are assigned and transferred to the new node.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8209) Exchange worker busy-spins after ForceRebalanceExchangeTask

[Alexey Goncharuk (JIRA)]

Alexey Goncharuk created IGNITE-8209:


 Summary: Exchange worker busy-spins after 
ForceRebalanceExchangeTask
 Key: IGNITE-8209
 URL: https://issues.apache.org/jira/browse/IGNITE-8209
 Project: Ignite
  Issue Type: Bug
Reporter: Alexey Goncharuk


Currently, we assign timeout=0 if exchange worker receives 
ForceRebalanceExchangeTask, however, assignment back to non-zero value is 
performed only if rebalance is finished, which results in exchange worker 
busy-spin.

Looks like we should re-assign timeout back to network timeout if timeout is 
zero and the queue is empty.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

[Dmitriy Setrakyan]

I am not convinced that the performance degradation is only due to the new
change that fixes the incorrect behavior. To my knowledge, there is also a
drop in memory-only mode. Can someone explain why do we have such a drop?

D.

On Tue, Apr 10, 2018 at 9:08 AM, Vladimir Ozerov 
wrote:

> 16% looks perfectly ok to me provided that we compare correct
> implementation with incorrect one.
>
> вт, 10 апр. 2018 г. в 18:24, Dmitriy Setrakyan :
>
> > Ilya, can we find out why pure in-memory scenario also had a performance
> > drop and which commit caused it? It should not be affected by changes in
> > persistence at all.
> >
> > D.
> >
> > On Tue, Apr 10, 2018 at 7:56 AM, Ilya Suntsov 
> > wrote:
> >
> > > Igniters,
> > >
> > > Looks like commit:
> > >
> > > d0adb61ecd9af0d9907e480ec747ea1465f97cd7 is the first bad commit
> > > > commit d0adb61ecd9af0d9907e480ec747ea1465f97cd7
> > > > Author: Ivan Rakov 
> > > > Date:   Tue Mar 27 20:11:52 2018 +0300
> > > > IGNITE-7754 WAL in LOG_ONLY mode doesn't execute fsync on
> > checkpoint
> > > > begin - Fixes #3656.
> > >
> > >
> > > was the cause of performance drop ( > 10% vs AI 2.4.0) on the following
> > > benchmarks (LOG_ONLY):
> > >
> > >- atomic-put  (16 %)
> > >- atomic-putAll (14 %)
> > >- tx-putAll (11 %)
> > >
> > > As I understand it is greater than initial assessment.
> > >
> > > Thoughts?
> > >
> > > 2018-03-27 20:13 GMT+03:00 Dmitry Pavlov :
> > >
> > > > Ivan, sure :)
> > > >
> > > > Thank you for this contribution, merged to master.
> > > >
> > > > вт, 27 мар. 2018 г. в 20:08, Ivan Rakov :
> > > >
> > > > > Dmitry,
> > > > >
> > > > > Firstly PR contained dirty fix for performance measurement, but now
> > it
> > > > > contains good fix. :) Sorry for inconvenience.
> > > > > I've renamed the PR.
> > > > >
> > > > > Best Regards,
> > > > > Ivan Rakov
> > > > >
> > > > > On 27.03.2018 19:40, Dmitry Pavlov wrote:
> > > > > > Hi Eduard, thank you for review.
> > > > > >
> > > > > > Hi Ivan,
> > > > > >
> > > > > > I'm confused on PR naming
> > > > > > https://github.com/apache/ignite/pull/3656
> > > > > >
> > > > > > Could you rename?
> > > > > >
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > > > >
> > > > > > вт, 27 мар. 2018 г. в 19:38, Eduard Shangareev <
> > > > > eduard.shangar...@gmail.com
> > > > > >> :
> > > > > >> Ivan, I have reviewed your changes, looks good.
> > > > > >>
> > > > > >> On Tue, Mar 27, 2018 at 2:56 PM, Ivan Rakov <
> > ivan.glu...@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >>> Igniters,
> > > > > >>>
> > > > > >>> I've completed development of https://issues.apache.org/jira
> > > > > >>> /browse/IGNITE-7754. TeamCity state is ok. Please, review my
> > > changes.
> > > > > >>> Please note that it will be possible to track time of WAL fsync
> > on
> > > > > >>> checkpoint begin by *walCpRecordFsyncDuration *metric in
> > > "Checkpoint
> > > > > >>> started" message.
> > > > > >>>
> > > > > >>> Also, I've created https://issues.apache.org/
> > > jira/browse/IGNITE-8057
> > > > > >> with
> > > > > >>> description of possible further improvement of WAL fsync on
> > > > checkpoint
> > > > > >>> begin.
> > > > > >>>
> > > > > >>> Best Regards,
> > > > > >>> Ivan Rakov
> > > > > >>>
> > > > > >>>
> > > > > >>> On 26.03.2018 23:45, Valentin Kulichenko wrote:
> > > > > >>>
> > > > >  Ivan,
> > > > > 
> > > > >  It's all good then :) Thanks!
> > > > > 
> > > > >  -Val
> > > > > 
> > > > >  On Mon, Mar 26, 2018 at 1:50 AM, Ivan Rakov <
> > > ivan.glu...@gmail.com>
> > > > >  wrote:
> > > > > 
> > > > >  Val,
> > > > > > There's no any sense to use WalMode.NONE in production
> > > environment,
> > > > > >> it's
> > > > > > kept for testing and debugging purposes (including possible
> > user
> > > > > > activities
> > > > > > like capacity planning).
> > > > > > We already print a warning at node start in case WalMode.NONE
> > is
> > > > set:
> > > > > >
> > > > > > U.quietAndWarn(log,"Started write-ahead log manager in NONE
> > mode,
> > > > > >
> > > > > >> persisted data may be lost in " +
> > > > > >>"a case of unexpected node failure. Make sure to
> > > deactivate
> > > > > the
> > > > > >> cluster before shutdown.");
> > > > > >>
> > > > > >> Best Regards,
> > > > > > Ivan Rakov
> > > > > >
> > > > > >
> > > > > > On 24.03.2018 1:40, Valentin Kulichenko wrote:
> > > > > >
> > > > > > Dmitry,
> > > > > >> Thanks for clarification. So it sounds like if we fix all
> > other
> > > > > modes
> > > > > >> as
> > > > > >> we
> > > > > >> discuss here, NONE would be the only one allowing
> corruption.
> > I
> > > > also
> > > > > >> don't
> > > > > >> see much sense in this and I think we should clearly state
> > this
> > > in
> > > > > the
> > 

Make TC Green in OSGI: IgniteKarafFeaturesInstallationTest

[Dmitry Pavlov]

Hi Raúl, Igniters,

Test related to OSGI/Karaf
(IgniteKarafFeaturesInstallationTest.testAllBundlesActiveAndFeaturesInstalled)
is currently failing
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=451522206339372479=%3Cdefault%3E=testDetails
with low success rate.

Recently Igniters have done 2 fixes for make this test passing (
https://issues.apache.org/jira/browse/IGNITE-7646 ,
https://issues.apache.org/jira/browse/IGNITE-7814 ) but test is failing
anyway.

Could you please step in and help to make this test green?

Sincerely,
Dmitriy Pavlov

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

[Dmitry Pavlov]

++1 from me for Vladimir's point

вт, 10 апр. 2018 г. в 19:08, Vladimir Ozerov :

> 16% looks perfectly ok to me provided that we compare correct
> implementation with incorrect one.
>
> вт, 10 апр. 2018 г. в 18:24, Dmitriy Setrakyan :
>
> > Ilya, can we find out why pure in-memory scenario also had a performance
> > drop and which commit caused it? It should not be affected by changes in
> > persistence at all.
> >
> > D.
> >
> > On Tue, Apr 10, 2018 at 7:56 AM, Ilya Suntsov 
> > wrote:
> >
> > > Igniters,
> > >
> > > Looks like commit:
> > >
> > > d0adb61ecd9af0d9907e480ec747ea1465f97cd7 is the first bad commit
> > > > commit d0adb61ecd9af0d9907e480ec747ea1465f97cd7
> > > > Author: Ivan Rakov 
> > > > Date:   Tue Mar 27 20:11:52 2018 +0300
> > > > IGNITE-7754 WAL in LOG_ONLY mode doesn't execute fsync on
> > checkpoint
> > > > begin - Fixes #3656.
> > >
> > >
> > > was the cause of performance drop ( > 10% vs AI 2.4.0) on the following
> > > benchmarks (LOG_ONLY):
> > >
> > >- atomic-put  (16 %)
> > >- atomic-putAll (14 %)
> > >- tx-putAll (11 %)
> > >
> > > As I understand it is greater than initial assessment.
> > >
> > > Thoughts?
> > >
> > > 2018-03-27 20:13 GMT+03:00 Dmitry Pavlov :
> > >
> > > > Ivan, sure :)
> > > >
> > > > Thank you for this contribution, merged to master.
> > > >
> > > > вт, 27 мар. 2018 г. в 20:08, Ivan Rakov :
> > > >
> > > > > Dmitry,
> > > > >
> > > > > Firstly PR contained dirty fix for performance measurement, but now
> > it
> > > > > contains good fix. :) Sorry for inconvenience.
> > > > > I've renamed the PR.
> > > > >
> > > > > Best Regards,
> > > > > Ivan Rakov
> > > > >
> > > > > On 27.03.2018 19:40, Dmitry Pavlov wrote:
> > > > > > Hi Eduard, thank you for review.
> > > > > >
> > > > > > Hi Ivan,
> > > > > >
> > > > > > I'm confused on PR naming
> > > > > > https://github.com/apache/ignite/pull/3656
> > > > > >
> > > > > > Could you rename?
> > > > > >
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > > > >
> > > > > > вт, 27 мар. 2018 г. в 19:38, Eduard Shangareev <
> > > > > eduard.shangar...@gmail.com
> > > > > >> :
> > > > > >> Ivan, I have reviewed your changes, looks good.
> > > > > >>
> > > > > >> On Tue, Mar 27, 2018 at 2:56 PM, Ivan Rakov <
> > ivan.glu...@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >>> Igniters,
> > > > > >>>
> > > > > >>> I've completed development of https://issues.apache.org/jira
> > > > > >>> /browse/IGNITE-7754. TeamCity state is ok. Please, review my
> > > changes.
> > > > > >>> Please note that it will be possible to track time of WAL fsync
> > on
> > > > > >>> checkpoint begin by *walCpRecordFsyncDuration *metric in
> > > "Checkpoint
> > > > > >>> started" message.
> > > > > >>>
> > > > > >>> Also, I've created https://issues.apache.org/
> > > jira/browse/IGNITE-8057
> > > > > >> with
> > > > > >>> description of possible further improvement of WAL fsync on
> > > > checkpoint
> > > > > >>> begin.
> > > > > >>>
> > > > > >>> Best Regards,
> > > > > >>> Ivan Rakov
> > > > > >>>
> > > > > >>>
> > > > > >>> On 26.03.2018 23:45, Valentin Kulichenko wrote:
> > > > > >>>
> > > > >  Ivan,
> > > > > 
> > > > >  It's all good then :) Thanks!
> > > > > 
> > > > >  -Val
> > > > > 
> > > > >  On Mon, Mar 26, 2018 at 1:50 AM, Ivan Rakov <
> > > ivan.glu...@gmail.com>
> > > > >  wrote:
> > > > > 
> > > > >  Val,
> > > > > > There's no any sense to use WalMode.NONE in production
> > > environment,
> > > > > >> it's
> > > > > > kept for testing and debugging purposes (including possible
> > user
> > > > > > activities
> > > > > > like capacity planning).
> > > > > > We already print a warning at node start in case WalMode.NONE
> > is
> > > > set:
> > > > > >
> > > > > > U.quietAndWarn(log,"Started write-ahead log manager in NONE
> > mode,
> > > > > >
> > > > > >> persisted data may be lost in " +
> > > > > >>"a case of unexpected node failure. Make sure to
> > > deactivate
> > > > > the
> > > > > >> cluster before shutdown.");
> > > > > >>
> > > > > >> Best Regards,
> > > > > > Ivan Rakov
> > > > > >
> > > > > >
> > > > > > On 24.03.2018 1:40, Valentin Kulichenko wrote:
> > > > > >
> > > > > > Dmitry,
> > > > > >> Thanks for clarification. So it sounds like if we fix all
> > other
> > > > > modes
> > > > > >> as
> > > > > >> we
> > > > > >> discuss here, NONE would be the only one allowing
> corruption.
> > I
> > > > also
> > > > > >> don't
> > > > > >> see much sense in this and I think we should clearly state
> > this
> > > in
> > > > > the
> > > > > >> doc,
> > > > > >> as well print out a warning if NONE mode is used.
> Eventually,
> > if
> > > > > it's
> > > > > >> confirmed that there are no reasonable use cases for it, we
> > can

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

[Vladimir Ozerov]

16% looks perfectly ok to me provided that we compare correct
implementation with incorrect one.

вт, 10 апр. 2018 г. в 18:24, Dmitriy Setrakyan :

> Ilya, can we find out why pure in-memory scenario also had a performance
> drop and which commit caused it? It should not be affected by changes in
> persistence at all.
>
> D.
>
> On Tue, Apr 10, 2018 at 7:56 AM, Ilya Suntsov 
> wrote:
>
> > Igniters,
> >
> > Looks like commit:
> >
> > d0adb61ecd9af0d9907e480ec747ea1465f97cd7 is the first bad commit
> > > commit d0adb61ecd9af0d9907e480ec747ea1465f97cd7
> > > Author: Ivan Rakov 
> > > Date:   Tue Mar 27 20:11:52 2018 +0300
> > > IGNITE-7754 WAL in LOG_ONLY mode doesn't execute fsync on
> checkpoint
> > > begin - Fixes #3656.
> >
> >
> > was the cause of performance drop ( > 10% vs AI 2.4.0) on the following
> > benchmarks (LOG_ONLY):
> >
> >- atomic-put  (16 %)
> >- atomic-putAll (14 %)
> >- tx-putAll (11 %)
> >
> > As I understand it is greater than initial assessment.
> >
> > Thoughts?
> >
> > 2018-03-27 20:13 GMT+03:00 Dmitry Pavlov :
> >
> > > Ivan, sure :)
> > >
> > > Thank you for this contribution, merged to master.
> > >
> > > вт, 27 мар. 2018 г. в 20:08, Ivan Rakov :
> > >
> > > > Dmitry,
> > > >
> > > > Firstly PR contained dirty fix for performance measurement, but now
> it
> > > > contains good fix. :) Sorry for inconvenience.
> > > > I've renamed the PR.
> > > >
> > > > Best Regards,
> > > > Ivan Rakov
> > > >
> > > > On 27.03.2018 19:40, Dmitry Pavlov wrote:
> > > > > Hi Eduard, thank you for review.
> > > > >
> > > > > Hi Ivan,
> > > > >
> > > > > I'm confused on PR naming
> > > > > https://github.com/apache/ignite/pull/3656
> > > > >
> > > > > Could you rename?
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > > вт, 27 мар. 2018 г. в 19:38, Eduard Shangareev <
> > > > eduard.shangar...@gmail.com
> > > > >> :
> > > > >> Ivan, I have reviewed your changes, looks good.
> > > > >>
> > > > >> On Tue, Mar 27, 2018 at 2:56 PM, Ivan Rakov <
> ivan.glu...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >>> Igniters,
> > > > >>>
> > > > >>> I've completed development of https://issues.apache.org/jira
> > > > >>> /browse/IGNITE-7754. TeamCity state is ok. Please, review my
> > changes.
> > > > >>> Please note that it will be possible to track time of WAL fsync
> on
> > > > >>> checkpoint begin by *walCpRecordFsyncDuration *metric in
> > "Checkpoint
> > > > >>> started" message.
> > > > >>>
> > > > >>> Also, I've created https://issues.apache.org/
> > jira/browse/IGNITE-8057
> > > > >> with
> > > > >>> description of possible further improvement of WAL fsync on
> > > checkpoint
> > > > >>> begin.
> > > > >>>
> > > > >>> Best Regards,
> > > > >>> Ivan Rakov
> > > > >>>
> > > > >>>
> > > > >>> On 26.03.2018 23:45, Valentin Kulichenko wrote:
> > > > >>>
> > > >  Ivan,
> > > > 
> > > >  It's all good then :) Thanks!
> > > > 
> > > >  -Val
> > > > 
> > > >  On Mon, Mar 26, 2018 at 1:50 AM, Ivan Rakov <
> > ivan.glu...@gmail.com>
> > > >  wrote:
> > > > 
> > > >  Val,
> > > > > There's no any sense to use WalMode.NONE in production
> > environment,
> > > > >> it's
> > > > > kept for testing and debugging purposes (including possible
> user
> > > > > activities
> > > > > like capacity planning).
> > > > > We already print a warning at node start in case WalMode.NONE
> is
> > > set:
> > > > >
> > > > > U.quietAndWarn(log,"Started write-ahead log manager in NONE
> mode,
> > > > >
> > > > >> persisted data may be lost in " +
> > > > >>"a case of unexpected node failure. Make sure to
> > deactivate
> > > > the
> > > > >> cluster before shutdown.");
> > > > >>
> > > > >> Best Regards,
> > > > > Ivan Rakov
> > > > >
> > > > >
> > > > > On 24.03.2018 1:40, Valentin Kulichenko wrote:
> > > > >
> > > > > Dmitry,
> > > > >> Thanks for clarification. So it sounds like if we fix all
> other
> > > > modes
> > > > >> as
> > > > >> we
> > > > >> discuss here, NONE would be the only one allowing corruption.
> I
> > > also
> > > > >> don't
> > > > >> see much sense in this and I think we should clearly state
> this
> > in
> > > > the
> > > > >> doc,
> > > > >> as well print out a warning if NONE mode is used. Eventually,
> if
> > > > it's
> > > > >> confirmed that there are no reasonable use cases for it, we
> can
> > > > >> deprecate
> > > > >> it.
> > > > >>
> > > > >> -Val
> > > > >>
> > > > >> On Fri, Mar 23, 2018 at 3:26 PM, Dmitry Pavlov <
> > > > dpavlov@gmail.com
> > > > >> wrote:
> > > > >>
> > > > >> Hi Val,
> > > > >>
> > > > >>> NONE means that the WAL log is disabled and not written at
> all.
> > > Use
> > > > >> of
> > > > >>> the
> > > > >>> mode is at your 

[GitHub] ignite pull request #3790: IGNITE-8042

[devozerov]

GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/3790

IGNITE-8042



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8042

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3790.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3790


commit 68d5f687479fc7151b55d2cbaa0b788a29c6c169
Author: devozerov 
Date:   2018-04-09T10:01:10Z

Implemented server-side part.

commit 41a9431b97cf4b94d77e45556b9d55f2cc1303e7
Author: devozerov 
Date:   2018-04-10T12:51:07Z

Merge branch 'master' into ignite-8042

commit 22272fd239b4b6b014119705ce84dbd092f89e2c
Author: devozerov 
Date:   2018-04-10T15:49:10Z

Tests.




---

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

[Dmitriy Setrakyan]

Ilya, can we find out why pure in-memory scenario also had a performance
drop and which commit caused it? It should not be affected by changes in
persistence at all.

D.

On Tue, Apr 10, 2018 at 7:56 AM, Ilya Suntsov  wrote:

> Igniters,
>
> Looks like commit:
>
> d0adb61ecd9af0d9907e480ec747ea1465f97cd7 is the first bad commit
> > commit d0adb61ecd9af0d9907e480ec747ea1465f97cd7
> > Author: Ivan Rakov 
> > Date:   Tue Mar 27 20:11:52 2018 +0300
> > IGNITE-7754 WAL in LOG_ONLY mode doesn't execute fsync on checkpoint
> > begin - Fixes #3656.
>
>
> was the cause of performance drop ( > 10% vs AI 2.4.0) on the following
> benchmarks (LOG_ONLY):
>
>- atomic-put  (16 %)
>- atomic-putAll (14 %)
>- tx-putAll (11 %)
>
> As I understand it is greater than initial assessment.
>
> Thoughts?
>
> 2018-03-27 20:13 GMT+03:00 Dmitry Pavlov :
>
> > Ivan, sure :)
> >
> > Thank you for this contribution, merged to master.
> >
> > вт, 27 мар. 2018 г. в 20:08, Ivan Rakov :
> >
> > > Dmitry,
> > >
> > > Firstly PR contained dirty fix for performance measurement, but now it
> > > contains good fix. :) Sorry for inconvenience.
> > > I've renamed the PR.
> > >
> > > Best Regards,
> > > Ivan Rakov
> > >
> > > On 27.03.2018 19:40, Dmitry Pavlov wrote:
> > > > Hi Eduard, thank you for review.
> > > >
> > > > Hi Ivan,
> > > >
> > > > I'm confused on PR naming
> > > > https://github.com/apache/ignite/pull/3656
> > > >
> > > > Could you rename?
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > вт, 27 мар. 2018 г. в 19:38, Eduard Shangareev <
> > > eduard.shangar...@gmail.com
> > > >> :
> > > >> Ivan, I have reviewed your changes, looks good.
> > > >>
> > > >> On Tue, Mar 27, 2018 at 2:56 PM, Ivan Rakov 
> > > wrote:
> > > >>
> > > >>> Igniters,
> > > >>>
> > > >>> I've completed development of https://issues.apache.org/jira
> > > >>> /browse/IGNITE-7754. TeamCity state is ok. Please, review my
> changes.
> > > >>> Please note that it will be possible to track time of WAL fsync on
> > > >>> checkpoint begin by *walCpRecordFsyncDuration *metric in
> "Checkpoint
> > > >>> started" message.
> > > >>>
> > > >>> Also, I've created https://issues.apache.org/
> jira/browse/IGNITE-8057
> > > >> with
> > > >>> description of possible further improvement of WAL fsync on
> > checkpoint
> > > >>> begin.
> > > >>>
> > > >>> Best Regards,
> > > >>> Ivan Rakov
> > > >>>
> > > >>>
> > > >>> On 26.03.2018 23:45, Valentin Kulichenko wrote:
> > > >>>
> > >  Ivan,
> > > 
> > >  It's all good then :) Thanks!
> > > 
> > >  -Val
> > > 
> > >  On Mon, Mar 26, 2018 at 1:50 AM, Ivan Rakov <
> ivan.glu...@gmail.com>
> > >  wrote:
> > > 
> > >  Val,
> > > > There's no any sense to use WalMode.NONE in production
> environment,
> > > >> it's
> > > > kept for testing and debugging purposes (including possible user
> > > > activities
> > > > like capacity planning).
> > > > We already print a warning at node start in case WalMode.NONE is
> > set:
> > > >
> > > > U.quietAndWarn(log,"Started write-ahead log manager in NONE mode,
> > > >
> > > >> persisted data may be lost in " +
> > > >>"a case of unexpected node failure. Make sure to
> deactivate
> > > the
> > > >> cluster before shutdown.");
> > > >>
> > > >> Best Regards,
> > > > Ivan Rakov
> > > >
> > > >
> > > > On 24.03.2018 1:40, Valentin Kulichenko wrote:
> > > >
> > > > Dmitry,
> > > >> Thanks for clarification. So it sounds like if we fix all other
> > > modes
> > > >> as
> > > >> we
> > > >> discuss here, NONE would be the only one allowing corruption. I
> > also
> > > >> don't
> > > >> see much sense in this and I think we should clearly state this
> in
> > > the
> > > >> doc,
> > > >> as well print out a warning if NONE mode is used. Eventually, if
> > > it's
> > > >> confirmed that there are no reasonable use cases for it, we can
> > > >> deprecate
> > > >> it.
> > > >>
> > > >> -Val
> > > >>
> > > >> On Fri, Mar 23, 2018 at 3:26 PM, Dmitry Pavlov <
> > > dpavlov@gmail.com
> > > >> wrote:
> > > >>
> > > >> Hi Val,
> > > >>
> > > >>> NONE means that the WAL log is disabled and not written at all.
> > Use
> > > >> of
> > > >>> the
> > > >>> mode is at your own risk. It is possible that restore state
> after
> > > the
> > > >>> crash
> > > >>> at the middle of checkpoint will not succeed. I do not see much
> > > sence
> > > >>> in
> > > >>> it, especially in production.
> > > >>>
> > > >>> BACKGROUND is full functional WAL mode, but allows some delay
> > > before
> > > >>> flush
> > > >>> to disk.
> > > >>>
> > > >>> Sincerely,
> > > >>> Dmitriy Pavlov
> > > >>>
> > > >>> сб, 24 мар. 2018 г. в 1:07, 

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

[Dmitry Pavlov]

Hi Ilya,

Thank you for checking Ignite performance.

Is it slower than our old default WAL mode 'Default'?

Sincerely,
Dmitriy Pavlov

вт, 10 апр. 2018 г. в 17:57, Ilya Suntsov :

> Igniters,
>
> Looks like commit:
>
> d0adb61ecd9af0d9907e480ec747ea1465f97cd7 is the first bad commit
> > commit d0adb61ecd9af0d9907e480ec747ea1465f97cd7
> > Author: Ivan Rakov 
> > Date:   Tue Mar 27 20:11:52 2018 +0300
> > IGNITE-7754 WAL in LOG_ONLY mode doesn't execute fsync on checkpoint
> > begin - Fixes #3656.
>
>
> was the cause of performance drop ( > 10% vs AI 2.4.0) on the following
> benchmarks (LOG_ONLY):
>
>- atomic-put  (16 %)
>- atomic-putAll (14 %)
>- tx-putAll (11 %)
>
> As I understand it is greater than initial assessment.
>
> Thoughts?
>
> 2018-03-27 20:13 GMT+03:00 Dmitry Pavlov :
>
> > Ivan, sure :)
> >
> > Thank you for this contribution, merged to master.
> >
> > вт, 27 мар. 2018 г. в 20:08, Ivan Rakov :
> >
> > > Dmitry,
> > >
> > > Firstly PR contained dirty fix for performance measurement, but now it
> > > contains good fix. :) Sorry for inconvenience.
> > > I've renamed the PR.
> > >
> > > Best Regards,
> > > Ivan Rakov
> > >
> > > On 27.03.2018 19:40, Dmitry Pavlov wrote:
> > > > Hi Eduard, thank you for review.
> > > >
> > > > Hi Ivan,
> > > >
> > > > I'm confused on PR naming
> > > > https://github.com/apache/ignite/pull/3656
> > > >
> > > > Could you rename?
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > вт, 27 мар. 2018 г. в 19:38, Eduard Shangareev <
> > > eduard.shangar...@gmail.com
> > > >> :
> > > >> Ivan, I have reviewed your changes, looks good.
> > > >>
> > > >> On Tue, Mar 27, 2018 at 2:56 PM, Ivan Rakov 
> > > wrote:
> > > >>
> > > >>> Igniters,
> > > >>>
> > > >>> I've completed development of https://issues.apache.org/jira
> > > >>> /browse/IGNITE-7754. TeamCity state is ok. Please, review my
> changes.
> > > >>> Please note that it will be possible to track time of WAL fsync on
> > > >>> checkpoint begin by *walCpRecordFsyncDuration *metric in
> "Checkpoint
> > > >>> started" message.
> > > >>>
> > > >>> Also, I've created
> https://issues.apache.org/jira/browse/IGNITE-8057
> > > >> with
> > > >>> description of possible further improvement of WAL fsync on
> > checkpoint
> > > >>> begin.
> > > >>>
> > > >>> Best Regards,
> > > >>> Ivan Rakov
> > > >>>
> > > >>>
> > > >>> On 26.03.2018 23:45, Valentin Kulichenko wrote:
> > > >>>
> > >  Ivan,
> > > 
> > >  It's all good then :) Thanks!
> > > 
> > >  -Val
> > > 
> > >  On Mon, Mar 26, 2018 at 1:50 AM, Ivan Rakov <
> ivan.glu...@gmail.com>
> > >  wrote:
> > > 
> > >  Val,
> > > > There's no any sense to use WalMode.NONE in production
> environment,
> > > >> it's
> > > > kept for testing and debugging purposes (including possible user
> > > > activities
> > > > like capacity planning).
> > > > We already print a warning at node start in case WalMode.NONE is
> > set:
> > > >
> > > > U.quietAndWarn(log,"Started write-ahead log manager in NONE mode,
> > > >
> > > >> persisted data may be lost in " +
> > > >>"a case of unexpected node failure. Make sure to
> deactivate
> > > the
> > > >> cluster before shutdown.");
> > > >>
> > > >> Best Regards,
> > > > Ivan Rakov
> > > >
> > > >
> > > > On 24.03.2018 1:40, Valentin Kulichenko wrote:
> > > >
> > > > Dmitry,
> > > >> Thanks for clarification. So it sounds like if we fix all other
> > > modes
> > > >> as
> > > >> we
> > > >> discuss here, NONE would be the only one allowing corruption. I
> > also
> > > >> don't
> > > >> see much sense in this and I think we should clearly state this
> in
> > > the
> > > >> doc,
> > > >> as well print out a warning if NONE mode is used. Eventually, if
> > > it's
> > > >> confirmed that there are no reasonable use cases for it, we can
> > > >> deprecate
> > > >> it.
> > > >>
> > > >> -Val
> > > >>
> > > >> On Fri, Mar 23, 2018 at 3:26 PM, Dmitry Pavlov <
> > > dpavlov@gmail.com
> > > >> wrote:
> > > >>
> > > >> Hi Val,
> > > >>
> > > >>> NONE means that the WAL log is disabled and not written at all.
> > Use
> > > >> of
> > > >>> the
> > > >>> mode is at your own risk. It is possible that restore state
> after
> > > the
> > > >>> crash
> > > >>> at the middle of checkpoint will not succeed. I do not see much
> > > sence
> > > >>> in
> > > >>> it, especially in production.
> > > >>>
> > > >>> BACKGROUND is full functional WAL mode, but allows some delay
> > > before
> > > >>> flush
> > > >>> to disk.
> > > >>>
> > > >>> Sincerely,
> > > >>> Dmitriy Pavlov
> > > >>>
> > > >>> сб, 24 мар. 2018 г. в 1:07, Valentin Kulichenko <
> > > >>> 

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

[Ilya Suntsov]

Igniters,

Looks like commit:

d0adb61ecd9af0d9907e480ec747ea1465f97cd7 is the first bad commit
> commit d0adb61ecd9af0d9907e480ec747ea1465f97cd7
> Author: Ivan Rakov 
> Date:   Tue Mar 27 20:11:52 2018 +0300
> IGNITE-7754 WAL in LOG_ONLY mode doesn't execute fsync on checkpoint
> begin - Fixes #3656.


was the cause of performance drop ( > 10% vs AI 2.4.0) on the following
benchmarks (LOG_ONLY):

   - atomic-put  (16 %)
   - atomic-putAll (14 %)
   - tx-putAll (11 %)

As I understand it is greater than initial assessment.

Thoughts?

2018-03-27 20:13 GMT+03:00 Dmitry Pavlov :

> Ivan, sure :)
>
> Thank you for this contribution, merged to master.
>
> вт, 27 мар. 2018 г. в 20:08, Ivan Rakov :
>
> > Dmitry,
> >
> > Firstly PR contained dirty fix for performance measurement, but now it
> > contains good fix. :) Sorry for inconvenience.
> > I've renamed the PR.
> >
> > Best Regards,
> > Ivan Rakov
> >
> > On 27.03.2018 19:40, Dmitry Pavlov wrote:
> > > Hi Eduard, thank you for review.
> > >
> > > Hi Ivan,
> > >
> > > I'm confused on PR naming
> > > https://github.com/apache/ignite/pull/3656
> > >
> > > Could you rename?
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > вт, 27 мар. 2018 г. в 19:38, Eduard Shangareev <
> > eduard.shangar...@gmail.com
> > >> :
> > >> Ivan, I have reviewed your changes, looks good.
> > >>
> > >> On Tue, Mar 27, 2018 at 2:56 PM, Ivan Rakov 
> > wrote:
> > >>
> > >>> Igniters,
> > >>>
> > >>> I've completed development of https://issues.apache.org/jira
> > >>> /browse/IGNITE-7754. TeamCity state is ok. Please, review my changes.
> > >>> Please note that it will be possible to track time of WAL fsync on
> > >>> checkpoint begin by *walCpRecordFsyncDuration *metric in "Checkpoint
> > >>> started" message.
> > >>>
> > >>> Also, I've created https://issues.apache.org/jira/browse/IGNITE-8057
> > >> with
> > >>> description of possible further improvement of WAL fsync on
> checkpoint
> > >>> begin.
> > >>>
> > >>> Best Regards,
> > >>> Ivan Rakov
> > >>>
> > >>>
> > >>> On 26.03.2018 23:45, Valentin Kulichenko wrote:
> > >>>
> >  Ivan,
> > 
> >  It's all good then :) Thanks!
> > 
> >  -Val
> > 
> >  On Mon, Mar 26, 2018 at 1:50 AM, Ivan Rakov 
> >  wrote:
> > 
> >  Val,
> > > There's no any sense to use WalMode.NONE in production environment,
> > >> it's
> > > kept for testing and debugging purposes (including possible user
> > > activities
> > > like capacity planning).
> > > We already print a warning at node start in case WalMode.NONE is
> set:
> > >
> > > U.quietAndWarn(log,"Started write-ahead log manager in NONE mode,
> > >
> > >> persisted data may be lost in " +
> > >>"a case of unexpected node failure. Make sure to deactivate
> > the
> > >> cluster before shutdown.");
> > >>
> > >> Best Regards,
> > > Ivan Rakov
> > >
> > >
> > > On 24.03.2018 1:40, Valentin Kulichenko wrote:
> > >
> > > Dmitry,
> > >> Thanks for clarification. So it sounds like if we fix all other
> > modes
> > >> as
> > >> we
> > >> discuss here, NONE would be the only one allowing corruption. I
> also
> > >> don't
> > >> see much sense in this and I think we should clearly state this in
> > the
> > >> doc,
> > >> as well print out a warning if NONE mode is used. Eventually, if
> > it's
> > >> confirmed that there are no reasonable use cases for it, we can
> > >> deprecate
> > >> it.
> > >>
> > >> -Val
> > >>
> > >> On Fri, Mar 23, 2018 at 3:26 PM, Dmitry Pavlov <
> > dpavlov@gmail.com
> > >> wrote:
> > >>
> > >> Hi Val,
> > >>
> > >>> NONE means that the WAL log is disabled and not written at all.
> Use
> > >> of
> > >>> the
> > >>> mode is at your own risk. It is possible that restore state after
> > the
> > >>> crash
> > >>> at the middle of checkpoint will not succeed. I do not see much
> > sence
> > >>> in
> > >>> it, especially in production.
> > >>>
> > >>> BACKGROUND is full functional WAL mode, but allows some delay
> > before
> > >>> flush
> > >>> to disk.
> > >>>
> > >>> Sincerely,
> > >>> Dmitriy Pavlov
> > >>>
> > >>> сб, 24 мар. 2018 г. в 1:07, Valentin Kulichenko <
> > >>> valentin.kuliche...@gmail.com>:
> > >>>
> > >>> I agree. In my view, any possibility to get a corrupted storage
> is
> > a
> > >>> bug
> > >>>
> >  which needs to be fixed.
> > 
> >  BTW, can someone explain semantics of NONE mode? What is the
> >  difference
> >  from BACKGROUND from user's perspective? Is there any particular
> > use
> >  case
> >  where it can be used?
> > 
> >  -Val
> > 
> >  On Fri, Mar 23, 2018 at 2:49 AM, Dmitry 

[GitHub] ignite pull request #3789: IGNITE-7999 Enhance performance of the thin JDBC ...

[tledkov-gridgain]

GitHub user tledkov-gridgain opened a pull request:

https://github.com/apache/ignite/pull/3789

IGNITE-7999  Enhance performance of the thin JDBC streaming mode 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7999

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3789.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3789


commit 2082930dd8358196bd2e8a3fabb0f89ac075584d
Author: tledkov-gridgain 
Date:   2018-04-05T15:23:12Z

ignite-7999: save the progress

commit 558cd63fbb85c33ab083d7ee33f9e042bdaabd95
Author: tledkov-gridgain 
Date:   2018-04-06T10:15:36Z

Merge branch '_master' into ignite-7999

commit 2bd9bf43261d6c5707c59de6065f0c300b1bb210
Author: tledkov-gridgain 
Date:   2018-04-06T13:57:13Z

ignite-7999: save the progress

commit 61a3b1a290c4c11f5ec25df1b86e14f4cc650c0e
Author: tledkov-gridgain 
Date:   2018-04-06T15:54:02Z

ignite-7999: save the progress

commit bfa967f81c0d6f02bc5b607de5a9ed7e37a2b918
Author: tledkov-gridgain 
Date:   2018-04-06T16:42:06Z

ignite-7999: save the progress

commit cbc7d18f52d51507a49343dd7ea5b36f87b6f58f
Author: tledkov-gridgain 
Date:   2018-04-09T11:13:42Z

ignite-7999: save the progress

commit 3a3a59ec75469b80bcb0200299b950f0324d5cc6
Author: tledkov-gridgain 
Date:   2018-04-09T11:35:16Z

ignite-7999: save the progress

commit b1f2141b29077ff9e27afcbeb3b2f7ec5a730bcd
Author: tledkov-gridgain 
Date:   2018-04-09T13:14:53Z

Merge branch '_master' into ignite-7999

commit cf04edaa0ae8751c3af6050d636b5b2d9714ceee
Author: tledkov-gridgain 
Date:   2018-04-09T13:31:52Z

ignite-7999: save the progress

commit 19fc8721a35f6145fe0d7aaf4702b9e52e4fb615
Author: tledkov-gridgain 
Date:   2018-04-09T14:08:23Z

Merge branch '_master' into ignite-7999

commit ead3e54df13e3d7d259a05bcdd5a2c7ebb2bebda
Author: tledkov-gridgain 
Date:   2018-04-10T14:35:01Z

ignite-7999: save the progress

commit dd9836d56ada5bf8473b458c7b98378973ab3e22
Author: tledkov-gridgain 
Date:   2018-04-10T14:41:31Z

Merge branch '_master' into ignite-7999




---

[GitHub] ignite pull request #3722: Ignite 7772

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3722


---

Re: IEP-18: Transparent Data Encryption

[Vladimir Ozerov]

Hi NIkolay,

Regarding system caches, rule of thumb here - do not use them. Keys should
be stored near cache.

As far as password:
1) Oracle auto-login wallet [1]
2) MySQL- password may be set inside configuration [2]

I do not think that any kind of prompts are needed here out of the box. May
be we could consider them in future, but at the moment it looks redundant
to me.

[1]
https://docs.oracle.com/cd/E11882_01/network.112/e40393/asotrans.htm#CHDCCHBH
[2]
https://dev.mysql.com/doc/refman/5.7/en/keyring-encrypted-file-plugin.html

On Mon, Apr 9, 2018 at 5:42 PM, Nikolay Izhikov  wrote:

> Hello, Vladimir.
>
> > 1) Why do you propose to store CEK in separate cache?
>
> All CEKs data should be available on all cluster nodes.
> We want to use system cache to get data synchronization feature "for free".
>
> > We consider storing any metadata in system caches as antipattern from
> our previous experience.
>
> Very interesting!
> Can you share your experience?
> I didn't know that.
> I think we can use any convinient storage for CEK.
> Current design doesn't couple to system caches, so we can change this part
> at any moment.
>
> > 2) I do not think that decryption process should require any
> "administrator" role and passwords.
> > Instead, we can have a kind of pluggable interface which will provide
> decrypted MEK on demand when it is needed.
> > This should be pre-configured in advance on server node(s).
> > AFAIK this is how a number of other vendors work.
>
> Can't agree with you, for now.
> Can you please send me some info about implementation you refer to?
> We study following papers to make current design:
>
> 1. ORACLE [1]
>
> To enable encryption one has to execute following statement:
>
> `ALTER SYSTEM SET ENCRYPTION KEY IDENTIFIED BY *clear_text_password*`
> Or after database server restart - `ALTER SYSTEM SET WALLET OPEN
> IDENTIFIED BY *clear_text_password*`
> So yes, administrator is involved into server restart.
> And no, it's not all automatic.
>
> 2. MSSQL [4] - AFAIK uses some Windows OS internal key storage to write
> and read master key.
> DB server process should have administrator(system) priveleges to access
> that storage.
>
> > Instead, we can have a kind of pluggable interface which will provide
> decrypted MEK on demand when it is needed.
>
> We, for sure, will have pluggable interface for providing MEK.
> But, from my point of view, default implementation should use some default
> java features.
> I think we should use java key store.
> It requires to have a clear text password to be loaded [3]
>
> [1] https://docs.oracle.com/cd/B19306_01/network.102/b14268/
> asotrans.htm#BABJJAIG
> [2] https://technet.microsoft.com/en-us/library/bb934049(v=sql.110).aspx
> [3] https://docs.oracle.com/javase/8/docs/api/java/
> security/KeyStore.html#load-java.io.InputStream-char:A-
> [4] https://technet.microsoft.com/en-us/library/bb934049(v=sql.110).aspx
>
> > 3) CEK decryption should not be tied to MEK decryption.
>
> MEK will be available from the moment it obtained to the node stop.
> So, we can decrypt existing or encrypt newly created CEK whenever it
> necessary.
>
> > 4) I do not think that SSL should be a strict requirement. It is up to
> the> user to asses the risks.
>
> I'm fully OK with it.
> Let's remove that restriction.
>
> В Пн, 09/04/2018 в 11:35 +0300, Vladimir Ozerov пишет:
> > Hi Nikolay,
> >
> > First of all thank you for excellent summary. Two-tiered key management
> is
> > well respected technique and makes perfect sense to me. However, several
> > questions regarding architecture arises:
>
> > 3) CEK decryption should not be tied to MEK decryption. Main reason - CEK
> > could be required during dynamic cache/table creation. So there should be
> > no coupling between activation and CEK processing.
>
> > 4) I do not think that SSL should be a strict requirement. It is up to
> the
> > user to asses the risks.
> >
> > On Fri, Apr 6, 2018 at 9:59 PM, Denis Magda  wrote:
> >
> > > Nikolay, Dmitriy R.,
> > >
> > > Thanks for the research and for writing down a summary in the IEP form.
> > >
> > > Please answer several high-level questions:
> > >
> > >- Is it necessary to have CEP keys for every cache? Not sure how
> all the
> > >keys will be managed if the user wants to encrypt 10-100 caches. Is
> it
> > >possible to use a single CEP by default with an option of having a
> > > unique
> > >one for a cache with more sensitive information?
> > >- It's not written, but I guess it would be up to me which caches to
> > >encrypt, right? In practice, you don't need to have all the data
> > > encrypted.
> > >Usually, companies look to hide personal, payments history, etc.
> > >- Should we think of procedures of CEP keys regeneration? A key can
> be
> > >lost or stolen.
> > >- Similar question goes for MEP key.
> > >
> > > --
> > > Denis
> > >
> > > On Thu, Apr 5, 2018 at 2:15 PM, Dmitriy Setrakyan <
> 

[jira] [Created] (IGNITE-8208) SQL TX: Do not check with TxLog rows having the same insert and update versions

[Igor Seliverstov (JIRA)]

Igor Seliverstov created IGNITE-8208:


 Summary: SQL TX: Do not check with TxLog rows having the same 
insert and update versions
 Key: IGNITE-8208
 URL: https://issues.apache.org/jira/browse/IGNITE-8208
 Project: Ignite
  Issue Type: Task
Reporter: Igor Seliverstov


Such rows are invisible for all readers (double-changed in scope of one tx).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8207) SQL TX: Cleanup own previous changes on update

[Igor Seliverstov (JIRA)]

Igor Seliverstov created IGNITE-8207:


 Summary: SQL TX: Cleanup own previous changes on update
 Key: IGNITE-8207
 URL: https://issues.apache.org/jira/browse/IGNITE-8207
 Project: Ignite
  Issue Type: Task
Reporter: Igor Seliverstov


During update process we're cleaning up rows, which became invisible for all 
readers.

Updating a row more than once we are free to cleanup previous own changes 
except last one. This means we shouldn't leave more than two rows per key with 
version equal to current tx for updates and one for delete operations.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8206) SQL TX: Rewrite MvccCursor

[Igor Seliverstov (JIRA)]

Igor Seliverstov created IGNITE-8206:


 Summary: SQL TX: Rewrite MvccCursor
 Key: IGNITE-8206
 URL: https://issues.apache.org/jira/browse/IGNITE-8206
 Project: Ignite
  Issue Type: Task
Reporter: Igor Seliverstov


Currently we materialize rows before passing Mvcc filter. It means we 
deserialize all rows, including invisible for snapshot.

We need to pass the filter to BPlusTree.find() method instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8205) Services are not redeployed when cluster is getting activated

[Denis Mekhanikov (JIRA)]

Denis Mekhanikov created IGNITE-8205:


 Summary: Services are not redeployed when cluster is getting 
activated
 Key: IGNITE-8205
 URL: https://issues.apache.org/jira/browse/IGNITE-8205
 Project: Ignite
  Issue Type: Improvement
Reporter: Denis Mekhanikov
 Fix For: 2.5
 Attachments: ServiceDeploymentOnActivationTest.java

Steps to reproduce:
 # Start a node
 # Activate the cluster
 # Deploy a service
 # Deactivate the cluster
 # Activate the cluster

Service is getting cancelled after the step 4, but not coming back after the 
step 5.

Test class with a reproducer is attached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3788: IGNITE-7824

[NSAmelchev]

GitHub user NSAmelchev reopened a pull request:

https://github.com/apache/ignite/pull/3788

IGNITE-7824

Fixed log message.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NSAmelchev/ignite ignite-7170

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3788.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3788


commit 11424e9eeefc8fe1b8575623c18a8d565a376ab9
Author: NSAmelchev 
Date:   2017-11-28T14:25:13Z

Merge remote-tracking branch 'refs/remotes/apache/master'

commit 0b16700731bda414b8d7921f2c098c5ad1b6540b
Author: NSAmelchev 
Date:   2018-01-19T09:46:31Z

Merge remote-tracking branch 'refs/remotes/apache/master'

commit 0de054fe0a1ae41db58a67d3387d08deaf6d22e2
Author: NSAmelchev 
Date:   2018-02-27T11:00:41Z

Merge remote-tracking branch 'apache/master'

commit 29cb0f44d25621d1e41e00d504c3e7bf44c1d735
Author: NSAmelchev 
Date:   2018-03-15T10:58:37Z

Merge pull request #20 from apache/master

merge

commit 9b0f16930a5e1da4ef3a528b7879cd1fca5307f7
Author: NSAmelchev 
Date:   2018-03-20T08:17:26Z

Merge pull request #21 from apache/master

Merge

commit 07dc37c806dcc9a9527f4b85c116d1bb6543effc
Author: NSAmelchev 
Date:   2018-04-10T13:33:27Z

fix log message




---

[GitHub] ignite pull request #3788: IGNITE-7170

[NSAmelchev]

Github user NSAmelchev closed the pull request at:

https://github.com/apache/ignite/pull/3788


---

[GitHub] ignite pull request #3788: IGNITE-7170

[NSAmelchev]

GitHub user NSAmelchev opened a pull request:

https://github.com/apache/ignite/pull/3788

IGNITE-7170

Fixed log message.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/NSAmelchev/ignite ignite-7170

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3788.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3788


commit 11424e9eeefc8fe1b8575623c18a8d565a376ab9
Author: NSAmelchev 
Date:   2017-11-28T14:25:13Z

Merge remote-tracking branch 'refs/remotes/apache/master'

commit 0b16700731bda414b8d7921f2c098c5ad1b6540b
Author: NSAmelchev 
Date:   2018-01-19T09:46:31Z

Merge remote-tracking branch 'refs/remotes/apache/master'

commit 0de054fe0a1ae41db58a67d3387d08deaf6d22e2
Author: NSAmelchev 
Date:   2018-02-27T11:00:41Z

Merge remote-tracking branch 'apache/master'

commit 29cb0f44d25621d1e41e00d504c3e7bf44c1d735
Author: NSAmelchev 
Date:   2018-03-15T10:58:37Z

Merge pull request #20 from apache/master

merge

commit 9b0f16930a5e1da4ef3a528b7879cd1fca5307f7
Author: NSAmelchev 
Date:   2018-03-20T08:17:26Z

Merge pull request #21 from apache/master

Merge

commit 07dc37c806dcc9a9527f4b85c116d1bb6543effc
Author: NSAmelchev 
Date:   2018-04-10T13:33:27Z

fix log message




---

Re: affinityRun/Call in C++

[Vladimir Ozerov]

Val,

They are simply not implemented yet. I am not aware of concrete plans to
support them.

On Mon, Apr 9, 2018 at 11:33 PM, Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Guys,
>
> Is there a way to run collocated compute job in C++? I can't find
> affinityRun and affinityCall method in C++ compute API, am I missing
> something? If we really don't have them, is there any particular reason for
> this and/or plans to add them?
>
> -Val
>

[GitHub] ignite pull request #3786: Ignite 2.4.4.b2

[AMashenkov]

GitHub user AMashenkov opened a pull request:

https://github.com/apache/ignite/pull/3786

Ignite 2.4.4.b2

For test purposes

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2.4.4.b2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3786.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3786


commit 915dd2966084d78f7b4f3d482e6bd25f860c1e23
Author: Alexey Goncharuk 
Date:   2018-01-31T08:22:26Z

IGNITE-7569 Fixed index rebuild future - Fixes #3454.

commit 8ea8609259039852ab0c26f26ac528c1ffae7c94
Author: Alexey Goncharuk 
Date:   2018-01-31T08:24:57Z

IGNITE-7577 Fixing public API active flag on baseline changes - Fixes #3455.

commit c8ce1f66e98b3174d771a3b801a2538499dc2c3d
Author: Ivan Rakov 
Date:   2018-01-31T09:51:09Z

IGNITE-7475 Improved VerifyBackupPartitionsTask to calculate partition 
hashes in parallel - Fixes #3407.

Signed-off-by: Alexey Goncharuk 

commit 258ff4299da20122d7c387cb8579264035c93c18
Author: Alexey Goncharuk 
Date:   2018-01-31T13:52:24Z

IGNITE-7573 Fixed full API tests to be compliant with baseline topology

commit 254ed3a9c32d092702a0461509bf867cbd7cdee6
Author: Alexey Kuznetsov 
Date:   2018-02-01T08:22:53Z

ignite-2.4.0 Update version.

(cherry picked from commit 2e43749)

commit c1a9c0a404d77fba08170bedf14844f87abe3028
Author: Alexey Goncharuk 
Date:   2018-02-01T10:17:28Z

IGNITE-7569 Fixing index rebuild future

commit e43799ce70cdbe03d9e206381d1d5138b820b075
Author: Alexey Goncharuk 
Date:   2018-02-01T13:39:17Z

IGNITE-7520 Provide util-methods to get baseline from context - Fixes #3431.

commit 8f5fc7cfb0624cf2048efad38dfff32f782116e8
Author: Sergey Chugunov 
Date:   2018-02-02T08:24:29Z

IGNITE-7580 Fix compatibilityMode flag consistency

This closes #3466

(cherry picked from commit 8f2045e)

commit d3ddd50cb2b889173176b6c47c9ff61410e1d909
Author: Ilya Lantukh 
Date:   2018-02-07T10:33:28Z

IGNITE-7514 Affinity assignment should be recalculated when primary node is 
not OWNER

(cherry picked from commit faf50f1)

commit d3745e9d0a3ff5a64fba494889b7e2605f3af6bb
Author: Alexey Goncharuk 
Date:   2018-02-07T18:10:32Z

IGNITE-7639 Fixed NPE

commit f7c16855ba802d9d47048521aec7e14285e4a281
Author: Pavel Kovalenko 
Date:   2018-02-09T13:55:15Z

IGNITE-7540 Prevent page memory metadata corruption during checkpoint and 
group destroying. - Fixes #3490.

Signed-off-by: Alexey Goncharuk 

commit c92f167fc491078f02b9f94fe89edafc2902ebc2
Author: ilantukh 
Date:   2018-02-14T12:40:13Z

Updated version in properties.

commit 1ecf348dd429cf7861b414e0e5a7776b72dba281
Author: Sergey Chugunov 
Date:   2018-02-16T13:21:12Z

IGNITE-7699 BinaryMetadata exchange should not be triggered if metadata was 
not updated - Fixes #3523.

Signed-off-by: Alexey Goncharuk 

(cherry-picked from commit bcd3881)

commit 2458bd08a5b501b3eeb5caf0ae6dcaa2bcccd915
Author: EdShangGG 
Date:   2018-02-16T13:29:49Z

IGNITE-7676 Add affinity version to snapshot plugin stub - Fixes #3510.

Signed-off-by: Alexey Goncharuk 
(cherry picked from commit b6d21fb)

commit bfdcda7a2a6b5cf64f15ed169d2beb886f131fac
Author: EdShangGG 
Date:   2018-02-12T16:36:30Z

IGNITE-7626 Unify code in test which cleans up persistence directories - 
Fixes #3477.

Signed-off-by: Alexey Goncharuk 
(cherry picked from commit a0997b9)

commit 2e92e0094b270aa8489e66d94bfcf15eadabfb4f
Author: EdShangGG 
Date:   2018-02-12T18:44:10Z

IGNITE-7626 Unify code in test which clean up persistence directories - 
Fixes #3512.

Signed-off-by: Alexey Goncharuk 
(cherry picked from commit 6f6f8dd)

commit 3f86c127c78065999663a4fc4eaedb5e5d4bee1c
Author: EdShangGG 
Date:   2018-02-12T18:26:31Z

compilation fix

commit 0b9322c566f9b464291854142ac02495bd1817e4
Author: gg-shq 
Date:   2018-02-07T11:28:04Z

IGNITE-6917: Implemented SQL COPY command.

commit c5e386ca96750213bddcd98d0af0c589fee476ca
Author: gg-shq 
Date:   2018-02-07T15:31:27Z

IGNITE-7586: Added COPY command into the JDBC example.

This closes #3485

commit d8203e2d81f8fbf0f7fbe5e710c9908f2fcb8307
Author: shq 
Date:   

Re: IGNITE-6252

[Dmitry Pavlov]

Hi,

It seems your question is related to code of Cassandra cache store.
According to git, author of this code is Igor Rudyak, and it seems Igor is
also watcher of ticket, where you were asking.

Igor,

could you please advice?

Sincerely,
Dmitriy Pavlov


вт, 10 апр. 2018 г. в 10:49, kotamrajuyashasvi :

> Hi
>
> Can some one reply to my comments in the following ticket or redirect to
> someone who can look into
> https://issues.apache.org/jira/browse/IGNITE-6252
>
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

[GitHub] ignite pull request #3785: IGNITE-8204

[alexpaschenko]

GitHub user alexpaschenko opened a pull request:

https://github.com/apache/ignite/pull/3785

IGNITE-8204

(cherry picked from commit cabdc52)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8204

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3785.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3785


commit 81fd25298f86d1f3f6d8576fe97511d9ce950210
Author: Alexander Paschenko 
Date:   2018-04-09T18:05:15Z

IGNITE-7712 testRestarts fix

(cherry picked from commit cabdc52)




---

[jira] [Created] (IGNITE-8204) Tests hang with lazy query flag on

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-8204:
---

 Summary: Tests hang with lazy query flag on
 Key: IGNITE-8204
 URL: https://issues.apache.org/jira/browse/IGNITE-8204
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.5


Affected tests:

{{IgniteCacheClientQueryReplicatedNodeRestartSelfTest.testRestart}}

{{IgniteCacheDistributedPartitionQueryNodeRestartsSelfTest.testJoinQueryUnstableTopology}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Remove cache groups in AI 3.0

[Dmitry Pavlov]

Hi Vladimir,

We can solve "too many fsyncs" or 'too many small files' by placing several
partitions of cache group in one file.

We don't need to get rid from cache groups in this case.

It is not trivial task, but it is doable. We need to create simplest FS for
paritition chunks inside one file.

Sincerely,
Dmitriy Pavlov

вт, 10 апр. 2018 г. в 12:31, Vladimir Ozerov :

> Dima,
>
> 1) Easy to understand for users
> AI 2.x: cluster -> cache group -> cache -> table
> AI 3.x: cluster -> cache(==table)
>
> 2) Fine grained cache management
> - MVCC on/off per-cache
> - WAL mode on/off per-cache
> - Data size per-cache
>
> 3) Performance:
> - Efficient scans are not possible with cache groups
> - Efficient destroy/DROP - O(N) now, O(1) afterwards
>
> "Huge refactoring" is not precise estimate. Let's think on how to do that
> instead of how not to do :-)
>
> On Tue, Apr 10, 2018 at 11:41 AM, Dmitriy Setrakyan  >
> wrote:
>
> > Vladimir, sounds like a huge refactoring. Other than "cache groups are
> > confusing", are we solving any other big issues with the new proposed
> > approach?
> >
> > (every time we try to refactor rebalancing, I get goose bumps)
> >
> > D.
> >
> > On Tue, Apr 10, 2018 at 1:32 AM, Vladimir Ozerov 
> > wrote:
> >
> > > Igniters,
> > >
> > > Cache groups were implemented for a sole purpose - to hide internal
> > > inefficiencies. Namely (add more if I missed something):
> > > 1) Excessive heap usage for affinity/partition data
> > > 2) Too much data files as we employ file-per-partition approach.
> > >
> > > These problems were resolved, but now cache groups are a great source
> of
> > > confusion both for users and us - hard to understand, no way to
> configure
> > > it in deterministic way. Should we resolve mentioned performance issues
> > we
> > > would never had cache groups. I propose to think we would it take for
> us
> > to
> > > get rid of cache groups.
> > >
> > > Please provide your inputs to suggestions below.
> > >
> > > 1) "Merge" partition data from different caches
> > > Consider that we start a new cache with the same affinity configuration
> > > (cache mode, partition number, affinity function) as some of already
> > > existing caches, Is it possible to re-use partition distribution and
> > > history of existing cache for a new cache? Think of it as a kind of
> > > automatic cache grouping which is transparent to the user. This would
> > > remove heap pressure. Also it could resolve our long-standing issue
> with
> > > FairAffinityFunction when tow caches with the same affinity
> configuration
> > > are not co-located when started on different topology versions.
> > >
> > > 2) Employ segment-extent based approach instead of file-per-partition
> > > - Every object (cache, index) reside in dedicated segment
> > > - Segment consists of extents (minimal allocation units)
> > > - Extents are allocated and deallocated as needed
> > > - *Ignite specific*: particular extent can be used by only one
> partition
> > > - Segments may be located in any number of data files we find
> convenient
> > > With this approach "too many fsyncs" problem goes away automatically.
> At
> > > the same time it would be possible to implement efficient rebalance
> still
> > > as partition data will be split across moderate number of extents, not
> > > chaotically.
> > >
> > > Once we have p.1 and p.2 ready cache groups could be removed, couldn't
> > > they?
> > >
> > > Vladimir.
> > >
> >
>

Re: IGNITE-6879

[Dmitry Pavlov]

Thank you, Roman.

Igniters,

IMO we should consider one more alternative - renaming of old module and
package names. Users, which prefer to stay on previous version will be
requiered to update their pom's. In the same time users which are ready to
migrate to spring data 2.0 will need to update methods naming.

Denis M, what would you say?

Sincerely,
Dmitriy Pavlov

вт, 10 апр. 2018 г. в 11:27, Роман Меерсон :

> Hi Dmitry!
>
> I`ve just commited new fix. I renamed package of new module to
> springdata20, it helps us to separate old implementation from new and also
> should fix all compilation errors.
>
> пн, 9 апр. 2018 г. в 23:54, Роман Меерсон :
>
>> Ok, I'll check it, but I haven't face this problem.
>> If I'll find same issue, what is the proper way? Renaming to something
>> like Ignite2QueryGenerator or module removing?
>> пн, 9 апр. 2018 г. в 23:40, Dmitry Pavlov :
>>
>>> There are 2 classes IgniteQueryGenerator with same package name. Ignite
>>> in Idea can't compile.
>>>
>>>
>>> пн, 9 апр. 2018 г., 21:38 Роман Меерсон :
>>>
 Hi Dmitry!
>>>
>>>
 Could you specify where you find conflict? Because I don’t have any.
 пн, 9 апр. 2018 г. в 21:09, Dmitry Pavlov :

> Hi Denis,
>
> could we support just one version instead of leaving compatible module?
>
> Sincerely,
> Dmitriy Pavlov
>
> пн, 9 апр. 2018 г. в 20:08, Dmitry Pavlov :
>
>>
>>
>> пн, 9 апр. 2018 г. в 20:07, Dmitry Pavlov :
>>
>>> Hi Roman,
>>>
>>> I've applied PR locally and I have class name conflict at least for
>>> org.apache.ignite.springdata.repository.query.IgniteQueryGenerator
>>>
>>> How could we solve it? Is it better to rename class for new plugin
>>> version?
>>>
>>> Sincerely,
>>> Dmitriy Pavlov
>>>
>>> пт, 6 апр. 2018 г. в 17:38, Dmitry Pavlov :
>>>
 Excellend picture. I remember about this change.

 If Denis M. would be able to look througt the changes faster than
 me, I can merge without detailed review.

 пт, 6 апр. 2018 г. в 16:15, Роман Меерсон :

> OK
>
> [image: 1486924635147168240.jpg]
>
>
> пт, 6 апр. 2018 г. в 17:08, Igor Sapego :
>
>> Hi,
>> Well, Dmitry has said he's going to merge it in 3-4 days 2 days
>> ago,
>> so I guess, the merge is going to happen in 1-2 days or so.
>>
>>
>> Best Regards,
>> Igor
>>
>> On Fri, Apr 6, 2018 at 3:48 PM, Роман Меерсон <
>> homich1...@gmail.com> wrote:
>>
>> > Hi all!
>> >
>> > As i see everything is awesome and there is no objections, so
>> when my PR
>> > would be merged?
>> >
>> > чт, 5 апр. 2018 г. в 18:58, Вячеслав Коптилин <
>> slava.kopti...@gmail.com>:
>> >
>> > > Thank you, Roman!
>> > >
>> > > 2018-04-05 17:49 GMT+03:00 Роман Меерсон <
>> homich1...@gmail.com>:
>> > >
>> > > > Hi Slava,
>> > > >
>> > > > Fixed
>> > > >
>> > > > чт, 5 апр. 2018 г. в 18:41, Вячеслав Коптилин <
>> > slava.kopti...@gmail.com
>> > > >:
>> > > >
>> > > > > Hi Roman,
>> > > > >
>> > > > > please take into account my comment
>> IgniteQueryGenerator.java
>> > > > > <
>> > > > >
>> https://reviews.ignite.apache.org/ignite/review/IGNT-CR-541?
>> > > > commentId=de43c65f-9ac7-4080-9904-aec119138c94=/
>> > > > modules/spring-data-2.0/src/main/java/org/apache/ignite/
>> > > > springdata/repository/query/IgniteQueryGenerator.java
>> > > > > >
>> > > > >
>> > > > > Best regards,
>> > > > > Slava.
>> > > > >
>> > > > > 2018-04-05 14:59 GMT+03:00 Роман Меерсон <
>> homich1...@gmail.com>:
>> > > > >
>> > > > > > Ok, so waiting for accept and commit
>> > > > > >
>> > > > > > чт, 5 апр. 2018 г. в 15:29, Alexey Kukushkin <
>> > > > kukushkinale...@gmail.com
>> > > > > >:
>> > > > > >
>> > > > > > > Roman,
>> > > > > > >
>> > > > > > > Just pay commiter's (Dmitry Pavlov will most likely
>> commit your
>> > > code)
>> > > > > > > attention to include the new test suite to TeamCity
>> > configuration.
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

[jira] [Created] (IGNITE-8203) Interrupting task can cause node fail with PersistenceStorageIOException.

[Ivan Daschinskiy (JIRA)]

Ivan Daschinskiy created IGNITE-8203:


 Summary: Interrupting task can cause node fail with 
PersistenceStorageIOException. 
 Key: IGNITE-8203
 URL: https://issues.apache.org/jira/browse/IGNITE-8203
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Affects Versions: 2.4
Reporter: Ivan Daschinskiy
 Fix For: 2.6
 Attachments: GridFailNodesOnCanceledTaskTest.java

Interrupting task with simple cache operations (i.e. get, put) can cause 
PersistenceStorageIOException. Main cause of this failure is lack of proper 
handling InterruptedException in FilePageStore.init() etc. This cause a throw 
ClosedByInterruptException by FileChannel.write() and so on. 

PersistenceStorageIOException is a critical failure and typically makes a node 
to stop.

A reproducer is attached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3760: IGNITE-8059 Integrate decision tree with partitio...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3760


---

Re: Remove cache groups in AI 3.0

[Vladimir Ozerov]

Dima,

1) Easy to understand for users
AI 2.x: cluster -> cache group -> cache -> table
AI 3.x: cluster -> cache(==table)

2) Fine grained cache management
- MVCC on/off per-cache
- WAL mode on/off per-cache
- Data size per-cache

3) Performance:
- Efficient scans are not possible with cache groups
- Efficient destroy/DROP - O(N) now, O(1) afterwards

"Huge refactoring" is not precise estimate. Let's think on how to do that
instead of how not to do :-)

On Tue, Apr 10, 2018 at 11:41 AM, Dmitriy Setrakyan 
wrote:

> Vladimir, sounds like a huge refactoring. Other than "cache groups are
> confusing", are we solving any other big issues with the new proposed
> approach?
>
> (every time we try to refactor rebalancing, I get goose bumps)
>
> D.
>
> On Tue, Apr 10, 2018 at 1:32 AM, Vladimir Ozerov 
> wrote:
>
> > Igniters,
> >
> > Cache groups were implemented for a sole purpose - to hide internal
> > inefficiencies. Namely (add more if I missed something):
> > 1) Excessive heap usage for affinity/partition data
> > 2) Too much data files as we employ file-per-partition approach.
> >
> > These problems were resolved, but now cache groups are a great source of
> > confusion both for users and us - hard to understand, no way to configure
> > it in deterministic way. Should we resolve mentioned performance issues
> we
> > would never had cache groups. I propose to think we would it take for us
> to
> > get rid of cache groups.
> >
> > Please provide your inputs to suggestions below.
> >
> > 1) "Merge" partition data from different caches
> > Consider that we start a new cache with the same affinity configuration
> > (cache mode, partition number, affinity function) as some of already
> > existing caches, Is it possible to re-use partition distribution and
> > history of existing cache for a new cache? Think of it as a kind of
> > automatic cache grouping which is transparent to the user. This would
> > remove heap pressure. Also it could resolve our long-standing issue with
> > FairAffinityFunction when tow caches with the same affinity configuration
> > are not co-located when started on different topology versions.
> >
> > 2) Employ segment-extent based approach instead of file-per-partition
> > - Every object (cache, index) reside in dedicated segment
> > - Segment consists of extents (minimal allocation units)
> > - Extents are allocated and deallocated as needed
> > - *Ignite specific*: particular extent can be used by only one partition
> > - Segments may be located in any number of data files we find convenient
> > With this approach "too many fsyncs" problem goes away automatically. At
> > the same time it would be possible to implement efficient rebalance still
> > as partition data will be split across moderate number of extents, not
> > chaotically.
> >
> > Once we have p.1 and p.2 ready cache groups could be removed, couldn't
> > they?
> >
> > Vladimir.
> >
>

Re: REST API and new authentication API

[Alexey Kuznetsov]

Dmitriy,

Yes, sound reasonable to add "authenticate" command and require token for
all subsequent commands.

Will update issue description.

On Tue, Apr 10, 2018 at 2:43 PM, Dmitriy Setrakyan 
wrote:

> On Tue, Apr 10, 2018 at 12:28 AM, Alexey Kuznetsov 
> wrote:
>
> > Dmitriy,
> >
> > Yes, because we have a command "Add new user" and this command can be
> > executed only with credentials of some "admin" user.
> >
> > It means, that in one command you need to specify name of new user and
> > "admin" credentials at the same time.?
>
>
> > If you have any ideas how we can handle this - I will be glad to discuss
> > it.
> >
>
> I am not sure if I agree with the approach you have suggested. In my view,
> we should have "authenticate" command, which should ask for the username
> and password. Once the user is authenticated and logged in, you should use
> the session token to perform all other commands. We should NOT be
> authenticating users on every command.
>
> If you follow this approach, then the command for adding a new user should
> require any authentication.
>
> Makes sense?
>
> D.
>



-- 
Alexey Kuznetsov

Re: Remove cache groups in AI 3.0

[Dmitriy Setrakyan]

Vladimir, sounds like a huge refactoring. Other than "cache groups are
confusing", are we solving any other big issues with the new proposed
approach?

(every time we try to refactor rebalancing, I get goose bumps)

D.

On Tue, Apr 10, 2018 at 1:32 AM, Vladimir Ozerov 
wrote:

> Igniters,
>
> Cache groups were implemented for a sole purpose - to hide internal
> inefficiencies. Namely (add more if I missed something):
> 1) Excessive heap usage for affinity/partition data
> 2) Too much data files as we employ file-per-partition approach.
>
> These problems were resolved, but now cache groups are a great source of
> confusion both for users and us - hard to understand, no way to configure
> it in deterministic way. Should we resolve mentioned performance issues we
> would never had cache groups. I propose to think we would it take for us to
> get rid of cache groups.
>
> Please provide your inputs to suggestions below.
>
> 1) "Merge" partition data from different caches
> Consider that we start a new cache with the same affinity configuration
> (cache mode, partition number, affinity function) as some of already
> existing caches, Is it possible to re-use partition distribution and
> history of existing cache for a new cache? Think of it as a kind of
> automatic cache grouping which is transparent to the user. This would
> remove heap pressure. Also it could resolve our long-standing issue with
> FairAffinityFunction when tow caches with the same affinity configuration
> are not co-located when started on different topology versions.
>
> 2) Employ segment-extent based approach instead of file-per-partition
> - Every object (cache, index) reside in dedicated segment
> - Segment consists of extents (minimal allocation units)
> - Extents are allocated and deallocated as needed
> - *Ignite specific*: particular extent can be used by only one partition
> - Segments may be located in any number of data files we find convenient
> With this approach "too many fsyncs" problem goes away automatically. At
> the same time it would be possible to implement efficient rebalance still
> as partition data will be split across moderate number of extents, not
> chaotically.
>
> Once we have p.1 and p.2 ready cache groups could be removed, couldn't
> they?
>
> Vladimir.
>

[GitHub] ignite pull request #3784: ignite-8058 fix flaky GridMarshallerMappingConsis...

[dmekhanikov]

GitHub user dmekhanikov opened a pull request:

https://github.com/apache/ignite/pull/3784

ignite-8058 fix flaky GridMarshallerMappingConsistencyTest



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8058

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3784.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3784


commit 1094389b0c072bee195326c449c21571976fc3a7
Author: Denis Mekhanikov 
Date:   2018-04-10T08:36:01Z

ignite-8058 fix flaky GridMarshallerMappingConsistencyTest




---

Remove cache groups in AI 3.0

[Vladimir Ozerov]

Igniters,

Cache groups were implemented for a sole purpose - to hide internal
inefficiencies. Namely (add more if I missed something):
1) Excessive heap usage for affinity/partition data
2) Too much data files as we employ file-per-partition approach.

These problems were resolved, but now cache groups are a great source of
confusion both for users and us - hard to understand, no way to configure
it in deterministic way. Should we resolve mentioned performance issues we
would never had cache groups. I propose to think we would it take for us to
get rid of cache groups.

Please provide your inputs to suggestions below.

1) "Merge" partition data from different caches
Consider that we start a new cache with the same affinity configuration
(cache mode, partition number, affinity function) as some of already
existing caches, Is it possible to re-use partition distribution and
history of existing cache for a new cache? Think of it as a kind of
automatic cache grouping which is transparent to the user. This would
remove heap pressure. Also it could resolve our long-standing issue with
FairAffinityFunction when tow caches with the same affinity configuration
are not co-located when started on different topology versions.

2) Employ segment-extent based approach instead of file-per-partition
- Every object (cache, index) reside in dedicated segment
- Segment consists of extents (minimal allocation units)
- Extents are allocated and deallocated as needed
- *Ignite specific*: particular extent can be used by only one partition
- Segments may be located in any number of data files we find convenient
With this approach "too many fsyncs" problem goes away automatically. At
the same time it would be possible to implement efficient rebalance still
as partition data will be split across moderate number of extents, not
chaotically.

Once we have p.1 and p.2 ready cache groups could be removed, couldn't they?

Vladimir.

Re: IGNITE-6879

[Роман Меерсон]

Hi Dmitry!

I`ve just commited new fix. I renamed package of new module to
springdata20, it helps us to separate old implementation from new and also
should fix all compilation errors.

пн, 9 апр. 2018 г. в 23:54, Роман Меерсон :

> Ok, I'll check it, but I haven't face this problem.
> If I'll find same issue, what is the proper way? Renaming to something
> like Ignite2QueryGenerator or module removing?
> пн, 9 апр. 2018 г. в 23:40, Dmitry Pavlov :
>
>> There are 2 classes IgniteQueryGenerator with same package name. Ignite
>> in Idea can't compile.
>>
>>
>> пн, 9 апр. 2018 г., 21:38 Роман Меерсон :
>>
>>> Hi Dmitry!
>>
>>
>>> Could you specify where you find conflict? Because I don’t have any.
>>> пн, 9 апр. 2018 г. в 21:09, Dmitry Pavlov :
>>>
 Hi Denis,

 could we support just one version instead of leaving compatible module?

 Sincerely,
 Dmitriy Pavlov

 пн, 9 апр. 2018 г. в 20:08, Dmitry Pavlov :

>
>
> пн, 9 апр. 2018 г. в 20:07, Dmitry Pavlov :
>
>> Hi Roman,
>>
>> I've applied PR locally and I have class name conflict at least for
>> org.apache.ignite.springdata.repository.query.IgniteQueryGenerator
>>
>> How could we solve it? Is it better to rename class for new plugin
>> version?
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>> пт, 6 апр. 2018 г. в 17:38, Dmitry Pavlov :
>>
>>> Excellend picture. I remember about this change.
>>>
>>> If Denis M. would be able to look througt the changes faster than
>>> me, I can merge without detailed review.
>>>
>>> пт, 6 апр. 2018 г. в 16:15, Роман Меерсон :
>>>
 OK

 [image: 1486924635147168240.jpg]


 пт, 6 апр. 2018 г. в 17:08, Igor Sapego :

> Hi,
> Well, Dmitry has said he's going to merge it in 3-4 days 2 days
> ago,
> so I guess, the merge is going to happen in 1-2 days or so.
>
>
> Best Regards,
> Igor
>
> On Fri, Apr 6, 2018 at 3:48 PM, Роман Меерсон <
> homich1...@gmail.com> wrote:
>
> > Hi all!
> >
> > As i see everything is awesome and there is no objections, so
> when my PR
> > would be merged?
> >
> > чт, 5 апр. 2018 г. в 18:58, Вячеслав Коптилин <
> slava.kopti...@gmail.com>:
> >
> > > Thank you, Roman!
> > >
> > > 2018-04-05 17:49 GMT+03:00 Роман Меерсон  >:
> > >
> > > > Hi Slava,
> > > >
> > > > Fixed
> > > >
> > > > чт, 5 апр. 2018 г. в 18:41, Вячеслав Коптилин <
> > slava.kopti...@gmail.com
> > > >:
> > > >
> > > > > Hi Roman,
> > > > >
> > > > > please take into account my comment
> IgniteQueryGenerator.java
> > > > > <
> > > > >
> https://reviews.ignite.apache.org/ignite/review/IGNT-CR-541?
> > > > commentId=de43c65f-9ac7-4080-9904-aec119138c94=/
> > > > modules/spring-data-2.0/src/main/java/org/apache/ignite/
> > > > springdata/repository/query/IgniteQueryGenerator.java
> > > > > >
> > > > >
> > > > > Best regards,
> > > > > Slava.
> > > > >
> > > > > 2018-04-05 14:59 GMT+03:00 Роман Меерсон <
> homich1...@gmail.com>:
> > > > >
> > > > > > Ok, so waiting for accept and commit
> > > > > >
> > > > > > чт, 5 апр. 2018 г. в 15:29, Alexey Kukushkin <
> > > > kukushkinale...@gmail.com
> > > > > >:
> > > > > >
> > > > > > > Roman,
> > > > > > >
> > > > > > > Just pay commiter's (Dmitry Pavlov will most likely
> commit your
> > > code)
> > > > > > > attention to include the new test suite to TeamCity
> > configuration.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: REST API and new authentication API

[Sergey Kozlov]

Hi

I a bit investigated the issue for REST authentication and found following
approaches:

1. Add authenticate command providing sessions token by login and password.
Any further requests will require that token.
Advantages:
 - Small changes for REST requests (just add token parameter)
Disadvantages:
 - New command for authentication
 - We need to store user sessions on the server side and manage them
(delete) if token life time reached.

2. Use HMAC (hash-based message authentication code) [1]. All requests
require to provide "sign" parameter generated by as has256 for parameters
string + secret key
Advantages:
 - No new command for authentication
Disadvantages:
 - we need to generate access + secret keys on the server side together
with username and password (two additional fields for user record).
 - logic to generate sign parameter on client side

1.
https://eclipsesource.com/blogs/2016/07/06/keyed-hash-message-authentication-code-in-rest-apis/


On Tue, Apr 10, 2018 at 10:43 AM, Dmitriy Setrakyan 
wrote:

> On Tue, Apr 10, 2018 at 12:28 AM, Alexey Kuznetsov 
> wrote:
>
> > Dmitriy,
> >
> > Yes, because we have a command "Add new user" and this command can be
> > executed only with credentials of some "admin" user.
> >
> > It means, that in one command you need to specify name of new user and
> > "admin" credentials at the same time.?
>
>
> > If you have any ideas how we can handle this - I will be glad to discuss
> > it.
> >
>
> I am not sure if I agree with the approach you have suggested. In my view,
> we should have "authenticate" command, which should ask for the username
> and password. Once the user is authenticated and logged in, you should use
> the session token to perform all other commands. We should NOT be
> authenticating users on every command.
>
> If you follow this approach, then the command for adding a new user should
> require any authentication.
>
> Makes sense?
>
> D.
>



-- 
Sergey Kozlov
GridGain Systems
www.gridgain.com

IGNITE-6252

[kotamrajuyashasvi]

Hi

Can some one reply to my comments in the following ticket or redirect to
someone who can look into
https://issues.apache.org/jira/browse/IGNITE-6252





--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: REST API and new authentication API

[Dmitriy Setrakyan]

On Tue, Apr 10, 2018 at 12:28 AM, Alexey Kuznetsov 
wrote:

> Dmitriy,
>
> Yes, because we have a command "Add new user" and this command can be
> executed only with credentials of some "admin" user.
>
> It means, that in one command you need to specify name of new user and
> "admin" credentials at the same time.?


> If you have any ideas how we can handle this - I will be glad to discuss
> it.
>

I am not sure if I agree with the approach you have suggested. In my view,
we should have "authenticate" command, which should ask for the username
and password. Once the user is authenticated and logged in, you should use
the session token to perform all other commands. We should NOT be
authenticating users on every command.

If you follow this approach, then the command for adding a new user should
require any authentication.

Makes sense?

D.

Re: REST API and new authentication API

[Alexey Kuznetsov]

Dmitriy,

Yes, because we have a command "Add new user" and this command can be
executed only with credentials of some "admin" user.

It means, that in one command you need to specify name of new user and
"admin" credentials at the same time.

If you have any ideas how we can handle this - I will be glad to discuss it.


On Tue, Apr 10, 2018 at 2:05 PM, Dmitriy Setrakyan 
wrote:

> Alexey, are you suggesting that we have "newUser" as command parameter,
> while "user" is also a valid command parameter?
>
> ⁣D.​
>
> On Apr 10, 2018, 12:00 AM, at 12:00 AM, Alexey Kuznetsov <
> akuznet...@apache.org> wrote:
> >I looked into code and I think that we could do the following:
> >
> >1) Use user and password for authentication.
> >2) Use newUser and newPassword for new authentication API (add, remove
> >and
> >update user).
> >3) Debug why sessionToken is null.
> >
> >Created issues:
> >https://issues.apache.org/jira/browse/IGNITE-8201
> >https://issues.apache.org/jira/browse/IGNITE-8202
> >
> >I will try to implement them in a couple of days.
> >
> >
> >On Tue, Apr 10, 2018 at 11:32 AM, Alexey Kuznetsov
> >
> >wrote:
> >
> >> Igniters,
> >>
> >> Recently new authentication API was added.
> >>
> >> I added support for it on REST, but several problems appeared:
> >>
> >> 1) "=login" and "=pwd" should be used
> >for
> >> credentials.
> >> May be we should use "=user" and " =pwd " instead?
> >> But this will lead to conflict with new authentication API:
> >>  For example add user:  http://localhost/ignite?cmd=*adduser*&*user*=
> >> sample&*password*=sample=ignite=ignite
> >>
> >> Any ideas how to resolve this?
> >>
> >> 2) For some reason when new authentication is enabled session toked
> >> returned as null
> >>
> >{"successStatus":0,*"sessionToken":null*,"error":null,"response":"2.5.0"}
> >>
> >> Session token can be used instead of adding user name and password to
> >> every request.
> >>
> >> Who can help with resolving this issue?
> >>
> >> --
> >> Alexey Kuznetsov
> >>
> >
> >
> >
> >--
> >Alexey Kuznetsov
>



-- 
Alexey Kuznetsov

Re: REST API and new authentication API

[Dmitriy Setrakyan]

Alexey, are you suggesting that we have "newUser" as command parameter, while 
"user" is also a valid command parameter? 

⁣D.​

On Apr 10, 2018, 12:00 AM, at 12:00 AM, Alexey Kuznetsov 
 wrote:
>I looked into code and I think that we could do the following:
>
>1) Use user and password for authentication.
>2) Use newUser and newPassword for new authentication API (add, remove
>and
>update user).
>3) Debug why sessionToken is null.
>
>Created issues:
>https://issues.apache.org/jira/browse/IGNITE-8201
>https://issues.apache.org/jira/browse/IGNITE-8202
>
>I will try to implement them in a couple of days.
>
>
>On Tue, Apr 10, 2018 at 11:32 AM, Alexey Kuznetsov
>
>wrote:
>
>> Igniters,
>>
>> Recently new authentication API was added.
>>
>> I added support for it on REST, but several problems appeared:
>>
>> 1) "=login" and "=pwd" should be used
>for
>> credentials.
>> May be we should use "=user" and " =pwd " instead?
>> But this will lead to conflict with new authentication API:
>>  For example add user:  http://localhost/ignite?cmd=*adduser*&*user*=
>> sample&*password*=sample=ignite=ignite
>>
>> Any ideas how to resolve this?
>>
>> 2) For some reason when new authentication is enabled session toked
>> returned as null
>>
>{"successStatus":0,*"sessionToken":null*,"error":null,"response":"2.5.0"}
>>
>> Session token can be used instead of adding user name and password to
>> every request.
>>
>> Who can help with resolving this issue?
>>
>> --
>> Alexey Kuznetsov
>>
>
>
>
>--
>Alexey Kuznetsov

Re: REST API and new authentication API

[Alexey Kuznetsov]

I looked into code and I think that we could do the following:

1) Use user and password for authentication.
2) Use newUser and newPassword for new authentication API (add, remove and
update user).
3) Debug why sessionToken is null.

Created issues:
https://issues.apache.org/jira/browse/IGNITE-8201
https://issues.apache.org/jira/browse/IGNITE-8202

I will try to implement them in a couple of days.


On Tue, Apr 10, 2018 at 11:32 AM, Alexey Kuznetsov 
wrote:

> Igniters,
>
> Recently new authentication API was added.
>
> I added support for it on REST, but several problems appeared:
>
> 1) "=login" and "=pwd" should be used for
> credentials.
> May be we should use "=user" and " =pwd " instead?
> But this will lead to conflict with new authentication API:
>  For example add user:  http://localhost/ignite?cmd=*adduser*&*user*=
> sample&*password*=sample=ignite=ignite
>
> Any ideas how to resolve this?
>
> 2) For some reason when new authentication is enabled session toked
> returned as null
> {"successStatus":0,*"sessionToken":null*,"error":null,"response":"2.5.0"}
>
> Session token can be used instead of adding user name and password to
> every request.
>
> Who can help with resolving this issue?
>
> --
> Alexey Kuznetsov
>



-- 
Alexey Kuznetsov

[jira] [Created] (IGNITE-8202) Debug why REST return null as sessionToken when authentication enabled

[Alexey Kuznetsov (JIRA)]

Alexey Kuznetsov created IGNITE-8202:


 Summary: Debug why REST return null as sessionToken when 
authentication enabled
 Key: IGNITE-8202
 URL: https://issues.apache.org/jira/browse/IGNITE-8202
 Project: Ignite
  Issue Type: Task
  Components: rest
Reporter: Alexey Kuznetsov
Assignee: Alexey Kuznetsov
 Fix For: 2.5






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8201) Refactor REST API for authentication

[Alexey Kuznetsov (JIRA)]

Alexey Kuznetsov created IGNITE-8201:


 Summary: Refactor REST API for authentication
 Key: IGNITE-8201
 URL: https://issues.apache.org/jira/browse/IGNITE-8201
 Project: Ignite
  Issue Type: Task
  Components: rest
Reporter: Alexey Kuznetsov
Assignee: Alexey Kuznetsov


1) Use user and password for authentication.
2) Use newUser and newPassword for new authentication API (add, remove and 
update user).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)