date:20190124

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *New stable failure of a flaky test in master 
IgniteClusterActivateDeactivateTestWithPersistenceAndMemoryReuse.testReActivateSimple_5_Servers_4_Clients_FromClient
 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=20411650201522285=%3Cdefault%3E=testDetails
 No changes in the build

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 09:32:51 25-01-2019

[MTCGA]: new failures in builds [2741149] needs to be handled

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *Recently contributed test failed in master NuGet.CacheTest 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6551101028080942383=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master NuGet.ComputeTest 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-3754909490222985251=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master NuGet.AspNetTest 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-4468304894809184645=%3Cdefault%3E=testDetails
 No changes in the build

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 07:02:51 25-01-2019

[MTCGA]: new failures in builds [2896735] needs to be handled

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_9.testReadThroughSkipStore 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-3064886136657669025=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_14.testReadThroughSkipStore 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=7059411073480725538=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_21.testReadThroughSkipStore 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-6110917073341866321=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_1.testReadThroughWithExpirePolicy 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=8504677670216970350=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_3.testReadThroughSkipStore 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=7690444991650716335=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_16.testReadThroughWithExpirePolicy 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-1137019029346570375=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_5.testReadThroughEvictionPolicy 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-5007445660852409761=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_16.testReadThroughEvictionPolicy 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-8104367150476499429=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_9.testReadThroughEvictionPolicy 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6348175937647450361=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_4.testReadThroughWithExpirePolicy 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-1628740259193127234=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_27.testReadThroughSkipStore 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-2651356594821188059=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_12.testReadThroughEvictionPolicy 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=5863301210755700221=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_4.testReadThroughSkipStore 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=5618552400597115659=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_31.testReadThroughExpirePolicyConfigured 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=3340167861315355968=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_22.testReadThroughSkipStore 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7004761306689221849=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_24.testReadThroughEvictionPolicy 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-5251644764142570174=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_30.testReadThroughExpirePolicyConfigured 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-3040001973028619726=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_11.testReadThroughEvictionPolicy 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-1285392420408875781=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master 
IgniteCacheReadThroughEvictionSelfTest_27.testReadThroughEvictionPolicy

[jira] [Created] (IGNITE-11070) Update RocketMQ dependency to 4.4.0

2019-01-24 Thread Roman Shtykh (JIRA)

Roman Shtykh created IGNITE-11070:
-

 Summary: Update RocketMQ dependency to 4.4.0
 Key: IGNITE-11070
 URL: https://issues.apache.org/jira/browse/IGNITE-11070
 Project: Ignite
  Issue Type: Task
Reporter: Roman Shtykh
Assignee: Roman Shtykh






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSSION] Control.sh global rework in apache ignite 3.0

2019-01-24 Thread Sergey

Hi!
Why should we choose between interactive and command mode?
I'd suggest that new utility doesn't matter of its name control.sh or visor
should have support of both modes.

Sergey Kosarev

чт, 24 янв. 2019 г., 23:02 Denis Magda dma...@apache.org:

> Sergey,
>
> Let's break and rebuild the visor from the ground. It's fine if it becomes
> incompatible with older versions.
>
> -
> Denis
>
>
> On Thu, Jan 24, 2019 at 7:05 AM Sergey Kozlov 
> wrote:
>
> > Denis
> >
> > I'm not sure that visorcmd can be refactored without incompatible changes
> > or significant changes for behaviour:
> > 1. Visorcmd starts as demon node and joins the cluster. The modern
> > utilities don't need spring config and just connect to TCP management
> port.
> > 2. Visorcmd is mostly an interactive tool but control.sh looks like *nix
> > regular command-line program.
> > 3. Visorcmd command set (IMO) is weird and has some commands that
> sometimes
> > are not about ignite (like deploy - copy file to remote nodes) or not
> clear
> > why we need them (mlist/mget/mcompact)
> >
> > I think we should define the root purpose for new utility (or for
> > refactored visorcmd)
> > From my standpoint they are following:
> >  - cluster status commands (topology, node status, configuration, etc)
> >  - cluster management commands (control.sh baseline commands + visrcmd
> > kill/stop nodes)
> >  - cache content commands (print/clear cache content)
> >  - cache management commands (create/stop/destroy/copy/rename etc cache)
> >  - service/job management commands (start/stop/restart service/job)
> >  - tx management (tx list, kill)
> >  - data consistency commands (idle_verify, crc_check, checkpoint)
> >  - statistics commands (cluster/cache metrics)
> >
> >
> >
> >
> > On Thu, Jan 24, 2019 at 12:12 PM Sergey Antonov <
> antonovserge...@gmail.com
> > >
> > wrote:
> >
> > > Alexey, Denis I agree yours points.
> > >
> > > I think we should rewrite visor.cmd from scala to java at first. And
> then
> > > merge control.sh into visor.cmd.
> > >
> > > чт, 24 янв. 2019 г. в 11:44, Alexey Kuznetsov :
> > >
> > > > I agree with Denis,
> > > >
> > > > How about to merge control.sh into Visor.CMD ?
> > > > And rewrite Visor.CMD from Scala to Java.
> > > >
> > > > What do you think?
> > > >
> > > > On Thu, Jan 24, 2019 at 4:41 AM Denis Magda 
> wrote:
> > > >
> > > > > Why don't we go in a reverse direction - instead of creating
> multiple
> > > > > scripts for different needs we incorporate all capabilities within
> > > > > visorcmd? Visor is an app/script that can be updated to meet the
> > > > > requirements of specific tools.
> > > > >
> > > > >
> > > > > -
> > > > > Denis
> > > > >
> > > > >
> > > > > On Wed, Jan 23, 2019 at 1:23 PM Sergey Kozlov <
> skoz...@gridgain.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks Sergey for the raising the question.
> > > > > >
> > > > > > What's about move actual/interesting commands from visorcmd and
> > > remove
> > > > > that
> > > > > > script at all?
> > > > > >
> > > > > > Also would good simplify the bash code of script and ideally
> > process
> > > > > > everyting inside java code?
> > > > > >
> > > > > > On Wed, Jan 23, 2019 at 7:38 PM Stanislav Lukyanov <
> > > > > stanlukya...@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I value strict compatibility rules very highly, and would be
> > happy
> > > if
> > > > > we
> > > > > > > never removed a public class (even if it is in the “internal”
> > > > package)
> > > > > > > in a minor release.
> > > > > > > Unfortunately, Ignite is far from that place for now. We don’t
> > have
> > > > any
> > > > > > > distinction between API and internal classes, don’t have
> > > > > > > plugin-only APIs, etc. All classes are public, everything is
> > > > accessible
> > > > > > to
> > > > > > > user code. We even refer to internal classes in public Javadoc
> > > > > > > (e.g. I recall mentions of IgniteUtils in examples here and
> > there).
> > > > > > > Considering this, moving CommandHandler from ignite-core to
> > > > > > > ignite-control-utility
> > > > > > > doesn't look that bad. It doesn’t differ to much from any other
> > > > change
> > > > > > > that removes or renames a class.
> > > > > > > There could be required changes with a higher compatibility
> > impact
> > > > but
> > > > > I
> > > > > > > don’t see them after a superficial glance.
> > > > > > >
> > > > > > > Stan
> > > > > > >
> > > > > > > From: Sergey Antonov
> > > > > > > Sent: 23 января 2019 г. 19:15
> > > > > > > To: dev@ignite.apache.org
> > > > > > > Subject: Re: [DISCUSSION] Control.sh global rework in apache
> > ignite
> > > > 3.0
> > > > > > >
> > > > > > > Stan, thank you for response!
> > > > > > >
> > > > > > > I my view we shouldn't make incompatible changes and switch
> > > > extendable
> > > > > > > classes (i.e. VisorDataTransferObject ->
> > IgniteDataTransferObject)
> > > > > > between
> > > > > > > minor releases. Therefore we couldn't rework utility in 2.8
> > >

Re: [DISCUSSION] Control.sh global rework in apache ignite 3.0

2019-01-24 Thread Denis Magda

Sergey,

Let's break and rebuild the visor from the ground. It's fine if it becomes
incompatible with older versions.

-
Denis


On Thu, Jan 24, 2019 at 7:05 AM Sergey Kozlov  wrote:

> Denis
>
> I'm not sure that visorcmd can be refactored without incompatible changes
> or significant changes for behaviour:
> 1. Visorcmd starts as demon node and joins the cluster. The modern
> utilities don't need spring config and just connect to TCP management port.
> 2. Visorcmd is mostly an interactive tool but control.sh looks like *nix
> regular command-line program.
> 3. Visorcmd command set (IMO) is weird and has some commands that sometimes
> are not about ignite (like deploy - copy file to remote nodes) or not clear
> why we need them (mlist/mget/mcompact)
>
> I think we should define the root purpose for new utility (or for
> refactored visorcmd)
> From my standpoint they are following:
>  - cluster status commands (topology, node status, configuration, etc)
>  - cluster management commands (control.sh baseline commands + visrcmd
> kill/stop nodes)
>  - cache content commands (print/clear cache content)
>  - cache management commands (create/stop/destroy/copy/rename etc cache)
>  - service/job management commands (start/stop/restart service/job)
>  - tx management (tx list, kill)
>  - data consistency commands (idle_verify, crc_check, checkpoint)
>  - statistics commands (cluster/cache metrics)
>
>
>
>
> On Thu, Jan 24, 2019 at 12:12 PM Sergey Antonov  >
> wrote:
>
> > Alexey, Denis I agree yours points.
> >
> > I think we should rewrite visor.cmd from scala to java at first. And then
> > merge control.sh into visor.cmd.
> >
> > чт, 24 янв. 2019 г. в 11:44, Alexey Kuznetsov :
> >
> > > I agree with Denis,
> > >
> > > How about to merge control.sh into Visor.CMD ?
> > > And rewrite Visor.CMD from Scala to Java.
> > >
> > > What do you think?
> > >
> > > On Thu, Jan 24, 2019 at 4:41 AM Denis Magda  wrote:
> > >
> > > > Why don't we go in a reverse direction - instead of creating multiple
> > > > scripts for different needs we incorporate all capabilities within
> > > > visorcmd? Visor is an app/script that can be updated to meet the
> > > > requirements of specific tools.
> > > >
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Wed, Jan 23, 2019 at 1:23 PM Sergey Kozlov 
> > > > wrote:
> > > >
> > > > > Thanks Sergey for the raising the question.
> > > > >
> > > > > What's about move actual/interesting commands from visorcmd and
> > remove
> > > > that
> > > > > script at all?
> > > > >
> > > > > Also would good simplify the bash code of script and ideally
> process
> > > > > everyting inside java code?
> > > > >
> > > > > On Wed, Jan 23, 2019 at 7:38 PM Stanislav Lukyanov <
> > > > stanlukya...@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > I value strict compatibility rules very highly, and would be
> happy
> > if
> > > > we
> > > > > > never removed a public class (even if it is in the “internal”
> > > package)
> > > > > > in a minor release.
> > > > > > Unfortunately, Ignite is far from that place for now. We don’t
> have
> > > any
> > > > > > distinction between API and internal classes, don’t have
> > > > > > plugin-only APIs, etc. All classes are public, everything is
> > > accessible
> > > > > to
> > > > > > user code. We even refer to internal classes in public Javadoc
> > > > > > (e.g. I recall mentions of IgniteUtils in examples here and
> there).
> > > > > > Considering this, moving CommandHandler from ignite-core to
> > > > > > ignite-control-utility
> > > > > > doesn't look that bad. It doesn’t differ to much from any other
> > > change
> > > > > > that removes or renames a class.
> > > > > > There could be required changes with a higher compatibility
> impact
> > > but
> > > > I
> > > > > > don’t see them after a superficial glance.
> > > > > >
> > > > > > Stan
> > > > > >
> > > > > > From: Sergey Antonov
> > > > > > Sent: 23 января 2019 г. 19:15
> > > > > > To: dev@ignite.apache.org
> > > > > > Subject: Re: [DISCUSSION] Control.sh global rework in apache
> ignite
> > > 3.0
> > > > > >
> > > > > > Stan, thank you for response!
> > > > > >
> > > > > > I my view we shouldn't make incompatible changes and switch
> > > extendable
> > > > > > classes (i.e. VisorDataTransferObject ->
> IgniteDataTransferObject)
> > > > > between
> > > > > > minor releases. Therefore we couldn't rework utility in 2.8
> > release.
> > > > > >
> > > > > > ср, 23 янв. 2019 г. в 18:48, Stanislav Lukyanov <
> > > > stanlukya...@gmail.com
> > > > > >:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Sounds good. I agree with all points so far.
> > > > > > >
> > > > > > > I don’t really see why to wait for 3.0 though.
> > > > > > > As long as the old commands work I think it’s fine to do all of
> > > that
> > > > > in a
> > > > > > > minor release.
> > > > > > >
> > > > > > > Even moving the code to a separate module is fine as all the
> > > classes
> > > > > are
> > > > > > > internal and

Re: CacheInterceptor ClassCastException in case of cache was updated from thin java client

Ivan,

There is no inconsistency between thick and thin clients.
All of them work with caches in binary mode, see ClientCacheRequest (thin)
and PlatformCache (thick) classes.

On Thu, Jan 24, 2019 at 10:26 PM Ivan Pavlukhina 
wrote:

> Sergey,
>
> There are couple of things which should be addressed:
> 1. Unnecessary deserialization.
> 2. Inconsistent behavior.
> 3. Unclear documentation.
>
> Deserialization is not free and in my mind should be avoided where
> possible. I think that if some feature (like interceptors) requires
> deserialization then it should be enabled explicitly and an impact should
> be clear to user. I can imagine a toggle “withAllowedDeserialization”.
>
> If there is an inconsistency between thick and thin clients it should be
> eliminated. I do not see a reason why behavior should be different.
>
> If something is a good thing but it is not intuitive it could be
> documented. But if there is s really good reason for it. Otherwise
> simplicity and consistency are better alias.
>
> > On 24 Jan 2019, at 17:42, Sergey Antonov 
> wrote:
> >
> > I think it's bad idea. This contract nowhere defined and it's not clear
> for
> > users.
> >
> > чт, 24 янв. 2019 г. в 17:18, Pavel Tupitsyn :
> >
> >> Yes
> >>
> >> On Thu, Jan 24, 2019 at 5:15 PM Sergey Antonov <
> antonovserge...@gmail.com>
> >> wrote:
> >>
> >>> Pavel,
> >>>
> >>> "Leave it as is, use instanceof."
> >>> You meant always use CacheInterceptor and in all
> methods
> >>> check, that passed arguments is BinaryObject?
> >>>
> >>> чт, 24 янв. 2019 г. в 17:10, Pavel Tupitsyn :
> >>>
>  I don't think we should complicate things. Leave it as is, use
> >>> instanceof.
>  The fact is - you can get anything, BinaryObject or any user class, so
> >> be
>  prepared.
>  Good example of older API is CacheEvent, which actually has oldValue()
> >>> and
>  newValue() as Object.
> 
>  Igniters, any other thoughts?
> 
> 
>  On Thu, Jan 24, 2019 at 2:16 PM Sergey Antonov <
> >>> antonovserge...@gmail.com>
>  wrote:
> 
> > Pavel, how about marker interface DeserializedValueCacheInterceptor?
> >> We
> > will deserialize data and pass it to cache interceptor, if
>  CacheInterceptor
> > implements marker interface.
> >
> > чт, 24 янв. 2019 г. в 13:41, Pavel Tupitsyn :
> >
> >> You are exactly right, generic parameters don't make much sense
> >> here.
> >> Ignite caches are not restricted to any type, and there is type
> >>> erasure
> > in
> >> Java so you have no runtime guarantees.
> >>
> >> Maybe Interceptor design should be improved (e.g. add a flag to
> >> force
> >> binary or non-binary mode),
> >> but Thin or Thick client connector logic is unrelated here.
> >> withKeepBinary() call is valid and should not depend on Interceptor
> >> presence or implementation.
> >>
> >> On Thu, Jan 24, 2019 at 1:17 PM Sergey Antonov <
> > antonovserge...@gmail.com>
> >> wrote:
> >>
> >>> Hi, Pavel,
> >>>
> >>> "Interceptor should support both modes, binary or not. Any code
> >> can
> > call
> >>> withKeepBinary(), this should be expected.
> >>> Just add if (x instanceof BinaryObject) and go from there. "
> >>> I don't agree. The cache interceptor[1] is a parametrized class
> >> and
>  you
> >>> couldn't pass multiple cache interceptors in cache configuration.
> >>> So
> > all
> >>> cache interceptors must have Object, Object parameters for
> >>> supporting
> >> both
> >>> modes: binary and deserialized. In this case parametrized class
> >> no
> > sense.
> >>>
> >>> [1]
> >>>
> >>>
> >>
> >
> 
> >>>
> >>
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheInterceptor.html
> >>>
> >>> чт, 24 янв. 2019 г. в 13:06, Pavel Tupitsyn <
> >> ptupit...@apache.org
>  :
> >>>
>  Hi Sergey,
> 
>  I don't think this is a bug.
> 
>  Thick or thin clients always work in binary mode on server
> >> side,
> >> because
>  you receive data in serialized form and there is no point in
> >>> deserializing
>  it.
>  Moreover, in most cases you don't have classes on the server,
> >> so
> > binary
>  mode is the only way.
> 
>  Interceptor should support both modes, binary or not. Any code
> >>> can
> > call
>  withKeepBinary(), this should be expected.
>  Just add if (x instanceof BinaryObject) and go from there.
> 
>  Thanks,
>  Pavel
> 
>  On Thu, Jan 24, 2019 at 12:38 PM Sergey Antonov <
> >>> antonovserge...@gmail.com
> >
>  wrote:
> 
> > I did a little investigation. In
>  o.a.i.i.p.p.c.c.ClientCacheRequest#cache()
> > enforced cache with keep binary. Why we should always work
> >>> binary
>  objects?
>

Re: CacheInterceptor ClassCastException in case of cache was updated from thin java client

2019-01-24 Thread Ivan Pavlukhina

Sergey,

There are couple of things which should be addressed:
1. Unnecessary deserialization.
2. Inconsistent behavior.
3. Unclear documentation.

Deserialization is not free and in my mind should be avoided where possible. I 
think that if some feature (like interceptors) requires deserialization then it 
should be enabled explicitly and an impact should be clear to user. I can 
imagine a toggle “withAllowedDeserialization”.

If there is an inconsistency between thick and thin clients it should be 
eliminated. I do not see a reason why behavior should be different.

If something is a good thing but it is not intuitive it could be documented. 
But if there is s really good reason for it. Otherwise simplicity and 
consistency are better alias.

> On 24 Jan 2019, at 17:42, Sergey Antonov  wrote:
> 
> I think it's bad idea. This contract nowhere defined and it's not clear for
> users.
> 
> чт, 24 янв. 2019 г. в 17:18, Pavel Tupitsyn :
> 
>> Yes
>> 
>> On Thu, Jan 24, 2019 at 5:15 PM Sergey Antonov 
>> wrote:
>> 
>>> Pavel,
>>> 
>>> "Leave it as is, use instanceof."
>>> You meant always use CacheInterceptor and in all methods
>>> check, that passed arguments is BinaryObject?
>>> 
>>> чт, 24 янв. 2019 г. в 17:10, Pavel Tupitsyn :
>>> 
 I don't think we should complicate things. Leave it as is, use
>>> instanceof.
 The fact is - you can get anything, BinaryObject or any user class, so
>> be
 prepared.
 Good example of older API is CacheEvent, which actually has oldValue()
>>> and
 newValue() as Object.
 
 Igniters, any other thoughts?
 
 
 On Thu, Jan 24, 2019 at 2:16 PM Sergey Antonov <
>>> antonovserge...@gmail.com>
 wrote:
 
> Pavel, how about marker interface DeserializedValueCacheInterceptor?
>> We
> will deserialize data and pass it to cache interceptor, if
 CacheInterceptor
> implements marker interface.
> 
> чт, 24 янв. 2019 г. в 13:41, Pavel Tupitsyn :
> 
>> You are exactly right, generic parameters don't make much sense
>> here.
>> Ignite caches are not restricted to any type, and there is type
>>> erasure
> in
>> Java so you have no runtime guarantees.
>> 
>> Maybe Interceptor design should be improved (e.g. add a flag to
>> force
>> binary or non-binary mode),
>> but Thin or Thick client connector logic is unrelated here.
>> withKeepBinary() call is valid and should not depend on Interceptor
>> presence or implementation.
>> 
>> On Thu, Jan 24, 2019 at 1:17 PM Sergey Antonov <
> antonovserge...@gmail.com>
>> wrote:
>> 
>>> Hi, Pavel,
>>> 
>>> "Interceptor should support both modes, binary or not. Any code
>> can
> call
>>> withKeepBinary(), this should be expected.
>>> Just add if (x instanceof BinaryObject) and go from there. "
>>> I don't agree. The cache interceptor[1] is a parametrized class
>> and
 you
>>> couldn't pass multiple cache interceptors in cache configuration.
>>> So
> all
>>> cache interceptors must have Object, Object parameters for
>>> supporting
>> both
>>> modes: binary and deserialized. In this case parametrized class
>> no
> sense.
>>> 
>>> [1]
>>> 
>>> 
>> 
> 
 
>>> 
>> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheInterceptor.html
>>> 
>>> чт, 24 янв. 2019 г. в 13:06, Pavel Tupitsyn <
>> ptupit...@apache.org
 :
>>> 
 Hi Sergey,
 
 I don't think this is a bug.
 
 Thick or thin clients always work in binary mode on server
>> side,
>> because
 you receive data in serialized form and there is no point in
>>> deserializing
 it.
 Moreover, in most cases you don't have classes on the server,
>> so
> binary
 mode is the only way.
 
 Interceptor should support both modes, binary or not. Any code
>>> can
> call
 withKeepBinary(), this should be expected.
 Just add if (x instanceof BinaryObject) and go from there.
 
 Thanks,
 Pavel
 
 On Thu, Jan 24, 2019 at 12:38 PM Sergey Antonov <
>>> antonovserge...@gmail.com
> 
 wrote:
 
> I did a little investigation. In
 o.a.i.i.p.p.c.c.ClientCacheRequest#cache()
> enforced cache with keep binary. Why we should always work
>>> binary
 objects?
> 
> чт, 24 янв. 2019 г. в 12:29, Sergey Antonov <
>> antonovserge...@gmail.com
 :
> 
>> Hello, Igniters!
>> 
>> I have ignite node with configured cache. The cache have
>>> cache
>> interceptor. I wiil got ClassCastException on cache
 interceptor,
>> If I
 put
>> some entry to the cache (without keepBinary) from thin java
> client.
>> 
>> I think it's a bug. I'd like to find out yours view!
>> 
>>

[MTCGA]: new failures in builds [2658036] needs to be handled

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *Recently contributed test failed in master tests: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase::testCacheGetWrongArgs 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-2711660524736740778=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master tests: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase::testCachePutWrongArgs 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7419076455473690169=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master tests: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase::testCacheClearKeyWrongArgs 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=1014161704039071695=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master tests: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase::testCacheClearKeysWrongArgs 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6066988179553985738=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master tests: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase::testCacheGetSizeWrongArgs 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=8039044665871263128=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master tests: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase::testContainsKeyWrongArgs 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=8278104022332781338=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master tests: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase::testCacheReplaceWrongArgs 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=2601421891678519225=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master tests: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase::testCacheGetAndPutIfAbsentWrongArgs
 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6918523075100081828=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master tests: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase::testCacheContainsKeysWrongArgs 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=8508115998050634179=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master tests: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase::testCacheGetAllWrongArgs 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=8010201874973788100=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master tests: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase::testRemoveIfEqualsWrongArgs 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=5613882862418653296=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master tests: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase::testCacheReplaceIfEqualsWrongArgs 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=7510292952369692624=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master tests: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase::testCacheGetAndRemoveWrongArgs 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6767768552589233912=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master tests: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase::testCacheGetAndPutWrongArgs 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=1431371577522230596=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master tests: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase: 
Apache\Ignite\Tests\CacheKeyValueOpsTestCase::testCacheRemoveKeyWrongArgs 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-165170373995072247=%3Cdefault%3E=testDetails

 *Recently contributed test failed in master tests:

Re: Baseline auto-adjust`s discuss

2019-01-24 Thread Vladimir Ozerov

Hi Anton,

This is great feature, but I am a bit confused about automatic disabling of
a feature during manual baseline adjustment. This may lead to unpleasant
situations when a user enabled auto-adjustment, then re-adjusted it
manually somehow (e.g. from some previously created script) so that
auto-adjustment disabling went unnoticed, then added more nodes hoping that
auto-baseline is still active, etc.

Instead, I would rather make manual and auto adjustment mutually exclusive
- baseline cannot be adjusted manually when auto mode is set, and vice
versa. If exception is thrown in that cases, administrators will always
know current behavior of the system.

As far as configuration, wouldn’t it be enough to have a single long value
as opposed to Boolean + long? Say, 0 - immediate auto adjustment, negative
- disabled, positive - auto adjustment after timeout.

Thoughts?

чт, 24 янв. 2019 г. в 18:33, Anton Kalashnikov :

>
> Hello, Igniters!
>
> Work on the Phase II of IEP-4 (Baseline topology) [1] has started. I want
> to start to discuss of implementation of "Baseline auto-adjust" [2].
>
> "Baseline auto-adjust" feature implements mechanism of auto-adjust
> baseline corresponding to current topology after event join/left was
> appeared. It is required because when a node left the grid and nobody would
> change baseline manually it can lead to lost data(when some more nodes left
> the grid on depends in backup factor) but permanent tracking of grid is not
> always possible/desirible. Looks like in many cases auto-adjust baseline
> after some timeout is very helpfull.
>
> Distributed metastore[3](it is already done):
>
> First of all it is required the ability to store configuration data
> consistently and cluster-wide. Ignite doesn't have any specific API for
> such configurations and we don't want to have many similar implementations
> of the same feature in our code. After some thoughts is was proposed to
> implement it as some kind of distributed metastorage that gives the ability
> to store any data in it.
> First implementation is based on existing local metastorage API for
> persistent clusters (in-memory clusters will store data in memory).
> Write/remove operation use Discovery SPI to send updates to the cluster, it
> guarantees updates order and the fact that all existing (alive) nodes have
> handled the update message. As a way to find out which node has the latest
> data there is a "version" value of distributed metastorage, which is
> basically . All updates history
> until some point in the past is stored along with the data, so when an
> outdated node connects to the cluster it will receive all the missing data
> and apply it locally. If there's not enough history stored or joining node
> is clear then it'll receive shapshot of distributed metastorage so there
> won't be inconsistencies.
>
> Baseline auto-adjust:
>
> Main scenario:
> - There is grid with the baseline is equal to the current topology
> - New node joins to grid or some node left(failed) the grid
> - New mechanism detects this event and it add task for changing
> baseline to queue with configured timeout
> - If new event are happened before baseline would be changed task
> would be removed from queue and new task will be added
> - When timeout are expired the task would try to set new baseline
> corresponded to current topology
>
> First of all we need to add two parameters[4]:
> - baselineAutoAdjustEnabled - enable/disable "Baseline
> auto-adjust" feature.
> - baselineAutoAdjustTimeout - timeout after which baseline should
> be changed.
>
> This parameters are cluster wide and can be changed in real time because
> it is based on "Distributed metastore". On first time this parameters would
> be initiated by corresponded parameters(initBaselineAutoAdjustEnabled,
> initBaselineAutoAdjustTimeout) from "Ignite Configuration". Init value
> valid only before first changing of it after  value would be changed it is
> stored in "Distributed metastore".
>
> Restrictions:
> - This mechanism handling events only on active grid
> - If baselineNodes != gridNodes on activate this feature would be
> disabled
> - If lost partitions was detected this feature would be disabled
> - If baseline was adjusted manually on baselineNodes != gridNodes
> this feature would be disabled
>
> Draft implementation you can find here[5]. Feel free to ask more details
> and make suggestions.
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-4+Baseline+topology+for+caches
> [2] https://issues.apache.org/jira/browse/IGNITE-8571
> [3] https://issues.apache.org/jira/browse/IGNITE-10640
> [4] https://issues.apache.org/jira/browse/IGNITE-8573
> [5] https://github.com/apache/ignite/pull/5907
>
> --
> Best regards,
> Anton Kalashnikov
>
>

[jira] [Created] (IGNITE-11069) MVCC: Refactor GridCacheMapEntry mvcc set/remove/lock methods and listeners

2019-01-24 Thread Roman Kondakov (JIRA)

Roman Kondakov created IGNITE-11069:
---

 Summary: MVCC: Refactor GridCacheMapEntry mvcc set/remove/lock 
methods and listeners  
 Key: IGNITE-11069
 URL: https://issues.apache.org/jira/browse/IGNITE-11069
 Project: Ignite
  Issue Type: Improvement
  Components: mvcc
Reporter: Roman Kondakov


We should extract common logic from {{mvccSet()}}, {{mvccRemove()}} and 
{{mvccLock()}} methods and corresponding MVCC listeners: 
{{MvccUpdateLockListener}}, {{MvccRemoveLockListener}} and 
{{MvccAcquireLockListener}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: SQL View with list of existing indexes

2019-01-24 Thread Юрий

Hi Vladimir,

Thanks for your comments,

1) Agree.
2) Ok.
3) We create number of index copies depend on query parallelism. But seems
you are right - it should be exposed on TABLES level.
4) Approx. inline size shouldn't be used here, due to the value depend on
node and not has single value.
5) Do we have a plans for some view with table columns? If yes, may be will
be better have just array with column order from the columns view. For
example you want to know which columns are indexed already. In case we will
have plain comma-separated form it can't be achieved.





чт, 24 янв. 2019 г. в 18:09, Vladimir Ozerov :

> Hi Yuriy,
>
> Please note that MySQL link is about SHOW command, which is a different
> beast. In general I think that PG approach is better as it allows user to
> get quick overview of index content without complex JOINs. I would start
> with plain single view and add columns view later if we found it useful. As
> far as view columns:
> 1) I would add both cache ID/name and cache group ID/name
> 2) Number of columns does not look as a useful info to me
> 3) Query parallelism is related to cache, not index, so it should be in
> IGNITE.TABLES view instead
> 4) Inline size is definitely useful metric. Not sure about approximate
> inline size
> 5) I would add list of columns in plain comma-separated form with ASC/DESC
> modifiers
>
> Thoughts?
>
> Vladimir.
>
> On Thu, Jan 24, 2019 at 3:52 PM Юрий  wrote:
>
> > Hi Igniters,
> >
> > As part of IEP-29: SQL management and monitoring
> > <
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-29%3A+SQL+management+and+monitoring
> > >
> > I'm going to implement SQL view with list of existing indexes.
> > I've investigate how it expose by ORACLE, MySQL and Postgres.
> > ORACLE -
> >
> >
> https://docs.oracle.com/en/database/oracle/oracle-database/18/refrn/ALL_INDEXES.html#GUID-E39825BA-70AC-45D8-AF30-C7FF561373B6
> >
> > MySQL - https://dev.mysql.com/doc/refman/8.0/en/show-index.html
> > Postgres - https://www.postgresql.org/docs/11/view-pg-indexes.html ,
> > https://www.postgresql.org/docs/11/catalog-pg-index.html
> >
> > All vendors have such views which show at least following information:
> > schema name   - Name of schema related to table and index.
> > table name- Name of table related to an index.
> > index name   - Name of index.
> > list of columns   - All columns and their order included into an
> > index.
> > collation - ASC or DESC sort for each columns.
> >
> > + many specific information which different form vendor to vendor.
> >
> > In our case such specific information could be at least:
> >
> >1. Owning cache ID   - not sure, but may
> be
> >useful to join with other our views.
> >2. number of columns at the index- just to know how many
> result
> >should be in columns view
> >3. query parallelism   - It's
> configuration
> >parameter show how many thread can be used to execute query.
> >4. inline size   - inline size
> >used for this index.
> >5. is affinity - boolean
> >parameter show that affinity key index
> >6. is pk- boolean
> >parameter show that PK index
> >7. approx recommended inline size- dynamically calculated
> >recommended inline size for this index to show required size to keep
> > whole
> >indexed columns as inlined.
> >
> >
> >
> > All vendors have different ways  to present information about index
> > columns:
> > PG - use array of index table columns and second array for collation each
> > of columns.
> > MySQL - each row in index view contains information about one of indexed
> > columsn with ther position at the index. So for one index there are many
> > columns.
> > ORACLE,  - use separate view where each of row present column included
> into
> > index with all required information and can be joined by schema, table
> and
> > index names.
> > ORACLE indexed columns view -
> >
> >
> https://docs.oracle.com/cd/B19306_01/server.102/b14237/statviews_1064.htm#i1577532
> > MySql -
> >
> > I propose use ORACLE way and have second view to represent column
> included
> > into indexes.
> >
> > In this case such view can have the following information:
> > schema name   - Name of schema related to table and index.
> > table name- Name of table related to an index.
> > index name   - Name of index.
> > column name- Name of column included into index.
> > column type  - Type of the column.
> > column position - Position of column within the index.
> > collation- Either the column is sorted descending or
> > ascending
> >
> > And can be joined with index view through schema, table and index names.
> >
> >
> >

Baseline auto-adjust`s discuss

2019-01-24 Thread Anton Kalashnikov

Hello, Igniters!

Work on the Phase II of IEP-4 (Baseline topology) [1] has started. I want to
start to discuss of implementation of "Baseline auto-adjust" [2].

"Baseline auto-adjust" feature implements mechanism of auto-adjust baseline
corresponding to current topology after event join/left was appeared. It is
required because when a node left the grid and nobody would change baseline
manually it can lead to lost data(when some more nodes left the grid on depends
in backup factor) but permanent tracking of grid is not always
possible/desirible. Looks like in many cases auto-adjust baseline after some
timeout is very helpfull.

Distributed metastore[3](it is already done):

First of all it is required the ability to store configuration data
consistently and cluster-wide. Ignite doesn't have any specific API for such
configurations and we don't want to have many similar implementations of the
same feature in our code. After some thoughts is was proposed to implement it
as some kind of distributed metastorage that gives the ability to store any
data in it.
First implementation is based on existing local metastorage API for persistent
clusters (in-memory clusters will store data in memory). Write/remove operation
use Discovery SPI to send updates to the cluster, it guarantees updates order
and the fact that all existing (alive) nodes have handled the update message.
As a way to find out which node has the latest data there is a "version" value
of distributed metastorage, which is basically . All updates history until some point in the past is stored along with
the data, so when an outdated node connects to the cluster it will receive all
the missing data and apply it locally. If there's not enough history stored or
joining node is clear then it'll receive shapshot of distributed metastorage so
there won't be inconsistencies.

Baseline auto-adjust:

Main scenario:
- There is grid with the baseline is equal to the current topology
- New node joins to grid or some node left(failed) the grid
- New mechanism detects this event and it add task for changing
baseline to queue with configured timeout
- If new event are happened before baseline would be changed task would
be removed from queue and new task will be added
- When timeout are expired the task would try to set new baseline
corresponded to current topology

First of all we need to add two parameters[4]:
- baselineAutoAdjustEnabled - enable/disable "Baseline auto-adjust"
feature.
- baselineAutoAdjustTimeout - timeout after which baseline should be
changed.

This parameters are cluster wide and can be changed in real time because it is
based on "Distributed metastore". On first time this parameters would be
initiated by corresponded parameters(initBaselineAutoAdjustEnabled,
initBaselineAutoAdjustTimeout) from "Ignite Configuration". Init value valid
only before first changing of it after value would be changed it is stored in
"Distributed metastore".

Restrictions:
- This mechanism handling events only on active grid
- If baselineNodes != gridNodes on activate this feature would be
disabled
- If lost partitions was detected this feature would be disabled
- If baseline was adjusted manually on baselineNodes != gridNodes this
feature would be disabled

Draft implementation you can find here[5]. Feel free to ask more details and
make suggestions.

[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-4+Baseline+topology+for+caches
[2] https://issues.apache.org/jira/browse/IGNITE-8571
[3] https://issues.apache.org/jira/browse/IGNITE-10640
[4] https://issues.apache.org/jira/browse/IGNITE-8573
[5] https://github.com/apache/ignite/pull/5907

--
Best regards,
Anton Kalashnikov

[jira] [Created] (IGNITE-11068) MVCC TX: Do not create TxLog and do not start vacuum workers in case there are no MVCC caches on the node.

2019-01-24 Thread Igor Seliverstov (JIRA)

Igor Seliverstov created IGNITE-11068:
-

 Summary: MVCC TX: Do not create TxLog and do not start vacuum 
workers in case there are no MVCC caches on the node.
 Key: IGNITE-11068
 URL: https://issues.apache.org/jira/browse/IGNITE-11068
 Project: Ignite
  Issue Type: Improvement
Reporter: Igor Seliverstov






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11067) MVCC: IoStatistics test failed on TC.

2019-01-24 Thread Andrew Mashenkov (JIRA)

Andrew Mashenkov created IGNITE-11067:
-

 Summary: MVCC: IoStatistics test failed on TC.
 Key: IGNITE-11067
 URL: https://issues.apache.org/jira/browse/IGNITE-11067
 Project: Ignite
  Issue Type: Test
  Components: mvcc
Reporter: Andrew Mashenkov


Next tests failed in MvccCache 9 suite on master:

IoStatisticsCachePersistenceSelfTest.testTransactonalCache 
 IoStatisticsCacheSelfTest.testTransactonalCache 
 IoStatisticsManagerSelfTest.testPersistentIOGlobalStat

Let's exclude these classes from Mvcc suite as they are already contain test 
methods for mvcc mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: SQL View with list of existing indexes

2019-01-24 Thread Vladimir Ozerov

Hi Yuriy,

Please note that MySQL link is about SHOW command, which is a different
beast. In general I think that PG approach is better as it allows user to
get quick overview of index content without complex JOINs. I would start
with plain single view and add columns view later if we found it useful. As
far as view columns:
1) I would add both cache ID/name and cache group ID/name
2) Number of columns does not look as a useful info to me
3) Query parallelism is related to cache, not index, so it should be in
IGNITE.TABLES view instead
4) Inline size is definitely useful metric. Not sure about approximate
inline size
5) I would add list of columns in plain comma-separated form with ASC/DESC
modifiers

Thoughts?

Vladimir.

On Thu, Jan 24, 2019 at 3:52 PM Юрий  wrote:

> Hi Igniters,
>
> As part of IEP-29: SQL management and monitoring
> <
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-29%3A+SQL+management+and+monitoring
> >
> I'm going to implement SQL view with list of existing indexes.
> I've investigate how it expose by ORACLE, MySQL and Postgres.
> ORACLE -
>
> https://docs.oracle.com/en/database/oracle/oracle-database/18/refrn/ALL_INDEXES.html#GUID-E39825BA-70AC-45D8-AF30-C7FF561373B6
>
> MySQL - https://dev.mysql.com/doc/refman/8.0/en/show-index.html
> Postgres - https://www.postgresql.org/docs/11/view-pg-indexes.html ,
> https://www.postgresql.org/docs/11/catalog-pg-index.html
>
> All vendors have such views which show at least following information:
> schema name   - Name of schema related to table and index.
> table name- Name of table related to an index.
> index name   - Name of index.
> list of columns   - All columns and their order included into an
> index.
> collation - ASC or DESC sort for each columns.
>
> + many specific information which different form vendor to vendor.
>
> In our case such specific information could be at least:
>
>1. Owning cache ID   - not sure, but may be
>useful to join with other our views.
>2. number of columns at the index- just to know how many result
>should be in columns view
>3. query parallelism   - It's configuration
>parameter show how many thread can be used to execute query.
>4. inline size   - inline size
>used for this index.
>5. is affinity - boolean
>parameter show that affinity key index
>6. is pk- boolean
>parameter show that PK index
>7. approx recommended inline size- dynamically calculated
>recommended inline size for this index to show required size to keep
> whole
>indexed columns as inlined.
>
>
>
> All vendors have different ways  to present information about index
> columns:
> PG - use array of index table columns and second array for collation each
> of columns.
> MySQL - each row in index view contains information about one of indexed
> columsn with ther position at the index. So for one index there are many
> columns.
> ORACLE,  - use separate view where each of row present column included into
> index with all required information and can be joined by schema, table and
> index names.
> ORACLE indexed columns view -
>
> https://docs.oracle.com/cd/B19306_01/server.102/b14237/statviews_1064.htm#i1577532
> MySql -
>
> I propose use ORACLE way and have second view to represent column included
> into indexes.
>
> In this case such view can have the following information:
> schema name   - Name of schema related to table and index.
> table name- Name of table related to an index.
> index name   - Name of index.
> column name- Name of column included into index.
> column type  - Type of the column.
> column position - Position of column within the index.
> collation- Either the column is sorted descending or
> ascending
>
> And can be joined with index view through schema, table and index names.
>
>
>
> What do you think about such approach and list of columns which could be
> included into the views?
>
> --
> Живи с улыбкой! :D
>

[jira] [Created] (IGNITE-11066) Start MetaStorage for write before activation

2019-01-24 Thread Ivan Bessonov (JIRA)

Ivan Bessonov created IGNITE-11066:
--

 Summary: Start MetaStorage for write before activation
 Key: IGNITE-11066
 URL: https://issues.apache.org/jira/browse/IGNITE-11066
 Project: Ignite
  Issue Type: Improvement
Reporter: Ivan Bessonov
Assignee: Ivan Bessonov


Distributed metastorage (IGNITE-10640) requires local metastorage to be 
writable to work properly. So early update messages will have to wait until 
this event which can hang the whole cluster for some time.

As a first step of fixing this problem it is proposed to start local 
metastorage in read-write mode before activation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: SQL View with list of existing indexes

2019-01-24 Thread Юрий

One additional thought which I figured out just now.

Seems approximately recommended inline size is not good choice to add to
the index view, due to value this parameter has different value for each
node. Even more, for non affinity node it always will be zero. So, seems it
should be excluded from my initial proposal.

чт, 24 янв. 2019 г. в 15:51, Юрий :

> Hi Igniters,
>
> As part of IEP-29: SQL management and monitoring
> 
> I'm going to implement SQL view with list of existing indexes.
> I've investigate how it expose by ORACLE, MySQL and Postgres.
> ORACLE -
> https://docs.oracle.com/en/database/oracle/oracle-database/18/refrn/ALL_INDEXES.html#GUID-E39825BA-70AC-45D8-AF30-C7FF561373B6
>
> MySQL - https://dev.mysql.com/doc/refman/8.0/en/show-index.html
> Postgres - https://www.postgresql.org/docs/11/view-pg-indexes.html ,
> https://www.postgresql.org/docs/11/catalog-pg-index.html
>
> All vendors have such views which show at least following information:
> schema name   - Name of schema related to table and index.
> table name- Name of table related to an index.
> index name   - Name of index.
> list of columns   - All columns and their order included into an
> index.
> collation - ASC or DESC sort for each columns.
>
> + many specific information which different form vendor to vendor.
>
> In our case such specific information could be at least:
>
>1. Owning cache ID   - not sure, but may
>be useful to join with other our views.
>2. number of columns at the index- just to know how many
>result should be in columns view
>3. query parallelism   - It's
>configuration parameter show how many thread can be used to execute query.
>4. inline size   - inline size
>used for this index.
>5. is affinity - boolean
>parameter show that affinity key index
>6. is pk- boolean
>parameter show that PK index
>7. approx recommended inline size- dynamically calculated
>recommended inline size for this index to show required size to keep whole
>indexed columns as inlined.
>
>
>
> All vendors have different ways  to present information about index
> columns:
> PG - use array of index table columns and second array for collation each
> of columns.
> MySQL - each row in index view contains information about one of indexed
> columsn with ther position at the index. So for one index there are many
> columns.
> ORACLE,  - use separate view where each of row present column included
> into index with all required information and can be joined by schema, table
> and index names.
> ORACLE indexed columns view -
> https://docs.oracle.com/cd/B19306_01/server.102/b14237/statviews_1064.htm#i1577532
> MySql -
>
> I propose use ORACLE way and have second view to represent column included
> into indexes.
>
> In this case such view can have the following information:
> schema name   - Name of schema related to table and index.
> table name- Name of table related to an index.
> index name   - Name of index.
> column name- Name of column included into index.
> column type  - Type of the column.
> column position - Position of column within the index.
> collation- Either the column is sorted descending or
> ascending
>
> And can be joined with index view through schema, table and index names.
>
>
>
> What do you think about such approach and list of columns which could be
> included into the views?
>
> --
> Живи с улыбкой! :D
>


-- 
Живи с улыбкой! :D

Re: [DISCUSSION] Control.sh global rework in apache ignite 3.0

2019-01-24 Thread Sergey Kozlov

Denis

I'm not sure that visorcmd can be refactored without incompatible changes
or significant changes for behaviour:
1. Visorcmd starts as demon node and joins the cluster. The modern
utilities don't need spring config and just connect to TCP management port.
2. Visorcmd is mostly an interactive tool but control.sh looks like *nix
regular command-line program.
3. Visorcmd command set (IMO) is weird and has some commands that sometimes
are not about ignite (like deploy - copy file to remote nodes) or not clear
why we need them (mlist/mget/mcompact)

I think we should define the root purpose for new utility (or for
refactored visorcmd)
>From my standpoint they are following:
 - cluster status commands (topology, node status, configuration, etc)
 - cluster management commands (control.sh baseline commands + visrcmd
kill/stop nodes)
 - cache content commands (print/clear cache content)
 - cache management commands (create/stop/destroy/copy/rename etc cache)
 - service/job management commands (start/stop/restart service/job)
 - tx management (tx list, kill)
 - data consistency commands (idle_verify, crc_check, checkpoint)
 - statistics commands (cluster/cache metrics)




On Thu, Jan 24, 2019 at 12:12 PM Sergey Antonov 
wrote:

> Alexey, Denis I agree yours points.
>
> I think we should rewrite visor.cmd from scala to java at first. And then
> merge control.sh into visor.cmd.
>
> чт, 24 янв. 2019 г. в 11:44, Alexey Kuznetsov :
>
> > I agree with Denis,
> >
> > How about to merge control.sh into Visor.CMD ?
> > And rewrite Visor.CMD from Scala to Java.
> >
> > What do you think?
> >
> > On Thu, Jan 24, 2019 at 4:41 AM Denis Magda  wrote:
> >
> > > Why don't we go in a reverse direction - instead of creating multiple
> > > scripts for different needs we incorporate all capabilities within
> > > visorcmd? Visor is an app/script that can be updated to meet the
> > > requirements of specific tools.
> > >
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Wed, Jan 23, 2019 at 1:23 PM Sergey Kozlov 
> > > wrote:
> > >
> > > > Thanks Sergey for the raising the question.
> > > >
> > > > What's about move actual/interesting commands from visorcmd and
> remove
> > > that
> > > > script at all?
> > > >
> > > > Also would good simplify the bash code of script and ideally process
> > > > everyting inside java code?
> > > >
> > > > On Wed, Jan 23, 2019 at 7:38 PM Stanislav Lukyanov <
> > > stanlukya...@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > I value strict compatibility rules very highly, and would be happy
> if
> > > we
> > > > > never removed a public class (even if it is in the “internal”
> > package)
> > > > > in a minor release.
> > > > > Unfortunately, Ignite is far from that place for now. We don’t have
> > any
> > > > > distinction between API and internal classes, don’t have
> > > > > plugin-only APIs, etc. All classes are public, everything is
> > accessible
> > > > to
> > > > > user code. We even refer to internal classes in public Javadoc
> > > > > (e.g. I recall mentions of IgniteUtils in examples here and there).
> > > > > Considering this, moving CommandHandler from ignite-core to
> > > > > ignite-control-utility
> > > > > doesn't look that bad. It doesn’t differ to much from any other
> > change
> > > > > that removes or renames a class.
> > > > > There could be required changes with a higher compatibility impact
> > but
> > > I
> > > > > don’t see them after a superficial glance.
> > > > >
> > > > > Stan
> > > > >
> > > > > From: Sergey Antonov
> > > > > Sent: 23 января 2019 г. 19:15
> > > > > To: dev@ignite.apache.org
> > > > > Subject: Re: [DISCUSSION] Control.sh global rework in apache ignite
> > 3.0
> > > > >
> > > > > Stan, thank you for response!
> > > > >
> > > > > I my view we shouldn't make incompatible changes and switch
> > extendable
> > > > > classes (i.e. VisorDataTransferObject -> IgniteDataTransferObject)
> > > > between
> > > > > minor releases. Therefore we couldn't rework utility in 2.8
> release.
> > > > >
> > > > > ср, 23 янв. 2019 г. в 18:48, Stanislav Lukyanov <
> > > stanlukya...@gmail.com
> > > > >:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Sounds good. I agree with all points so far.
> > > > > >
> > > > > > I don’t really see why to wait for 3.0 though.
> > > > > > As long as the old commands work I think it’s fine to do all of
> > that
> > > > in a
> > > > > > minor release.
> > > > > >
> > > > > > Even moving the code to a separate module is fine as all the
> > classes
> > > > are
> > > > > > internal and unlikely to be used in the wild.
> > > > > > On paper it’s an incompatible change, of course, but I think in
> > this
> > > > case
> > > > > > it’s fine.
> > > > > >
> > > > > > My 2 cents,
> > > > > > Stan
> > > > > >
> > > > > > From: Sergey Antonov
> > > > > > Sent: 23 января 2019 г. 17:10
> > > > > > To: dev@ignite.apache.org
> > > > > > Subject: [DISCUSSION] Control.sh global rework in apache ignite
> 3.0
> > > > > >
> > > > > > Hello, Igniters!
> > > > >

Re: CacheInterceptor ClassCastException in case of cache was updated from thin java client

I think it's bad idea. This contract nowhere defined and it's not clear for
users.

чт, 24 янв. 2019 г. в 17:18, Pavel Tupitsyn :

> Yes
>
> On Thu, Jan 24, 2019 at 5:15 PM Sergey Antonov 
> wrote:
>
> > Pavel,
> >
> > "Leave it as is, use instanceof."
> > You meant always use CacheInterceptor and in all methods
> > check, that passed arguments is BinaryObject?
> >
> > чт, 24 янв. 2019 г. в 17:10, Pavel Tupitsyn :
> >
> > > I don't think we should complicate things. Leave it as is, use
> > instanceof.
> > > The fact is - you can get anything, BinaryObject or any user class, so
> be
> > > prepared.
> > > Good example of older API is CacheEvent, which actually has oldValue()
> > and
> > > newValue() as Object.
> > >
> > > Igniters, any other thoughts?
> > >
> > >
> > > On Thu, Jan 24, 2019 at 2:16 PM Sergey Antonov <
> > antonovserge...@gmail.com>
> > > wrote:
> > >
> > > > Pavel, how about marker interface DeserializedValueCacheInterceptor?
> We
> > > > will deserialize data and pass it to cache interceptor, if
> > > CacheInterceptor
> > > > implements marker interface.
> > > >
> > > > чт, 24 янв. 2019 г. в 13:41, Pavel Tupitsyn :
> > > >
> > > > > You are exactly right, generic parameters don't make much sense
> here.
> > > > > Ignite caches are not restricted to any type, and there is type
> > erasure
> > > > in
> > > > > Java so you have no runtime guarantees.
> > > > >
> > > > > Maybe Interceptor design should be improved (e.g. add a flag to
> force
> > > > > binary or non-binary mode),
> > > > > but Thin or Thick client connector logic is unrelated here.
> > > > > withKeepBinary() call is valid and should not depend on Interceptor
> > > > > presence or implementation.
> > > > >
> > > > > On Thu, Jan 24, 2019 at 1:17 PM Sergey Antonov <
> > > > antonovserge...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi, Pavel,
> > > > > >
> > > > > > "Interceptor should support both modes, binary or not. Any code
> can
> > > > call
> > > > > > withKeepBinary(), this should be expected.
> > > > > > Just add if (x instanceof BinaryObject) and go from there. "
> > > > > > I don't agree. The cache interceptor[1] is a parametrized class
> and
> > > you
> > > > > > couldn't pass multiple cache interceptors in cache configuration.
> > So
> > > > all
> > > > > > cache interceptors must have Object, Object parameters for
> > supporting
> > > > > both
> > > > > > modes: binary and deserialized. In this case parametrized class
> no
> > > > sense.
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheInterceptor.html
> > > > > >
> > > > > > чт, 24 янв. 2019 г. в 13:06, Pavel Tupitsyn <
> ptupit...@apache.org
> > >:
> > > > > >
> > > > > > > Hi Sergey,
> > > > > > >
> > > > > > > I don't think this is a bug.
> > > > > > >
> > > > > > > Thick or thin clients always work in binary mode on server
> side,
> > > > > because
> > > > > > > you receive data in serialized form and there is no point in
> > > > > > deserializing
> > > > > > > it.
> > > > > > > Moreover, in most cases you don't have classes on the server,
> so
> > > > binary
> > > > > > > mode is the only way.
> > > > > > >
> > > > > > > Interceptor should support both modes, binary or not. Any code
> > can
> > > > call
> > > > > > > withKeepBinary(), this should be expected.
> > > > > > > Just add if (x instanceof BinaryObject) and go from there.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Pavel
> > > > > > >
> > > > > > > On Thu, Jan 24, 2019 at 12:38 PM Sergey Antonov <
> > > > > > antonovserge...@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I did a little investigation. In
> > > > > > > o.a.i.i.p.p.c.c.ClientCacheRequest#cache()
> > > > > > > > enforced cache with keep binary. Why we should always work
> > binary
> > > > > > > objects?
> > > > > > > >
> > > > > > > > чт, 24 янв. 2019 г. в 12:29, Sergey Antonov <
> > > > > antonovserge...@gmail.com
> > > > > > >:
> > > > > > > >
> > > > > > > > > Hello, Igniters!
> > > > > > > > >
> > > > > > > > > I have ignite node with configured cache. The cache have
> > cache
> > > > > > > > > interceptor. I wiil got ClassCastException on cache
> > > interceptor,
> > > > > If I
> > > > > > > put
> > > > > > > > > some entry to the cache (without keepBinary) from thin java
> > > > client.
> > > > > > > > >
> > > > > > > > > I think it's a bug. I'd like to find out yours view!
> > > > > > > > >
> > > > > > > > > Also I made JIRA ticket with reproducer [1].
> > > > > > > > >
> > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10789
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > BR, Sergey Antonov
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > BR, Sergey Antonov
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > BR, Sergey Antonov
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > BR,

[jira] [Created] (IGNITE-11064) Add documentation for enabling cache statistics only on appropriate nodes.

2019-01-24 Thread Stanilovsky Evgeny (JIRA)

Stanilovsky Evgeny created IGNITE-11064:
---

 Summary: Add documentation for enabling cache statistics only on 
appropriate nodes.
 Key: IGNITE-11064
 URL: https://issues.apache.org/jira/browse/IGNITE-11064
 Project: Ignite
  Issue Type: Improvement
  Components: documentation
Affects Versions: 2.7
Reporter: Stanilovsky Evgeny
 Fix For: 2.8


System property: IGNITE_DISCOVERY_DISABLE_CACHE_METRICS_UPDATE (false default) 
will disable statistic collecting even if statisticsEnabled flag is true.
[IGNITE-10172|https://issues.apache.org/jira/browse/IGNITE-10172]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: CacheInterceptor ClassCastException in case of cache was updated from thin java client

Yes

On Thu, Jan 24, 2019 at 5:15 PM Sergey Antonov 
wrote:

> Pavel,
>
> "Leave it as is, use instanceof."
> You meant always use CacheInterceptor and in all methods
> check, that passed arguments is BinaryObject?
>
> чт, 24 янв. 2019 г. в 17:10, Pavel Tupitsyn :
>
> > I don't think we should complicate things. Leave it as is, use
> instanceof.
> > The fact is - you can get anything, BinaryObject or any user class, so be
> > prepared.
> > Good example of older API is CacheEvent, which actually has oldValue()
> and
> > newValue() as Object.
> >
> > Igniters, any other thoughts?
> >
> >
> > On Thu, Jan 24, 2019 at 2:16 PM Sergey Antonov <
> antonovserge...@gmail.com>
> > wrote:
> >
> > > Pavel, how about marker interface DeserializedValueCacheInterceptor? We
> > > will deserialize data and pass it to cache interceptor, if
> > CacheInterceptor
> > > implements marker interface.
> > >
> > > чт, 24 янв. 2019 г. в 13:41, Pavel Tupitsyn :
> > >
> > > > You are exactly right, generic parameters don't make much sense here.
> > > > Ignite caches are not restricted to any type, and there is type
> erasure
> > > in
> > > > Java so you have no runtime guarantees.
> > > >
> > > > Maybe Interceptor design should be improved (e.g. add a flag to force
> > > > binary or non-binary mode),
> > > > but Thin or Thick client connector logic is unrelated here.
> > > > withKeepBinary() call is valid and should not depend on Interceptor
> > > > presence or implementation.
> > > >
> > > > On Thu, Jan 24, 2019 at 1:17 PM Sergey Antonov <
> > > antonovserge...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi, Pavel,
> > > > >
> > > > > "Interceptor should support both modes, binary or not. Any code can
> > > call
> > > > > withKeepBinary(), this should be expected.
> > > > > Just add if (x instanceof BinaryObject) and go from there. "
> > > > > I don't agree. The cache interceptor[1] is a parametrized class and
> > you
> > > > > couldn't pass multiple cache interceptors in cache configuration.
> So
> > > all
> > > > > cache interceptors must have Object, Object parameters for
> supporting
> > > > both
> > > > > modes: binary and deserialized. In this case parametrized class no
> > > sense.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheInterceptor.html
> > > > >
> > > > > чт, 24 янв. 2019 г. в 13:06, Pavel Tupitsyn  >:
> > > > >
> > > > > > Hi Sergey,
> > > > > >
> > > > > > I don't think this is a bug.
> > > > > >
> > > > > > Thick or thin clients always work in binary mode on server side,
> > > > because
> > > > > > you receive data in serialized form and there is no point in
> > > > > deserializing
> > > > > > it.
> > > > > > Moreover, in most cases you don't have classes on the server, so
> > > binary
> > > > > > mode is the only way.
> > > > > >
> > > > > > Interceptor should support both modes, binary or not. Any code
> can
> > > call
> > > > > > withKeepBinary(), this should be expected.
> > > > > > Just add if (x instanceof BinaryObject) and go from there.
> > > > > >
> > > > > > Thanks,
> > > > > > Pavel
> > > > > >
> > > > > > On Thu, Jan 24, 2019 at 12:38 PM Sergey Antonov <
> > > > > antonovserge...@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I did a little investigation. In
> > > > > > o.a.i.i.p.p.c.c.ClientCacheRequest#cache()
> > > > > > > enforced cache with keep binary. Why we should always work
> binary
> > > > > > objects?
> > > > > > >
> > > > > > > чт, 24 янв. 2019 г. в 12:29, Sergey Antonov <
> > > > antonovserge...@gmail.com
> > > > > >:
> > > > > > >
> > > > > > > > Hello, Igniters!
> > > > > > > >
> > > > > > > > I have ignite node with configured cache. The cache have
> cache
> > > > > > > > interceptor. I wiil got ClassCastException on cache
> > interceptor,
> > > > If I
> > > > > > put
> > > > > > > > some entry to the cache (without keepBinary) from thin java
> > > client.
> > > > > > > >
> > > > > > > > I think it's a bug. I'd like to find out yours view!
> > > > > > > >
> > > > > > > > Also I made JIRA ticket with reproducer [1].
> > > > > > > >
> > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10789
> > > > > > > >
> > > > > > > > --
> > > > > > > > BR, Sergey Antonov
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > BR, Sergey Antonov
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > BR, Sergey Antonov
> > > > >
> > > >
> > >
> > >
> > > --
> > > BR, Sergey Antonov
> > >
> >
>
>
> --
> BR, Sergey Antonov
>

Re: CacheInterceptor ClassCastException in case of cache was updated from thin java client

Pavel,

"Leave it as is, use instanceof."
You meant always use CacheInterceptor and in all methods
check, that passed arguments is BinaryObject?

чт, 24 янв. 2019 г. в 17:10, Pavel Tupitsyn :

> I don't think we should complicate things. Leave it as is, use instanceof.
> The fact is - you can get anything, BinaryObject or any user class, so be
> prepared.
> Good example of older API is CacheEvent, which actually has oldValue() and
> newValue() as Object.
>
> Igniters, any other thoughts?
>
>
> On Thu, Jan 24, 2019 at 2:16 PM Sergey Antonov 
> wrote:
>
> > Pavel, how about marker interface DeserializedValueCacheInterceptor? We
> > will deserialize data and pass it to cache interceptor, if
> CacheInterceptor
> > implements marker interface.
> >
> > чт, 24 янв. 2019 г. в 13:41, Pavel Tupitsyn :
> >
> > > You are exactly right, generic parameters don't make much sense here.
> > > Ignite caches are not restricted to any type, and there is type erasure
> > in
> > > Java so you have no runtime guarantees.
> > >
> > > Maybe Interceptor design should be improved (e.g. add a flag to force
> > > binary or non-binary mode),
> > > but Thin or Thick client connector logic is unrelated here.
> > > withKeepBinary() call is valid and should not depend on Interceptor
> > > presence or implementation.
> > >
> > > On Thu, Jan 24, 2019 at 1:17 PM Sergey Antonov <
> > antonovserge...@gmail.com>
> > > wrote:
> > >
> > > > Hi, Pavel,
> > > >
> > > > "Interceptor should support both modes, binary or not. Any code can
> > call
> > > > withKeepBinary(), this should be expected.
> > > > Just add if (x instanceof BinaryObject) and go from there. "
> > > > I don't agree. The cache interceptor[1] is a parametrized class and
> you
> > > > couldn't pass multiple cache interceptors in cache configuration. So
> > all
> > > > cache interceptors must have Object, Object parameters for supporting
> > > both
> > > > modes: binary and deserialized. In this case parametrized class no
> > sense.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheInterceptor.html
> > > >
> > > > чт, 24 янв. 2019 г. в 13:06, Pavel Tupitsyn :
> > > >
> > > > > Hi Sergey,
> > > > >
> > > > > I don't think this is a bug.
> > > > >
> > > > > Thick or thin clients always work in binary mode on server side,
> > > because
> > > > > you receive data in serialized form and there is no point in
> > > > deserializing
> > > > > it.
> > > > > Moreover, in most cases you don't have classes on the server, so
> > binary
> > > > > mode is the only way.
> > > > >
> > > > > Interceptor should support both modes, binary or not. Any code can
> > call
> > > > > withKeepBinary(), this should be expected.
> > > > > Just add if (x instanceof BinaryObject) and go from there.
> > > > >
> > > > > Thanks,
> > > > > Pavel
> > > > >
> > > > > On Thu, Jan 24, 2019 at 12:38 PM Sergey Antonov <
> > > > antonovserge...@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > I did a little investigation. In
> > > > > o.a.i.i.p.p.c.c.ClientCacheRequest#cache()
> > > > > > enforced cache with keep binary. Why we should always work binary
> > > > > objects?
> > > > > >
> > > > > > чт, 24 янв. 2019 г. в 12:29, Sergey Antonov <
> > > antonovserge...@gmail.com
> > > > >:
> > > > > >
> > > > > > > Hello, Igniters!
> > > > > > >
> > > > > > > I have ignite node with configured cache. The cache have cache
> > > > > > > interceptor. I wiil got ClassCastException on cache
> interceptor,
> > > If I
> > > > > put
> > > > > > > some entry to the cache (without keepBinary) from thin java
> > client.
> > > > > > >
> > > > > > > I think it's a bug. I'd like to find out yours view!
> > > > > > >
> > > > > > > Also I made JIRA ticket with reproducer [1].
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10789
> > > > > > >
> > > > > > > --
> > > > > > > BR, Sergey Antonov
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > BR, Sergey Antonov
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > BR, Sergey Antonov
> > > >
> > >
> >
> >
> > --
> > BR, Sergey Antonov
> >
>


-- 
BR, Sergey Antonov

[jira] [Created] (IGNITE-11062) Calculating Compute Usage section contains confusing numbers

2019-01-24 Thread Denis Mekhanikov (JIRA)

Denis Mekhanikov created IGNITE-11062:
-

 Summary: Calculating Compute Usage section contains confusing 
numbers
 Key: IGNITE-11062
 URL: https://issues.apache.org/jira/browse/IGNITE-11062
 Project: Ignite
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.7
Reporter: Denis Mekhanikov
Assignee: Prachi Garg


[Calculating Compute 
Usage|https://apacheignite.readme.io/docs/capacity-planning#section-calculating-compute-usage]
 section on the capacity planning page contains pretty confusing results. One 
may get an impression, that SQL queries are faster than cache API, and SQL 
should be used everywhere it's possible. This is pretty far from the truth, so 
it's better to rework this section.

Plus the provided link doesn't contain the mentioned results. It's better to 
specify a link to the following page and get benchmark results from it: 
[https://www.gridgain.com/resources/benchmarks/gridgain-benchmarks-results]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11063) IgniteClusterActivateDeactivateTestWithPersistence#testDeactivateDuringEvictionAndRebalance has NPE inside of it.

2019-01-24 Thread Ivan Bessonov (JIRA)

Ivan Bessonov created IGNITE-11063:
--

 Summary: 
IgniteClusterActivateDeactivateTestWithPersistence#testDeactivateDuringEvictionAndRebalance
 has NPE inside of it.
 Key: IGNITE-11063
 URL: https://issues.apache.org/jira/browse/IGNITE-11063
 Project: Ignite
  Issue Type: Test
Reporter: Ivan Bessonov


{code:java}
[2019-01-24 
16:23:19,561][ERROR][sys-#221%cache.IgniteClusterActivateDeactivateTestWithPersistence2%][PartitionsEvictManager]
 Partition eviction failed, this can cause grid hang.
java.lang.NullPointerException
at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:668)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1106)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:910)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask.run(PartitionsEvictManager.java:415)
at 
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6873)
at 
org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

{code}
{code:java}
 [2019-01-24 
16:28:30,862][ERROR][sys-#221%cache.IgniteClusterActivateDeactivateTestWithPersistence2%][PartitionsEvictManager]
 Partition eviction failed, this can cause grid hang. 
java.lang.NullPointerException at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:2566)
 at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:2544)
 at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:2122)
 at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:634)
 at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:4391)
 at 
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:652)
 at 
org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1106)
 at 
org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:910)
 at 
org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask.run(PartitionsEvictManager.java:415)
 at 
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6873)
 at 
org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
 at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) 
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
at java.lang.Thread.run(Thread.java:748)
{code}
There errors are caused by the lack of synchronization between partitions 
eviction and deactivation. Test is green despite all the errors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: CacheInterceptor ClassCastException in case of cache was updated from thin java client

I don't think we should complicate things. Leave it as is, use instanceof.
The fact is - you can get anything, BinaryObject or any user class, so be
prepared.
Good example of older API is CacheEvent, which actually has oldValue() and
newValue() as Object.

Igniters, any other thoughts?


On Thu, Jan 24, 2019 at 2:16 PM Sergey Antonov 
wrote:

> Pavel, how about marker interface DeserializedValueCacheInterceptor? We
> will deserialize data and pass it to cache interceptor, if CacheInterceptor
> implements marker interface.
>
> чт, 24 янв. 2019 г. в 13:41, Pavel Tupitsyn :
>
> > You are exactly right, generic parameters don't make much sense here.
> > Ignite caches are not restricted to any type, and there is type erasure
> in
> > Java so you have no runtime guarantees.
> >
> > Maybe Interceptor design should be improved (e.g. add a flag to force
> > binary or non-binary mode),
> > but Thin or Thick client connector logic is unrelated here.
> > withKeepBinary() call is valid and should not depend on Interceptor
> > presence or implementation.
> >
> > On Thu, Jan 24, 2019 at 1:17 PM Sergey Antonov <
> antonovserge...@gmail.com>
> > wrote:
> >
> > > Hi, Pavel,
> > >
> > > "Interceptor should support both modes, binary or not. Any code can
> call
> > > withKeepBinary(), this should be expected.
> > > Just add if (x instanceof BinaryObject) and go from there. "
> > > I don't agree. The cache interceptor[1] is a parametrized class and you
> > > couldn't pass multiple cache interceptors in cache configuration. So
> all
> > > cache interceptors must have Object, Object parameters for supporting
> > both
> > > modes: binary and deserialized. In this case parametrized class no
> sense.
> > >
> > > [1]
> > >
> > >
> >
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheInterceptor.html
> > >
> > > чт, 24 янв. 2019 г. в 13:06, Pavel Tupitsyn :
> > >
> > > > Hi Sergey,
> > > >
> > > > I don't think this is a bug.
> > > >
> > > > Thick or thin clients always work in binary mode on server side,
> > because
> > > > you receive data in serialized form and there is no point in
> > > deserializing
> > > > it.
> > > > Moreover, in most cases you don't have classes on the server, so
> binary
> > > > mode is the only way.
> > > >
> > > > Interceptor should support both modes, binary or not. Any code can
> call
> > > > withKeepBinary(), this should be expected.
> > > > Just add if (x instanceof BinaryObject) and go from there.
> > > >
> > > > Thanks,
> > > > Pavel
> > > >
> > > > On Thu, Jan 24, 2019 at 12:38 PM Sergey Antonov <
> > > antonovserge...@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > I did a little investigation. In
> > > > o.a.i.i.p.p.c.c.ClientCacheRequest#cache()
> > > > > enforced cache with keep binary. Why we should always work binary
> > > > objects?
> > > > >
> > > > > чт, 24 янв. 2019 г. в 12:29, Sergey Antonov <
> > antonovserge...@gmail.com
> > > >:
> > > > >
> > > > > > Hello, Igniters!
> > > > > >
> > > > > > I have ignite node with configured cache. The cache have cache
> > > > > > interceptor. I wiil got ClassCastException on cache interceptor,
> > If I
> > > > put
> > > > > > some entry to the cache (without keepBinary) from thin java
> client.
> > > > > >
> > > > > > I think it's a bug. I'd like to find out yours view!
> > > > > >
> > > > > > Also I made JIRA ticket with reproducer [1].
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10789
> > > > > >
> > > > > > --
> > > > > > BR, Sergey Antonov
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > BR, Sergey Antonov
> > > > >
> > > >
> > >
> > >
> > > --
> > > BR, Sergey Antonov
> > >
> >
>
>
> --
> BR, Sergey Antonov
>

[jira] [Created] (IGNITE-11065) System view for caches.

2019-01-24 Thread Stanilovsky Evgeny (JIRA)

Stanilovsky Evgeny created IGNITE-11065:
---

 Summary: System view for caches.
 Key: IGNITE-11065
 URL: https://issues.apache.org/jira/browse/IGNITE-11065
 Project: Ignite
  Issue Type: Improvement
  Components: documentation
Affects Versions: 2.7
Reporter: Stanilovsky Evgeny
 Fix For: 2.8


All this fields accessible from  _IGNITE.CACHES_ view:

{code:java}
SELECT NAME, CACHE_ID, CACHE_TYPE, GROUP_ID, GROUP_NAME, " +
"CACHE_MODE, ATOMICITY_MODE, IS_ONHEAP_CACHE_ENABLED, 
IS_COPY_ON_READ, IS_LOAD_PREVIOUS_VALUE, " +
"IS_READ_FROM_BACKUP, PARTITION_LOSS_POLICY, NODE_FILTER, 
TOPOLOGY_VALIDATOR, IS_EAGER_TTL, " +
"WRITE_SYNCHRONIZATION_MODE, IS_INVALIDATE, IS_EVENTS_DISABLED, 
IS_STATISTICS_ENABLED, " +
"IS_MANAGEMENT_ENABLED, BACKUPS, AFFINITY, AFFINITY_MAPPER, " +
"REBALANCE_MODE, REBALANCE_BATCH_SIZE, REBALANCE_TIMEOUT, 
REBALANCE_DELAY, REBALANCE_THROTTLE, " +
"REBALANCE_BATCHES_PREFETCH_COUNT, REBALANCE_ORDER, " +
"EVICTION_FILTER, EVICTION_POLICY_FACTORY, " +
"IS_NEAR_CACHE_ENABLED, NEAR_CACHE_EVICTION_POLICY_FACTORY, 
NEAR_CACHE_START_SIZE, " +
"DEFAULT_LOCK_TIMEOUT, CACHE_INTERCEPTOR, CACHE_STORE_FACTORY, 
" +
"IS_STORE_KEEP_BINARY, IS_READ_THROUGH, IS_WRITE_THROUGH, " +
"IS_WRITE_BEHIND_ENABLED, WRITE_BEHIND_COALESCING, 
WRITE_BEHIND_FLUSH_SIZE, " +
"WRITE_BEHIND_FLUSH_FREQUENCY, WRITE_BEHIND_FLUSH_THREAD_COUNT, 
WRITE_BEHIND_FLUSH_BATCH_SIZE, " +
"MAX_CONCURRENT_ASYNC_OPERATIONS, CACHE_LOADER_FACTORY, 
CACHE_WRITER_FACTORY, EXPIRY_POLICY_FACTORY, " +
"IS_SQL_ESCAPE_ALL, SQL_SCHEMA, SQL_INDEX_MAX_INLINE_SIZE, 
IS_SQL_ONHEAP_CACHE_ENABLED, " +
"SQL_ONHEAP_CACHE_MAX_SIZE, QUERY_DETAILS_METRICS_SIZE, 
QUERY_PARALLELISM, MAX_QUERY_ITERATORS_COUNT, " +
"DATA_REGION_NAME FROM IGNITE.CACHES
{code}

appropriate ticket 
[IGNITE-9500|https://issues.apache.org/jira/browse/IGNITE-9500]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11060) Add documentation about CacheInterceptor.

2019-01-24 Thread Sergey Antonov (JIRA)

Sergey Antonov created IGNITE-11060:
---

 Summary: Add documentation about CacheInterceptor.
 Key: IGNITE-11060
 URL: https://issues.apache.org/jira/browse/IGNITE-11060
 Project: Ignite
  Issue Type: Improvement
  Components: documentation
Reporter: Sergey Antonov
Assignee: Artem Budnikov
 Fix For: 2.8


I didn't find documentation about CacheInterceptor in 
[https://apacheignite.readme.io/]

(search request [https://apacheignite.readme.io/v2.7/search?q=interceptor] )

I think we should document this feature.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Ignite index corruption issue -> unrecoverable cluster

Denis, Whan do you think about a more general idea of creating FAQs for
Ignite users?

What if experts will once place their answer in a wiki page and then
develop answers for frequent problems.

And before diving into researching each problem, experienced community
members will ask users to check the FAQ first?

Sincerely,
Dmitriy Pavlov

P.S. here is an article, Apache guides have reference to
http://www.catb.org/~esr/faqs/smart-questions.html - one from required
actions from users is to search for information.

чт, 24 янв. 2019 г. в 01:55, Denis Magda :

> Another data/index corruption issue:
>
> https://stackoverflow.com/questions/54295401/ignite-transaction-failure-not-recoverable-with-persistance
>
> It's suggested to clean index.bin to be able to recover the cluster. Folks,
> let's prepare a list of actions to do if a cluster becomes unrecoverable
> due to data or index corruption issue. What should we do depending on an
> exception:
>
>- Remove index.bin if X or Y or Z
>- etc
>
>
> --
> Denis Magda
>
>
> On Sun, Dec 30, 2018 at 10:06 AM Denis Magda  wrote:
>
> > Ignite SQL and memory experts,
> >
> > The following issue was reported on SO:
> >
> >
> https://stackoverflow.com/questions/53979106/ignite-corruptedtreeexception-leads-to-cluster-failure
> >
> > The stack trace starts with the message below, more details are in that
> > forum:
> >
> > [SEVERE][data-streamer-stripe-2-#15][GridDhtAtomicCache] 
> > Unexpected exception during cache update
> > org.h2.message.DbException: General error: "class
> >
> org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
> > Runtime failure on row: Row@75ab6623[ key: CacheKey [idHash=242632156,
> > hash=-841684964, parentId=-8607237606486310912, hour=9,
> > id=-8607237528489033728, date=2018-09-09 00:00:00.0], val: CacheValue
> > [idHash=843227122, hash=-801894604, 
> >
> > Let's see if it's addressed in the latest release. Also, the user asked a
> > reasonable question - how to recover? Yes, it's possible to use snapshots
> > of GridGain if they are created before but I remember some discussions
> > around a recovery tool.
> >
> > --
> > Denis
> >
>

[jira] [Created] (IGNITE-11061) Сopyright still points out 2018

2019-01-24 Thread Pavel Voronkin (JIRA)

Pavel Voronkin created IGNITE-11061:
---

 Summary: Сopyright still points out 2018
 Key: IGNITE-11061
 URL: https://issues.apache.org/jira/browse/IGNITE-11061
 Project: Ignite
  Issue Type: Bug
Reporter: Pavel Voronkin






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11059) Print information about pending locks queue in case of dht local tx timeout.

2019-01-24 Thread Alexei Scherbakov (JIRA)

Alexei Scherbakov created IGNITE-11059:
--

 Summary: Print information about pending locks queue in case of 
dht local tx timeout.
 Key: IGNITE-11059
 URL: https://issues.apache.org/jira/browse/IGNITE-11059
 Project: Ignite
  Issue Type: Improvement
Reporter: Alexei Scherbakov
 Fix For: 2.8


Currently in case of dht local tx timeout it's hard to understand which keys 
was not locked.

Addtional information should be printed in log on timeout containing 
information about pending keys:

key, tx info holding a lock (xid, label if present)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

SQL View with list of existing indexes

2019-01-24 Thread Юрий

Hi Igniters,

As part of IEP-29: SQL management and monitoring

I'm going to implement SQL view with list of existing indexes.
I've investigate how it expose by ORACLE, MySQL and Postgres.
ORACLE -
https://docs.oracle.com/en/database/oracle/oracle-database/18/refrn/ALL_INDEXES.html#GUID-E39825BA-70AC-45D8-AF30-C7FF561373B6

MySQL - https://dev.mysql.com/doc/refman/8.0/en/show-index.html
Postgres - https://www.postgresql.org/docs/11/view-pg-indexes.html ,
https://www.postgresql.org/docs/11/catalog-pg-index.html

All vendors have such views which show at least following information:
schema name - Name of schema related to table and index.
table name- Name of table related to an index.
index name - Name of index.
list of columns - All columns and their order included into an
index.
collation - ASC or DESC sort for each columns.

+ many specific information which different form vendor to vendor.

In our case such specific information could be at least:

1. Owning cache ID - not sure, but may be
useful to join with other our views.
2. number of columns at the index- just to know how many result
should be in columns view
3. query parallelism - It's configuration
parameter show how many thread can be used to execute query.
4. inline size - inline size
used for this index.
5. is affinity - boolean
parameter show that affinity key index
6. is pk- boolean
parameter show that PK index
7. approx recommended inline size- dynamically calculated
recommended inline size for this index to show required size to keep whole
indexed columns as inlined.

All vendors have different ways to present information about index columns:
PG - use array of index table columns and second array for collation each
of columns.
MySQL - each row in index view contains information about one of indexed
columsn with ther position at the index. So for one index there are many
columns.
ORACLE, - use separate view where each of row present column included into
index with all required information and can be joined by schema, table and
index names.
ORACLE indexed columns view -
https://docs.oracle.com/cd/B19306_01/server.102/b14237/statviews_1064.htm#i1577532
MySql -

I propose use ORACLE way and have second view to represent column included
into indexes.

In this case such view can have the following information:
schema name - Name of schema related to table and index.
table name- Name of table related to an index.
index name - Name of index.
column name- Name of column included into index.
column type - Type of the column.
column position - Position of column within the index.
collation- Either the column is sorted descending or
ascending

And can be joined with index view through schema, table and index names.

What do you think about such approach and list of columns which could be
included into the views?

--
Живи с улыбкой! :D

Re: Review and merge IGNITE-10921

Count on my support if nobody else would like to help with the merge.
Eventually, I will pick up.

ср, 23 янв. 2019 г. в 19:53, Stanislav Lukyanov :

> Hi Igniters,
>
> Can anyone please review and merge IGNITE-10921?
> Artur Muradimov contributed the change, I did a review myself and now we
> need a committer to finish this.
>
> Thanks,
> Stan
>
>

[jira] [Created] (IGNITE-11058) Possible OOM due to large discard queue in TcpDiscoverySpi

2019-01-24 Thread Alexei Scherbakov (JIRA)

Alexei Scherbakov created IGNITE-11058:
--

 Summary: Possible OOM due to large discard queue in TcpDiscoverySpi
 Key: IGNITE-11058
 URL: https://issues.apache.org/jira/browse/IGNITE-11058
 Project: Ignite
  Issue Type: Improvement
Reporter: Alexei Scherbakov
 Fix For: 2.8


Currently it's necessary to store every ensured (marked with 
TcpDiscoveryEnsureDelivery annotation) in pending message queue until it's 
discarded from coordinator for implementing guaranteed delivery, otherwise if 
subsequent nodes will fail while forwarding message the guarantee couldn't be 
fulfilled.

On large topologies with active changes the queue may contain many very large 
messages causing heap usage bursts and possible OOM.

Possible solution:
 # off-load pending messages payload to off-heap or even on disk.
 # store messages in serialized form for avoiding JVM Object overhead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Greeting

2019-01-24 Thread Artur Muradimov

Hi,

My ticket IGNITE-10921 passed review and ready to merge as I understanding.
Can anyone merge that or I miss something?

On Tue, Jan 22, 2019 at 10:27 PM Dmitriy Pavlov  wrote:

> Hi Artur,
>
> You're in,  now you can assign an issue to yourself.
>
> Thank you for interest in Apache Ignite and welcome to the Apache Ignite
> Community.
>
> Following is a couple of links should help during joining to the community
> Short 'How to start' guide
>
> https://github.com/apache/ignite/blob/master/CONTRIBUTING.md#contributing-to-apache-ignite
>
> and full guide version
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
>
> Should you have any questions please do not hesitate to contact Ignite
> developers here.
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 22 янв. 2019 г. в 22:21, Artur Muradimov :
>
> > Community,
> >
> > I am Artur Muradimov and I want to contribute Apache Ignite.
> >
> > I had a time to watch the project and I think, already ready to make some
> > JIRA ticket.
> > I want to start from the issue IGNITE-10921
> >
> > Please grant me contributer privileges to make possible assign.
> > My jira login: arturmuradimov77
> >
>

Re: CacheInterceptor ClassCastException in case of cache was updated from thin java client

Pavel, how about marker interface DeserializedValueCacheInterceptor? We
will deserialize data and pass it to cache interceptor, if CacheInterceptor
implements marker interface.

чт, 24 янв. 2019 г. в 13:41, Pavel Tupitsyn :

> You are exactly right, generic parameters don't make much sense here.
> Ignite caches are not restricted to any type, and there is type erasure in
> Java so you have no runtime guarantees.
>
> Maybe Interceptor design should be improved (e.g. add a flag to force
> binary or non-binary mode),
> but Thin or Thick client connector logic is unrelated here.
> withKeepBinary() call is valid and should not depend on Interceptor
> presence or implementation.
>
> On Thu, Jan 24, 2019 at 1:17 PM Sergey Antonov 
> wrote:
>
> > Hi, Pavel,
> >
> > "Interceptor should support both modes, binary or not. Any code can call
> > withKeepBinary(), this should be expected.
> > Just add if (x instanceof BinaryObject) and go from there. "
> > I don't agree. The cache interceptor[1] is a parametrized class and you
> > couldn't pass multiple cache interceptors in cache configuration. So all
> > cache interceptors must have Object, Object parameters for supporting
> both
> > modes: binary and deserialized. In this case parametrized class no sense.
> >
> > [1]
> >
> >
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheInterceptor.html
> >
> > чт, 24 янв. 2019 г. в 13:06, Pavel Tupitsyn :
> >
> > > Hi Sergey,
> > >
> > > I don't think this is a bug.
> > >
> > > Thick or thin clients always work in binary mode on server side,
> because
> > > you receive data in serialized form and there is no point in
> > deserializing
> > > it.
> > > Moreover, in most cases you don't have classes on the server, so binary
> > > mode is the only way.
> > >
> > > Interceptor should support both modes, binary or not. Any code can call
> > > withKeepBinary(), this should be expected.
> > > Just add if (x instanceof BinaryObject) and go from there.
> > >
> > > Thanks,
> > > Pavel
> > >
> > > On Thu, Jan 24, 2019 at 12:38 PM Sergey Antonov <
> > antonovserge...@gmail.com
> > > >
> > > wrote:
> > >
> > > > I did a little investigation. In
> > > o.a.i.i.p.p.c.c.ClientCacheRequest#cache()
> > > > enforced cache with keep binary. Why we should always work binary
> > > objects?
> > > >
> > > > чт, 24 янв. 2019 г. в 12:29, Sergey Antonov <
> antonovserge...@gmail.com
> > >:
> > > >
> > > > > Hello, Igniters!
> > > > >
> > > > > I have ignite node with configured cache. The cache have cache
> > > > > interceptor. I wiil got ClassCastException on cache interceptor,
> If I
> > > put
> > > > > some entry to the cache (without keepBinary) from thin java client.
> > > > >
> > > > > I think it's a bug. I'd like to find out yours view!
> > > > >
> > > > > Also I made JIRA ticket with reproducer [1].
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10789
> > > > >
> > > > > --
> > > > > BR, Sergey Antonov
> > > > >
> > > >
> > > >
> > > > --
> > > > BR, Sergey Antonov
> > > >
> > >
> >
> >
> > --
> > BR, Sergey Antonov
> >
>


-- 
BR, Sergey Antonov

Re: CacheInterceptor ClassCastException in case of cache was updated from thin java client

Pavel, I agree your's points about working on server side in binary mode,
but we should add ability to proccessing data in deserialized form too. Of
course, it will add performance drop because of serialization round trip
(serialized -> deserialize -> serialized).

One of possible scenario: User wrote application on Clojure, but he want
use Ignite SQL with indexes. For solving problem he created pojo classes
and store them in the cache. The cache interceptor converts java -> clojure
in onGet() method, and clojure -> java in onBeforePut()/onBeforeRemove()
methods.

чт, 24 янв. 2019 г. в 13:16, Sergey Antonov :

> Hi, Pavel,
>
> "Interceptor should support both modes, binary or not. Any code can call
> withKeepBinary(), this should be expected.
> Just add if (x instanceof BinaryObject) and go from there. "
> I don't agree. The cache interceptor[1] is a parametrized class and you
> couldn't pass multiple cache interceptors in cache configuration. So all
> cache interceptors must have Object, Object parameters for supporting both
> modes: binary and deserialized. In this case parametrized class no sense.
>
> [1]
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheInterceptor.html
>
> чт, 24 янв. 2019 г. в 13:06, Pavel Tupitsyn :
>
>> Hi Sergey,
>>
>> I don't think this is a bug.
>>
>> Thick or thin clients always work in binary mode on server side, because
>> you receive data in serialized form and there is no point in deserializing
>> it.
>> Moreover, in most cases you don't have classes on the server, so binary
>> mode is the only way.
>>
>> Interceptor should support both modes, binary or not. Any code can call
>> withKeepBinary(), this should be expected.
>> Just add if (x instanceof BinaryObject) and go from there.
>>
>> Thanks,
>> Pavel
>>
>> On Thu, Jan 24, 2019 at 12:38 PM Sergey Antonov <
>> antonovserge...@gmail.com>
>> wrote:
>>
>> > I did a little investigation. In
>> o.a.i.i.p.p.c.c.ClientCacheRequest#cache()
>> > enforced cache with keep binary. Why we should always work binary
>> objects?
>> >
>> > чт, 24 янв. 2019 г. в 12:29, Sergey Antonov > >:
>> >
>> > > Hello, Igniters!
>> > >
>> > > I have ignite node with configured cache. The cache have cache
>> > > interceptor. I wiil got ClassCastException on cache interceptor, If I
>> put
>> > > some entry to the cache (without keepBinary) from thin java client.
>> > >
>> > > I think it's a bug. I'd like to find out yours view!
>> > >
>> > > Also I made JIRA ticket with reproducer [1].
>> > >
>> > > [1] https://issues.apache.org/jira/browse/IGNITE-10789
>> > >
>> > > --
>> > > BR, Sergey Antonov
>> > >
>> >
>> >
>> > --
>> > BR, Sergey Antonov
>> >
>>
>
>
> --
> BR, Sergey Antonov
>


-- 
BR, Sergey Antonov

Re: CacheInterceptor ClassCastException in case of cache was updated from thin java client

You are exactly right, generic parameters don't make much sense here.
Ignite caches are not restricted to any type, and there is type erasure in
Java so you have no runtime guarantees.

Maybe Interceptor design should be improved (e.g. add a flag to force
binary or non-binary mode),
but Thin or Thick client connector logic is unrelated here.
withKeepBinary() call is valid and should not depend on Interceptor
presence or implementation.

On Thu, Jan 24, 2019 at 1:17 PM Sergey Antonov 
wrote:

> Hi, Pavel,
>
> "Interceptor should support both modes, binary or not. Any code can call
> withKeepBinary(), this should be expected.
> Just add if (x instanceof BinaryObject) and go from there. "
> I don't agree. The cache interceptor[1] is a parametrized class and you
> couldn't pass multiple cache interceptors in cache configuration. So all
> cache interceptors must have Object, Object parameters for supporting both
> modes: binary and deserialized. In this case parametrized class no sense.
>
> [1]
>
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheInterceptor.html
>
> чт, 24 янв. 2019 г. в 13:06, Pavel Tupitsyn :
>
> > Hi Sergey,
> >
> > I don't think this is a bug.
> >
> > Thick or thin clients always work in binary mode on server side, because
> > you receive data in serialized form and there is no point in
> deserializing
> > it.
> > Moreover, in most cases you don't have classes on the server, so binary
> > mode is the only way.
> >
> > Interceptor should support both modes, binary or not. Any code can call
> > withKeepBinary(), this should be expected.
> > Just add if (x instanceof BinaryObject) and go from there.
> >
> > Thanks,
> > Pavel
> >
> > On Thu, Jan 24, 2019 at 12:38 PM Sergey Antonov <
> antonovserge...@gmail.com
> > >
> > wrote:
> >
> > > I did a little investigation. In
> > o.a.i.i.p.p.c.c.ClientCacheRequest#cache()
> > > enforced cache with keep binary. Why we should always work binary
> > objects?
> > >
> > > чт, 24 янв. 2019 г. в 12:29, Sergey Antonov  >:
> > >
> > > > Hello, Igniters!
> > > >
> > > > I have ignite node with configured cache. The cache have cache
> > > > interceptor. I wiil got ClassCastException on cache interceptor, If I
> > put
> > > > some entry to the cache (without keepBinary) from thin java client.
> > > >
> > > > I think it's a bug. I'd like to find out yours view!
> > > >
> > > > Also I made JIRA ticket with reproducer [1].
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-10789
> > > >
> > > > --
> > > > BR, Sergey Antonov
> > > >
> > >
> > >
> > > --
> > > BR, Sergey Antonov
> > >
> >
>
>
> --
> BR, Sergey Antonov
>

Re: CacheInterceptor ClassCastException in case of cache was updated from thin java client

Hi, Pavel,

"Interceptor should support both modes, binary or not. Any code can call
withKeepBinary(), this should be expected.
Just add if (x instanceof BinaryObject) and go from there. "
I don't agree. The cache interceptor[1] is a parametrized class and you
couldn't pass multiple cache interceptors in cache configuration. So all
cache interceptors must have Object, Object parameters for supporting both
modes: binary and deserialized. In this case parametrized class no sense.

[1]
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/CacheInterceptor.html

чт, 24 янв. 2019 г. в 13:06, Pavel Tupitsyn :

> Hi Sergey,
>
> I don't think this is a bug.
>
> Thick or thin clients always work in binary mode on server side, because
> you receive data in serialized form and there is no point in deserializing
> it.
> Moreover, in most cases you don't have classes on the server, so binary
> mode is the only way.
>
> Interceptor should support both modes, binary or not. Any code can call
> withKeepBinary(), this should be expected.
> Just add if (x instanceof BinaryObject) and go from there.
>
> Thanks,
> Pavel
>
> On Thu, Jan 24, 2019 at 12:38 PM Sergey Antonov  >
> wrote:
>
> > I did a little investigation. In
> o.a.i.i.p.p.c.c.ClientCacheRequest#cache()
> > enforced cache with keep binary. Why we should always work binary
> objects?
> >
> > чт, 24 янв. 2019 г. в 12:29, Sergey Antonov :
> >
> > > Hello, Igniters!
> > >
> > > I have ignite node with configured cache. The cache have cache
> > > interceptor. I wiil got ClassCastException on cache interceptor, If I
> put
> > > some entry to the cache (without keepBinary) from thin java client.
> > >
> > > I think it's a bug. I'd like to find out yours view!
> > >
> > > Also I made JIRA ticket with reproducer [1].
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-10789
> > >
> > > --
> > > BR, Sergey Antonov
> > >
> >
> >
> > --
> > BR, Sergey Antonov
> >
>


-- 
BR, Sergey Antonov

Re: CacheInterceptor ClassCastException in case of cache was updated from thin java client

Hi Sergey,

I don't think this is a bug.

Thick or thin clients always work in binary mode on server side, because
you receive data in serialized form and there is no point in deserializing
it.
Moreover, in most cases you don't have classes on the server, so binary
mode is the only way.

Interceptor should support both modes, binary or not. Any code can call
withKeepBinary(), this should be expected.
Just add if (x instanceof BinaryObject) and go from there.

Thanks,
Pavel

On Thu, Jan 24, 2019 at 12:38 PM Sergey Antonov 
wrote:

> I did a little investigation. In o.a.i.i.p.p.c.c.ClientCacheRequest#cache()
> enforced cache with keep binary. Why we should always work binary objects?
>
> чт, 24 янв. 2019 г. в 12:29, Sergey Antonov :
>
> > Hello, Igniters!
> >
> > I have ignite node with configured cache. The cache have cache
> > interceptor. I wiil got ClassCastException on cache interceptor, If I put
> > some entry to the cache (without keepBinary) from thin java client.
> >
> > I think it's a bug. I'd like to find out yours view!
> >
> > Also I made JIRA ticket with reproducer [1].
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-10789
> >
> > --
> > BR, Sergey Antonov
> >
>
>
> --
> BR, Sergey Antonov
>

Re: [MTCGA]: new failures in builds [2872965] needs to be handled

Hi, it is probably related to
https://issues.apache.org/jira/browse/IGNITE-10822 Test was failing before.
I've muted it.

ср, 23 янв. 2019 г. в 12:33, :

> Hi Igniters,
>
>  I've detected some new issue on TeamCity to be handled. You are more than
> welcomed to help.
>
>  If your changes can lead to this failure(s): We're grateful that you were
> a volunteer to make the contribution to this project, but things change and
> you may no longer be able to finalize your contribution.
>  Could you respond to this email and indicate if you wish to continue and
> fix test failures or step down and some committer may revert you commit.
>
>  *New stable failure of a flaky test in master
> SysPropWalDeltaConsistencyTest.testPutRemoveMultinode
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=8000156635398196642=%3Cdefault%3E=testDetails
>  Changes may lead to failure were done by
>  - skosarev
> https://ci.ignite.apache.org/viewModification.html?modId=872921
>  - kondakov87
> https://ci.ignite.apache.org/viewModification.html?modId=872922
>  - y.chief
> https://ci.ignite.apache.org/viewModification.html?modId=872934
>  - zaleslaw.sin
> https://ci.ignite.apache.org/viewModification.html?modId=872912
>
>  - Here's a reminder of what contributors were agreed to do
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
>  - Should you have any questions please contact
> dev@ignite.apache.org
>
> Best Regards,
> Apache Ignite TeamCity Bot
> https://github.com/apache/ignite-teamcity-bot
> Notification generated at 12:32:51 23-01-2019
>

Re: [MTCGA]: new failures in builds [2872963] needs to be handled

Hi, it is an obvious and stable new test failure. This issue is probably
coming from

Roman Kondakov (rkondakov)
3 files
IGNITE-10753 : MVCC: Await all queries are done before checking versions
are cleaned. This closes #5786.
Sergey Kosarev (skosarev)
20 files
IGNITE-6564  Correct calculation of size and keySize for cluster cache
metrics.

Roman, Sergey,

Please chime in.

Sincerely,
Dmitriy Pavlov



ср, 23 янв. 2019 г. в 12:13, :

> Hi Igniters,
>
>  I've detected some new issue on TeamCity to be handled. You are more than
> welcomed to help.
>
>  If your changes can lead to this failure(s): We're grateful that you were
> a volunteer to make the contribution to this project, but things change and
> you may no longer be able to finalize your contribution.
>  Could you respond to this email and indicate if you wish to continue and
> fix test failures or step down and some committer may revert you commit.
>
>  *New stable failure of a flaky test in master
> CacheMetricsEntitiesCountTest.testEnitiesCount
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-9212566675620168727=%3Cdefault%3E=testDetails
>  Changes may lead to failure were done by
>  - skosarev
> https://ci.ignite.apache.org/viewModification.html?modId=872921
>  - kondakov87
> https://ci.ignite.apache.org/viewModification.html?modId=872922
>  - y.chief
> https://ci.ignite.apache.org/viewModification.html?modId=872934
>  - zaleslaw.sin
> https://ci.ignite.apache.org/viewModification.html?modId=872912
>
>  - Here's a reminder of what contributors were agreed to do
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
>  - Should you have any questions please contact
> dev@ignite.apache.org
>
> Best Regards,
> Apache Ignite TeamCity Bot
> https://github.com/apache/ignite-teamcity-bot
> Notification generated at 12:02:49 23-01-2019
>

Re: CacheInterceptor ClassCastException in case of cache was updated from thin java client

I did a little investigation. In o.a.i.i.p.p.c.c.ClientCacheRequest#cache()
enforced cache with keep binary. Why we should always work binary objects?

чт, 24 янв. 2019 г. в 12:29, Sergey Antonov :

> Hello, Igniters!
>
> I have ignite node with configured cache. The cache have cache
> interceptor. I wiil got ClassCastException on cache interceptor, If I put
> some entry to the cache (without keepBinary) from thin java client.
>
> I think it's a bug. I'd like to find out yours view!
>
> Also I made JIRA ticket with reproducer [1].
>
> [1] https://issues.apache.org/jira/browse/IGNITE-10789
>
> --
> BR, Sergey Antonov
>


-- 
BR, Sergey Antonov

CacheInterceptor ClassCastException in case of cache was updated from thin java client

Hello, Igniters!

I have ignite node with configured cache. The cache have cache interceptor.
I wiil got ClassCastException on cache interceptor, If I put some entry to
the cache (without keepBinary) from thin java client.

I think it's a bug. I'd like to find out yours view!

Also I made JIRA ticket with reproducer [1].

[1] https://issues.apache.org/jira/browse/IGNITE-10789

-- 
BR, Sergey Antonov

Re: [DISCUSSION] Control.sh global rework in apache ignite 3.0