Re: Apache Ignite RPM packaging (Stage II)

[Dmitriy Setrakyan]

Petr,

I am confused. Do we already have Debian packages?

D.

On Tue, Mar 27, 2018 at 5:10 AM, Petr Ivanov  wrote:

> Hi, Igniters!
>
>
> Here are some news on our RPM packages initiative.
>
> 1. I’ve finished preliminary developing of Stage II version of RPM
> packages [1]. Main “new feature” is — split design. Also I’ve added
> package.sh script for automating package building process which will help
> organise corresponding builds in TC as well as simplify process for
> developers who wishes to have custom packages.
> PR#3703 [2] is ready for review. Denis, in order to catch up with Apache
> Ignite 2.5 release, I’d greatly appreciate your help in finding reviewer.
> 2. With the help of ASF INFRA team, we now have RPM [3] and DEB [4]
> repositories on Apache Bintray. Though they are already prepared for
> hosting RPM and DEB packages respectively, and there is a way of linking
> them to apache.org/dist/ignite page, there is possible alternative in
> storing there only plain directory layout corresponding to each repository
> type (RPM and DEB) and manage this layout (repodata, distributions,
> versions, etc.) by ourselves, having more control over repositories but
> lacking some simplicity of deploying new releases. WDYT? Should we try
> Cassandra approach? They are storing their DEB packages as I described
> above [5].
>
> Also — a question arose while I was working on this issue: which OSes (and
> which versions of each) are we going to support (if we are going) in terms
> of step-by-step list? Currently RPM packages are tested only with latest
> CentOS (and, respectively — RHEL), but there are a lot more RPM-based
> distributives [6] some of which are more o less popular among OS community
> (ALT, Fedora, openSUSE, etc.).
>
>
> [1] https://issues.apache.org/jira/browse/IGNITE-7647
> [2] https://github.com/apache/ignite/pull/3703
> [3] https://bintray.com/apache/ignite-rpm
> [4] https://bintray.com/apache/ignite-deb
> [5] https://bintray.com/apache/cassandra/debian#files/
> [6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_distributions
>
>
>
>
> > On 15 Mar 2018, at 22:15, Petr Ivanov  wrote:
> >
> > I suppose that most everything if not all from libs/options will go to
> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
> > More precise lib selection (if something from optional would better to
> have in core package) will be discussed right after preliminary split
> architecture agreement.
> >
> >
> >
> >> On 15 Mar 2018, at 22:11, Dmitry Pavlov  wrote:
> >>
> >> I like idea of keeping simple system of modules, so +1 from me.
> >>
> >> Where optional libs (e.g Direct IO plugin) would be included, would it
> be
> >> core or optional?
> >>
> >> чт, 15 мар. 2018 г. в 22:09, Denis Magda :
> >>
> 
> >
> > How big would be a final core module?
>  Around 30M. Can be shrinked to ~15M if separate Visor and create it’s
> own
>  package.
> >>>
> >>>
> >>> Guys, 30 vs 280M is a hge difference.  I would agree with Petr and
> >>> propose the simplest modular system:
> >>>
> >>>  - core module that includes basic Ignite capabilities including SQL,
> >>>  compute grid, service grid, k/v
> >>>  - optional module hosts the rest - ML, streamers integration (kafka,
> >>>  flink), kubernetes, etc.
> >>>
> >>> What do you think?
> >>>
> >>> --
> >>> Denis
> >>>
> >>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov 
> wrote:
> >>>
>  *DEB package
> 
> 
> > On 15 Mar 2018, at 10:35, Petr Ivanov  wrote:
> >
> > Considering that DEV package for now is almost platform independent
> >>> (its
>  a java application more or less), that package will work almost on any
>  DEB-based linux, including but not limited to Ubuntu, Debian, etc.
> > The only restriction is existence of systemctl (systemd) service
> >>> manager
>  — we are dependent on it.
> >
> > Thats why, for instance, our RPM repository is called simply ‘rpm’
> and
>  package has no arch or dist suffix — it will work on CentOS, RHEL,
> >>> Fedora,
>  etc. with presence of aforementioned systemd.
> >
> >
> >
> >> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan 
>  wrote:
> >>
> >> Will Debian package work for Ubuntu?
> >>
> >> D.
> >>
> >> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov 
>  wrote:
> >>
> >>> Not a problem, rather nuisance. Also, when we will move to official
> >>> repositories, there can be a problem from OS community.
> >>>
> >>> Concerning DEB packages — I plan to use RPM as base for DEB package
>  build
> >>> (package layout / install scripts) for speeding up things and
> >>> excluding
> >>> possible duplication and desynchronisation, so its a matter of ’sit
>  and do’
> >>> rather then some technical research. Thats 

Re: MTCGA: Hot failures

[Dmitriy Setrakyan]

Dmitriy,

I think it would make sense to create a ticket for every single one of
these, or at least try to group them in to several tickets. What do you
think?

D.

On Tue, Mar 27, 2018 at 5:17 AM, Dmitry Pavlov 
wrote:

> Hi Igniters,
>
> I’ve filtered report of master failures to find most failing tests.
> Following is 5 failing suites and their frequently failed test with links
> to TC.
>
> Igniters, please reply who would like to pick up
>
> - test failures context research,
>
> - tickets creation
>
> - and (the best option) fixing these fails.
>
> Sincerely,
>
> Dmitriy Pavlov
>
>   Basic [2]
>  IgniteTests24Java8_Basic2=%3Cdefault%3E=buildTypeStatusDiv>
>
> IgniteServiceConfigVariationsFullApiTestSuite:
> IgniteServiceConfigVariationsFullApiTest.testDeploy (master fail rate
> 23,1%)
>  IgniteTests24Java8=-6001195444279699378=%
> 3Cdefault%3E=testDetails>
>
> IgniteServiceConfigVariationsFullApiTestSuite:
> IgniteServiceConfigVariationsFullApiTest.testClusterSingletonDeploy
> (master
> fail rate 20,8%)
>  IgniteTests24Java8=-7595768653681126954=%
> 3Cdefault%3E=testDetails>
>
> IgniteServiceConfigVariationsFullApiTestSuite:
> IgniteServiceConfigVariationsFullApiTest.testKeyAffinityDeploy (master
> fail
> rate 20,8%)
>  IgniteTests24Java8=1677053649107278613=%
> 3Cdefault%3E=testDetails>
>
> IgniteServiceConfigVariationsFullApiTestSuite:
> IgniteServiceConfigVariationsFullApiTest.testMultipleDeploy (master fail
> rate 17,5%)
>  IgniteTests24Java8=-7956361514936000460=%
> 3Cdefault%3E=testDetails>
>
> IgniteServiceConfigVariationsFullApiTestSuite:
> IgniteServiceConfigVariationsFullApiTest.testNodeSingletonDeploy (master
> fail rate 16,7%)
>  IgniteTests24Java8=1909279554207447487=%
> 3Cdefault%3E=testDetails>
>
>
>
>Ignite Start Nodes
>  IgniteTests24Java8_IgniteStartNodes=%3Cdefault%3E=
> buildTypeStatusDiv>
>
>  IgniteStartStopRestartTestSuite:
> IgniteProjectionStartStopRestartSelfTest.testStartFiveNodesInTwoCalls
> (master fail rate 12,1%)
>  IgniteTests24Java8=6814497542781613621=%
> 3Cdefault%3E=testDetails>
>
>  IgniteStartStopRestartTestSuite:
> IgniteProjectionStartStopRestartSelfTest.testStopNodes (master fail rate
> 10,1%)
>  IgniteTests24Java8=1179745277331816127=%
> 3Cdefault%3E=testDetails>
>
>  IgniteStartStopRestartTestSuite:
> IgniteProjectionStartStopRestartSelfTest.testStopNodesByIds (master fail
> rate 10,1%)
>  IgniteTests24Java8=-842929918974855010=%
> 3Cdefault%3E=testDetails>
>
>   Binary Objects (Simple Mapper Basic)
>  IgniteTests24Java8_BinaryObjectsSimpleMapperBasic
> =%3Cdefault%3E=buildTypeStatusDiv>
>
> IgniteBinarySimpleNameMapperBasicTestSuite:
> GridMarshallerMappingConsistencyTest.testPersistedMappingsSharedOnJoin
> (master fail rate 14,3%)
>  IgniteTests24Java8=408332895682007527=%
> 3Cdefault%3E=testDetails>
>
> Ignite Data Structures
>  IgniteTests24Java8_IgniteDataStructures=%3Cdefault%3E=
> buildTypeStatusDiv>
>
>
>IgniteCacheDataStructuresSelfTestSuite:
> IgnitePartitionedCountDownLatchSelfTest.testLatchMultinode1 (master fail
> rate 12,4%)
>  IgniteTests24Java8=677264269171099154=%
> 3Cdefault%3E=testDetails>
>
>IgniteCacheDataStructuresSelfTestSuite:
> IgniteClientDataStructuresTest.testSequence (master fail rate 11,9%)
>  IgniteTests24Java8=8124188160733602720=%
> 3Cdefault%3E=testDetails>
>
>IgniteCacheDataStructuresSelfTestSuite:
> IgniteClientDataStructuresTest.testReentrantLock (master fail rate 10,3%)
>  IgniteTests24Java8=-4280806541597260987=%
> 3Cdefault%3E=testDetails>
>
> Activate | Deactivate Cluster
>  IgniteTests24Java8_ActivateDeactivateCluster=%3Cdefault%3E=
> buildTypeStatusDiv>
>
>
> IgniteStandByClusterSuite:
> CacheBaselineTopologyTest.testPrimaryLeftAndClusterRestart (master fail
> rate 37,1%)
>  IgniteTests24Java8=6798733272445954906=%
> 3Cdefault%3E=testDetails>
>
> IgniteStandByClusterSuite:
> CacheBaselineTopologyTest.testBaselineTopologyChangesFromClient (master
> 

Re: Server stores cache data on-heap if client has near cache - IGNITE-4662

[Dmitry Pavlov]

Its kind of improvement. I'm not sure if it can reduce overall used memory,
but probably will decrease GC pauses.

ср, 28 мар. 2018 г. в 0:09, Vyacheslav Daradur :

> Aleksey, thank you, it's clear now.
>
> If this task is really about moving reader's information to off-heap:
> do we have issues with the current implementation (e.g. at restarting)
> or this task is kind of improvement?
>
> On Tue, Mar 27, 2018 at 11:32 PM, ALEKSEY KUZNETSOV
>  wrote:
> > When client node performs get it stores information about itself on
> server node, so-called 'reader'.
> > If another node updates cache entry, server node uses 'reader'
> info(because it tracks client info) to find client and update cache entry
> in client's near cache.
> >
> > PS. obviously, there may be plenty 'reader' nodes
> >
> >
> > Отправлено с iPhone
> >
> >> 27 марта 2018 г., в 22:41, Vyacheslav Daradur 
> написал(а):
> >>
> >> Dmitry, thanks, but I'm not sure that I correctly understood you.
> >>
>  Every get operation is tracked and enlisted on server node.
> >>
> >> What does it mean? Is it about metrics?
> >>
> >>> On Tue, Mar 27, 2018 at 6:29 PM, Dmitry Pavlov 
> wrote:
> >>> Hi Vyacheslav,
> >>>
> >>>
> >>> Task is still actual. It is another question if it is a priority or
> not.
> >>>
> >>>
> >>> This is a task is about following case: a client node has near cache.
> Every
> >>> get operation is tracked and enlisted on server node. The server node
> must
> >>> trace the operation and accordingly store the results to Java Heap
> Memory.
> >>> Currently it is on-heap, and task is about idea to make it off-heap.
> >>>
> >>>
> >>> Sincerely,
> >>>
> >>> Dmitriy Pavlov
> >>>
> >>> пт, 23 мар. 2018 г. в 13:13, Dmitry Pavlov :
> >>>
>  Hi Igniters,
> 
>  Alexey, please step in. I did not fully understand issue description,
> and
>  can't identify if data is still not moved from onheap.
> 
>  I suggest to consider it as outdated if we don't have any additional
> info.
> 
>  Sincerely,
>  Dmitriy Pavlov
> 
>  чт, 22 мар. 2018 г. в 10:48, Vyacheslav Daradur  >:
> 
> > Hi, Igniters!
> >
> > Look like it's outdated and can be closed.
> >
> > Could someone comment if this ticket [1] is still valid?
> >
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-4662
> >
> > On Wed, Jun 28, 2017 at 2:38 AM, Valentin Kulichenko
> >  wrote:
> >> I'm not sure this ticket is valid for 2.0. Semen, can you comment?
> >>
> >> -Val
> >>
> >> On Tue, Jun 27, 2017 at 1:14 AM, Vyacheslav Daradur <
> > daradu...@gmail.com>
> >> wrote:
> >>
> >>> Hi Igniters.
> >>>
> >>> I have some questions according to this task:
> >>>
> >>> 1. Does the method: GridCacheMapEntry#evictInternal do the
> >>> eviction(on-heap
> >>> -> off-heap)?
> >>> 2. Is CacheOffheapEvictionManager responsible for managing the
> >>> eviction(on-heap -> off-heap)? (if not, then who is?)
> >>> 3. At what moment the eviction(on-heap -> off-heap) is called?
> >>>
> >>>
> >>> --
> >>> Best Regards, Vyacheslav D.
> >
> >
> >
> > --
> > Best Regards, Vyacheslav D.
> >>
> >>
> >>
> >> --
> >> Best Regards, Vyacheslav D.
>
>
>
> --
> Best Regards, Vyacheslav D.
>

Re: Server stores cache data on-heap if client has near cache - IGNITE-4662

[Dmitry Pavlov]

Hi Alexey, thank you for stepping in and explaining.

вт, 27 мар. 2018 г. в 23:32, ALEKSEY KUZNETSOV :

> When client node performs get it stores information about itself on server
> node, so-called 'reader'.
> If another node updates cache entry, server node uses 'reader'
> info(because it tracks client info) to find client and update cache entry
> in client's near cache.
>
> PS. obviously, there may be plenty 'reader' nodes
>
>
> Отправлено с iPhone
>
> > 27 марта 2018 г., в 22:41, Vyacheslav Daradur 
> написал(а):
> >
> > Dmitry, thanks, but I'm not sure that I correctly understood you.
> >
> >>> Every get operation is tracked and enlisted on server node.
> >
> > What does it mean? Is it about metrics?
> >
> >> On Tue, Mar 27, 2018 at 6:29 PM, Dmitry Pavlov 
> wrote:
> >> Hi Vyacheslav,
> >>
> >>
> >> Task is still actual. It is another question if it is a priority or not.
> >>
> >>
> >> This is a task is about following case: a client node has near cache.
> Every
> >> get operation is tracked and enlisted on server node. The server node
> must
> >> trace the operation and accordingly store the results to Java Heap
> Memory.
> >> Currently it is on-heap, and task is about idea to make it off-heap.
> >>
> >>
> >> Sincerely,
> >>
> >> Dmitriy Pavlov
> >>
> >> пт, 23 мар. 2018 г. в 13:13, Dmitry Pavlov :
> >>
> >>> Hi Igniters,
> >>>
> >>> Alexey, please step in. I did not fully understand issue description,
> and
> >>> can't identify if data is still not moved from onheap.
> >>>
> >>> I suggest to consider it as outdated if we don't have any additional
> info.
> >>>
> >>> Sincerely,
> >>> Dmitriy Pavlov
> >>>
> >>> чт, 22 мар. 2018 г. в 10:48, Vyacheslav Daradur :
> >>>
>  Hi, Igniters!
> 
>  Look like it's outdated and can be closed.
> 
>  Could someone comment if this ticket [1] is still valid?
> 
> 
>  [1] https://issues.apache.org/jira/browse/IGNITE-4662
> 
>  On Wed, Jun 28, 2017 at 2:38 AM, Valentin Kulichenko
>   wrote:
> > I'm not sure this ticket is valid for 2.0. Semen, can you comment?
> >
> > -Val
> >
> > On Tue, Jun 27, 2017 at 1:14 AM, Vyacheslav Daradur <
>  daradu...@gmail.com>
> > wrote:
> >
> >> Hi Igniters.
> >>
> >> I have some questions according to this task:
> >>
> >> 1. Does the method: GridCacheMapEntry#evictInternal do the
> >> eviction(on-heap
> >> -> off-heap)?
> >> 2. Is CacheOffheapEvictionManager responsible for managing the
> >> eviction(on-heap -> off-heap)? (if not, then who is?)
> >> 3. At what moment the eviction(on-heap -> off-heap) is called?
> >>
> >>
> >> --
> >> Best Regards, Vyacheslav D.
> 
> 
> 
>  --
>  Best Regards, Vyacheslav D.
> >
> >
> >
> > --
> > Best Regards, Vyacheslav D.
>

Re: Server stores cache data on-heap if client has near cache - IGNITE-4662

[Vyacheslav Daradur]

Aleksey, thank you, it's clear now.

If this task is really about moving reader's information to off-heap:
do we have issues with the current implementation (e.g. at restarting)
or this task is kind of improvement?

On Tue, Mar 27, 2018 at 11:32 PM, ALEKSEY KUZNETSOV
 wrote:
> When client node performs get it stores information about itself on server 
> node, so-called 'reader'.
> If another node updates cache entry, server node uses 'reader' info(because 
> it tracks client info) to find client and update cache entry in client's near 
> cache.
>
> PS. obviously, there may be plenty 'reader' nodes
>
>
> Отправлено с iPhone
>
>> 27 марта 2018 г., в 22:41, Vyacheslav Daradur  
>> написал(а):
>>
>> Dmitry, thanks, but I'm not sure that I correctly understood you.
>>
 Every get operation is tracked and enlisted on server node.
>>
>> What does it mean? Is it about metrics?
>>
>>> On Tue, Mar 27, 2018 at 6:29 PM, Dmitry Pavlov  
>>> wrote:
>>> Hi Vyacheslav,
>>>
>>>
>>> Task is still actual. It is another question if it is a priority or not.
>>>
>>>
>>> This is a task is about following case: a client node has near cache. Every
>>> get operation is tracked and enlisted on server node. The server node must
>>> trace the operation and accordingly store the results to Java Heap Memory.
>>> Currently it is on-heap, and task is about idea to make it off-heap.
>>>
>>>
>>> Sincerely,
>>>
>>> Dmitriy Pavlov
>>>
>>> пт, 23 мар. 2018 г. в 13:13, Dmitry Pavlov :
>>>
 Hi Igniters,

 Alexey, please step in. I did not fully understand issue description, and
 can't identify if data is still not moved from onheap.

 I suggest to consider it as outdated if we don't have any additional info.

 Sincerely,
 Dmitriy Pavlov

 чт, 22 мар. 2018 г. в 10:48, Vyacheslav Daradur :

> Hi, Igniters!
>
> Look like it's outdated and can be closed.
>
> Could someone comment if this ticket [1] is still valid?
>
>
> [1] https://issues.apache.org/jira/browse/IGNITE-4662
>
> On Wed, Jun 28, 2017 at 2:38 AM, Valentin Kulichenko
>  wrote:
>> I'm not sure this ticket is valid for 2.0. Semen, can you comment?
>>
>> -Val
>>
>> On Tue, Jun 27, 2017 at 1:14 AM, Vyacheslav Daradur <
> daradu...@gmail.com>
>> wrote:
>>
>>> Hi Igniters.
>>>
>>> I have some questions according to this task:
>>>
>>> 1. Does the method: GridCacheMapEntry#evictInternal do the
>>> eviction(on-heap
>>> -> off-heap)?
>>> 2. Is CacheOffheapEvictionManager responsible for managing the
>>> eviction(on-heap -> off-heap)? (if not, then who is?)
>>> 3. At what moment the eviction(on-heap -> off-heap) is called?
>>>
>>>
>>> --
>>> Best Regards, Vyacheslav D.
>
>
>
> --
> Best Regards, Vyacheslav D.
>>
>>
>>
>> --
>> Best Regards, Vyacheslav D.



-- 
Best Regards, Vyacheslav D.

Re: Server stores cache data on-heap if client has near cache - IGNITE-4662

[ALEKSEY KUZNETSOV]

When client node performs get it stores information about itself on server 
node, so-called 'reader'.
If another node updates cache entry, server node uses 'reader' info(because it 
tracks client info) to find client and update cache entry in client's near 
cache.

PS. obviously, there may be plenty 'reader' nodes


Отправлено с iPhone

> 27 марта 2018 г., в 22:41, Vyacheslav Daradur  
> написал(а):
> 
> Dmitry, thanks, but I'm not sure that I correctly understood you.
> 
>>> Every get operation is tracked and enlisted on server node.
> 
> What does it mean? Is it about metrics?
> 
>> On Tue, Mar 27, 2018 at 6:29 PM, Dmitry Pavlov  wrote:
>> Hi Vyacheslav,
>> 
>> 
>> Task is still actual. It is another question if it is a priority or not.
>> 
>> 
>> This is a task is about following case: a client node has near cache. Every
>> get operation is tracked and enlisted on server node. The server node must
>> trace the operation and accordingly store the results to Java Heap Memory.
>> Currently it is on-heap, and task is about idea to make it off-heap.
>> 
>> 
>> Sincerely,
>> 
>> Dmitriy Pavlov
>> 
>> пт, 23 мар. 2018 г. в 13:13, Dmitry Pavlov :
>> 
>>> Hi Igniters,
>>> 
>>> Alexey, please step in. I did not fully understand issue description, and
>>> can't identify if data is still not moved from onheap.
>>> 
>>> I suggest to consider it as outdated if we don't have any additional info.
>>> 
>>> Sincerely,
>>> Dmitriy Pavlov
>>> 
>>> чт, 22 мар. 2018 г. в 10:48, Vyacheslav Daradur :
>>> 
 Hi, Igniters!
 
 Look like it's outdated and can be closed.
 
 Could someone comment if this ticket [1] is still valid?
 
 
 [1] https://issues.apache.org/jira/browse/IGNITE-4662
 
 On Wed, Jun 28, 2017 at 2:38 AM, Valentin Kulichenko
  wrote:
> I'm not sure this ticket is valid for 2.0. Semen, can you comment?
> 
> -Val
> 
> On Tue, Jun 27, 2017 at 1:14 AM, Vyacheslav Daradur <
 daradu...@gmail.com>
> wrote:
> 
>> Hi Igniters.
>> 
>> I have some questions according to this task:
>> 
>> 1. Does the method: GridCacheMapEntry#evictInternal do the
>> eviction(on-heap
>> -> off-heap)?
>> 2. Is CacheOffheapEvictionManager responsible for managing the
>> eviction(on-heap -> off-heap)? (if not, then who is?)
>> 3. At what moment the eviction(on-heap -> off-heap) is called?
>> 
>> 
>> --
>> Best Regards, Vyacheslav D.
 
 
 
 --
 Best Regards, Vyacheslav D.
> 
> 
> 
> -- 
> Best Regards, Vyacheslav D.

Re: Service grid redesign

[Vyacheslav Daradur]

Hi, Denis Mekhanikov!

As far as I know, Ignite services are based on IgniteCache and we have
all its features. We can use listeners or continuous queries for
deployment synchronizations.

Why do you want using the discovery layer for that?

One more thing: we can use baseline approach for services, that means
*IgniteService.deploy()* returns ready to work service after
deployment on baseline nodes and deploy to other nodes on demand, for
example when deployed service's loading will be hight.

About versioning, maybe there is sense to extend public API:
IgniteServices.service(name, *version*)?

At first deployment, we can compute service's hashcode (just for an
example) and store it, after new deployment request for services with
an existing name we will compute new service's hashcode and compare
them if they have different hashcodes that we will deploy new service
as service with a different version.


On Fri, Mar 23, 2018 at 10:03 PM, Denis Magda  wrote:
> Denis,
>
> Thanks for the extensive analysis. There is a vast room for optimizations
> on the service grid side.
>
> Yakov, Sam, Alex G.,
>
> How do you like the idea of the usage of discovery protocol for the service
> grid system messages exchange? Any pitfalls?
>
>
> --
> Denis
>
>
> On Fri, Mar 23, 2018 at 8:01 AM, Denis Mekhanikov 
> wrote:
>
>> Igniters,
>>
>> I'd like to start a discussion on Ignite service grid redesign.
>> We have a number of problems in our current architecture, that have to be
>> addressed.
>>
>> Here are the most severe ones:
>>
>> One of them is lack of guarantee, that service is successfully deployed and
>> ready for work by the time, when *IgniteService.deploy*()* methods return.
>> Furthermore, if an exception is thrown from *Service.init() *method, then
>> the deploying side is not able to receive it, or even understand, that
>> service is in unusable state.
>> So, you may end up in such situation, when you deployed a service without
>> receiving any errors, then called a service's method, and hung indefinitely
>> on this invocation.
>> JIRA ticket: https://issues.apache.org/jira/browse/IGNITE-3392
>>
>> Another problem is locking during service deployment on unstable topology.
>> This issue is caused by missing updates in continuous query listeners on
>> the internal cache.
>> It is hard to reproduce, but it happens sometimes. We shouldn't allow such
>> possibility, that deployment methods hang without saying anything.
>> JIRA ticket: https://issues.apache.org/jira/browse/IGNITE-6259
>>
>> I think, we should change the deployment procedure to make it more
>> reliable.
>> Moving from operating over internal replicated service cache to sending
>> custom discovery events seems to be a good idea.
>> Service deployment may trigger a discovery event, that will make chosen
>> nodes deploy the service, and the same event will notify other nodes about
>> the deployed service instances.
>> It will eliminate the need for distributed transactions on the internal
>> replicated system cache, and make the service deployment protocol more
>> transparent.
>>
>> There are a few points, that should be taken into account though.
>>
>> First of all, we can't wait for services to be deployed and initialised in
>> the discovery thread.
>> So, we need to make notification about service deployment result
>> asynchronous, presumably over communication protocol.
>> I can think of a procedure similar to the current exchange protocol, when
>> service deployment is initialised with an initial discovery message,
>> followed by asynchronous notifications from the hosting servers over
>> communication. And finally, one more discovery message will notify all
>> nodes about the service deployment result and location of the deployed
>> service instances. Coordinator will be responsible for collecting of the
>> deployment results in this scheme.
>>
>> Another problem is failover in case, when some nodes fail during deployment
>> or further work.
>> The following cases should be handled:
>>
>>1. coordinator failure during deployment;
>>2. failure of nodes, that were chosen to host the service, during
>>deployment;
>>3. failure of nodes, that contain deployed services, after the
>>deployment.
>>
>> The first case may be resolved by either continuation of deployment with a
>> new coordinator, or by cancelling it.
>> The second case will require another node to be chosen and notified. Maybe
>> another discovery message will be needed.
>> The third case will require redeployment, so coordinator should track
>> topology changes and redeploy failed services.
>>
>> Another good improvement would be service versioning. This matter was
>> already discussed in another thread:
>> http://apache-ignite-developers.2346864.n4.nabble.com/Service-versioning-
>> td20858.html
>> Let's resume this discussion and state the final decision here.
>> This feature is closely connected to peer class loading, which is 

Re: Server stores cache data on-heap if client has near cache - IGNITE-4662

[Vyacheslav Daradur]

Dmitry, thanks, but I'm not sure that I correctly understood you.

>> Every get operation is tracked and enlisted on server node.

What does it mean? Is it about metrics?

On Tue, Mar 27, 2018 at 6:29 PM, Dmitry Pavlov  wrote:
> Hi Vyacheslav,
>
>
> Task is still actual. It is another question if it is a priority or not.
>
>
> This is a task is about following case: a client node has near cache. Every
> get operation is tracked and enlisted on server node. The server node must
> trace the operation and accordingly store the results to Java Heap Memory.
> Currently it is on-heap, and task is about idea to make it off-heap.
>
>
> Sincerely,
>
> Dmitriy Pavlov
>
> пт, 23 мар. 2018 г. в 13:13, Dmitry Pavlov :
>
>> Hi Igniters,
>>
>> Alexey, please step in. I did not fully understand issue description, and
>> can't identify if data is still not moved from onheap.
>>
>> I suggest to consider it as outdated if we don't have any additional info.
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>> чт, 22 мар. 2018 г. в 10:48, Vyacheslav Daradur :
>>
>>> Hi, Igniters!
>>>
>>> Look like it's outdated and can be closed.
>>>
>>> Could someone comment if this ticket [1] is still valid?
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/IGNITE-4662
>>>
>>> On Wed, Jun 28, 2017 at 2:38 AM, Valentin Kulichenko
>>>  wrote:
>>> > I'm not sure this ticket is valid for 2.0. Semen, can you comment?
>>> >
>>> > -Val
>>> >
>>> > On Tue, Jun 27, 2017 at 1:14 AM, Vyacheslav Daradur <
>>> daradu...@gmail.com>
>>> > wrote:
>>> >
>>> >> Hi Igniters.
>>> >>
>>> >> I have some questions according to this task:
>>> >>
>>> >> 1. Does the method: GridCacheMapEntry#evictInternal do the
>>> >> eviction(on-heap
>>> >> -> off-heap)?
>>> >> 2. Is CacheOffheapEvictionManager responsible for managing the
>>> >> eviction(on-heap -> off-heap)? (if not, then who is?)
>>> >> 3. At what moment the eviction(on-heap -> off-heap) is called?
>>> >>
>>> >>
>>> >> --
>>> >> Best Regards, Vyacheslav D.
>>> >>
>>>
>>>
>>>
>>> --
>>> Best Regards, Vyacheslav D.
>>>
>>



-- 
Best Regards, Vyacheslav D.

[GitHub] ignite pull request #3709: -

[daradurvs]

GitHub user daradurvs opened a pull request:

https://github.com/apache/ignite/pull/3709

-



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/daradurvs/ignite ignite-5979

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3709.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3709






---

[GitHub] ignite pull request #3708: IGNITE-4193: Added transaction support for ODBC

[isapego]

GitHub user isapego opened a pull request:

https://github.com/apache/ignite/pull/3708

IGNITE-4193: Added transaction support for ODBC



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4193

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3708.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3708


commit 3155f6bc8884f61d786dc4ef24c4925fd10fc8c5
Author: sboikov 
Date:   2017-10-25T09:39:23Z

Merge remote-tracking branch 'remotes/origin/master' into ignite-3478

# Conflicts:
#   modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/atomic/GridDhtAtomicCache.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/RowStore.java
#   
modules/yardstick/src/main/java/org/apache/ignite/yardstick/IgniteBenchmarkArguments.java

commit 00bd4794ab33725d9bca22eaf1b29bb3f0e71c2b
Author: sboikov 
Date:   2017-10-25T09:40:14Z

Merge remote-tracking branch 'remotes/origin/master' into ignite-3478

# Conflicts:
#   modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/atomic/GridDhtAtomicCache.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/RowStore.java
#   
modules/yardstick/src/main/java/org/apache/ignite/yardstick/IgniteBenchmarkArguments.java

commit 6150f3a0ad310810606ec5bafbd007804808ff25
Author: sboikov 
Date:   2017-10-25T12:15:56Z

ignite-3478 Mvcc support for sql indexes

commit 2a2a2c4593bdf1708894ccbe5031708a60b097e7
Author: sboikov 
Date:   2017-10-26T08:15:34Z

Merge remote-tracking branch 'remotes/origin/master' into ignite-3478

commit 980517fb1a905447471b53a8fdae1eea46331c7d
Author: sboikov 
Date:   2017-10-26T10:11:02Z

ignite-3478 Tests with persistence enabled.

commit 987a57e3371c4f82fc51706539746fcd2736a034
Author: sboikov 
Date:   2017-10-26T13:07:18Z

ignite-3478 Support for cache groups.

commit e37dfa3b9a96d13ba38dc4c564ebeb4d0bccefa1
Author: sboikov 
Date:   2017-10-27T07:53:33Z

Merge remote-tracking branch 'remotes/origin/master' into ignite-3478

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/managers/discovery/GridDiscoveryManager.java

commit 39f7ae974a222b63c6a91b1df6cf669266a3e911
Author: sboikov 
Date:   2017-10-27T07:58:14Z

Merge remote-tracking branch 'remotes/origin/master' into ignite-3478

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/managers/discovery/GridDiscoveryManager.java

commit 68b61f6f59279f2ea01f152018a33b0f878f75b4
Author: sboikov 
Date:   2017-10-27T08:22:20Z

ignite-3478 Added missed serialVersionUID

commit f8c5cc5dcce9c2219b68c923221e901cae35733b
Author: sboikov 
Date:   2017-10-27T08:47:24Z

ignite-3478 Fixed compilation

commit b04849ea6e1313b0e9d8b2646b08914f8cfa3a7b
Author: sboikov 
Date:   2017-10-27T10:40:26Z

ignite-3478 Fix tests

commit d46a039506be32c1b472903525392fd3500e9c13
Author: sboikov 
Date:   2017-10-27T11:52:13Z

ignite-3478 Fix tests

commit 6466adf54f017219a16cf9835ee0214853db269e
Author: sboikov 
Date:   2017-10-27T12:14:39Z

Merge remote-tracking branch 'remotes/origin/master' into ignite-3478

commit ecdeff85ebc63dfee02635113dbfcad98d235a31
Author: sboikov 
Date:   2017-10-27T14:09:47Z

ignite-3478 add test

commit 84f5c0f2d8d48be207b9094fc3812f31f1c12b15
Author: Igor Seliverstov 
Date:   2017-11-02T12:38:20Z

Merge branch 'master' into ignite-3478

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/reader/StandaloneGridKernalContext.java

commit 066f0aed9484160a39e75e4e3c4a92b285707e9e
Author: Igor Seliverstov 
Date:   2017-11-08T09:47:14Z

IGNITE-6709 Support mvcc filter for H2TreeIndex.findFirstOrLast

commit e1d80187a17b9a926511cc9fb316896a3c60e3d1
Author: Igor Seliverstov 
Date:   2017-11-09T16:03:11Z

Merge branch 'master' into ignite-3478

commit 694b94c6ad9fe119fddecc72aa67bbf49413e10b
Author: Igor Seliverstov 
Date:   2017-11-10T08:34:50Z

Merge branch 'master' into ignite-3478

commit 23cae6695db34d8369abb395001ce3e1b88a6394
Author: devozerov 
Date:   2017-11-10T09:55:15Z

Minors.

commit fdfcd0b74665f23386c85e559b7d2abe3493a939
Author: devozerov 

[GitHub] ignite pull request #3618: IGNITE-7791: add test case for delayed partition ...

[Mmuzaf]

Github user Mmuzaf closed the pull request at:

https://github.com/apache/ignite/pull/3618


---

Re[2]: Welcome

[Протченко Алексей]

Hello all

Thank you, Alexey!


>Вторник, 27 марта 2018, 21:57 +06:00 от Alexey Goncharuk 
>:
>
>Hello Alexey,
>
>Welcome to the Ignite community! I've added you to the contributors list,
>you should be now able to assign the JIRA ticket.
>
>--AG
>
>2018-03-27 18:42 GMT+03:00 Spider (Alexey) < spiderru5...@gmail.com >:
>
>> Hello Ignite Community!
>>
>> My name is Aleksey. I want to contribute to Apache Ignite and want to
>> start with this issue - IGNITE-8049 but I can't assign it to me, my JIRA
>> username astelmak. Any help on this will be appreciated.
>>
>> Thanks!
>>
>>

[GitHub] ignite pull request #3656: IGNITE-7754 Fix for FSYNC on archivation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3656


---

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

[Dmitry Pavlov]

Ivan, sure :)

Thank you for this contribution, merged to master.

вт, 27 мар. 2018 г. в 20:08, Ivan Rakov :

> Dmitry,
>
> Firstly PR contained dirty fix for performance measurement, but now it
> contains good fix. :) Sorry for inconvenience.
> I've renamed the PR.
>
> Best Regards,
> Ivan Rakov
>
> On 27.03.2018 19:40, Dmitry Pavlov wrote:
> > Hi Eduard, thank you for review.
> >
> > Hi Ivan,
> >
> > I'm confused on PR naming
> > https://github.com/apache/ignite/pull/3656
> >
> > Could you rename?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > вт, 27 мар. 2018 г. в 19:38, Eduard Shangareev <
> eduard.shangar...@gmail.com
> >> :
> >> Ivan, I have reviewed your changes, looks good.
> >>
> >> On Tue, Mar 27, 2018 at 2:56 PM, Ivan Rakov 
> wrote:
> >>
> >>> Igniters,
> >>>
> >>> I've completed development of https://issues.apache.org/jira
> >>> /browse/IGNITE-7754. TeamCity state is ok. Please, review my changes.
> >>> Please note that it will be possible to track time of WAL fsync on
> >>> checkpoint begin by *walCpRecordFsyncDuration *metric in "Checkpoint
> >>> started" message.
> >>>
> >>> Also, I've created https://issues.apache.org/jira/browse/IGNITE-8057
> >> with
> >>> description of possible further improvement of WAL fsync on checkpoint
> >>> begin.
> >>>
> >>> Best Regards,
> >>> Ivan Rakov
> >>>
> >>>
> >>> On 26.03.2018 23:45, Valentin Kulichenko wrote:
> >>>
>  Ivan,
> 
>  It's all good then :) Thanks!
> 
>  -Val
> 
>  On Mon, Mar 26, 2018 at 1:50 AM, Ivan Rakov 
>  wrote:
> 
>  Val,
> > There's no any sense to use WalMode.NONE in production environment,
> >> it's
> > kept for testing and debugging purposes (including possible user
> > activities
> > like capacity planning).
> > We already print a warning at node start in case WalMode.NONE is set:
> >
> > U.quietAndWarn(log,"Started write-ahead log manager in NONE mode,
> >
> >> persisted data may be lost in " +
> >>"a case of unexpected node failure. Make sure to deactivate
> the
> >> cluster before shutdown.");
> >>
> >> Best Regards,
> > Ivan Rakov
> >
> >
> > On 24.03.2018 1:40, Valentin Kulichenko wrote:
> >
> > Dmitry,
> >> Thanks for clarification. So it sounds like if we fix all other
> modes
> >> as
> >> we
> >> discuss here, NONE would be the only one allowing corruption. I also
> >> don't
> >> see much sense in this and I think we should clearly state this in
> the
> >> doc,
> >> as well print out a warning if NONE mode is used. Eventually, if
> it's
> >> confirmed that there are no reasonable use cases for it, we can
> >> deprecate
> >> it.
> >>
> >> -Val
> >>
> >> On Fri, Mar 23, 2018 at 3:26 PM, Dmitry Pavlov <
> dpavlov@gmail.com
> >> wrote:
> >>
> >> Hi Val,
> >>
> >>> NONE means that the WAL log is disabled and not written at all. Use
> >> of
> >>> the
> >>> mode is at your own risk. It is possible that restore state after
> the
> >>> crash
> >>> at the middle of checkpoint will not succeed. I do not see much
> sence
> >>> in
> >>> it, especially in production.
> >>>
> >>> BACKGROUND is full functional WAL mode, but allows some delay
> before
> >>> flush
> >>> to disk.
> >>>
> >>> Sincerely,
> >>> Dmitriy Pavlov
> >>>
> >>> сб, 24 мар. 2018 г. в 1:07, Valentin Kulichenko <
> >>> valentin.kuliche...@gmail.com>:
> >>>
> >>> I agree. In my view, any possibility to get a corrupted storage is
> a
> >>> bug
> >>>
>  which needs to be fixed.
> 
>  BTW, can someone explain semantics of NONE mode? What is the
>  difference
>  from BACKGROUND from user's perspective? Is there any particular
> use
>  case
>  where it can be used?
> 
>  -Val
> 
>  On Fri, Mar 23, 2018 at 2:49 AM, Dmitry Pavlov <
> >> dpavlov@gmail.com
>  wrote:
> 
>  Hi Ivan,
> 
> > IMO we have to add extra FSYNCS for BACKGROUND WAL. Agree?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пт, 23 мар. 2018 г. в 12:23, Ivan Rakov :
> >
> > Igniters, there's another important question about this matter.
> >
> >> Do we want to add extra FSYNCS for BACKGROUND WAL mode? I think
> >> that
> >> we
>  have to do it: it will cause similar performance drop, but if we
> 
> > consider LOG_ONLY broken without these fixes, BACKGROUND is
> broken
> >> as
> >> well.
> > Best Regards,
> >> Ivan Rakov
> >>
> >> On 23.03.2018 10:27, Ivan Rakov wrote:
> >>
> >> Fixes are quite simple.
> >>> I expect them to be merged in master in a 

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

[Ivan Rakov]

Dmitry,

Firstly PR contained dirty fix for performance measurement, but now it 
contains good fix. :) Sorry for inconvenience.

I've renamed the PR.

Best Regards,
Ivan Rakov

On 27.03.2018 19:40, Dmitry Pavlov wrote:

Hi Eduard, thank you for review.

Hi Ivan,

I'm confused on PR naming
https://github.com/apache/ignite/pull/3656

Could you rename?

Sincerely,
Dmitriy Pavlov

вт, 27 мар. 2018 г. в 19:38, Eduard Shangareev  wrote:


Igniters,

I've completed development of https://issues.apache.org/jira
/browse/IGNITE-7754. TeamCity state is ok. Please, review my changes.
Please note that it will be possible to track time of WAL fsync on
checkpoint begin by *walCpRecordFsyncDuration *metric in "Checkpoint
started" message.

Also, I've created https://issues.apache.org/jira/browse/IGNITE-8057

with

description of possible further improvement of WAL fsync on checkpoint
begin.

Best Regards,
Ivan Rakov


On 26.03.2018 23:45, Valentin Kulichenko wrote:


Ivan,

It's all good then :) Thanks!

-Val

On Mon, Mar 26, 2018 at 1:50 AM, Ivan Rakov 
wrote:

Val,

There's no any sense to use WalMode.NONE in production environment,

it's

kept for testing and debugging purposes (including possible user
activities
like capacity planning).
We already print a warning at node start in case WalMode.NONE is set:

U.quietAndWarn(log,"Started write-ahead log manager in NONE mode,


persisted data may be lost in " +
   "a case of unexpected node failure. Make sure to deactivate the
cluster before shutdown.");

Best Regards,

Ivan Rakov


On 24.03.2018 1:40, Valentin Kulichenko wrote:

Dmitry,

Thanks for clarification. So it sounds like if we fix all other modes

as

we
discuss here, NONE would be the only one allowing corruption. I also
don't
see much sense in this and I think we should clearly state this in the
doc,
as well print out a warning if NONE mode is used. Eventually, if it's
confirmed that there are no reasonable use cases for it, we can
deprecate
it.

-Val

On Fri, Mar 23, 2018 at 3:26 PM, Dmitry Pavlov :

I agree. In my view, any possibility to get a corrupted storage is a
bug


which needs to be fixed.

BTW, can someone explain semantics of NONE mode? What is the
difference
from BACKGROUND from user's perspective? Is there any particular use
case
where it can be used?

-Val

On Fri, Mar 23, 2018 at 2:49 AM, Dmitry Pavlov <

dpavlov@gmail.com

wrote:

Hi Ivan,


IMO we have to add extra FSYNCS for BACKGROUND WAL. Agree?

Sincerely,
Dmitriy Pavlov

пт, 23 мар. 2018 г. в 12:23, Ivan Rakov :

Igniters, there's another important question about this matter.


Do we want to add extra FSYNCS for BACKGROUND WAL mode? I think

that

we

have to do it: it will cause similar performance drop, but if we


consider LOG_ONLY broken without these fixes, BACKGROUND is broken

as

well.

Best Regards,

Ivan Rakov

On 23.03.2018 10:27, Ivan Rakov wrote:

Fixes are quite simple.

I expect them to be merged in master in a week in worst case.

Best Regards,
Ivan Rakov

On 22.03.2018 17:49, Denis Magda wrote:

Ivan,

How quick are you going to merge the fix into the master? Many
persistence
related optimizations have already stacked up. Probably, we can

release

them sooner if the community agrees.


--

Denis

On Thu, Mar 22, 2018 at 5:22 AM, Ivan Rakov <

ivan.glu...@gmail.com>

wrote:

Thanks all!

We seem to have reached a consensus on this issue. I'll just

add

necessary
fsyncs under IGNITE-7754.

Best Regards,
Ivan Rakov


On 22.03.2018 15:13, Ilya Lantukh wrote:

+1 for fixing LOG_ONLY. If current implementation doesn't
protect


from

data

corruption, it doesn't make sence.

On Wed, Mar 21, 2018 at 10:38 PM, Denis Magda <

dma...@apache.org>

wrote:

+1 for the fix of LOG_ONLY

On Wed, Mar 21, 2018 at 11:23 AM, Alexey Goncharuk <

alexey.goncha...@gmail.com> wrote:

+1 for fixing LOG_ONLY to enforce corruption safety given the
provided

performance results.

2018-03-21 18:20 GMT+03:00 Vladimir Ozerov <

voze...@gridgain.com

:

+1 for accepting drop in LOG_ONLY. 7% is not that much and

not a

drop

at

all, provided that we fixing a bug. I.e. should we implement

it

correctly

in the first place we would never notice any "drop".

I do not understand why someone would like to use current

broken

mode.

On Wed, Mar 21, 2018 at 6:11 PM, Dmitry 

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

[Dmitry Pavlov]

Hi Eduard, thank you for review.

Hi Ivan,

I'm confused on PR naming
https://github.com/apache/ignite/pull/3656

Could you rename?

Sincerely,
Dmitriy Pavlov

вт, 27 мар. 2018 г. в 19:38, Eduard Shangareev :

> Ivan, I have reviewed your changes, looks good.
>
> On Tue, Mar 27, 2018 at 2:56 PM, Ivan Rakov  wrote:
>
> > Igniters,
> >
> > I've completed development of https://issues.apache.org/jira
> > /browse/IGNITE-7754. TeamCity state is ok. Please, review my changes.
> > Please note that it will be possible to track time of WAL fsync on
> > checkpoint begin by *walCpRecordFsyncDuration *metric in "Checkpoint
> > started" message.
> >
> > Also, I've created https://issues.apache.org/jira/browse/IGNITE-8057
> with
> > description of possible further improvement of WAL fsync on checkpoint
> > begin.
> >
> > Best Regards,
> > Ivan Rakov
> >
> >
> > On 26.03.2018 23:45, Valentin Kulichenko wrote:
> >
> >> Ivan,
> >>
> >> It's all good then :) Thanks!
> >>
> >> -Val
> >>
> >> On Mon, Mar 26, 2018 at 1:50 AM, Ivan Rakov 
> >> wrote:
> >>
> >> Val,
> >>>
> >>> There's no any sense to use WalMode.NONE in production environment,
> it's
> >>> kept for testing and debugging purposes (including possible user
> >>> activities
> >>> like capacity planning).
> >>> We already print a warning at node start in case WalMode.NONE is set:
> >>>
> >>> U.quietAndWarn(log,"Started write-ahead log manager in NONE mode,
> >>>
>  persisted data may be lost in " +
>    "a case of unexpected node failure. Make sure to deactivate the
>  cluster before shutdown.");
> 
>  Best Regards,
> >>> Ivan Rakov
> >>>
> >>>
> >>> On 24.03.2018 1:40, Valentin Kulichenko wrote:
> >>>
> >>> Dmitry,
> 
>  Thanks for clarification. So it sounds like if we fix all other modes
> as
>  we
>  discuss here, NONE would be the only one allowing corruption. I also
>  don't
>  see much sense in this and I think we should clearly state this in the
>  doc,
>  as well print out a warning if NONE mode is used. Eventually, if it's
>  confirmed that there are no reasonable use cases for it, we can
>  deprecate
>  it.
> 
>  -Val
> 
>  On Fri, Mar 23, 2018 at 3:26 PM, Dmitry Pavlov  >
>  wrote:
> 
>  Hi Val,
> 
> > NONE means that the WAL log is disabled and not written at all. Use
> of
> > the
> > mode is at your own risk. It is possible that restore state after the
> > crash
> > at the middle of checkpoint will not succeed. I do not see much sence
> > in
> > it, especially in production.
> >
> > BACKGROUND is full functional WAL mode, but allows some delay before
> > flush
> > to disk.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > сб, 24 мар. 2018 г. в 1:07, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com>:
> >
> > I agree. In my view, any possibility to get a corrupted storage is a
> > bug
> >
> >> which needs to be fixed.
> >>
> >> BTW, can someone explain semantics of NONE mode? What is the
> >> difference
> >> from BACKGROUND from user's perspective? Is there any particular use
> >> case
> >> where it can be used?
> >>
> >> -Val
> >>
> >> On Fri, Mar 23, 2018 at 2:49 AM, Dmitry Pavlov <
> dpavlov@gmail.com
> >> >
> >> wrote:
> >>
> >> Hi Ivan,
> >>
> >>> IMO we have to add extra FSYNCS for BACKGROUND WAL. Agree?
> >>>
> >>> Sincerely,
> >>> Dmitriy Pavlov
> >>>
> >>> пт, 23 мар. 2018 г. в 12:23, Ivan Rakov :
> >>>
> >>> Igniters, there's another important question about this matter.
> >>>
>  Do we want to add extra FSYNCS for BACKGROUND WAL mode? I think
> that
> 
>  we
> >>>
> >> have to do it: it will cause similar performance drop, but if we
> >>
> >>> consider LOG_ONLY broken without these fixes, BACKGROUND is broken
> as
> 
>  well.
> >>>
> >>> Best Regards,
>  Ivan Rakov
> 
>  On 23.03.2018 10:27, Ivan Rakov wrote:
> 
>  Fixes are quite simple.
> > I expect them to be merged in master in a week in worst case.
> >
> > Best Regards,
> > Ivan Rakov
> >
> > On 22.03.2018 17:49, Denis Magda wrote:
> >
> > Ivan,
> >>
> >> How quick are you going to merge the fix into the master? Many
> >> persistence
> >> related optimizations have already stacked up. Probably, we can
> >>
> >> release
> >
>  them sooner if the community agrees.
> 
> > --
> >> Denis
> >>
> >> On Thu, Mar 22, 2018 at 5:22 AM, Ivan Rakov <
> >>
> >> ivan.glu...@gmail.com>
> >
>  wrote:

Re: Code inspection

[Dmitry Pavlov]

Hi Petr,

Could you please take inspections and run it on AI code base in
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_InspectionsCore_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv
?

Sincerely,
Dmitriy Pavlov

вт, 27 мар. 2018 г. в 19:27, Dmitry Pavlov :

> Alexey, thank you for bring this topic to top.
>
> What do you think about committing this inspections into Ignite code base?
>
> What can be our next steps after demonstrating CI check is possible
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_InspectionsCore_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv
>  ?
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 27 мар. 2018 г. в 15:28, Alexey Goncharuk  >:
>
>> Bumping up.
>>
>> Attached is my local inspections profile exported from Idea. Let's run
>> the first iteration and check if it differs significantly from other
>> community members.
>>
>> --AG
>>
>> 2018-03-19 16:39 GMT+03:00 Petr Ivanov :
>>
>>> Filed https://issues.apache.org/jira/browse/IGNITE-7985 <
>>> https://issues.apache.org/jira/browse/IGNITE-7985> [1].
>>
>>
>>>
>>>
>>>
>>> > On 18 Mar 2018, at 00:56, Dmitry Pavlov  wrote:
>>> >
>>> > Hello Petr,
>>> >
>>> > Many members of the community would appreciate such additional code
>>> control, and it's a pity that no one made this happen. Agree?
>>> >
>>> > Could you please pick up this activity?
>>> >
>>> > It might be an idea to create 'IDEA Inspections' step to be run in
>>> parallel with 'Build Apache Ignite'. WDYT? Would it work?
>>> >
>>> > Sincerely,
>>> > Dmitriy Pavlov
>>> >
>>>
>> > https://confluence.jetbrains.com/display/TCD10/Inspections <
>>> https://confluence.jetbrains.com/display/TCD10/Inspections>
>>> >
>>> >
>>> > пн, 12 мар. 2018 г. в 14:37, Dmitry Pavlov >> >:
>>
>>
>>> > Hi Dmitriy,
>>> >
>>> > would you pick up this activity?
>>> >
>>> > Sincerely,
>>> > Dmitriy Pavlov
>>> >
>>>
>> > вт, 6 мар. 2018 г. в 14:09, Dmitry Pavlov >> >:
>>
>>
>>> > What I can suggest now it is to take XML file with existing as is from
>>> previous topic (I remember someone in community already prepared settings)
>>> and set up TeamCity Run configuration as part of Run All Basic Tests (per
>>> commit basis).
>>> >
>>> > If we don’t have XML, I suggest to enable build-in Idea inspections
>>> 'as is' on TeamCity and iteratively improve it according to found issues.
>>> >
>>> > Dmitriy G., would you prepare PR and proof-of-concept TC run
>>> configuration?
>>> >
>>> > As discussion became really active, I think that means community is
>>> interested in static code checks.
>>> >
>>>
>> > вт, 6 мар. 2018 г. в 14:08, Dmitry Pavlov >> >:
>>
>>
>>> > I was thinking about some quick check, which will automatically
>>> require minimum runs. Now, any committer can push changes to the master,
>>> which break not only the inspection and style, but even the compilation. If
>>> this control would be automatic, it can allow us make codebase better quite
>>> fast. But I am afraid it is not realistic.
>>> >
>>> >
>>> >
>>>
>> > вт, 6 мар. 2018 г. в 13:42, Petr Ivanov > mr.wei...@gmail.com>>:
>>
>>
>>> > Sonar is powerful, yes, but it’s power in thoroughness. I.e. it does
>>> its job well in cases of leisurely post-build analysis.
>>> >
>>> > I’d suggest we use it (if we will use it) in the following scenarios:
>>> >  — some basic checks Sonar profile for Blocker bugs (it is fast) —
>>> something that cannot be passed to master;
>>> >  — nightly or even weekly run with Full Sonar profile (600+ checks
>>> from Firebug, Codestyle, Coverage, etc.) for regression and overall code
>>> quality improvement goals.
>>> >
>>> > Did not quite get you about push-to-master prohibition. Can you
>>> explain scenario in more details?
>>> >
>>> >
>>> > > On 6 Mar 2018, at 13:27, Dmitry Pavlov >> > wrote:
>>> > >
>>> > > Petr, I've heard Sonar is powerful tool.
>>> > >
>>> > > Would it help us to prohibit commits to master w/o test run / too
>>> much
>>> > > failed tests / too much inspection errors appeared?
>>> > >
>>>
>> > > вт, 6 мар. 2018 г. в 13:22, Alexey Goncharuk <
>>> alexey.goncha...@gmail.com >:
>>
>>
>>> > >
>>> > >> Dmitriy,
>>> > >>
>>> > >> I like this idea a lot. For example, the inspection profile should
>>> have
>>> > >> inspection 'Anonymous class can be converted to lambda' disabled
>>> because
>>> > >> quite a lot of such classes can be sent over the network (although
>>> even
>>> > >> anonymous classes are discourage for such purposes).
>>> > >>
>>> > >> I believe we can start with sharing somehow one of the profiles and
>>> then
>>> > >> iteratively improving it until the community is 

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

[Eduard Shangareev]

Ivan, I have reviewed your changes, looks good.

On Tue, Mar 27, 2018 at 2:56 PM, Ivan Rakov  wrote:

> Igniters,
>
> I've completed development of https://issues.apache.org/jira
> /browse/IGNITE-7754. TeamCity state is ok. Please, review my changes.
> Please note that it will be possible to track time of WAL fsync on
> checkpoint begin by *walCpRecordFsyncDuration *metric in "Checkpoint
> started" message.
>
> Also, I've created https://issues.apache.org/jira/browse/IGNITE-8057 with
> description of possible further improvement of WAL fsync on checkpoint
> begin.
>
> Best Regards,
> Ivan Rakov
>
>
> On 26.03.2018 23:45, Valentin Kulichenko wrote:
>
>> Ivan,
>>
>> It's all good then :) Thanks!
>>
>> -Val
>>
>> On Mon, Mar 26, 2018 at 1:50 AM, Ivan Rakov 
>> wrote:
>>
>> Val,
>>>
>>> There's no any sense to use WalMode.NONE in production environment, it's
>>> kept for testing and debugging purposes (including possible user
>>> activities
>>> like capacity planning).
>>> We already print a warning at node start in case WalMode.NONE is set:
>>>
>>> U.quietAndWarn(log,"Started write-ahead log manager in NONE mode,
>>>
 persisted data may be lost in " +
   "a case of unexpected node failure. Make sure to deactivate the
 cluster before shutdown.");

 Best Regards,
>>> Ivan Rakov
>>>
>>>
>>> On 24.03.2018 1:40, Valentin Kulichenko wrote:
>>>
>>> Dmitry,

 Thanks for clarification. So it sounds like if we fix all other modes as
 we
 discuss here, NONE would be the only one allowing corruption. I also
 don't
 see much sense in this and I think we should clearly state this in the
 doc,
 as well print out a warning if NONE mode is used. Eventually, if it's
 confirmed that there are no reasonable use cases for it, we can
 deprecate
 it.

 -Val

 On Fri, Mar 23, 2018 at 3:26 PM, Dmitry Pavlov 
 wrote:

 Hi Val,

> NONE means that the WAL log is disabled and not written at all. Use of
> the
> mode is at your own risk. It is possible that restore state after the
> crash
> at the middle of checkpoint will not succeed. I do not see much sence
> in
> it, especially in production.
>
> BACKGROUND is full functional WAL mode, but allows some delay before
> flush
> to disk.
>
> Sincerely,
> Dmitriy Pavlov
>
> сб, 24 мар. 2018 г. в 1:07, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
>
> I agree. In my view, any possibility to get a corrupted storage is a
> bug
>
>> which needs to be fixed.
>>
>> BTW, can someone explain semantics of NONE mode? What is the
>> difference
>> from BACKGROUND from user's perspective? Is there any particular use
>> case
>> where it can be used?
>>
>> -Val
>>
>> On Fri, Mar 23, 2018 at 2:49 AM, Dmitry Pavlov > >
>> wrote:
>>
>> Hi Ivan,
>>
>>> IMO we have to add extra FSYNCS for BACKGROUND WAL. Agree?
>>>
>>> Sincerely,
>>> Dmitriy Pavlov
>>>
>>> пт, 23 мар. 2018 г. в 12:23, Ivan Rakov :
>>>
>>> Igniters, there's another important question about this matter.
>>>
 Do we want to add extra FSYNCS for BACKGROUND WAL mode? I think that

 we
>>>
>> have to do it: it will cause similar performance drop, but if we
>>
>>> consider LOG_ONLY broken without these fixes, BACKGROUND is broken as

 well.
>>>
>>> Best Regards,
 Ivan Rakov

 On 23.03.2018 10:27, Ivan Rakov wrote:

 Fixes are quite simple.
> I expect them to be merged in master in a week in worst case.
>
> Best Regards,
> Ivan Rakov
>
> On 22.03.2018 17:49, Denis Magda wrote:
>
> Ivan,
>>
>> How quick are you going to merge the fix into the master? Many
>> persistence
>> related optimizations have already stacked up. Probably, we can
>>
>> release
>
 them sooner if the community agrees.

> --
>> Denis
>>
>> On Thu, Mar 22, 2018 at 5:22 AM, Ivan Rakov <
>>
>> ivan.glu...@gmail.com>
>
 wrote:
>>
>>> Thanks all!
>>
>>> We seem to have reached a consensus on this issue. I'll just add
>>> necessary
>>> fsyncs under IGNITE-7754.
>>>
>>> Best Regards,
>>> Ivan Rakov
>>>
>>>
>>> On 22.03.2018 15:13, Ilya Lantukh wrote:
>>>
>>> +1 for fixing LOG_ONLY. If current implementation doesn't
>>> protect
>>>
>> from
>>
>>> data

> corruption, it doesn't make sence.

 On Wed, Mar 21, 2018 at 10:38 

Re: Rebalancing - how to make it faster

[Denis Magda]

Ilya, granted you all the required permissions. Please let me know if you
still have troubles with the wiki.

--
Denis

On Tue, Mar 27, 2018 at 8:56 AM, Ilya Lantukh  wrote:

> Unfortunately, I don't have permission to create page for IEP on wiki.
> Denis, can you grant it? My username is ilantukh.
>
> On Mon, Mar 26, 2018 at 8:04 PM, Anton Vinogradov  wrote:
>
> > >> It is impossible to disable WAL only for certain partitions without
> > >> completely overhauling design of Ignite storage mechanism. Right now
> we
> > can
> > >> afford only to change WAL mode per cache group.
> >
> > Cache group rebalancing is a one cache rebalancing, and then this cache
> > ("cache group") can be presented as a set of virtual caches.
> > So, there is no issues for initial rebalancing.
> > Lets disable WAL on initial rebalancing.
> >
> > 2018-03-26 16:46 GMT+03:00 Ilya Lantukh :
> >
> > > Dmitry,
> > > It is impossible to disable WAL only for certain partitions without
> > > completely overhauling design of Ignite storage mechanism. Right now we
> > can
> > > afford only to change WAL mode per cache group.
> > >
> > > The idea is to disable WAL when node doesn't have any partition in
> OWNING
> > > state, which means it doesn't have any consistent data and won't be
> able
> > to
> > > restore from WAL anyway. I don't see any potential use for WAL on such
> > > node, but we can keep a configurable parameter indicating can we
> > > automatically disable WAL in such case or not.
> > >
> > > On Fri, Mar 23, 2018 at 10:40 PM, Dmitry Pavlov  >
> > > wrote:
> > >
> > > > Denis, as I understood, there is and idea to exclude only rebalanced
> > > > partition(s) data. All other data will go to the WAL.
> > > >
> > > > Ilya, please correct me if I'm wrong.
> > > >
> > > > пт, 23 мар. 2018 г. в 22:15, Denis Magda :
> > > >
> > > > > Ilya,
> > > > >
> > > > > That's a decent boost (5-20%) even having WAL enabled. Not sure
> that
> > we
> > > > > should stake on the WAL "off" mode here because if the whole
> cluster
> > > goes
> > > > > down, it's then the data consistency is questionable. As an
> > architect,
> > > I
> > > > > wouldn't disable WAL for the sake of rebalancing; it's too risky.
> > > > >
> > > > > If you agree, then let's create the IEP. This way it will be easier
> > to
> > > > > track this endeavor. BTW, are you already ready to release any
> > > > > optimizations in 2.5 that is being discussed in a separate thread?
> > > > >
> > > > > --
> > > > > Denis
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Mar 23, 2018 at 6:37 AM, Ilya Lantukh <
> ilant...@gridgain.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Denis,
> > > > > >
> > > > > > > - Don't you want to aggregate the tickets under an IEP?
> > > > > > Yes, I think so.
> > > > > >
> > > > > > > - Does it mean we're going to update our B+Tree implementation?
> > Any
> > > > > ideas
> > > > > > how risky it is?
> > > > > > One of tickets that I created (
> > > > > > https://issues.apache.org/jira/browse/IGNITE-7935) involves
> B+Tree
> > > > > > modification, but I am not planning to do it in the nearest
> future.
> > > It
> > > > > > shouldn't affect existing tree operations, only introduce new
> ones
> > > > > (putAll,
> > > > > > invokeAll, removeAll).
> > > > > >
> > > > > > > - Any chance you had a prototype that shows performance
> > > optimizations
> > > > > the
> > > > > > approach you are suggesting to take?
> > > > > > I have a prototype for simplest improvements (
> > > > https://issues.apache.org/
> > > > > > jira/browse/IGNITE-8019 & https://issues.apache.org/
> > > > > > jira/browse/IGNITE-8018)
> > > > > > - together they increase throughput by 5-20%, depending on
> > > > configuration
> > > > > > and environment. Also, I've tested different WAL modes -
> switching
> > > from
> > > > > > LOG_ONLY to NONE gives over 100% boost - this is what I expect
> from
> > > > > > https://issues.apache.org/jira/browse/IGNITE-8017.
> > > > > >
> > > > > > On Thu, Mar 22, 2018 at 9:48 PM, Denis Magda 
> > > > wrote:
> > > > > >
> > > > > > > Ilya,
> > > > > > >
> > > > > > > That's outstanding research and summary. Thanks for spending
> your
> > > > time
> > > > > on
> > > > > > > this.
> > > > > > >
> > > > > > > Not sure I have enough expertise to challenge your approach,
> but
> > it
> > > > > > sounds
> > > > > > > 100% reasonable to me. As side notes:
> > > > > > >
> > > > > > >- Don't you want to aggregate the tickets under an IEP?
> > > > > > >- Does it mean we're going to update our B+Tree
> > implementation?
> > > > Any
> > > > > > >ideas how risky it is?
> > > > > > >- Any chance you had a prototype that shows performance
> > > > > optimizations
> > > > > > of
> > > > > > >the approach you are suggesting to take?
> > > > > > >
> > > > > > > --
> > > > > > > Denis
> > > > > > >
> > > > > > > On Thu, Mar 22, 2018 at 

Re: Code inspection

[Dmitry Pavlov]

Alexey, thank you for bring this topic to top.

What do you think about committing this inspections into Ignite code base?

What can be our next steps after demonstrating CI check is possible
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_InspectionsCore_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv
 ?

Sincerely,
Dmitriy Pavlov

вт, 27 мар. 2018 г. в 15:28, Alexey Goncharuk :

> Bumping up.
>
> Attached is my local inspections profile exported from Idea. Let's run the
> first iteration and check if it differs significantly from other community
> members.
>
> --AG
>
> 2018-03-19 16:39 GMT+03:00 Petr Ivanov :
>
>> Filed https://issues.apache.org/jira/browse/IGNITE-7985 <
>> https://issues.apache.org/jira/browse/IGNITE-7985> [1].
>
>
>>
>>
>>
>> > On 18 Mar 2018, at 00:56, Dmitry Pavlov  wrote:
>> >
>> > Hello Petr,
>> >
>> > Many members of the community would appreciate such additional code
>> control, and it's a pity that no one made this happen. Agree?
>> >
>> > Could you please pick up this activity?
>> >
>> > It might be an idea to create 'IDEA Inspections' step to be run in
>> parallel with 'Build Apache Ignite'. WDYT? Would it work?
>> >
>> > Sincerely,
>> > Dmitriy Pavlov
>> >
>>
> > https://confluence.jetbrains.com/display/TCD10/Inspections <
>> https://confluence.jetbrains.com/display/TCD10/Inspections>
>> >
>> >
>> > пн, 12 мар. 2018 г. в 14:37, Dmitry Pavlov > >:
>
>
>> > Hi Dmitriy,
>> >
>> > would you pick up this activity?
>> >
>> > Sincerely,
>> > Dmitriy Pavlov
>> >
>>
> > вт, 6 мар. 2018 г. в 14:09, Dmitry Pavlov > >:
>
>
>> > What I can suggest now it is to take XML file with existing as is from
>> previous topic (I remember someone in community already prepared settings)
>> and set up TeamCity Run configuration as part of Run All Basic Tests (per
>> commit basis).
>> >
>> > If we don’t have XML, I suggest to enable build-in Idea inspections 'as
>> is' on TeamCity and iteratively improve it according to found issues.
>> >
>> > Dmitriy G., would you prepare PR and proof-of-concept TC run
>> configuration?
>> >
>> > As discussion became really active, I think that means community is
>> interested in static code checks.
>> >
>>
> > вт, 6 мар. 2018 г. в 14:08, Dmitry Pavlov > >:
>
>
>> > I was thinking about some quick check, which will automatically require
>> minimum runs. Now, any committer can push changes to the master, which
>> break not only the inspection and style, but even the compilation. If this
>> control would be automatic, it can allow us make codebase better quite
>> fast. But I am afraid it is not realistic.
>> >
>> >
>> >
>>
> > вт, 6 мар. 2018 г. в 13:42, Petr Ivanov  mr.wei...@gmail.com>>:
>
>
>> > Sonar is powerful, yes, but it’s power in thoroughness. I.e. it does
>> its job well in cases of leisurely post-build analysis.
>> >
>> > I’d suggest we use it (if we will use it) in the following scenarios:
>> >  — some basic checks Sonar profile for Blocker bugs (it is fast) —
>> something that cannot be passed to master;
>> >  — nightly or even weekly run with Full Sonar profile (600+ checks from
>> Firebug, Codestyle, Coverage, etc.) for regression and overall code quality
>> improvement goals.
>> >
>> > Did not quite get you about push-to-master prohibition. Can you explain
>> scenario in more details?
>> >
>> >
>> > > On 6 Mar 2018, at 13:27, Dmitry Pavlov > > wrote:
>> > >
>> > > Petr, I've heard Sonar is powerful tool.
>> > >
>> > > Would it help us to prohibit commits to master w/o test run / too much
>> > > failed tests / too much inspection errors appeared?
>> > >
>>
> > > вт, 6 мар. 2018 г. в 13:22, Alexey Goncharuk <
>> alexey.goncha...@gmail.com >:
>
>
>> > >
>> > >> Dmitriy,
>> > >>
>> > >> I like this idea a lot. For example, the inspection profile should
>> have
>> > >> inspection 'Anonymous class can be converted to lambda' disabled
>> because
>> > >> quite a lot of such classes can be sent over the network (although
>> even
>> > >> anonymous classes are discourage for such purposes).
>> > >>
>> > >> I believe we can start with sharing somehow one of the profiles and
>> then
>> > >> iteratively improving it until the community is satisfied with the
>> result.
>> > >>
>> > >> Thoughts?
>> > >>
>>
> > >> 2018-03-06 12:06 GMT+03:00 Petr Ivanov  mr.wei...@gmail.com>>:
>
>
>> > >>
>> > >>> We can use Sonar as instrument for code analysis and test coverage
>> > >>> inspections.
>> > >>>
>> > >>>
>> > >>>
>> >  On 6 Mar 2018, at 11:28, Dmitriy Govorukhin <
>> > >>> dmitriy.govoruk...@gmail.com >
>> wrote:
>> > 
>> >  Dmitriy,

Re: What's about releasing Ignite 2.5 a bit earlier?

[Denis Magda]

Andrey, it's great to see you in a role of 2.5 release engineer! Thanks for
stepping in.

--
Denis



On Tue, Mar 27, 2018 at 8:42 AM, Andrey Gura  wrote:

> Igniters,
>
> I'm ready to take care of Apache Ignite 2.5 release.
>
> I've started preparing wiki pages. There are many fixes in master
> branch so I think that we should limit scope of the release soon (e.g.
> next week) because stabilization and QA activities will also require
> time.
> I'll keep the community informed about all release preparation steps.
>
>
> On Tue, Mar 27, 2018 at 3:07 AM, Denis Magda  wrote:
> > Good, hope that everyone interested has enough time to share an opinion.
> >
> > As a summary of this discussion, the community decided to release Ignite
> > 2.5 on April 30. Most of the changes and features are already in the
> master.
> >
> > Who is ready to become a release manager of 2.5? We need to prepare a
> > respective wiki page and outline the milestones (code freeze, QA, vote,
> > release).
> >
> > --
> > Denis
> >
> > On Sat, Mar 24, 2018 at 4:38 PM, Yury Babak  wrote:
> >
> >> Hi,
> >>
> >> We already implemented LSQR for linear regression and SVM(support vector
> >> machine) algorithms. Also we implement new distributed Datasets.
> >>
> >> And we want to adapt all our algorithms to this new dataset API. So
> from my
> >> point of view we have enough time for those tasks.
> >>
> >> So I think that we could release Apache Ignite 2.5 at 30 Apr.
> >>
> >> Regards,
> >> Yury
> >>
> >>
> >>
> >> --
> >> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >>
>

Re: Partition recovery issue on partition loss.

[Dmitry Pavlov]

Denis, it seems noone is working.

чт, 22 мар. 2018 г. в 21:26, Denis Magda :

> Igniters,
>
> Is anybody working on this bug? There is a high chance we can add a fix to
> 2.5 if the community agrees to release it earlier.
>
> --
> Denis
>
> On Thu, Mar 15, 2018 at 11:04 AM, Denis Magda  wrote:
>
> > I dared to set fix version to 2.5 and increased the severity. It's
> > important to fix the race since we've just released the partition loss
> > functionality in 2.4 and it's already broken.
> >
> > Andrey, please keep us posted. If you didn't fix it, we would need to
> find
> > another contributor.
> >
> > --
> > Denis
> >
> > On Thu, Mar 15, 2018 at 7:29 AM, Dmitry Pavlov 
> > wrote:
> >
> >> Hi Andrew Mashenkov,
> >>
> >> would you like to pick up issue?
> >>
> >> Sincerely,
> >> Dmitriy Pavlov
> >>
> >> чт, 15 мар. 2018 г. в 6:23, Dmitriy Setrakyan :
> >>
> >> > Completely agree, we must fix this. I like the proposed design. We
> >> should
> >> > also specify that resetLostPartitions() method should return true and
> >> > false.
> >> >
> >> > Val, do you mind updating the ticket with new design?
> >> > https://issues.apache.org/jira/browse/IGNITE-7832
> >> >
> >> > D.
> >> >
> >> > On Tue, Mar 13, 2018 at 5:31 PM, Valentin Kulichenko <
> >> > valentin.kuliche...@gmail.com> wrote:
> >> >
> >> > > This indeed looks like a bigger issue. Basically, there is no clear
> >> way
> >> > (or
> >> > > no way at all) to synchronize code that listens to partition loss
> >> event,
> >> > > and the code that calls resetLostPartitions() method. Example
> >> scenario:
> >> > >
> >> > > 1. Cache is configured with 3rd party persistence.
> >> > > 2. One or more nodes fail causing loss of several partitions in
> >> memory.
> >> > > 3. Ignite blocks access to those partitions according to partition
> >> loss
> >> > > policy and fires an event.
> >> > > 4. Application listens to the event and starts reloading the data
> from
> >> > > store.
> >> > > 5. When reloading is complete, application calls
> >> resetLostPartitions() to
> >> > > restore access.
> >> > > 6. Nodes fail again causing another partition loss, new event is
> >> fired.
> >> > >
> >> > > There is race between steps 5 and 6. If 2nd failure happens BEFORE
> >> > > resetLostPartitions() is called, we end up with inconsistent data.
> >> > >
> >> > > I believe the only way to fix this is to add corresponding topology
> >> > version
> >> > > to partition loss event, and also add it as a parameter for
> >> > > resetLostPartitions().
> >> > > This way if resetLostPartitions() is invoked with a version that is
> >> not
> >> > the
> >> > > latest anymore, the invocation will be ignored.
> >> > >
> >> > > The only problem with this approach  is that topology version itself
> >> is
> >> > > currently not a part of public API. It needs to be properly exposed
> >> there
> >> > > first.
> >> > >
> >> > > -Val
> >> > >
> >> > > On Mon, Mar 12, 2018 at 1:07 PM, Denis Magda 
> >> wrote:
> >> > >
> >> > > > Just in case here is you can find the present documentation:
> >> > > >
> >> >
> https://apacheignite.readme.io/docs/cache-modes#partition-loss-policies
> >> > > >
> >> > > > Let us know what needs to be updated once the issues reported by
> you
> >> > are
> >> > > > addressed.
> >> > > >
> >> > > > --
> >> > > > Denis
> >> > > >
> >> > > > On Mon, Mar 12, 2018 at 3:33 AM, Andrey Mashenkov <
> >> > > > andrey.mashen...@gmail.com> wrote:
> >> > > >
> >> > > > > Hi Igniters,
> >> > > > >
> >> > > > > I've found we no documentation how user can recover cache from
> >> > > cacheStore
> >> > > > > in case of partition loss.
> >> > > > > Ignite provides some instruments (methods and events) that
> should
> >> > help
> >> > > > user
> >> > > > > to solve this problem,
> >> > > > > but looks like these instruments have an architecture lack.
> >> > > > >
> >> > > > > The first one is an usability issue. Ignite provides partition
> >> loss
> >> > > event
> >> > > > > to user can handle this, but Ignite fires an event per
> partition.
> >> > > > > Why we can't have an event with list of lost partitions?
> >> > > > >
> >> > > > > The second one is a bug. Ignite.resetLostPartitions() method
> >> doesn't
> >> > > care
> >> > > > > about what topology version recovered partitions belonged to.
> >> > > > > Tthere is a race, when user call this method after a node was
> >> failed,
> >> > > but
> >> > > > > right before Ignite fire an event.
> >> > > > > So, it is possible state of just lost partitions will be reseted
> >> > > > > unexpectedly.
> >> > > > >
> >> > > > >
> >> > > > > I've created a ticket for this [1] and think we should rethink
> the
> >> > > > > architecture of the partition recovery mechanics and improve
> >> > > > documentation.
> >> > > > > Any thoughts?
> >> > > > >
> >> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-7832
> >> > > > >
> >> > > > >
> >> > > > > 

Re: Test 150 clients migrated to `Cache [6]` suite

[Dmitry Pavlov]

Hi Igniters,

I've deleted this run config.

Sincerely,
Dmitriy Pavlov

пт, 23 мар. 2018 г. в 13:50, Petr Ivanov :

> No objections then.
>
>
>
> > On 23 Mar 2018, at 13:03, Dmitry Pavlov  wrote:
> >
> > Hi Dmitriy,
> >
> >  Thank you for this improvement.
> >
> > Hi Petr,
> >
> >  IMO there is no need to keep Obsolete suites for so long time because
> > Igniters ususally do not use test history (excepting TC Jedi who helps
> with
> > monitoring). Moreover, test history itself is global, so still no need to
> > keep suites too long time.
> >
> > I think it is reasonable to notify before changes because someone may
> have
> > created unique parameters combination in suite, so it is not desirable to
> > delete it without notification (as it was for zIgnores & reproducing
> > suite).
> >
> > So let's wait for some time for objections, and if noone objects, we'll
> > remove.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пт, 23 мар. 2018 г. в 12:07, Petr Ivanov :
> >
> >> Nice work!
> >>
> >>
> >> Yet, I have to ask to keep '~[Obsolete] 150 Clients’ build configuration
> >> for at least month from latest run until its build history is removed by
> >> autoclean — it’ll be nice to keep that test history.
> >>
> >>
> >>
> >>> On 23 Mar 2018, at 11:53, Дмитрий Рябов  wrote:
> >>>
> >>> Hello Igniters!
> >>>
> >>>
> >>>
> >>> I migrated test `IgniteCache150ClientsTest` from `150 Clients` suite
> to `
> >>> Cache [6]`, because `150 Clients` contains only one test class and
> makes
> >>> ~6min excess Ignite build.
> >>>
> >>>
> >>>
> >>> So, test suites `150 Clients` and `~[Obsolete] 150 Clients` will be
> >> deleted
> >>> soon.
> >>
> >>
>
>

[GitHub] ignite pull request #3707: IGNITE-8017

[ilantukh]

GitHub user ilantukh opened a pull request:

https://github.com/apache/ignite/pull/3707

IGNITE-8017



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8017

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3707.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3707






---

Re: Welcome

[Alexey Goncharuk]

Hello Alexey,

Welcome to the Ignite community! I've added you to the contributors list,
you should be now able to assign the JIRA ticket.

--AG

2018-03-27 18:42 GMT+03:00 Spider (Alexey) :

> Hello Ignite Community!
>
> My name is Aleksey. I want to contribute to Apache Ignite and want to
> start with this issue - IGNITE-8049 but I can't assign it to me, my JIRA
> username astelmak. Any help on this will be appreciated.
>
> Thanks!
>
>

Re: Rebalancing - how to make it faster

[Ilya Lantukh]

Unfortunately, I don't have permission to create page for IEP on wiki.
Denis, can you grant it? My username is ilantukh.

On Mon, Mar 26, 2018 at 8:04 PM, Anton Vinogradov  wrote:

> >> It is impossible to disable WAL only for certain partitions without
> >> completely overhauling design of Ignite storage mechanism. Right now we
> can
> >> afford only to change WAL mode per cache group.
>
> Cache group rebalancing is a one cache rebalancing, and then this cache
> ("cache group") can be presented as a set of virtual caches.
> So, there is no issues for initial rebalancing.
> Lets disable WAL on initial rebalancing.
>
> 2018-03-26 16:46 GMT+03:00 Ilya Lantukh :
>
> > Dmitry,
> > It is impossible to disable WAL only for certain partitions without
> > completely overhauling design of Ignite storage mechanism. Right now we
> can
> > afford only to change WAL mode per cache group.
> >
> > The idea is to disable WAL when node doesn't have any partition in OWNING
> > state, which means it doesn't have any consistent data and won't be able
> to
> > restore from WAL anyway. I don't see any potential use for WAL on such
> > node, but we can keep a configurable parameter indicating can we
> > automatically disable WAL in such case or not.
> >
> > On Fri, Mar 23, 2018 at 10:40 PM, Dmitry Pavlov 
> > wrote:
> >
> > > Denis, as I understood, there is and idea to exclude only rebalanced
> > > partition(s) data. All other data will go to the WAL.
> > >
> > > Ilya, please correct me if I'm wrong.
> > >
> > > пт, 23 мар. 2018 г. в 22:15, Denis Magda :
> > >
> > > > Ilya,
> > > >
> > > > That's a decent boost (5-20%) even having WAL enabled. Not sure that
> we
> > > > should stake on the WAL "off" mode here because if the whole cluster
> > goes
> > > > down, it's then the data consistency is questionable. As an
> architect,
> > I
> > > > wouldn't disable WAL for the sake of rebalancing; it's too risky.
> > > >
> > > > If you agree, then let's create the IEP. This way it will be easier
> to
> > > > track this endeavor. BTW, are you already ready to release any
> > > > optimizations in 2.5 that is being discussed in a separate thread?
> > > >
> > > > --
> > > > Denis
> > > >
> > > >
> > > >
> > > > On Fri, Mar 23, 2018 at 6:37 AM, Ilya Lantukh  >
> > > > wrote:
> > > >
> > > > > Denis,
> > > > >
> > > > > > - Don't you want to aggregate the tickets under an IEP?
> > > > > Yes, I think so.
> > > > >
> > > > > > - Does it mean we're going to update our B+Tree implementation?
> Any
> > > > ideas
> > > > > how risky it is?
> > > > > One of tickets that I created (
> > > > > https://issues.apache.org/jira/browse/IGNITE-7935) involves B+Tree
> > > > > modification, but I am not planning to do it in the nearest future.
> > It
> > > > > shouldn't affect existing tree operations, only introduce new ones
> > > > (putAll,
> > > > > invokeAll, removeAll).
> > > > >
> > > > > > - Any chance you had a prototype that shows performance
> > optimizations
> > > > the
> > > > > approach you are suggesting to take?
> > > > > I have a prototype for simplest improvements (
> > > https://issues.apache.org/
> > > > > jira/browse/IGNITE-8019 & https://issues.apache.org/
> > > > > jira/browse/IGNITE-8018)
> > > > > - together they increase throughput by 5-20%, depending on
> > > configuration
> > > > > and environment. Also, I've tested different WAL modes - switching
> > from
> > > > > LOG_ONLY to NONE gives over 100% boost - this is what I expect from
> > > > > https://issues.apache.org/jira/browse/IGNITE-8017.
> > > > >
> > > > > On Thu, Mar 22, 2018 at 9:48 PM, Denis Magda 
> > > wrote:
> > > > >
> > > > > > Ilya,
> > > > > >
> > > > > > That's outstanding research and summary. Thanks for spending your
> > > time
> > > > on
> > > > > > this.
> > > > > >
> > > > > > Not sure I have enough expertise to challenge your approach, but
> it
> > > > > sounds
> > > > > > 100% reasonable to me. As side notes:
> > > > > >
> > > > > >- Don't you want to aggregate the tickets under an IEP?
> > > > > >- Does it mean we're going to update our B+Tree
> implementation?
> > > Any
> > > > > >ideas how risky it is?
> > > > > >- Any chance you had a prototype that shows performance
> > > > optimizations
> > > > > of
> > > > > >the approach you are suggesting to take?
> > > > > >
> > > > > > --
> > > > > > Denis
> > > > > >
> > > > > > On Thu, Mar 22, 2018 at 8:38 AM, Ilya Lantukh <
> > ilant...@gridgain.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > I've spent some time analyzing performance of rebalancing
> > process.
> > > > The
> > > > > > > initial goal was to understand, what limits it's throughput,
> > > because
> > > > it
> > > > > > is
> > > > > > > significantly slower than network and storage device can
> > > > theoretically
> > > > > > > handle.
> > > > > > >
> > > > > 

Re: What's about releasing Ignite 2.5 a bit earlier?

[Andrey Gura]

Igniters,

I'm ready to take care of Apache Ignite 2.5 release.

I've started preparing wiki pages. There are many fixes in master
branch so I think that we should limit scope of the release soon (e.g.
next week) because stabilization and QA activities will also require
time.
I'll keep the community informed about all release preparation steps.


On Tue, Mar 27, 2018 at 3:07 AM, Denis Magda  wrote:
> Good, hope that everyone interested has enough time to share an opinion.
>
> As a summary of this discussion, the community decided to release Ignite
> 2.5 on April 30. Most of the changes and features are already in the master.
>
> Who is ready to become a release manager of 2.5? We need to prepare a
> respective wiki page and outline the milestones (code freeze, QA, vote,
> release).
>
> --
> Denis
>
> On Sat, Mar 24, 2018 at 4:38 PM, Yury Babak  wrote:
>
>> Hi,
>>
>> We already implemented LSQR for linear regression and SVM(support vector
>> machine) algorithms. Also we implement new distributed Datasets.
>>
>> And we want to adapt all our algorithms to this new dataset API. So from my
>> point of view we have enough time for those tasks.
>>
>> So I think that we could release Apache Ignite 2.5 at 30 Apr.
>>
>> Regards,
>> Yury
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>>

Welcome

[Spider (Alexey)]

Hello Ignite Community!

My name is Aleksey. I want to contribute to Apache Ignite and want to 
start with this issue - IGNITE-8049 but I can't assign it to me, my JIRA 
username astelmak. Any help on this will be appreciated.

Thanks!

Re: Server stores cache data on-heap if client has near cache - IGNITE-4662

[Dmitry Pavlov]

Hi Vyacheslav,


Task is still actual. It is another question if it is a priority or not.


This is a task is about following case: a client node has near cache. Every
get operation is tracked and enlisted on server node. The server node must
trace the operation and accordingly store the results to Java Heap Memory.
Currently it is on-heap, and task is about idea to make it off-heap.


Sincerely,

Dmitriy Pavlov

пт, 23 мар. 2018 г. в 13:13, Dmitry Pavlov :

> Hi Igniters,
>
> Alexey, please step in. I did not fully understand issue description, and
> can't identify if data is still not moved from onheap.
>
> I suggest to consider it as outdated if we don't have any additional info.
>
> Sincerely,
> Dmitriy Pavlov
>
> чт, 22 мар. 2018 г. в 10:48, Vyacheslav Daradur :
>
>> Hi, Igniters!
>>
>> Look like it's outdated and can be closed.
>>
>> Could someone comment if this ticket [1] is still valid?
>>
>>
>> [1] https://issues.apache.org/jira/browse/IGNITE-4662
>>
>> On Wed, Jun 28, 2017 at 2:38 AM, Valentin Kulichenko
>>  wrote:
>> > I'm not sure this ticket is valid for 2.0. Semen, can you comment?
>> >
>> > -Val
>> >
>> > On Tue, Jun 27, 2017 at 1:14 AM, Vyacheslav Daradur <
>> daradu...@gmail.com>
>> > wrote:
>> >
>> >> Hi Igniters.
>> >>
>> >> I have some questions according to this task:
>> >>
>> >> 1. Does the method: GridCacheMapEntry#evictInternal do the
>> >> eviction(on-heap
>> >> -> off-heap)?
>> >> 2. Is CacheOffheapEvictionManager responsible for managing the
>> >> eviction(on-heap -> off-heap)? (if not, then who is?)
>> >> 3. At what moment the eviction(on-heap -> off-heap) is called?
>> >>
>> >>
>> >> --
>> >> Best Regards, Vyacheslav D.
>> >>
>>
>>
>>
>> --
>> Best Regards, Vyacheslav D.
>>
>

[jira] [Created] (IGNITE-8060) Sqline creating tables on client nodes works incorrect in case if node shutdown

[Andrey Aleksandrov (JIRA)]

Andrey Aleksandrov created IGNITE-8060:
--

 Summary: Sqline creating tables on client nodes works incorrect in 
case if node shutdown
 Key: IGNITE-8060
 URL: https://issues.apache.org/jira/browse/IGNITE-8060
 Project: Ignite
  Issue Type: Bug
  Components: jdbc
Affects Versions: 2.4
Reporter: Andrey Aleksandrov


For reproducing:

You should start one local server and one local client nodes and follow the 
instructions:

1)Connect to client node:

sqlline.bat --color=true --verbose=true -u jdbc:ignite:thin://127.0.0.1:10801

2)Create new table on client node:

CREATE TABLE City (id LONG PRIMARY KEY, name VARCHAR)WITH "template=replicated";

3)Check that table exists from server node:

!tables

On this step table should be shown in the response.

4)Drop the client node

5)Create new client node

6)Connect to new client node:

sqlline.bat --color=true --verbose=true -u jdbc:ignite:thin://127.0.0.1:10801

7)Check that table exists from server node:

!tables

*On this step there is no "city" table in the list.*

8)Try to drop the table:
DROP TABLE City;
java.sql.SQLException: Table doesn't exist: CITY
 at 
org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:671)
 at 
org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute0(JdbcThinStatement.java:130)
 at 
org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute(JdbcThinStatement.java:299)
 at sqlline.Commands.execute(Commands.java:823)
 at sqlline.Commands.sql(Commands.java:733)
 at sqlline.SqlLine.dispatch(SqlLine.java:795)
 at sqlline.SqlLine.begin(SqlLine.java:668)
 at sqlline.SqlLine.start(SqlLine.java:373)
 at sqlline.SqlLine.main(SqlLine.java:265)



9)Try to create new table:
CREATE TABLE City (id LONG PRIMARY KEY, name VARCHAR)WITH "template=replicated";

java.sql.SQLException: Table already exists: CITY
 at 
org.apache.ignite.internal.jdbc.thin.JdbcThinConnection.sendRequest(JdbcThinConnection.java:671)
 at 
org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute0(JdbcThinStatement.java:130)
 at 
org.apache.ignite.internal.jdbc.thin.JdbcThinStatement.execute(JdbcThinStatement.java:299)
 at sqlline.Commands.execute(Commands.java:823)
 at sqlline.Commands.sql(Commands.java:733)
 at sqlline.SqlLine.dispatch(SqlLine.java:795)
 at sqlline.SqlLine.begin(SqlLine.java:668)
 at sqlline.SqlLine.start(SqlLine.java:373)
 at sqlline.SqlLine.main(SqlLine.java:265)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8059) Integrate decision tree with partition based dataset

[Anton Dmitriev (JIRA)]

Anton Dmitriev created IGNITE-8059:
--

 Summary: Integrate decision tree with partition based dataset
 Key: IGNITE-8059
 URL: https://issues.apache.org/jira/browse/IGNITE-8059
 Project: Ignite
  Issue Type: Improvement
  Components: ml
Reporter: Anton Dmitriev
Assignee: Anton Dmitriev


A partition based dataset (new underlying infrastructure component) was added 
as part of IGNITE-7437 and now we need to adopt decision tree algorithm to work 
on top of this infrastructure. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3706: Ignite 6807: validation of local queries on clien...

[pavel-kuznetsov]

GitHub user pavel-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/3706

Ignite 6807: validation of local queries on client

Added more meaningful handling of execution local queries on client node.
Local queries on client's local cache are still allowed.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6807

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3706.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3706


commit 7b3c8187b10cab6663fb8314b2f0ef599525bd8c
Author: Pavel Kuznetsov 
Date:   2018-03-23T13:00:06Z

IGNITE-6807: Negative test for local query on client.

commit f4377b5398e4000ed04c587fe2fc4c4970204992
Author: Pavel Kuznetsov 
Date:   2018-03-26T10:46:20Z

IGNITE-6807: Updated tests.

commit d5801b5684f629d0913d0c5913173f3107f43614
Author: Pavel Kuznetsov 
Date:   2018-03-26T10:48:29Z

IGNITE-6807: Added validation for local queries on client nodes.

commit ada6b7030b70e0f8f3dd719a5f7d3a2f9f505588
Author: Pavel Kuznetsov 
Date:   2018-03-26T14:33:03Z

IGNITE-6807: Better test.

commit cd8f1308e1c0b6a9bd397dd613fa2c8d8d6c092c
Author: Pavel Kuznetsov 
Date:   2018-03-26T14:49:29Z

IGNITE-6807: Cosmetic.

commit 6978ba7b1718001c646925b7d768dbcf91be03c4
Author: Pavel Kuznetsov 
Date:   2018-03-27T13:35:27Z

IGNITE-6807: Added positive test. Updated comments/style.

commit 9ada7256da1e054a6d485e099365756ecbc53d38
Author: Pavel Kuznetsov 
Date:   2018-03-27T13:58:30Z

Merge remote-tracking branch 'origin/master' into ignite-6807




---

[GitHub] ignite pull request #3567: IGNITE-7774 Missing Google Cloud libraries at bin...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3567


---

[jira] [Created] (IGNITE-8058) GridMarshallerMappingConsistencyTest.testPersistedMappingsSharedOnJoin is flaky in master

[Dmitriy Pavlov (JIRA)]

Dmitriy Pavlov created IGNITE-8058:
--

 Summary: 
GridMarshallerMappingConsistencyTest.testPersistedMappingsSharedOnJoin  is 
flaky in master
 Key: IGNITE-8058
 URL: https://issues.apache.org/jira/browse/IGNITE-8058
 Project: Ignite
  Issue Type: Test
Reporter: Dmitriy Pavlov
Assignee: Denis Mekhanikov


https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=408332895682007527=%3Cdefault%3E=testDetails

Test introduced in [IGNITE-7794] is flaky, fail rate ~33%

{noformat}
org.apache.ignite.IgniteException: Can not perform the operation because the 
cluster is inactive. Note, that the cluster is considered inactive by default 
if Ignite Persistent Store is used to let all the nodes join the cluster. To 
activate the cluster call Ignite.active(true).
at 
org.apache.ignite.marshaller.GridMarshallerMappingConsistencyTest.testPersistedMappingsSharedOnJoin(GridMarshallerMappingConsistencyTest.java:160)
{noformat}
Please fix. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: .NET: Add "authenticationEnabled" flag to IgntieConfiguration

[Dmitry Pavlov]

Hi Pavel,

.NET core tests are passing now.

Let me sincerely thank you for helping the community in keeping TC to be
green.

Best Regards,
Dmitriy Pavlov

вт, 27 мар. 2018 г. в 10:55, Pavel Tupitsyn :

> IGNITE-8034 done, I have added configuration flag.
>
> However, I still have concerns about property naming:
> Is this authentication going to be used for thin clients only, or for
> classic client and server nodes as well?
>
> On Tue, Mar 27, 2018 at 9:50 AM, Dmitry Pavlov 
> wrote:
>
> > Hi Folks,
> >
> > Test is still failing on TC.
> > https://ci.ignite.apache.org/viewLog.html?buildId=1160178;
> > tab=buildResultsDiv=IgniteTests24Java8_
> > IgnitePlatformNetCoreLinux
> >
> > I suggest to mute it until fix is ready.Test is running in Basic Run All
> > subsets, so each commit will genereate TC failure email. What do you
> think?
> >
> > Sincerely.
> > Dmitriy Pavlov.
> >
> > вт, 27 мар. 2018 г. в 2:22, Denis Magda :
> >
> > > Taras,
> > >
> > > Please document the authentication part on a 2.5 version of the binary
> > > protocol page on readme.io. We need to share it with the guys who are
> > > developing Node.JS client right now.
> > >
> > > --
> > > Denis
> > >
> > > On Mon, Mar 26, 2018 at 2:55 AM, Taras Ledkov 
> > > wrote:
> > >
> > > > I had implement user credential store according with the previous
> > > > discussion about user authentication.
> > > >
> > > > Now JDBC thin, and ODBC support user authentication. We haven't
> > > > implemented it for all thin client because protocols are not similar.
> > > >
> > > > I see two ways to implement authentication for thin client protocol:
> > > > - You implement authentication on server side and at the .NET;
> > > > - When java thin client [1] is merged I implement authentication for
> > thin
> > > > protocol & for java thin client.
> > > >
> > > > I'll add documentation for user authentication ASAP. Please feel free
> > to
> > > > contact if you need more info till documentation is added.
> > > >
> > > > [1]. https://issues.apache.org/jira/browse/IGNITE-7421
> > > >
> > > >
> > > > On 26.03.2018 9:56, Pavel Tupitsyn wrote:
> > > >
> > > >> I've started this task, and the property name combined with lack of
> > > >> javadoc
> > > >> seems confusing and misleading:
> > > >>
> > > >> * Turns out this authentication is only for thin clients
> > > >> * Not clear how to configure and use it, even after digging through
> > Jira
> > > >> and devlist
> > > >>
> > > >> How do I write test to ensure it works?
> > > >>
> > > >> Thanks,
> > > >> Pavel
> > > >>
> > > >> On Fri, Mar 23, 2018 at 6:44 PM, Pavel Tupitsyn <
> ptupit...@apache.org
> > >
> > > >> wrote:
> > > >>
> > > >> Thanks, got it, will do.
> > > >>>
> > > >>> On Fri, Mar 23, 2018 at 4:36 PM, Dmitry Pavlov <
> > dpavlov@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>> Hi Pavel,
> > > 
> > >  Related ticket is
> https://issues.apache.org/jira/browse/IGNITE-7436
> > > 
> > >  Sincerely,
> > >  Dmitriy Pavlov
> > > 
> > >  пт, 23 мар. 2018 г. в 16:24, Pavel Tupitsyn  >:
> > > 
> > >  Please provide description in IGNITE-8034 and link Java-side
> ticket
> > > >
> > >  there.
> > > 
> > > > On Fri, Mar 23, 2018 at 4:23 PM, Pavel Tupitsyn <
> > > ptupit...@apache.org>
> > > > wrote:
> > > >
> > > > Hi Vladimir,
> > > >>
> > > >> Can you provide more details?
> > > >> * What does it do?
> > > >> * Do we need to only propagate the flag to .NET or do anything
> > else?
> > > >> * Related ticket?
> > > >>
> > > >> Thanks,
> > > >> Pavel
> > > >>
> > > >> On Fri, Mar 23, 2018 at 2:25 PM, Vladimir Ozerov <
> > > >>
> > > > voze...@gridgain.com>
> > > 
> > > > wrote:
> > > >>
> > > >> Pavel,
> > > >>>
> > > >>> We introduced new flag
> IgniteConfiguration.authenticationEnabled
> > > >>>
> > > >> recently.
> > > >
> > > >> Would you mind adding it to IgniteConfigutation.cs [1]?
> > > >>>
> > > >>> Vladimir.
> > > >>>
> > > >>> [1] https://issues.apache.org/jira/browse/IGNITE-8034
> > > >>>
> > > >>>
> > > >>
> > > >>>
> > > > --
> > > > Taras Ledkov
> > > > Mail-To: tled...@gridgain.com
> > > >
> > > >
> > >
> >
>

Re: Code inspection

[Alexey Goncharuk]

Bumping up.

Attached is my local inspections profile exported from Idea. Let's run the
first iteration and check if it differs significantly from other community
members.

--AG

2018-03-19 16:39 GMT+03:00 Petr Ivanov :

> Filed https://issues.apache.org/jira/browse/IGNITE-7985 <
> https://issues.apache.org/jira/browse/IGNITE-7985> [1].
>
>
>
> > On 18 Mar 2018, at 00:56, Dmitry Pavlov  wrote:
> >
> > Hello Petr,
> >
> > Many members of the community would appreciate such additional code
> control, and it's a pity that no one made this happen. Agree?
> >
> > Could you please pick up this activity?
> >
> > It might be an idea to create 'IDEA Inspections' step to be run in
> parallel with 'Build Apache Ignite'. WDYT? Would it work?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > https://confluence.jetbrains.com/display/TCD10/Inspections <
> https://confluence.jetbrains.com/display/TCD10/Inspections>
> >
> >
> > пн, 12 мар. 2018 г. в 14:37, Dmitry Pavlov  >:
> > Hi Dmitriy,
> >
> > would you pick up this activity?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > вт, 6 мар. 2018 г. в 14:09, Dmitry Pavlov  >:
> > What I can suggest now it is to take XML file with existing as is from
> previous topic (I remember someone in community already prepared settings)
> and set up TeamCity Run configuration as part of Run All Basic Tests (per
> commit basis).
> >
> > If we don’t have XML, I suggest to enable build-in Idea inspections 'as
> is' on TeamCity and iteratively improve it according to found issues.
> >
> > Dmitriy G., would you prepare PR and proof-of-concept TC run
> configuration?
> >
> > As discussion became really active, I think that means community is
> interested in static code checks.
> >
> > вт, 6 мар. 2018 г. в 14:08, Dmitry Pavlov  >:
> > I was thinking about some quick check, which will automatically require
> minimum runs. Now, any committer can push changes to the master, which
> break not only the inspection and style, but even the compilation. If this
> control would be automatic, it can allow us make codebase better quite
> fast. But I am afraid it is not realistic.
> >
> >
> >
> > вт, 6 мар. 2018 г. в 13:42, Petr Ivanov >:
> > Sonar is powerful, yes, but it’s power in thoroughness. I.e. it does its
> job well in cases of leisurely post-build analysis.
> >
> > I’d suggest we use it (if we will use it) in the following scenarios:
> >  — some basic checks Sonar profile for Blocker bugs (it is fast) —
> something that cannot be passed to master;
> >  — nightly or even weekly run with Full Sonar profile (600+ checks from
> Firebug, Codestyle, Coverage, etc.) for regression and overall code quality
> improvement goals.
> >
> > Did not quite get you about push-to-master prohibition. Can you explain
> scenario in more details?
> >
> >
> > > On 6 Mar 2018, at 13:27, Dmitry Pavlov > wrote:
> > >
> > > Petr, I've heard Sonar is powerful tool.
> > >
> > > Would it help us to prohibit commits to master w/o test run / too much
> > > failed tests / too much inspection errors appeared?
> > >
> > > вт, 6 мар. 2018 г. в 13:22, Alexey Goncharuk <
> alexey.goncha...@gmail.com >:
> > >
> > >> Dmitriy,
> > >>
> > >> I like this idea a lot. For example, the inspection profile should
> have
> > >> inspection 'Anonymous class can be converted to lambda' disabled
> because
> > >> quite a lot of such classes can be sent over the network (although
> even
> > >> anonymous classes are discourage for such purposes).
> > >>
> > >> I believe we can start with sharing somehow one of the profiles and
> then
> > >> iteratively improving it until the community is satisfied with the
> result.
> > >>
> > >> Thoughts?
> > >>
> > >> 2018-03-06 12:06 GMT+03:00 Petr Ivanov >:
> > >>
> > >>> We can use Sonar as instrument for code analysis and test coverage
> > >>> inspections.
> > >>>
> > >>>
> > >>>
> >  On 6 Mar 2018, at 11:28, Dmitriy Govorukhin <
> > >>> dmitriy.govoruk...@gmail.com >
> wrote:
> > 
> >  Dmitriy,
> > 
> >  As I understood, preview topic was of static code analysis in
> general.
> >  In this topic, I want to discuss only idea inspection rule.
> >  In future, of course, we can expаnd this rule to the TeamCity build.
> > 
> >  On Tue, Mar 6, 2018 at 11:16 AM, Nikolay Izhikov <
> nizhi...@apache.org >
> >  wrote:
> > 
> > > Hello, Igniters.
> > >
> > > +1 to automatic code style tools.
> > >
> > > Let's make it already!
> > > Do we have a ticket for it?
> > >
> > > Related discussion -
> > >> 

[GitHub] ignite pull request #3682: IGNITE-7902 Enable either swap space or persisten...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3682


---

Re: Memory usage per cache

[Andrey Kuznetsov]

I apologize for the previous message sent in hurry. It's imposible to
measure the difference between 'precise' and 'estimated' page memory usage
per cache unless we fully implement approach 2. Current model allows to
store objects from several caches in a single page.

2018-03-23 22:21 GMT+03:00 Andrey Kuznetsov :

> Denis,
>
> I'll need to conduct some experiments to estimate the difference. And the
> answer will depend on numerous parameters: object sizes, number of caches,
> that share the same data region and so on.
>
> пт, 23 марта 2018, 21:53 Denis Magda :
>
>> Andrey,
>>
>> How good will be the estimate if we go for 1. and utilize pagesFillFactor
>> somehow? In other words, how big can be a difference between 100% precise
>> calculation you the approach you're suggesting?
>>
>> --
>> Denis
>>
>>


-- 
Best regards,
  Andrey Kuznetsov.

MTCGA: Hot failures

[Dmitry Pavlov]

Hi Igniters,

I’ve filtered report of master failures to find most failing tests.
Following is 5 failing suites and their frequently failed test with links
to TC.

Igniters, please reply who would like to pick up

- test failures context research,

- tickets creation

- and (the best option) fixing these fails.

Sincerely,

Dmitriy Pavlov

  Basic [2]


IgniteServiceConfigVariationsFullApiTestSuite:
IgniteServiceConfigVariationsFullApiTest.testDeploy (master fail rate 23,1%)


IgniteServiceConfigVariationsFullApiTestSuite:
IgniteServiceConfigVariationsFullApiTest.testClusterSingletonDeploy (master
fail rate 20,8%)


IgniteServiceConfigVariationsFullApiTestSuite:
IgniteServiceConfigVariationsFullApiTest.testKeyAffinityDeploy (master fail
rate 20,8%)


IgniteServiceConfigVariationsFullApiTestSuite:
IgniteServiceConfigVariationsFullApiTest.testMultipleDeploy (master fail
rate 17,5%)


IgniteServiceConfigVariationsFullApiTestSuite:
IgniteServiceConfigVariationsFullApiTest.testNodeSingletonDeploy (master
fail rate 16,7%)




   Ignite Start Nodes


 IgniteStartStopRestartTestSuite:
IgniteProjectionStartStopRestartSelfTest.testStartFiveNodesInTwoCalls
(master fail rate 12,1%)


 IgniteStartStopRestartTestSuite:
IgniteProjectionStartStopRestartSelfTest.testStopNodes (master fail rate
10,1%)


 IgniteStartStopRestartTestSuite:
IgniteProjectionStartStopRestartSelfTest.testStopNodesByIds (master fail
rate 10,1%)


  Binary Objects (Simple Mapper Basic)


IgniteBinarySimpleNameMapperBasicTestSuite:
GridMarshallerMappingConsistencyTest.testPersistedMappingsSharedOnJoin
(master fail rate 14,3%)


Ignite Data Structures



   IgniteCacheDataStructuresSelfTestSuite:
IgnitePartitionedCountDownLatchSelfTest.testLatchMultinode1 (master fail
rate 12,4%)


   IgniteCacheDataStructuresSelfTestSuite:
IgniteClientDataStructuresTest.testSequence (master fail rate 11,9%)


   IgniteCacheDataStructuresSelfTestSuite:
IgniteClientDataStructuresTest.testReentrantLock (master fail rate 10,3%)


Activate | Deactivate Cluster



IgniteStandByClusterSuite:
CacheBaselineTopologyTest.testPrimaryLeftAndClusterRestart (master fail
rate 37,1%)


IgniteStandByClusterSuite:
CacheBaselineTopologyTest.testBaselineTopologyChangesFromClient (master
fail rate 24,9%)


IgniteStandByClusterSuite:
IgniteClusterActivateDeactivateTest.testClientReconnectClusterDeactivateInProgress
(master fail rate 12,8%)


   IgniteStandByClusterSuite:
JoinInActiveNodeToActiveCluster.testJoinClientStaticCacheConfigurationOnJoin
(master fail rate 12,8%)

Re: Apache Ignite RPM packaging (Stage II)

[Petr Ivanov]

Hi, Igniters!


Here are some news on our RPM packages initiative.

1. I’ve finished preliminary developing of Stage II version of RPM packages 
[1]. Main “new feature” is — split design. Also I’ve added package.sh script 
for automating package building process which will help organise corresponding 
builds in TC as well as simplify process for developers who wishes to have 
custom packages.
PR#3703 [2] is ready for review. Denis, in order to catch up with Apache Ignite 
2.5 release, I’d greatly appreciate your help in finding reviewer.
2. With the help of ASF INFRA team, we now have RPM [3] and DEB [4] 
repositories on Apache Bintray. Though they are already prepared for hosting 
RPM and DEB packages respectively, and there is a way of linking them to 
apache.org/dist/ignite page, there is possible alternative in storing there 
only plain directory layout corresponding to each repository type (RPM and DEB) 
and manage this layout (repodata, distributions, versions, etc.) by ourselves, 
having more control over repositories but lacking some simplicity of deploying 
new releases. WDYT? Should we try Cassandra approach? They are storing their 
DEB packages as I described above [5].

Also — a question arose while I was working on this issue: which OSes (and 
which versions of each) are we going to support (if we are going) in terms of 
step-by-step list? Currently RPM packages are tested only with latest CentOS 
(and, respectively — RHEL), but there are a lot more RPM-based distributives 
[6] some of which are more o less popular among OS community (ALT, Fedora, 
openSUSE, etc.).


[1] https://issues.apache.org/jira/browse/IGNITE-7647
[2] https://github.com/apache/ignite/pull/3703
[3] https://bintray.com/apache/ignite-rpm
[4] https://bintray.com/apache/ignite-deb
[5] https://bintray.com/apache/cassandra/debian#files/
[6] https://en.wikipedia.org/wiki/Category:RPM-based_Linux_distributions




> On 15 Mar 2018, at 22:15, Petr Ivanov  wrote:
> 
> I suppose that most everything if not all from libs/options will go to 
> OPTIONAL (I’d call it simply ‘apache-ignite-libs').
> More precise lib selection (if something from optional would better to have 
> in core package) will be discussed right after preliminary split architecture 
> agreement.
> 
> 
> 
>> On 15 Mar 2018, at 22:11, Dmitry Pavlov  wrote:
>> 
>> I like idea of keeping simple system of modules, so +1 from me.
>> 
>> Where optional libs (e.g Direct IO plugin) would be included, would it be
>> core or optional?
>> 
>> чт, 15 мар. 2018 г. в 22:09, Denis Magda :
>> 
 
> 
> How big would be a final core module?
 Around 30M. Can be shrinked to ~15M if separate Visor and create it’s own
 package.
>>> 
>>> 
>>> Guys, 30 vs 280M is a hge difference.  I would agree with Petr and
>>> propose the simplest modular system:
>>> 
>>>  - core module that includes basic Ignite capabilities including SQL,
>>>  compute grid, service grid, k/v
>>>  - optional module hosts the rest - ML, streamers integration (kafka,
>>>  flink), kubernetes, etc.
>>> 
>>> What do you think?
>>> 
>>> --
>>> Denis
>>> 
>>> On Thu, Mar 15, 2018 at 12:36 AM, Petr Ivanov  wrote:
>>> 
 *DEB package
 
 
> On 15 Mar 2018, at 10:35, Petr Ivanov  wrote:
> 
> Considering that DEV package for now is almost platform independent
>>> (its
 a java application more or less), that package will work almost on any
 DEB-based linux, including but not limited to Ubuntu, Debian, etc.
> The only restriction is existence of systemctl (systemd) service
>>> manager
 — we are dependent on it.
> 
> Thats why, for instance, our RPM repository is called simply ‘rpm’ and
 package has no arch or dist suffix — it will work on CentOS, RHEL,
>>> Fedora,
 etc. with presence of aforementioned systemd.
> 
> 
> 
>> On 15 Mar 2018, at 07:57, Dmitriy Setrakyan 
 wrote:
>> 
>> Will Debian package work for Ubuntu?
>> 
>> D.
>> 
>> On Wed, Mar 14, 2018 at 9:52 PM, Petr Ivanov 
 wrote:
>> 
>>> Not a problem, rather nuisance. Also, when we will move to official
>>> repositories, there can be a problem from OS community.
>>> 
>>> Concerning DEB packages — I plan to use RPM as base for DEB package
 build
>>> (package layout / install scripts) for speeding up things and
>>> excluding
>>> possible duplication and desynchronisation, so its a matter of ’sit
 and do’
>>> rather then some technical research. Thats why I rose discussion
>>> about
>>> future package architecture, so that after agreement I'm be able to
 pack
>>> both RPM and DEB identically.
>>> 
>>> Yet, if you insist, I can create DEB package according to current RPM
>>> layout in no time.
>>> 
>>> 
>>> 
 On 15 Mar 2018, 

[GitHub] ignite pull request #3705: ignite-8044: IgniteQueryGenerator.getOptions() me...

[sk0x50]

GitHub user sk0x50 opened a pull request:

https://github.com/apache/ignite/pull/3705

ignite-8044: IgniteQueryGenerator.getOptions() method should properly 
handle empty list of parameters.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8044

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3705.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3705


commit 2c34cdea77371dbff4daeeabd5d1551c9ec9a4d1
Author: Slava Koptilin 
Date:   2018-03-27T12:04:14Z

ignite-8044: added handling of an empty parameter list




---

[GitHub] ignite pull request #2803: IGNITE-6557: Fluky test fixed.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2803


---

[GitHub] ignite pull request #3704: IGNITE-6879 and migration to spring-data 2.0.5.RE...

[homich1991]

GitHub user homich1991 opened a pull request:

https://github.com/apache/ignite/pull/3704

IGNITE-6879 and migration to spring-data 2.0.5.RELEASE

Fixed IGNITE-6879
Migrated to spring-data 2.0.5.RELEASE
Migrated to spring 5.0.4.RELEASE
Fixed overrided methods naming
--

was problem with merging master in previous one 
https://github.com/apache/ignite/pull/3628, so created this one

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/homich1991/ignite ignite-6879

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3704.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3704


commit ae280108fad94d42a04d8b955262178ef6a365dd
Author: Roman_Meerson 
Date:   2018-03-13T15:32:46Z

Fixed IGNITE-6879
Migrated to spring-data 2.0.5.RELEASE
Migrated to spring 5.0.4.RELEASE
Fixed overrided methods naming

commit f090873961609dc0177172b0ea0a05efc4c462f8
Author: Roman_Meerson 
Date:   2018-03-13T18:34:17Z

Fixed tests

commit 2b16d5224290f81cd7949643c8c98b8a4e2b3c4f
Author: Roman_Meerson 
Date:   2018-03-13T20:46:19Z

Fixed sorting problem (added UNSORTED option)
Fixed bean loading order. Now repositories are loaded only via factory and 
not by them selves.

commit 4ca587129cc91d0a10642ab88e8469bdec4a5d92
Author: Roman_Meerson 
Date:   2018-03-22T18:44:17Z

Fixed comments on PR

commit 62f57f920eb3e57f036e664d3334dc9f3f6f599d
Author: Roman_Meerson 
Date:   2018-03-25T15:10:54Z

Fixed comments on PR

commit e4c443198d52b70912d4ce06fb74c1acd726dad3
Author: Roman_Meerson 
Date:   2018-03-25T18:30:55Z

Fixed spring data examples

commit df3fc87a815b816ad599a9872d479b3d63cd028f
Author: Roman_Meerson 
Date:   2018-03-25T18:34:50Z

Fixed check style comments in ConditionFalse class

commit 1ab3d63cceff5646f9d9d24cec952bd02615a818
Author: Roman_Meerson 
Date:   2018-03-25T19:25:24Z

Return null instead of exception throwing

commit b118dc23b65d2d6a28512ec19fa90cb779ffc793
Author: Roman_Meerson 
Date:   2018-03-27T11:56:54Z

Merge branch 'master' into ignite-6879




---

Re: Reconsider default WAL mode: we need something between LOG_ONLY and FSYNC

[Ivan Rakov]

Igniters,

I've completed development of 
https://issues.apache.org/jira/browse/IGNITE-7754. TeamCity state is ok. 
Please, review my changes.
Please note that it will be possible to track time of WAL fsync on 
checkpoint begin by *walCpRecordFsyncDuration *metric in "Checkpoint 
started" message.


Also, I've created https://issues.apache.org/jira/browse/IGNITE-8057 
with description of possible further improvement of WAL fsync on 
checkpoint begin.


Best Regards,
Ivan Rakov

On 26.03.2018 23:45, Valentin Kulichenko wrote:

Ivan,

It's all good then :) Thanks!

-Val

On Mon, Mar 26, 2018 at 1:50 AM, Ivan Rakov  wrote:


Val,

There's no any sense to use WalMode.NONE in production environment, it's
kept for testing and debugging purposes (including possible user activities
like capacity planning).
We already print a warning at node start in case WalMode.NONE is set:

U.quietAndWarn(log,"Started write-ahead log manager in NONE mode,

persisted data may be lost in " +
  "a case of unexpected node failure. Make sure to deactivate the
cluster before shutdown.");


Best Regards,
Ivan Rakov


On 24.03.2018 1:40, Valentin Kulichenko wrote:


Dmitry,

Thanks for clarification. So it sounds like if we fix all other modes as
we
discuss here, NONE would be the only one allowing corruption. I also don't
see much sense in this and I think we should clearly state this in the
doc,
as well print out a warning if NONE mode is used. Eventually, if it's
confirmed that there are no reasonable use cases for it, we can deprecate
it.

-Val

On Fri, Mar 23, 2018 at 3:26 PM, Dmitry Pavlov 
wrote:

Hi Val,

NONE means that the WAL log is disabled and not written at all. Use of
the
mode is at your own risk. It is possible that restore state after the
crash
at the middle of checkpoint will not succeed. I do not see much sence in
it, especially in production.

BACKGROUND is full functional WAL mode, but allows some delay before
flush
to disk.

Sincerely,
Dmitriy Pavlov

сб, 24 мар. 2018 г. в 1:07, Valentin Kulichenko <
valentin.kuliche...@gmail.com>:

I agree. In my view, any possibility to get a corrupted storage is a bug

which needs to be fixed.

BTW, can someone explain semantics of NONE mode? What is the difference
from BACKGROUND from user's perspective? Is there any particular use
case
where it can be used?

-Val

On Fri, Mar 23, 2018 at 2:49 AM, Dmitry Pavlov 
wrote:

Hi Ivan,

IMO we have to add extra FSYNCS for BACKGROUND WAL. Agree?

Sincerely,
Dmitriy Pavlov

пт, 23 мар. 2018 г. в 12:23, Ivan Rakov :

Igniters, there's another important question about this matter.

Do we want to add extra FSYNCS for BACKGROUND WAL mode? I think that


we

have to do it: it will cause similar performance drop, but if we

consider LOG_ONLY broken without these fixes, BACKGROUND is broken as


well.


Best Regards,
Ivan Rakov

On 23.03.2018 10:27, Ivan Rakov wrote:


Fixes are quite simple.
I expect them to be merged in master in a week in worst case.

Best Regards,
Ivan Rakov

On 22.03.2018 17:49, Denis Magda wrote:


Ivan,

How quick are you going to merge the fix into the master? Many
persistence
related optimizations have already stacked up. Probably, we can


release

them sooner if the community agrees.

--
Denis

On Thu, Mar 22, 2018 at 5:22 AM, Ivan Rakov <


ivan.glu...@gmail.com>

wrote:

Thanks all!

We seem to have reached a consensus on this issue. I'll just add
necessary
fsyncs under IGNITE-7754.

Best Regards,
Ivan Rakov


On 22.03.2018 15:13, Ilya Lantukh wrote:

+1 for fixing LOG_ONLY. If current implementation doesn't
protect

from

data

corruption, it doesn't make sence.

On Wed, Mar 21, 2018 at 10:38 PM, Denis Magda <


dma...@apache.org>

wrote:

+1 for the fix of LOG_ONLY


On Wed, Mar 21, 2018 at 11:23 AM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

+1 for fixing LOG_ONLY to enforce corruption safety given the
provided


performance results.

2018-03-21 18:20 GMT+03:00 Vladimir Ozerov <


voze...@gridgain.com

:

+1 for accepting drop in LOG_ONLY. 7% is not that much and

not a

drop

at
all, provided that we fixing a bug. I.e. should we implement


it

correctly

in the first place we would never notice any "drop".


I do not understand why someone would like to use current


broken

mode.

On Wed, Mar 21, 2018 at 6:11 PM, Dmitry Pavlov

wrote:

Hi, I think option 1 is better. As Val said any mode that


allows

corruption

does not make much sense.

What Ivan mentioned here as drop, in relation to old mode


DEFAULT

(FSYNC

now), is still significant perfromance boost.


Sincerely,
Dmitriy Pavlov

ср, 21 мар. 2018 г. в 17:56, Ivan Rakov <


ivan.glu...@gmail.com

:

I've attached benchmark results to the JIRA ticket.

We observe ~7% drop in "fair" LOG_ONLY_SAFE mode,


independent

of

WAL

compaction enabled flag. It's pretty significant drop: WAL
compaction
itself gives only ~3% 

[jira] [Created] (IGNITE-8057) Execute WAL fsync preventively before checkpoint begin

[Ivan Rakov (JIRA)]

Ivan Rakov created IGNITE-8057:
--

 Summary: Execute WAL fsync preventively before checkpoint begin
 Key: IGNITE-8057
 URL: https://issues.apache.org/jira/browse/IGNITE-8057
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 2.5
Reporter: Ivan Rakov


After fix of IGNITE-7754, we execute explicit WAL fsync on checkpoint begin:
{noformat}
if (hasPages) {
assert cpPtr != null;

tracker.onWalCpRecordFsyncStart();

// Sync log outside the checkpoint write lock.
cctx.wal().flush(cpPtr, true);

tracker.onWalCpRecordFsyncEnd();
{noformat}
It's executed outside of checkpoint write lock. However, it still can decrease 
overall throughput by suspending writing of dirty pages by checkpoint threads.
We can decrease time of this fsync by executing it preemptevely, before 
acquiring checkpoint write lock. 
We should prioritize this ticket if value of walCpRecordFsyncDuration metric in 
"Checkpoint started" message will be too big.
Note: it's possible to give a fsync hint to WAL manager in single-writer mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3628: IGNITE-6879 and migration to spring-data 2.0.5.RE...

[homich1991]

Github user homich1991 closed the pull request at:

https://github.com/apache/ignite/pull/3628


---

[GitHub] ignite pull request #3703: IGNITE-7647

[vveider]

GitHub user vveider opened a pull request:

https://github.com/apache/ignite/pull/3703

IGNITE-7647



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7647

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3703.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3703


commit 795d8ac6b56162b5f4e4c4897496c753a41f6347
Author: Ivanov Petr 
Date:   2018-03-20T15:26:54Z

IGNITE-7647




---

[jira] [Created] (IGNITE-8056) Ignite Cassandra integration doesn't correctly handle Inet4Address and Inet6Address objects

[Nick Jacobsen (JIRA)]

Nick Jacobsen created IGNITE-8056:
-

 Summary: Ignite Cassandra integration doesn't correctly handle 
Inet4Address and Inet6Address objects
 Key: IGNITE-8056
 URL: https://issues.apache.org/jira/browse/IGNITE-8056
 Project: Ignite
  Issue Type: Bug
  Components: cache, cassandra
Affects Versions: 2.4
Reporter: Nick Jacobsen


PropertyMappingHelper.getCassandraType correctly handles the inet <-> 
java.net.InetAddress type, but not it's sub classes of java.net.Inet4Address or 
java.net.Inet6Address.  As there is no standard way to directly create an 
InetAddress, this makes it very difficult to use an inet data types for keys in 
cassandra.

 

In general, shouldn't Cassandra types handle any subclasses of supported 
classes?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8055) Sqline command !tables works incorrect for client node

[Andrey Aleksandrov (JIRA)]

Andrey Aleksandrov created IGNITE-8055:
--

 Summary: Sqline command !tables works incorrect for client node
 Key: IGNITE-8055
 URL: https://issues.apache.org/jira/browse/IGNITE-8055
 Project: Ignite
  Issue Type: Bug
  Components: jdbc
Affects Versions: 2.4
Reporter: Andrey Aleksandrov


For reproducing:

You should start one local server and one local client nodes and follow the 
instructions:


1)Connect to server node:

sqlline.bat --color=true --verbose=true -u jdbc:ignite:thin://127.0.0.1:10800

2)Create new table on server node:

CREATE TABLE City (id LONG PRIMARY KEY, name VARCHAR)WITH "template=replicated";

3)Check that table exists from server node:

!tables

On this step table should be shown in the response.

4)Connect to client node:

sqlline.bat --color=true --verbose=true -u jdbc:ignite:thin://127.0.0.1:10801

5)Check that table exists from server node:

!tables

*On this step there is no "city" table in the list.*

Next commands work from client node as well:
SELECT * FROM City
DROP TABLE City
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Data compression design proposal

[Anton Vinogradov]

>> page compression at rebalancing is a good idea even is we have problems
with storing on disc.
BTW, do we have or going to have rebalancing based on pages streaming
instead of entries streaming?

2018-03-27 3:03 GMT+03:00 Dmitriy Setrakyan :

> AG,
>
> I would also ask about the compression itself. How and where do we store
> the compression meta information? We cannot be compressing every page
> separately, it will not be effective. However, if we try to store the
> compression metadata, how do we make other nodes aware of it? Has this been
> discussed?
>
> D.
>
> On Mon, Mar 26, 2018 at 8:53 AM, Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
> > Guys,
> >
> > How does this fit the PageMemory concept? Currently it assumes that the
> > size of the page in memory and the size of the page on disk is the same,
> so
> > only per-entry level compression within a page makes sense.
> >
> > If you compress a whole page, how do you calculate the page offset in the
> > target data file?
> >
> > --AG
> >
> > 2018-03-26 17:39 GMT+03:00 Vladimir Ozerov :
> >
> > > Gents,
> > >
> > > If I understood the idea correctly, the proposal is to compress pages
> on
> > > eviction and decompress them on read from disk. Is it correct?
> > >
> > > On Mon, Mar 26, 2018 at 5:13 PM, Anton Vinogradov 
> wrote:
> > >
> > > > + 1 to Taras's vision.
> > > >
> > > > Compression on eviction is a good case to store more.
> > > > Pages at memory always hot a real system, so complession in memory
> will
> > > > definetely slowdown the system, I think.
> > > >
> > > > Anyway, we can split issue to "on eviction compression" and to
> > "in-memory
> > > > compression".
> > > >
> > > >
> > > > 2018-03-06 12:14 GMT+03:00 Taras Ledkov :
> > > >
> > > > > Hi,
> > > > >
> > > > > I guess page level compression make sense on page loading /
> eviction.
> > > > > In this case we can decrease I/O operation and performance boost
> can
> > be
> > > > > reached.
> > > > > What is goal for in-memory compression? Holds about 2-5x data in
> > memory
> > > > > with performance drop?
> > > > >
> > > > > Also please clarify the case with compression/decompression for hot
> > and
> > > > > cold pages.
> > > > > Is it right for your approach:
> > > > > 1. Hot pages are always decompressed in memory because many
> > read/write
> > > > > operations touch ones.
> > > > > 2. So we can compress only cold pages.
> > > > >
> > > > > So the way is suitable when the hot data size << available RAM
> size.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > >
> > > > > On 05.03.2018 20:18, Vyacheslav Daradur wrote:
> > > > >
> > > > >> Hi Igniters!
> > > > >>
> > > > >> I’d like to do next step in our data compression discussion [1].
> > > > >>
> > > > >> Most Igniters vote for per-data-page compression.
> > > > >>
> > > > >> I’d like to accumulate  main theses to start implementation:
> > > > >> - page will be compressed with the dictionary-based approach
> > (e.g.LZV)
> > > > >> - page will be compressed in batch mode (not on every change)
> > > > >> - page compression should been initiated by an event, for
> example, a
> > > > >> page’s free space drops below 20%
> > > > >> - compression process will be under page write lock
> > > > >>
> > > > >> Vladimir Ozerov has written:
> > > > >>
> > > > >>> What we do not understand yet:
> > > >  1) Granularity of compression algorithm.
> > > >  1.1) It could be per-entry - i.e. we compress the whole entry
> > > content,
> > > >  but
> > > >  respect boundaries between entries. E.g.: before -
> > > [ENTRY_1][ENTRY_2],
> > > >  after - [COMPRESSED_ENTRY_1][COMPRESSED_ENTRY_2] (as opposed to
> > > >  [COMPRESSED ENTRY_1 and ENTRY_2]).
> > > >  v1.2) Or it could be per-field - i.e. we compress fields, but
> > > respect
> > > >  binary
> > > >  object layout. First approach is simple, straightforward, and
> will
> > > > give
> > > >  acceptable compression rate, but we will have to compress the
> > whole
> > > >  binary
> > > >  object on every field access, what may ruin our SQL performance.
> > > > Second
> > > >  approach is more complex, we are not sure about it's compression
> > > rate,
> > > >  but
> > > >  as BinaryObject structure is preserved, we will still have fast
> > > >  constant-time per-field access.
> > > > 
> > > > >>> I think there are advantages in both approaches and we will be
> able
> > > to
> > > > >> compare different approaches and algorithms after prototype
> > > > >> implementation.
> > > > >>
> > > > >> Main approach in brief:
> > > > >> 1) When page’s free space drops below 20% will be triggered
> > > compression
> > > > >> event
> > > > >> 2) Page will be locked by write lock
> > > > >> 3) Page will be passed to page’s compressor implementation
> > > > >> 4) Page will be replaced by compressed page
> > > > >>
> > > > >> Whole object or a field 

Re: Optimize GridLongList serialization

[Александр Меньшиков]

The ticket is created:
https://issues.apache.org/jira/browse/IGNITE-8054

2018-03-27 10:04 GMT+03:00 Dmitriy Setrakyan :

> Sorry, Dmitiry, I meant Ignite, not GridGain (memories of pre-apache
> coding). I am assuming that Alex Menshikov was referring to GridLongList
> class in Ignite.
>
> D.
>
> On Mon, Mar 26, 2018 at 11:52 PM, Dmitry Pavlov 
> wrote:
>
> > Hi Dmitriy, did you mean GridList?
> >
> > I don't understand what does it mean GridGain compress.
> >
> > вт, 27 мар. 2018 г. в 3:06, Dmitriy Setrakyan :
> >
> > > Thanks, Alex.
> > >
> > > GridGain automatically compresses all the internal types. Somehow it
> > looks
> > > like the GridLongList may have been mixed. Can you please file a ticket
> > for
> > > 2.5 release?
> > >
> > > D.
> > >
> > > On Mon, Mar 26, 2018 at 4:55 AM, Александр Меньшиков <
> > sharple...@gmail.com
> > > >
> > > wrote:
> > >
> > > > I investigated network loading and found that a big part of internal
> > data
> > > > inside messages is `GridLongList`.
> > > > It is a part of `GridDhtTxFinishRequest`,
> > > > `GridDhtAtomicDeferredUpdateResponse`, `GridDhtAtomicUpdateRequest`,
> > > > `GridNearAtomicFullUpdateRequest` and `NearCacheUpdates`.
> > > >
> > > > So I think it has the sense to optimize `GridLongList` serialization.
> > > >
> > > >
> > > > Here we serialize all elements and don't take into account `idx`
> value:
> > > >
> > > > ```
> > > >
> > > > @Override public boolean writeTo(ByteBuffer buf, MessageWriter
> writer)
> > {
> > > >
> > > > writer.setBuffer(buf);
> > > >
> > > >
> > > >
> > > > if (!writer.isHeaderWritten()) {
> > > >
> > > > if (!writer.writeHeader(directType(), fieldsCount()))
> > > >
> > > > return false;
> > > >
> > > >
> > > >
> > > > writer.onHeaderWritten();
> > > >
> > > > }
> > > >
> > > >
> > > >
> > > > switch (writer.state()) {
> > > >
> > > > case 0:
> > > >
> > > > if (!writer.writeLongArray("arr", arr))
> > > >
> > > > return false;
> > > >
> > > >
> > > >
> > > > writer.incrementState();
> > > >
> > > >
> > > >
> > > > case 1:
> > > >
> > > > if (!writer.writeInt("idx", idx))
> > > >
> > > > return false;
> > > >
> > > >
> > > >
> > > > writer.incrementState();
> > > >
> > > >
> > > >
> > > > }
> > > >
> > > >
> > > >
> > > > return true;
> > > >
> > > > }
> > > >
> > > > ```
> > > >
> > > >
> > > >
> > > > Which is not happening in another serialization method in the same
> > class:
> > > >
> > > >
> > > >
> > > > ```
> > > >
> > > > public static void writeTo(DataOutput out, @Nullable GridLongList
> list)
> > > > throws IOException {
> > > >
> > > > out.writeInt(list != null ? list.idx : -1);
> > > >
> > > >
> > > >
> > > > if (list != null) {
> > > >
> > > > for (int i = 0; i < list.idx; i++)
> > > >
> > > > out.writeLong(list.arr[i]);
> > > >
> > > > }
> > > >
> > > > }
> > > >
> > > > ```
> > > >
> > > >
> > > > So, we can simply reduce messages size by sending only a valuable
> part
> > of
> > > > the array.
> > > > If you don't mind I will create an issue in Jira for this.
> > > >
> > > >
> > > > By the way, `long` is a huge type. As I see in most cases
> > `GridLongList`
> > > > uses for counters.
> > > > And I have checked the possibility of compress `long` into smaller
> > types
> > > as
> > > > `int`, `short` or `byte` in test
> > > > `GridCacheInterceptorAtomicRebalanceTest` (took it by random).
> > > > And found out that all `long` in`GridLongList` can be cast to `int`
> and
> > > 70%
> > > > of them to shorts.
> > > > Such conversion is quite fast about 1.1 (ns) per element (I have
> > checked
> > > it
> > > > by JMH test).
> > > >
> > > >
> > > >
> > > > Of course, there are a lot of ways to compress data,
> > > > but I know proprietary GridGain plug-in has different `MessageWriter`
> > > > implementation.
> > > > So maybe it is unnecessary and some compression already exists in
> this
> > > > proprietary plug-in.
> > > > Does someone know something about it?
> > > >
> > >
> >
>

[jira] [Created] (IGNITE-8054) Let serialize only valuable part of GridLongList

[Alexander Menshikov (JIRA)]

Alexander Menshikov created IGNITE-8054:
---

 Summary: Let serialize only valuable part of GridLongList
 Key: IGNITE-8054
 URL: https://issues.apache.org/jira/browse/IGNITE-8054
 Project: Ignite
  Issue Type: Bug
  Components: messaging
Affects Versions: 2.4
Reporter: Alexander Menshikov
 Fix For: 2.5


Here in GridLongList we serialize all elements and don't take into account 
`idx` value:


{code:java}
@Override public boolean writeTo(ByteBuffer buf, MessageWriter writer) { 

    writer.setBuffer(buf); 

  

    if (!writer.isHeaderWritten()) { 

    if (!writer.writeHeader(directType(), fieldsCount())) 

    return false; 

  

    writer.onHeaderWritten(); 

    } 

  

    switch (writer.state()) { 

    case 0: 
    if (!writer.writeLongArray("arr", arr)) 

    return false; 

  

    writer.incrementState(); 

  

    case 1: 

    if (!writer.writeInt("idx", idx)) 

    return false; 

  

    writer.incrementState(); 

  

    } 

  

    return true; 

    } {code}

Which is not happening in another serialization method in the same class: 


{code:java}
public static void writeTo(DataOutput out, @Nullable GridLongList list) throws 
IOException { 

    out.writeInt(list != null ? list.idx : -1); 

  

    if (list != null) { 

    for (int i = 0; i < list.idx; i++) 

    out.writeLong(list.arr[i]); 

    } 

} {code}

So, we can simply reduce messages size by sending only a valuable part of the 
array.

 

 

I created this issue according to a discussion on the mailing list:

http://apache-ignite-developers.2346864.n4.nabble.com/Optimize-GridLongList-serialization-td28571.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3702: IGNITE-7884

[devozerov]

GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/3702

IGNITE-7884



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7884

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3702.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3702


commit 6b220002c3f3983219c4e09bd39622637038da41
Author: devozerov 
Date:   2018-03-27T08:23:30Z

WIP on bulk load.

commit 6a967d2fddea265c101cb183e176ffbfeec1c5eb
Author: devozerov 
Date:   2018-03-27T08:32:41Z

Minor.




---

[GitHub] ignite pull request #3696: IGNITE-8034 .NET: Add IgniteConfiguration.Authent...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3696


---

Re: .NET: Add "authenticationEnabled" flag to IgntieConfiguration

[Pavel Tupitsyn]

IGNITE-8034 done, I have added configuration flag.

However, I still have concerns about property naming:
Is this authentication going to be used for thin clients only, or for
classic client and server nodes as well?

On Tue, Mar 27, 2018 at 9:50 AM, Dmitry Pavlov 
wrote:

> Hi Folks,
>
> Test is still failing on TC.
> https://ci.ignite.apache.org/viewLog.html?buildId=1160178;
> tab=buildResultsDiv=IgniteTests24Java8_
> IgnitePlatformNetCoreLinux
>
> I suggest to mute it until fix is ready.Test is running in Basic Run All
> subsets, so each commit will genereate TC failure email. What do you think?
>
> Sincerely.
> Dmitriy Pavlov.
>
> вт, 27 мар. 2018 г. в 2:22, Denis Magda :
>
> > Taras,
> >
> > Please document the authentication part on a 2.5 version of the binary
> > protocol page on readme.io. We need to share it with the guys who are
> > developing Node.JS client right now.
> >
> > --
> > Denis
> >
> > On Mon, Mar 26, 2018 at 2:55 AM, Taras Ledkov 
> > wrote:
> >
> > > I had implement user credential store according with the previous
> > > discussion about user authentication.
> > >
> > > Now JDBC thin, and ODBC support user authentication. We haven't
> > > implemented it for all thin client because protocols are not similar.
> > >
> > > I see two ways to implement authentication for thin client protocol:
> > > - You implement authentication on server side and at the .NET;
> > > - When java thin client [1] is merged I implement authentication for
> thin
> > > protocol & for java thin client.
> > >
> > > I'll add documentation for user authentication ASAP. Please feel free
> to
> > > contact if you need more info till documentation is added.
> > >
> > > [1]. https://issues.apache.org/jira/browse/IGNITE-7421
> > >
> > >
> > > On 26.03.2018 9:56, Pavel Tupitsyn wrote:
> > >
> > >> I've started this task, and the property name combined with lack of
> > >> javadoc
> > >> seems confusing and misleading:
> > >>
> > >> * Turns out this authentication is only for thin clients
> > >> * Not clear how to configure and use it, even after digging through
> Jira
> > >> and devlist
> > >>
> > >> How do I write test to ensure it works?
> > >>
> > >> Thanks,
> > >> Pavel
> > >>
> > >> On Fri, Mar 23, 2018 at 6:44 PM, Pavel Tupitsyn  >
> > >> wrote:
> > >>
> > >> Thanks, got it, will do.
> > >>>
> > >>> On Fri, Mar 23, 2018 at 4:36 PM, Dmitry Pavlov <
> dpavlov@gmail.com>
> > >>> wrote:
> > >>>
> > >>> Hi Pavel,
> > 
> >  Related ticket is https://issues.apache.org/jira/browse/IGNITE-7436
> > 
> >  Sincerely,
> >  Dmitriy Pavlov
> > 
> >  пт, 23 мар. 2018 г. в 16:24, Pavel Tupitsyn :
> > 
> >  Please provide description in IGNITE-8034 and link Java-side ticket
> > >
> >  there.
> > 
> > > On Fri, Mar 23, 2018 at 4:23 PM, Pavel Tupitsyn <
> > ptupit...@apache.org>
> > > wrote:
> > >
> > > Hi Vladimir,
> > >>
> > >> Can you provide more details?
> > >> * What does it do?
> > >> * Do we need to only propagate the flag to .NET or do anything
> else?
> > >> * Related ticket?
> > >>
> > >> Thanks,
> > >> Pavel
> > >>
> > >> On Fri, Mar 23, 2018 at 2:25 PM, Vladimir Ozerov <
> > >>
> > > voze...@gridgain.com>
> > 
> > > wrote:
> > >>
> > >> Pavel,
> > >>>
> > >>> We introduced new flag IgniteConfiguration.authenticationEnabled
> > >>>
> > >> recently.
> > >
> > >> Would you mind adding it to IgniteConfigutation.cs [1]?
> > >>>
> > >>> Vladimir.
> > >>>
> > >>> [1] https://issues.apache.org/jira/browse/IGNITE-8034
> > >>>
> > >>>
> > >>
> > >>>
> > > --
> > > Taras Ledkov
> > > Mail-To: tled...@gridgain.com
> > >
> > >
> >
>

[jira] [Created] (IGNITE-8053) Exception during checkpoint concurrent changes in topology

[Sergey Kosarev (JIRA)]

Sergey Kosarev created IGNITE-8053:
--

 Summary: Exception during checkpoint concurrent changes in topology
 Key: IGNITE-8053
 URL: https://issues.apache.org/jira/browse/IGNITE-8053
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Affects Versions: 2.4
Reporter: Sergey Kosarev






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Optimize GridLongList serialization

[Dmitriy Setrakyan]

Sorry, Dmitiry, I meant Ignite, not GridGain (memories of pre-apache
coding). I am assuming that Alex Menshikov was referring to GridLongList
class in Ignite.

D.

On Mon, Mar 26, 2018 at 11:52 PM, Dmitry Pavlov 
wrote:

> Hi Dmitriy, did you mean GridList?
>
> I don't understand what does it mean GridGain compress.
>
> вт, 27 мар. 2018 г. в 3:06, Dmitriy Setrakyan :
>
> > Thanks, Alex.
> >
> > GridGain automatically compresses all the internal types. Somehow it
> looks
> > like the GridLongList may have been mixed. Can you please file a ticket
> for
> > 2.5 release?
> >
> > D.
> >
> > On Mon, Mar 26, 2018 at 4:55 AM, Александр Меньшиков <
> sharple...@gmail.com
> > >
> > wrote:
> >
> > > I investigated network loading and found that a big part of internal
> data
> > > inside messages is `GridLongList`.
> > > It is a part of `GridDhtTxFinishRequest`,
> > > `GridDhtAtomicDeferredUpdateResponse`, `GridDhtAtomicUpdateRequest`,
> > > `GridNearAtomicFullUpdateRequest` and `NearCacheUpdates`.
> > >
> > > So I think it has the sense to optimize `GridLongList` serialization.
> > >
> > >
> > > Here we serialize all elements and don't take into account `idx` value:
> > >
> > > ```
> > >
> > > @Override public boolean writeTo(ByteBuffer buf, MessageWriter writer)
> {
> > >
> > > writer.setBuffer(buf);
> > >
> > >
> > >
> > > if (!writer.isHeaderWritten()) {
> > >
> > > if (!writer.writeHeader(directType(), fieldsCount()))
> > >
> > > return false;
> > >
> > >
> > >
> > > writer.onHeaderWritten();
> > >
> > > }
> > >
> > >
> > >
> > > switch (writer.state()) {
> > >
> > > case 0:
> > >
> > > if (!writer.writeLongArray("arr", arr))
> > >
> > > return false;
> > >
> > >
> > >
> > > writer.incrementState();
> > >
> > >
> > >
> > > case 1:
> > >
> > > if (!writer.writeInt("idx", idx))
> > >
> > > return false;
> > >
> > >
> > >
> > > writer.incrementState();
> > >
> > >
> > >
> > > }
> > >
> > >
> > >
> > > return true;
> > >
> > > }
> > >
> > > ```
> > >
> > >
> > >
> > > Which is not happening in another serialization method in the same
> class:
> > >
> > >
> > >
> > > ```
> > >
> > > public static void writeTo(DataOutput out, @Nullable GridLongList list)
> > > throws IOException {
> > >
> > > out.writeInt(list != null ? list.idx : -1);
> > >
> > >
> > >
> > > if (list != null) {
> > >
> > > for (int i = 0; i < list.idx; i++)
> > >
> > > out.writeLong(list.arr[i]);
> > >
> > > }
> > >
> > > }
> > >
> > > ```
> > >
> > >
> > > So, we can simply reduce messages size by sending only a valuable part
> of
> > > the array.
> > > If you don't mind I will create an issue in Jira for this.
> > >
> > >
> > > By the way, `long` is a huge type. As I see in most cases
> `GridLongList`
> > > uses for counters.
> > > And I have checked the possibility of compress `long` into smaller
> types
> > as
> > > `int`, `short` or `byte` in test
> > > `GridCacheInterceptorAtomicRebalanceTest` (took it by random).
> > > And found out that all `long` in`GridLongList` can be cast to `int` and
> > 70%
> > > of them to shorts.
> > > Such conversion is quite fast about 1.1 (ns) per element (I have
> checked
> > it
> > > by JMH test).
> > >
> > >
> > >
> > > Of course, there are a lot of ways to compress data,
> > > but I know proprietary GridGain plug-in has different `MessageWriter`
> > > implementation.
> > > So maybe it is unnecessary and some compression already exists in this
> > > proprietary plug-in.
> > > Does someone know something about it?
> > >
> >
>

Re: Optimize GridLongList serialization

[Dmitry Pavlov]

Hi Dmitriy, did you mean GridList?

I don't understand what does it mean GridGain compress.

вт, 27 мар. 2018 г. в 3:06, Dmitriy Setrakyan :

> Thanks, Alex.
>
> GridGain automatically compresses all the internal types. Somehow it looks
> like the GridLongList may have been mixed. Can you please file a ticket for
> 2.5 release?
>
> D.
>
> On Mon, Mar 26, 2018 at 4:55 AM, Александр Меньшиков  >
> wrote:
>
> > I investigated network loading and found that a big part of internal data
> > inside messages is `GridLongList`.
> > It is a part of `GridDhtTxFinishRequest`,
> > `GridDhtAtomicDeferredUpdateResponse`, `GridDhtAtomicUpdateRequest`,
> > `GridNearAtomicFullUpdateRequest` and `NearCacheUpdates`.
> >
> > So I think it has the sense to optimize `GridLongList` serialization.
> >
> >
> > Here we serialize all elements and don't take into account `idx` value:
> >
> > ```
> >
> > @Override public boolean writeTo(ByteBuffer buf, MessageWriter writer) {
> >
> > writer.setBuffer(buf);
> >
> >
> >
> > if (!writer.isHeaderWritten()) {
> >
> > if (!writer.writeHeader(directType(), fieldsCount()))
> >
> > return false;
> >
> >
> >
> > writer.onHeaderWritten();
> >
> > }
> >
> >
> >
> > switch (writer.state()) {
> >
> > case 0:
> >
> > if (!writer.writeLongArray("arr", arr))
> >
> > return false;
> >
> >
> >
> > writer.incrementState();
> >
> >
> >
> > case 1:
> >
> > if (!writer.writeInt("idx", idx))
> >
> > return false;
> >
> >
> >
> > writer.incrementState();
> >
> >
> >
> > }
> >
> >
> >
> > return true;
> >
> > }
> >
> > ```
> >
> >
> >
> > Which is not happening in another serialization method in the same class:
> >
> >
> >
> > ```
> >
> > public static void writeTo(DataOutput out, @Nullable GridLongList list)
> > throws IOException {
> >
> > out.writeInt(list != null ? list.idx : -1);
> >
> >
> >
> > if (list != null) {
> >
> > for (int i = 0; i < list.idx; i++)
> >
> > out.writeLong(list.arr[i]);
> >
> > }
> >
> > }
> >
> > ```
> >
> >
> > So, we can simply reduce messages size by sending only a valuable part of
> > the array.
> > If you don't mind I will create an issue in Jira for this.
> >
> >
> > By the way, `long` is a huge type. As I see in most cases `GridLongList`
> > uses for counters.
> > And I have checked the possibility of compress `long` into smaller types
> as
> > `int`, `short` or `byte` in test
> > `GridCacheInterceptorAtomicRebalanceTest` (took it by random).
> > And found out that all `long` in`GridLongList` can be cast to `int` and
> 70%
> > of them to shorts.
> > Such conversion is quite fast about 1.1 (ns) per element (I have checked
> it
> > by JMH test).
> >
> >
> >
> > Of course, there are a lot of ways to compress data,
> > but I know proprietary GridGain plug-in has different `MessageWriter`
> > implementation.
> > So maybe it is unnecessary and some compression already exists in this
> > proprietary plug-in.
> > Does someone know something about it?
> >
>

Re: .NET: Add "authenticationEnabled" flag to IgntieConfiguration

[Dmitry Pavlov]

Hi Folks,

Test is still failing on TC.
https://ci.ignite.apache.org/viewLog.html?buildId=1160178=buildResultsDiv=IgniteTests24Java8_IgnitePlatformNetCoreLinux

I suggest to mute it until fix is ready.Test is running in Basic Run All
subsets, so each commit will genereate TC failure email. What do you think?

Sincerely.
Dmitriy Pavlov.

вт, 27 мар. 2018 г. в 2:22, Denis Magda :

> Taras,
>
> Please document the authentication part on a 2.5 version of the binary
> protocol page on readme.io. We need to share it with the guys who are
> developing Node.JS client right now.
>
> --
> Denis
>
> On Mon, Mar 26, 2018 at 2:55 AM, Taras Ledkov 
> wrote:
>
> > I had implement user credential store according with the previous
> > discussion about user authentication.
> >
> > Now JDBC thin, and ODBC support user authentication. We haven't
> > implemented it for all thin client because protocols are not similar.
> >
> > I see two ways to implement authentication for thin client protocol:
> > - You implement authentication on server side and at the .NET;
> > - When java thin client [1] is merged I implement authentication for thin
> > protocol & for java thin client.
> >
> > I'll add documentation for user authentication ASAP. Please feel free to
> > contact if you need more info till documentation is added.
> >
> > [1]. https://issues.apache.org/jira/browse/IGNITE-7421
> >
> >
> > On 26.03.2018 9:56, Pavel Tupitsyn wrote:
> >
> >> I've started this task, and the property name combined with lack of
> >> javadoc
> >> seems confusing and misleading:
> >>
> >> * Turns out this authentication is only for thin clients
> >> * Not clear how to configure and use it, even after digging through Jira
> >> and devlist
> >>
> >> How do I write test to ensure it works?
> >>
> >> Thanks,
> >> Pavel
> >>
> >> On Fri, Mar 23, 2018 at 6:44 PM, Pavel Tupitsyn 
> >> wrote:
> >>
> >> Thanks, got it, will do.
> >>>
> >>> On Fri, Mar 23, 2018 at 4:36 PM, Dmitry Pavlov 
> >>> wrote:
> >>>
> >>> Hi Pavel,
> 
>  Related ticket is https://issues.apache.org/jira/browse/IGNITE-7436
> 
>  Sincerely,
>  Dmitriy Pavlov
> 
>  пт, 23 мар. 2018 г. в 16:24, Pavel Tupitsyn :
> 
>  Please provide description in IGNITE-8034 and link Java-side ticket
> >
>  there.
> 
> > On Fri, Mar 23, 2018 at 4:23 PM, Pavel Tupitsyn <
> ptupit...@apache.org>
> > wrote:
> >
> > Hi Vladimir,
> >>
> >> Can you provide more details?
> >> * What does it do?
> >> * Do we need to only propagate the flag to .NET or do anything else?
> >> * Related ticket?
> >>
> >> Thanks,
> >> Pavel
> >>
> >> On Fri, Mar 23, 2018 at 2:25 PM, Vladimir Ozerov <
> >>
> > voze...@gridgain.com>
> 
> > wrote:
> >>
> >> Pavel,
> >>>
> >>> We introduced new flag IgniteConfiguration.authenticationEnabled
> >>>
> >> recently.
> >
> >> Would you mind adding it to IgniteConfigutation.cs [1]?
> >>>
> >>> Vladimir.
> >>>
> >>> [1] https://issues.apache.org/jira/browse/IGNITE-8034
> >>>
> >>>
> >>
> >>>
> > --
> > Taras Ledkov
> > Mail-To: tled...@gridgain.com
> >
> >
>