[GitHub] ignite pull request #2425: IGNITE-5097 BinaryMarshaller should write ints in...

[daradurvs]

GitHub user daradurvs opened a pull request:

https://github.com/apache/ignite/pull/2425

IGNITE-5097 BinaryMarshaller should write ints in "varint" encoding where 
it makes sense



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/daradurvs/ignite ignite-5736-module

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2425.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2425


commit efbac32e650d603e18ea405bee626589ac237f61
Author: Vyacheslav Daradur 
Date:   2017-08-09T14:05:29Z

IGNITE-5732 Provide API to test compatibility with old releases (#20)

commit 0a7ffcce6527111a68568da0a6d22d4a1eb306a4
Author: Vyacheslav Daradur 
Date:   2017-08-09T14:09:08Z

IGNITE-5097 BinaryMarshaller should write ints in "varint" encoding where 
it makes sense (#21)

commit 6b34d93609bc2087b77d7b44a0665ea160965e4e
Author: Vyacheslav Daradur 
Date:   2017-08-09T17:08:50Z

ignite-5736: added VarintArraysSizeCompatibilityTest




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2418: [IGNITE-6002] Make log message more pretty when i...

[sarutak]

GitHub user sarutak opened a pull request:

https://github.com/apache/ignite/pull/2418

[IGNITE-6002] Make log message more pretty when instance name is null

When instance name is null, log message containing "null" is shown like as 
follows.

```
[09-08-2017 14:49:38][INFO ][grid-timeout-worker-#15%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=de0743ec, name=null, uptime=00:01:00:007]
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sarutak/ignite IGNITE-6002

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2418.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2418


commit cb1d8bafb343d4365f3b47552ecdac9a3b0aea21
Author: Kousuke Saruta 
Date:   2017-08-09T06:28:35Z

Fix log message to make it more pretty




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2202: IGNITE-5076: improved multi-threaded start of nod...

[sk0x50]

Github user sk0x50 closed the pull request at:

https://github.com/apache/ignite/pull/2202


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2194: IGNITE-5076: improved multi-threaded start of nod...

[sk0x50]

Github user sk0x50 closed the pull request at:

https://github.com/apache/ignite/pull/2194


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2259: Ignite 5562 Assertions in TCP discovery SPI when ...

[sk0x50]

Github user sk0x50 closed the pull request at:

https://github.com/apache/ignite/pull/2259


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-6003) Detecting low memory on ignite node startup

[Sergey Chugunov (JIRA)]

Sergey Chugunov created IGNITE-6003:
---

 Summary: Detecting low memory on ignite node startup
 Key: IGNITE-6003
 URL: https://issues.apache.org/jira/browse/IGNITE-6003
 Project: Ignite
  Issue Type: Improvement
Reporter: Sergey Chugunov


h2. Motivation
Starting from version 2.0 Ignite keeps all data offheap and allocates it on 
startup based on MemoryPolicy settings.
So several Ignite nodes started on the same machine may allocate an order of 
magnitude more memory than physically available so when putting pressure on 
such cluster will cause the machine to hang.

h2. Acceptance Criteria
# Reliable cross-platform algorithm to detect low memory is developed.
# Ignite node fails to start with descriptive error message when the algorithm 
detects low memory which may cause hang.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6004) SQL: add "inlineSize" to QuerySqlField annotation

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-6004:
---

 Summary: SQL: add "inlineSize" to QuerySqlField annotation
 Key: IGNITE-6004
 URL: https://issues.apache.org/jira/browse/IGNITE-6004
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.1
Reporter: Vladimir Ozerov
Assignee: Taras Ledkov
 Fix For: 2.2


"inlineSize" is very important optimization allowing to avoid excessive data 
page reads when using index. However, it can only be set through 
{{QueryEntity}} -> {{QueryIndex}} chain, and cannot be set through 
{{\@QuerySqlField}} annotation.

Let's fix that and add {{int indexInlineSize}} parameter to that annotation.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: IGNITE-5994: IgniteInternalCache.invokeAsync().get() can return null

[Yakov Zhdanov]

Replied in ticket. Also posting here.

Currently invokeAsync() is defined as public  IgniteFuture
invokeAsync(.).

If we need EntryProcessorResult to be returned from Future then it should
be public  IgniteFuture invokeAsync(.). I
am not sure why we have this difference in sync/async counterpart.
Probably, Vladimir Ozerov
 or Taras
Ledkov

can
provide some info.

I also think that we will not changing the public API in 2.0. Let's leave
it as is and then fix in next major version.

--Yakov

2017-08-08 20:05 GMT+03:00 Александр Меньшиков :

> Hello, I found one small bug.
>
> JIRA: https://issues.apache.org/jira/browse/IGNITE-5994
>
> The problem is IgniteInternalCache.invokeAsync().get() will return null if
> EntryProcessor return null. But the IgniteInternalCache.invoke() will
> return EntryProcessorResult in the same situation (with null inside). It
> can be optimization, and I guess we have to change invoke()'s result in
> that case. Or it's a bug, and we have to change the result of
> invokeAsync().
>
> If EntryProcessor returns not null everything is okay.
>
> It doesn't relate with IgniteCache.invoke*.
>

[GitHub] ignite pull request #2414: IGNITE-5993

[devozerov]

Github user devozerov closed the pull request at:

https://github.com/apache/ignite/pull/2414


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [IGNITE-5717] improvements of MemoryPolicy default size

[Sergey Chugunov]

Folks,

I filed a ticket [1] to address the concern of starting too many nodes.
Please review it.

[1] https://issues.apache.org/jira/browse/IGNITE-6003

Thanks,
Sergey.

On Mon, Aug 7, 2017 at 12:53 PM, Sergey Chugunov 
wrote:

> Dmitriy,
>
> When Ignite node "allocates memory" it actually just reserves a chunk in
> its address space, almost no physical RAM is used.
>
> I can easily start half a dozen of ignite nodes with current defaults on
> my laptop with only 16 Gigs of RAM; and each node will "allocate" around 12
> Gigs; 72 gigabytes in total.
> The laptop will do easily with it so far I don't stream any data to the
> grid.
>
> But when I put some pressure to the grid, massive swapping of memory pages
> will show up as OS begins trying to keep a huge amount of pages of
> different processes in memory.
>
> So indicator "we are running out of memory" just doesn't work here.
>
> Thanks,
> Sergey.
>
> On Fri, Aug 4, 2017 at 1:01 PM,  wrote:
>
>> But why? We allocate the memory, so we should know when it runs out. What
>> am i missing?
>>
>> ⁣D.​
>>
>> On Aug 4, 2017, 11:55 AM, at 11:55 AM, Sergey Chugunov <
>> sergey.chugu...@gmail.com> wrote:
>> >I used GC and java only as an example, they are not applicable to
>> >Ignite
>> >case where we manage offheap memory.
>> >
>> >My point is that there is no easy way to implement this feature in
>> >Ignite,
>> >and more time is needed to properly design it and account for all
>> >risks.
>> >
>> >Thanks,
>> >Sergey.
>> >
>> >On Fri, Aug 4, 2017 at 12:44 PM,  wrote:
>> >
>> >> Hang on. I thought we were talking about offheap size, GC should not
>> >be
>> >> relevant. Am I wrong?
>> >>
>> >> ⁣D.​
>> >>
>> >> On Aug 4, 2017, 11:38 AM, at 11:38 AM, Sergey Chugunov <
>> >> sergey.chugu...@gmail.com> wrote:
>> >> >Do you see an obvious way of implementing it?
>> >> >
>> >> >In java there is a heap and GC working on it. And for instance, it
>> >is
>> >> >possible to make a decision to throw an OOM based on some gc
>> >metrics.
>> >> >
>> >> >I may be wrong but I don't see a mechanism in Ignite to use it right
>> >> >away
>> >> >for such purposes.
>> >> >And implementing something without thorough planning brings huge
>> >risk
>> >> >of
>> >> >false positives with nodes stopping when they don't have to.
>> >> >
>> >> >That's why I think it must be implemented and intensively tested as
>> >> >part of
>> >> >a separate ticket.
>> >> >
>> >> >Thanks,
>> >> >Sergey.
>> >> >
>> >> >On Fri, Aug 4, 2017 at 12:18 PM,  wrote:
>> >> >
>> >> >> Without #3, the #1 and #2 make little sense.
>> >> >>
>> >> >> Why is #3 so difficult?
>> >> >>
>> >> >> ⁣D.​
>> >> >>
>> >> >> On Aug 4, 2017, 10:46 AM, at 10:46 AM, Sergey Chugunov <
>> >> >> sergey.chugu...@gmail.com> wrote:
>> >> >> >Dmitriy,
>> >> >> >
>> >> >> >Last item makes perfect sense to me, one may think of it as an
>> >> >> >"OutOfMemoryException" in java.
>> >> >> >However, it looks like such feature requires considerable efforts
>> >to
>> >> >> >properly design and implement it, so I would propose to create a
>> >> >> >separate
>> >> >> >ticket and agree upon target version for it.
>> >> >> >
>> >> >> >Items #1 and #2 will be implemented under IGNITE-5717. Makes
>> >sense?
>> >> >> >
>> >> >> >Thanks,
>> >> >> >Sergey.
>> >> >> >
>> >> >> >On Thu, Aug 3, 2017 at 4:34 AM, Dmitriy Setrakyan
>> >> >> >
>> >> >> >wrote:
>> >> >> >
>> >> >> >> Here is what we should do:
>> >> >> >>
>> >> >> >>1. Pick an acceptable number. Does not matter if it is 10%
>> >or
>> >> >50%.
>> >> >> >>2. Print the allocated memory in *BOLD* letters into the
>> >log.
>> >> >> >>3. Make sure that Ignite server never hangs due to the low
>> >> >memory
>> >> >> >issue.
>> >> >> >>We should sense it and kick the node out automatically,
>> >again
>> >> >with
>> >> >> >a
>> >> >> >> *BOLD*
>> >> >> >>message in the log.
>> >> >> >>
>> >> >> >>  Is this possible?
>> >> >> >>
>> >> >> >> D.
>> >> >> >>
>> >> >> >> On Wed, Aug 2, 2017 at 6:09 PM, Vladimir Ozerov
>> >> >> >
>> >> >> >> wrote:
>> >> >> >>
>> >> >> >> > My proposal is 10% instead of 80%.
>> >> >> >> >
>> >> >> >> > ср, 2 авг. 2017 г. в 18:54, Denis Magda :
>> >> >> >> >
>> >> >> >> > > Vladimir, Dmitriy P.,
>> >> >> >> > >
>> >> >> >> > > Please see inline
>> >> >> >> > >
>> >> >> >> > > > On Aug 2, 2017, at 7:20 AM, Vladimir Ozerov
>> >> >> >
>> >> >> >> > > wrote:
>> >> >> >> > > >
>> >> >> >> > > > Denis,
>> >> >> >> > > >
>> >> >> >> > > > The reason is that product should not hang user's
>> >computer.
>> >> >How
>> >> >> >else
>> >> >> >> > this
>> >> >> >> > > > could be explained? I am developer. I start Ignite, 1
>> >node,
>> >> >2
>> >> >> >nodes,
>> >> >> >> X
>> >> >> >> > > > nodes, observe how they join topology. Add one key, 10
>> >keys,
>> >> >1M
>> >> >> >keys.
>> >> >> >> > > Then

[GitHub] ignite pull request #2384: IGNITE-5897: Fix session init/end logic. This fix...

[nizhikov]

Github user nizhikov closed the pull request at:

https://github.com/apache/ignite/pull/2384


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: IgniteUtils#currentTimeMillis

[Yakov Zhdanov]

Nickolay, IgniteUtils#currentTimeMillis() is some kind of an old heritage.
I guess nobody remembers when this method has been introduced. I agree that
we can use System.currentTimeMillis(). I would suggest you file a ticket
and replace this method calls with System.currentTimeMillis(). Sounds good?

As far as reliable elapsed time measurement I agree with you that
nanoTime() is better here, but it is definitely not a reason for mentioned
failure, since that test is launched in single JVM on a machine that most
probably does not do any ntp syncs during the test to make Ignite's timeout
machinery fail.

Please file a ticket to switch Ignite's timeouts to nanoTime() at some
point.

--Yakov

[jira] [Created] (IGNITE-6005) [Test failed] GridCachePartitionedDataStructuresFailoverSelfTest.testSemaphoreNonFailoverSafe

[Eduard Shangareev (JIRA)]

Eduard Shangareev created IGNITE-6005:
-

 Summary: [Test failed] 
GridCachePartitionedDataStructuresFailoverSelfTest.testSemaphoreNonFailoverSafe
 Key: IGNITE-6005
 URL: https://issues.apache.org/jira/browse/IGNITE-6005
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.1
Reporter: Eduard Shangareev
 Fix For: 2.2


Example of fail
https://ci.ignite.apache.org/viewLog.html?buildId=762788=buildResultsDiv=Ignite20Tests_IgniteDataStrucutures

Typical problem is 
{code}
org.apache.ignite.IgniteInterruptedException: Failed to wait for asynchronous 
operation permit (thread got interrupted).
at org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:805)
at org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:803)
at 
org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:961)
at 
org.apache.ignite.internal.processors.datastructures.GridCacheSemaphoreImpl.close(GridCacheSemaphoreImpl.java:1026)
at 
org.apache.ignite.internal.processors.cache.datastructures.GridCacheAbstractDataStructuresFailoverSelfTest.testSemaphoreNonFailoverSafe(GridCacheAbstractDataStructuresFailoverSelfTest.java:458)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:176)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2000)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:132)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1915)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.InterruptedException: null
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter.asyncOpAcquire(GridCacheAdapter.java:4314)
at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter.asyncOp(GridCacheAdapter.java:4177)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.colocated.GridDhtColocatedCache.getAsync(GridDhtColocatedCache.java:196)
at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter.get0(GridCacheAdapter.java:4509)
at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4490)
at 
org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1324)
at 
org.apache.ignite.internal.processors.cache.GridCacheProxyImpl.get(GridCacheProxyImpl.java:329)
at 
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$5.applyx(DataStructuresProcessor.java:635)
at 
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.retryTopologySafe(DataStructuresProcessor.java:1519)
at 
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.removeDataStructure(DataStructuresProcessor.java:629)
at 
org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.removeSemaphore(DataStructuresProcessor.java:1188)
at 
org.apache.ignite.internal.processors.datastructures.GridCacheSemaphoreImpl.close(GridCacheSemaphoreImpl.java:1023)
at 
org.apache.ignite.internal.processors.cache.datastructures.GridCacheAbstractDataStructuresFailoverSelfTest.testSemaphoreNonFailoverSafe(GridCacheAbstractDataStructuresFailoverSelfTest.java:458)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:176)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2000)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:132)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1915)
at java.lang.Thread.run(Thread.java:745)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2419: IGNITE-5452 GridTimeoutProcessor can hang on stop

[dmekhanikov]

GitHub user dmekhanikov opened a pull request:

https://github.com/apache/ignite/pull/2419

IGNITE-5452 GridTimeoutProcessor can hang on stop

Created for testing

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5452-1.7

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2419.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2419


commit d5e15af76044cf65385672f8528d48ecdeca3cb6
Author: Pavel Tupitsyn 
Date:   2016-11-02T09:02:00Z

IGNITE-4121 .NET: add ScanQuery example

commit 74f8308d10fc011c00e52efcdb315b35cc79e60a
Author: Pavel Tupitsyn 
Date:   2016-11-02T12:59:15Z

IGNITE-4123 .NET: Remove [Serializable] from Employee in examples

commit 92fff630fbf36c82f93bbd9ddd53d11bed44e772
Author: devozerov 
Date:   2016-11-02T14:50:51Z

Restored services compatibility.

commit a62a0136d295486d95c6e2ab5bba88270d831753
Author: dkarachentsev 
Date:   2016-11-02T16:07:45Z

GG-11655 - Fix merge

commit 348593986b56ddfcec4a4455e49d9b279eae4dc8
Author: devozerov 
Date:   2016-11-05T10:28:03Z

Merge branch 'ignite-1.7.3' into ignite-1.7.4

commit 175da6b7e394dd76c27d5155ff98a5b2ef03bb9d
Author: tledkov-gridgain 
Date:   2016-11-07T06:16:58Z

IGNITE-3432:  check data/meta cache names are different for different IGFS 
instances. This closes #1201

commit ead15193899d08f41491166003cabed0560f0c59
Author: Pavel Tupitsyn 
Date:   2016-11-07T07:49:03Z

IGNITE-4028 Get rid of OP_META in PlatformAbstractTarget

This closes #1192

commit 40ef2f5ae42826fe8fd077e3013e8f55c8512bdd
Author: Dmitriy Govorukhin 
Date:   2016-11-07T09:09:41Z

ignite-4178 support permission builder

commit df670c7d64046d282c053f296c47a4743c58c8b1
Author: Pavel Tupitsyn 
Date:   2016-11-07T09:40:00Z

IGNITE-4118 .NET: Optimistic transaction example

This closes #1200

commit 474f22fda4c7cf4d7b2623c451cd7c10f0d8c636
Author: Pavel Tupitsyn 
Date:   2016-11-07T09:55:20Z

IGNITE-4119 .NET: add TransactionDeadlockException

commit fc7ce5a4d72145f2e8a86debeda264ef0a5b37e3
Author: isapego 
Date:   2016-11-07T10:26:05Z

IGNITE-4090: Added flags so stdint and limits can be used in C++.

commit a98804a249496ba9bafbc96daa7aaf25b3d36724
Author: Igor Sapego 
Date:   2016-11-07T11:00:00Z

IGNITE-4113: Added tests. Added Statement::Set/GetAttribute.

commit b1c7c9bb95c900083702d0ba0362edf3aea5a7b4
Author: sboikov 
Date:   2016-11-07T12:40:36Z

GG-11360 - Implement SQL queries cancellation
Fix for commit 80abd1b: for distributed joins need always send cancel 
request.

commit 319014de075c80fb15e58172cc24e35ce16b56cf
Author: Pavel Tupitsyn 
Date:   2016-11-07T14:53:40Z

IGNITE-4132 .NET: Improve BinaryConfiguration documentation

commit 950bad474ef29f9b808e74034c49a69d57eb2740
Author: dkarachentsev 
Date:   2016-11-08T11:03:34Z

GG-11655 - Restore service compatibility with releases before 1.5.30.

commit 3d19bfc2b66574e3945ce17c7a4dfe77d0070b8d
Author: dkarachentsev 
Date:   2016-11-08T11:04:36Z

Merge remote-tracking branch 'origin/ignite-1.6.11' into ignite-1.6.11

commit 1612b6d66fed032182a41e90da71e6b986ae087b
Author: Pavel Tupitsyn 
Date:   2016-11-08T11:07:54Z

.NET: Fix minor analysis warnings

commit e821dc0083003bc81058b1cb223d8a8a2ee44daf
Author: Dmitriy Govorukhin 
Date:   2016-11-08T12:09:21Z

IGNITE-2079 (revert commit) GridCacheIoManager eats exception trail if it 
falls into the directed case

commit c2c82ca44befe4570325dd6cf2ba885e0d90596c
Author: Dmitriy Govorukhin 
Date:   2016-11-08T12:10:10Z

Merge remote-tracking branch 'professional/ignite-1.6.11' into ignite-1.6.11

commit 865bbcf0f41a0c4944e0928f1758d43a0eae82c5
Author: Dmitriy Govorukhin 
Date:   2016-11-08T12:18:29Z

Revert "Merge remote-tracking branch 'professional/ignite-1.6.11' into 
ignite-1.6.11"

This reverts commit c2c82ca44befe4570325dd6cf2ba885e0d90596c, reversing
changes made to e821dc0083003bc81058b1cb223d8a8a2ee44daf.

commit 9726421ff9efb2b19813b2fd6ad27a3728b5ab1a
Author: Dmitriy Govorukhin 
Date:   2016-11-08T12:59:00Z

  Revert  Revert  Merge remote-tracking branch 'professional/ignite-1.6.11'

commit 5a3a1960fff1dcf32961c45c0ba5149d6748d2fc
Author: Igor Sapego 
Date:   2016-11-08T14:36:35Z

 

Re: IgniteUtils#currentTimeMillis

[Vladimir Ozerov]

Java could do some heavy stuff when doing currentTimeMillis, depending on
the platform or vendor. I remember I saw some articles about performance
issues caused by currentTimeMillis (something about high contention on
certain OS configuration). So I do not see a reason why we should remove
it. It is not very good in terms of resolution, but AFAIK we do need it
anyway.

On Wed, Aug 9, 2017 at 2:41 PM, Yakov Zhdanov  wrote:

> Nickolay, IgniteUtils#currentTimeMillis() is some kind of an old heritage.
> I guess nobody remembers when this method has been introduced. I agree that
> we can use System.currentTimeMillis(). I would suggest you file a ticket
> and replace this method calls with System.currentTimeMillis(). Sounds good?
>
> As far as reliable elapsed time measurement I agree with you that
> nanoTime() is better here, but it is definitely not a reason for mentioned
> failure, since that test is launched in single JVM on a machine that most
> probably does not do any ntp syncs during the test to make Ignite's timeout
> machinery fail.
>
> Please file a ticket to switch Ignite's timeouts to nanoTime() at some
> point.
>
> --Yakov
>

Re: IgniteUtils#currentTimeMillis

[Dmitry Pavlov]

It seems System.currentTimeMillis () is now in intrinsic list. This means
on modern JVMs performance penalty will not be so significiant.

Nickolay, could you please raise standalone ticket for U.currentTimeMillis
() ?

Could you also please check if system.nanoTime / system.currentTimeMs can
fix https://issues.apache.org/jira/browse/IGNITE-5963 When you create a PR,
I can start several run for Ignite Cache 6 suite to check if issue is still
reprodacible.

ср, 9 авг. 2017 г. в 14:41, Yakov Zhdanov :

> Nickolay, IgniteUtils#currentTimeMillis() is some kind of an old heritage.
> I guess nobody remembers when this method has been introduced. I agree that
> we can use System.currentTimeMillis(). I would suggest you file a ticket
> and replace this method calls with System.currentTimeMillis(). Sounds good?
>
> As far as reliable elapsed time measurement I agree with you that
> nanoTime() is better here, but it is definitely not a reason for mentioned
> failure, since that test is launched in single JVM on a machine that most
> probably does not do any ntp syncs during the test to make Ignite's timeout
> machinery fail.
>
> Please file a ticket to switch Ignite's timeouts to nanoTime() at some
> point.
>
> --Yakov
>

Re: IgniteUtils#currentTimeMillis

[Dmitry Pavlov]

Vladimir, could we check it using benchmarks? Internet contains a lot of
articles about this issue. But do we know if it is still actual for new VMs?

ср, 9 авг. 2017 г. в 14:50, Dmitry Pavlov :

> It seems System.currentTimeMillis () is now in intrinsic list. This means
> on modern JVMs performance penalty will not be so significiant.
>
> Nickolay, could you please raise standalone ticket for U.currentTimeMillis
> () ?
>
> Could you also please check if system.nanoTime / system.currentTimeMs can
> fix https://issues.apache.org/jira/browse/IGNITE-5963 When you create a
> PR, I can start several run for Ignite Cache 6 suite to check if issue is
> still reprodacible.
>
> ср, 9 авг. 2017 г. в 14:41, Yakov Zhdanov :
>
>> Nickolay, IgniteUtils#currentTimeMillis() is some kind of an old heritage.
>> I guess nobody remembers when this method has been introduced. I agree
>> that
>> we can use System.currentTimeMillis(). I would suggest you file a ticket
>> and replace this method calls with System.currentTimeMillis(). Sounds
>> good?
>>
>> As far as reliable elapsed time measurement I agree with you that
>> nanoTime() is better here, but it is definitely not a reason for mentioned
>> failure, since that test is launched in single JVM on a machine that most
>> probably does not do any ntp syncs during the test to make Ignite's
>> timeout
>> machinery fail.
>>
>> Please file a ticket to switch Ignite's timeouts to nanoTime() at some
>> point.
>>
>> --Yakov
>>
>

Re: IgniteUtils#currentTimeMillis

[Николай Ижиков]

Hello, Dima.

> Nickolay, could you please raise standalone ticket for U.currentTimeMillis
() ?

Yes, I can.

> https://issues.apache.org/jira/browse/IGNITE-5963

I will try to make quick fix for issue without change Ignite core code.
So fix will not depend to U.currentTimeMillis() implementation.


2017-08-09 14:50 GMT+03:00 Dmitry Pavlov :

> It seems System.currentTimeMillis () is now in intrinsic list. This means
> on modern JVMs performance penalty will not be so significiant.
>
> Nickolay, could you please raise standalone ticket for U.currentTimeMillis
> () ?
>
> Could you also please check if system.nanoTime / system.currentTimeMs can
> fix https://issues.apache.org/jira/browse/IGNITE-5963 When you create a
> PR,
> I can start several run for Ignite Cache 6 suite to check if issue is still
> reprodacible.
>
> ср, 9 авг. 2017 г. в 14:41, Yakov Zhdanov :
>
> > Nickolay, IgniteUtils#currentTimeMillis() is some kind of an old
> heritage.
> > I guess nobody remembers when this method has been introduced. I agree
> that
> > we can use System.currentTimeMillis(). I would suggest you file a ticket
> > and replace this method calls with System.currentTimeMillis(). Sounds
> good?
> >
> > As far as reliable elapsed time measurement I agree with you that
> > nanoTime() is better here, but it is definitely not a reason for
> mentioned
> > failure, since that test is launched in single JVM on a machine that most
> > probably does not do any ntp syncs during the test to make Ignite's
> timeout
> > machinery fail.
> >
> > Please file a ticket to switch Ignite's timeouts to nanoTime() at some
> > point.
> >
> > --Yakov
> >
>



-- 
Nikolay Izhikov
nizhikov@gmail.com

Re: IgniteUtils#currentTimeMillis

[Николай Ижиков]

Vladimir,

> There is nothing wrong with U.currentTimeMillis() at the
moment.

I think we can't rely on the return value for time measurement.
Is it true? Is it OK for you?

It very counterintuitive for me as newcomer.


2017-08-09 14:55 GMT+03:00 Vladimir Ozerov :

> You cannot check it with benchmarks, because behavior of this method will
> vary between different JVMs, OSes and hardware. It can be different even
> with the same OS depending on it's settings. Again - let's just avoid
> unnecessary work. There is nothing wrong with U.currentTimeMillis() at the
> moment.
>
> On Wed, Aug 9, 2017 at 2:52 PM, Dmitry Pavlov 
> wrote:
>
> > Vladimir, could we check it using benchmarks? Internet contains a lot of
> > articles about this issue. But do we know if it is still actual for new
> > VMs?
> >
> > ср, 9 авг. 2017 г. в 14:50, Dmitry Pavlov :
> >
> > > It seems System.currentTimeMillis () is now in intrinsic list. This
> means
> > > on modern JVMs performance penalty will not be so significiant.
> > >
> > > Nickolay, could you please raise standalone ticket for
> > U.currentTimeMillis
> > > () ?
> > >
> > > Could you also please check if system.nanoTime / system.currentTimeMs
> can
> > > fix https://issues.apache.org/jira/browse/IGNITE-5963 When you create
> a
> > > PR, I can start several run for Ignite Cache 6 suite to check if issue
> is
> > > still reprodacible.
> > >
> > > ср, 9 авг. 2017 г. в 14:41, Yakov Zhdanov :
> > >
> > >> Nickolay, IgniteUtils#currentTimeMillis() is some kind of an old
> > heritage.
> > >> I guess nobody remembers when this method has been introduced. I agree
> > >> that
> > >> we can use System.currentTimeMillis(). I would suggest you file a
> ticket
> > >> and replace this method calls with System.currentTimeMillis(). Sounds
> > >> good?
> > >>
> > >> As far as reliable elapsed time measurement I agree with you that
> > >> nanoTime() is better here, but it is definitely not a reason for
> > mentioned
> > >> failure, since that test is launched in single JVM on a machine that
> > most
> > >> probably does not do any ntp syncs during the test to make Ignite's
> > >> timeout
> > >> machinery fail.
> > >>
> > >> Please file a ticket to switch Ignite's timeouts to nanoTime() at some
> > >> point.
> > >>
> > >> --Yakov
> > >>
> > >
> >
>



-- 
Nikolay Izhikov
nizhikov@gmail.com

Re: IgniteUtils#currentTimeMillis

[Dmitriy Setrakyan]

On Wed, Aug 9, 2017 at 4:59 AM, Николай Ижиков 
wrote:

> Vladimir,
>
> > There is nothing wrong with U.currentTimeMillis() at the
> moment.
>
> I think we can't rely on the return value for time measurement.
> Is it true? Is it OK for you?
>
> It very counterintuitive for me as newcomer.
>

I agree with Vladimir. There is nothing broken, yet we are trying to fix
something. However, we are definitely running a risk of breaking something,
if we "fix" it. I would leave this method alone.

Nikolai, if you believe that the method is not reliable, write a test that
will demonstrate it.


>
> 2017-08-09 14:55 GMT+03:00 Vladimir Ozerov :
>
> > You cannot check it with benchmarks, because behavior of this method will
> > vary between different JVMs, OSes and hardware. It can be different even
> > with the same OS depending on it's settings. Again - let's just avoid
> > unnecessary work. There is nothing wrong with U.currentTimeMillis() at
> the
> > moment.
> >
> > On Wed, Aug 9, 2017 at 2:52 PM, Dmitry Pavlov 
> > wrote:
> >
> > > Vladimir, could we check it using benchmarks? Internet contains a lot
> of
> > > articles about this issue. But do we know if it is still actual for new
> > > VMs?
> > >
> > > ср, 9 авг. 2017 г. в 14:50, Dmitry Pavlov :
> > >
> > > > It seems System.currentTimeMillis () is now in intrinsic list. This
> > means
> > > > on modern JVMs performance penalty will not be so significiant.
> > > >
> > > > Nickolay, could you please raise standalone ticket for
> > > U.currentTimeMillis
> > > > () ?
> > > >
> > > > Could you also please check if system.nanoTime / system.currentTimeMs
> > can
> > > > fix https://issues.apache.org/jira/browse/IGNITE-5963 When you
> create
> > a
> > > > PR, I can start several run for Ignite Cache 6 suite to check if
> issue
> > is
> > > > still reprodacible.
> > > >
> > > > ср, 9 авг. 2017 г. в 14:41, Yakov Zhdanov :
> > > >
> > > >> Nickolay, IgniteUtils#currentTimeMillis() is some kind of an old
> > > heritage.
> > > >> I guess nobody remembers when this method has been introduced. I
> agree
> > > >> that
> > > >> we can use System.currentTimeMillis(). I would suggest you file a
> > ticket
> > > >> and replace this method calls with System.currentTimeMillis().
> Sounds
> > > >> good?
> > > >>
> > > >> As far as reliable elapsed time measurement I agree with you that
> > > >> nanoTime() is better here, but it is definitely not a reason for
> > > mentioned
> > > >> failure, since that test is launched in single JVM on a machine that
> > > most
> > > >> probably does not do any ntp syncs during the test to make Ignite's
> > > >> timeout
> > > >> machinery fail.
> > > >>
> > > >> Please file a ticket to switch Ignite's timeouts to nanoTime() at
> some
> > > >> point.
> > > >>
> > > >> --Yakov
> > > >>
> > > >
> > >
> >
>
>
>
> --
> Nikolay Izhikov
> nizhikov@gmail.com
>

[jira] [Created] (IGNITE-6007) GridCacheBinaryObjectMetadataExchangeMultinodeTest.testSequentialUpdatesNoConflicts fails sometimes

[Ilya Lantukh (JIRA)]

Ilya Lantukh created IGNITE-6007:


 Summary: 
GridCacheBinaryObjectMetadataExchangeMultinodeTest.testSequentialUpdatesNoConflicts
 fails sometimes
 Key: IGNITE-6007
 URL: https://issues.apache.org/jira/browse/IGNITE-6007
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.1
Reporter: Ilya Lantukh
 Fix For: 2.2


{noformat}
java.lang.NullPointerException
at 
org.apache.ignite.internal.processors.cache.binary.GridCacheBinaryObjectMetadataExchangeMultinodeTest.testSequentialUpdatesNoConflicts(GridCacheBinaryObjectMetadataExchangeMultinodeTest.java:289)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at junit.framework.TestCase.runTest(TestCase.java:176)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2000)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:132)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1915)
at java.lang.Thread.run(Thread.java:745)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: IgniteUtils#currentTimeMillis

[Dmitry Pavlov]

+1 to Nikolay, this is very often question from newcomers.
It is not clear that current* method may return outdated value from some
moment in past.

Nikolay, how long outdated value can be returned by method?

ср, 9 авг. 2017 г. в 15:00, Николай Ижиков :

> Vladimir,
>
> > There is nothing wrong with U.currentTimeMillis() at the
> moment.
>
> I think we can't rely on the return value for time measurement.
> Is it true? Is it OK for you?
>
> It very counterintuitive for me as newcomer.
>
>
> 2017-08-09 14:55 GMT+03:00 Vladimir Ozerov :
>
> > You cannot check it with benchmarks, because behavior of this method will
> > vary between different JVMs, OSes and hardware. It can be different even
> > with the same OS depending on it's settings. Again - let's just avoid
> > unnecessary work. There is nothing wrong with U.currentTimeMillis() at
> the
> > moment.
> >
> > On Wed, Aug 9, 2017 at 2:52 PM, Dmitry Pavlov 
> > wrote:
> >
> > > Vladimir, could we check it using benchmarks? Internet contains a lot
> of
> > > articles about this issue. But do we know if it is still actual for new
> > > VMs?
> > >
> > > ср, 9 авг. 2017 г. в 14:50, Dmitry Pavlov :
> > >
> > > > It seems System.currentTimeMillis () is now in intrinsic list. This
> > means
> > > > on modern JVMs performance penalty will not be so significiant.
> > > >
> > > > Nickolay, could you please raise standalone ticket for
> > > U.currentTimeMillis
> > > > () ?
> > > >
> > > > Could you also please check if system.nanoTime / system.currentTimeMs
> > can
> > > > fix https://issues.apache.org/jira/browse/IGNITE-5963 When you
> create
> > a
> > > > PR, I can start several run for Ignite Cache 6 suite to check if
> issue
> > is
> > > > still reprodacible.
> > > >
> > > > ср, 9 авг. 2017 г. в 14:41, Yakov Zhdanov :
> > > >
> > > >> Nickolay, IgniteUtils#currentTimeMillis() is some kind of an old
> > > heritage.
> > > >> I guess nobody remembers when this method has been introduced. I
> agree
> > > >> that
> > > >> we can use System.currentTimeMillis(). I would suggest you file a
> > ticket
> > > >> and replace this method calls with System.currentTimeMillis().
> Sounds
> > > >> good?
> > > >>
> > > >> As far as reliable elapsed time measurement I agree with you that
> > > >> nanoTime() is better here, but it is definitely not a reason for
> > > mentioned
> > > >> failure, since that test is launched in single JVM on a machine that
> > > most
> > > >> probably does not do any ntp syncs during the test to make Ignite's
> > > >> timeout
> > > >> machinery fail.
> > > >>
> > > >> Please file a ticket to switch Ignite's timeouts to nanoTime() at
> some
> > > >> point.
> > > >>
> > > >> --Yakov
> > > >>
> > > >
> > >
> >
>
>
>
> --
> Nikolay Izhikov
> nizhikov@gmail.com
>

[jira] [Created] (IGNITE-6008) CacheRemoveAllSelfTest.testRemoveAll fails sometimes

[Ilya Lantukh (JIRA)]

Ilya Lantukh created IGNITE-6008:


 Summary: CacheRemoveAllSelfTest.testRemoveAll fails sometimes
 Key: IGNITE-6008
 URL: https://issues.apache.org/jira/browse/IGNITE-6008
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.1
Reporter: Ilya Lantukh
 Fix For: 2.2


{noformat}
[2017-08-09 01:18:15,172][ERROR][main][root] Test failed.
junit.framework.AssertionFailedError: Local size: 58
On heap: 58
Off heap: 58
Primary: 58
Backup: 14 expected:<0> but was:<58>
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.TestCase.assertEquals(TestCase.java:401)
at 
org.apache.ignite.internal.processors.cache.CacheRemoveAllSelfTest.testRemoveAll(CacheRemoveAllSelfTest.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:176)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2000)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:132)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1915)
at java.lang.Thread.run(Thread.java:745)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6009) CacheExamplesSelfTest.testCacheSemaphoreExample fails sometimes due to timeout

[Ilya Lantukh (JIRA)]

Ilya Lantukh created IGNITE-6009:


 Summary: CacheExamplesSelfTest.testCacheSemaphoreExample fails 
sometimes due to timeout
 Key: IGNITE-6009
 URL: https://issues.apache.org/jira/browse/IGNITE-6009
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.1
Reporter: Ilya Lantukh
 Fix For: 2.2






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6010) ZookeeperIpFinderTest.testFourNodesKillRestartZookeeper fails sometimes

[Ilya Lantukh (JIRA)]

Ilya Lantukh created IGNITE-6010:


 Summary: ZookeeperIpFinderTest.testFourNodesKillRestartZookeeper 
fails sometimes
 Key: IGNITE-6010
 URL: https://issues.apache.org/jira/browse/IGNITE-6010
 Project: Ignite
  Issue Type: Bug
  Components: zookeeper
Affects Versions: 2.1
Reporter: Ilya Lantukh
 Fix For: 2.2


{noformat}
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:55)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.Assert.assertTrue(Assert.java:31)
at junit.framework.TestCase.assertTrue(TestCase.java:201)
at 
org.apache.ignite.spi.discovery.tcp.ipfinder.zk.ZookeeperIpFinderTest.testFourNodesKillRestartZookeeper(ZookeeperIpFinderTest.java:365)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: IgniteUtils#currentTimeMillis

[Vladimir Ozerov]

Guys,

Are you really suggesting change the product for newcomer needs? This is
not an argument for the change. Can someone explain me what is currently
broken from product's perspective? Yes, we can get stale values for some
time, we know that. Does it break something?

On Wed, Aug 9, 2017 at 3:11 PM, Dmitry Pavlov  wrote:

> +1 to Nikolay, this is very often question from newcomers.
> It is not clear that current* method may return outdated value from some
> moment in past.
>
> Nikolay, how long outdated value can be returned by method?
>
> ср, 9 авг. 2017 г. в 15:00, Николай Ижиков :
>
> > Vladimir,
> >
> > > There is nothing wrong with U.currentTimeMillis() at the
> > moment.
> >
> > I think we can't rely on the return value for time measurement.
> > Is it true? Is it OK for you?
> >
> > It very counterintuitive for me as newcomer.
> >
> >
> > 2017-08-09 14:55 GMT+03:00 Vladimir Ozerov :
> >
> > > You cannot check it with benchmarks, because behavior of this method
> will
> > > vary between different JVMs, OSes and hardware. It can be different
> even
> > > with the same OS depending on it's settings. Again - let's just avoid
> > > unnecessary work. There is nothing wrong with U.currentTimeMillis() at
> > the
> > > moment.
> > >
> > > On Wed, Aug 9, 2017 at 2:52 PM, Dmitry Pavlov 
> > > wrote:
> > >
> > > > Vladimir, could we check it using benchmarks? Internet contains a lot
> > of
> > > > articles about this issue. But do we know if it is still actual for
> new
> > > > VMs?
> > > >
> > > > ср, 9 авг. 2017 г. в 14:50, Dmitry Pavlov :
> > > >
> > > > > It seems System.currentTimeMillis () is now in intrinsic list. This
> > > means
> > > > > on modern JVMs performance penalty will not be so significiant.
> > > > >
> > > > > Nickolay, could you please raise standalone ticket for
> > > > U.currentTimeMillis
> > > > > () ?
> > > > >
> > > > > Could you also please check if system.nanoTime /
> system.currentTimeMs
> > > can
> > > > > fix https://issues.apache.org/jira/browse/IGNITE-5963 When you
> > create
> > > a
> > > > > PR, I can start several run for Ignite Cache 6 suite to check if
> > issue
> > > is
> > > > > still reprodacible.
> > > > >
> > > > > ср, 9 авг. 2017 г. в 14:41, Yakov Zhdanov :
> > > > >
> > > > >> Nickolay, IgniteUtils#currentTimeMillis() is some kind of an old
> > > > heritage.
> > > > >> I guess nobody remembers when this method has been introduced. I
> > agree
> > > > >> that
> > > > >> we can use System.currentTimeMillis(). I would suggest you file a
> > > ticket
> > > > >> and replace this method calls with System.currentTimeMillis().
> > Sounds
> > > > >> good?
> > > > >>
> > > > >> As far as reliable elapsed time measurement I agree with you that
> > > > >> nanoTime() is better here, but it is definitely not a reason for
> > > > mentioned
> > > > >> failure, since that test is launched in single JVM on a machine
> that
> > > > most
> > > > >> probably does not do any ntp syncs during the test to make
> Ignite's
> > > > >> timeout
> > > > >> machinery fail.
> > > > >>
> > > > >> Please file a ticket to switch Ignite's timeouts to nanoTime() at
> > some
> > > > >> point.
> > > > >>
> > > > >> --Yakov
> > > > >>
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Nikolay Izhikov
> > nizhikov@gmail.com
> >
>

Re: IgniteUtils#currentTimeMillis

[Николай Ижиков]

Vladimir,

As far as I can understand behaviour of U.currentTimeMillis() breaks
transaction timeout test:

https://issues.apache.org/jira/browse/IGNITE-5963

IgniteOptimisticTxSuspendResumeTest#testTxTimeoutOnSuspend :

```
final Transaction tx = ignite.transactions().txStart(OPTIMISTIC, isolation,
TX_TIMEOUT, 0);

cache.put(1, 1);

Thread.sleep(TX_TIMEOUT * 2);

GridTestUtils.assertThrowsWithCause(new Callable() {
@Override public Object call() throws Exception {
tx.suspend();

return null;
}
}, TransactionTimeoutException.class);
```
Timeout checked like that:

IgniteTxAdapter#remainingTime

```
@Override public long remainingTime() {
if (timeout() <= 0)
return 0;

long timeLeft = timeout() - (U.currentTimeMillis() - startTime());

return timeLeft <= 0 ? -1 : timeLeft;

}

```


2017-08-09 15:26 GMT+03:00 Vladimir Ozerov :

> Guys,
>
> Are you really suggesting change the product for newcomer needs? This is
> not an argument for the change. Can someone explain me what is currently
> broken from product's perspective? Yes, we can get stale values for some
> time, we know that. Does it break something?
>
> On Wed, Aug 9, 2017 at 3:11 PM, Dmitry Pavlov 
> wrote:
>
> > +1 to Nikolay, this is very often question from newcomers.
> > It is not clear that current* method may return outdated value from some
> > moment in past.
> >
> > Nikolay, how long outdated value can be returned by method?
> >
> > ср, 9 авг. 2017 г. в 15:00, Николай Ижиков :
> >
> > > Vladimir,
> > >
> > > > There is nothing wrong with U.currentTimeMillis() at the
> > > moment.
> > >
> > > I think we can't rely on the return value for time measurement.
> > > Is it true? Is it OK for you?
> > >
> > > It very counterintuitive for me as newcomer.
> > >
> > >
> > > 2017-08-09 14:55 GMT+03:00 Vladimir Ozerov :
> > >
> > > > You cannot check it with benchmarks, because behavior of this method
> > will
> > > > vary between different JVMs, OSes and hardware. It can be different
> > even
> > > > with the same OS depending on it's settings. Again - let's just avoid
> > > > unnecessary work. There is nothing wrong with U.currentTimeMillis()
> at
> > > the
> > > > moment.
> > > >
> > > > On Wed, Aug 9, 2017 at 2:52 PM, Dmitry Pavlov  >
> > > > wrote:
> > > >
> > > > > Vladimir, could we check it using benchmarks? Internet contains a
> lot
> > > of
> > > > > articles about this issue. But do we know if it is still actual for
> > new
> > > > > VMs?
> > > > >
> > > > > ср, 9 авг. 2017 г. в 14:50, Dmitry Pavlov :
> > > > >
> > > > > > It seems System.currentTimeMillis () is now in intrinsic list.
> This
> > > > means
> > > > > > on modern JVMs performance penalty will not be so significiant.
> > > > > >
> > > > > > Nickolay, could you please raise standalone ticket for
> > > > > U.currentTimeMillis
> > > > > > () ?
> > > > > >
> > > > > > Could you also please check if system.nanoTime /
> > system.currentTimeMs
> > > > can
> > > > > > fix https://issues.apache.org/jira/browse/IGNITE-5963 When you
> > > create
> > > > a
> > > > > > PR, I can start several run for Ignite Cache 6 suite to check if
> > > issue
> > > > is
> > > > > > still reprodacible.
> > > > > >
> > > > > > ср, 9 авг. 2017 г. в 14:41, Yakov Zhdanov :
> > > > > >
> > > > > >> Nickolay, IgniteUtils#currentTimeMillis() is some kind of an
> old
> > > > > heritage.
> > > > > >> I guess nobody remembers when this method has been introduced. I
> > > agree
> > > > > >> that
> > > > > >> we can use System.currentTimeMillis(). I would suggest you file
> a
> > > > ticket
> > > > > >> and replace this method calls with System.currentTimeMillis().
> > > Sounds
> > > > > >> good?
> > > > > >>
> > > > > >> As far as reliable elapsed time measurement I agree with you
> that
> > > > > >> nanoTime() is better here, but it is definitely not a reason for
> > > > > mentioned
> > > > > >> failure, since that test is launched in single JVM on a machine
> > that
> > > > > most
> > > > > >> probably does not do any ntp syncs during the test to make
> > Ignite's
> > > > > >> timeout
> > > > > >> machinery fail.
> > > > > >>
> > > > > >> Please file a ticket to switch Ignite's timeouts to nanoTime()
> at
> > > some
> > > > > >> point.
> > > > > >>
> > > > > >> --Yakov
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Nikolay Izhikov
> > > nizhikov@gmail.com
> > >
> >
>



-- 
Nikolay Izhikov
nizhikov@gmail.com

Re: IgniteUtils#currentTimeMillis

[Vladimir Ozerov]

I would propose to either increase TX_TIMEOUT or sleep multiplier to make
test more reliable.

On Wed, Aug 9, 2017 at 3:32 PM, Николай Ижиков 
wrote:

> Vladimir,
>
> As far as I can understand behaviour of U.currentTimeMillis() breaks
> transaction timeout test:
>
> https://issues.apache.org/jira/browse/IGNITE-5963
>
> IgniteOptimisticTxSuspendResumeTest#testTxTimeoutOnSuspend :
>
> ```
> final Transaction tx = ignite.transactions().txStart(OPTIMISTIC,
> isolation,
> TX_TIMEOUT, 0);
>
> cache.put(1, 1);
>
> Thread.sleep(TX_TIMEOUT * 2);
>
> GridTestUtils.assertThrowsWithCause(new Callable() {
> @Override public Object call() throws Exception {
> tx.suspend();
>
> return null;
> }
> }, TransactionTimeoutException.class);
> ```
> Timeout checked like that:
>
> IgniteTxAdapter#remainingTime
>
> ```
> @Override public long remainingTime() {
> if (timeout() <= 0)
> return 0;
>
> long timeLeft = timeout() - (U.currentTimeMillis() - startTime());
>
> return timeLeft <= 0 ? -1 : timeLeft;
>
> }
>
> ```
>
>
> 2017-08-09 15:26 GMT+03:00 Vladimir Ozerov :
>
> > Guys,
> >
> > Are you really suggesting change the product for newcomer needs? This is
> > not an argument for the change. Can someone explain me what is currently
> > broken from product's perspective? Yes, we can get stale values for some
> > time, we know that. Does it break something?
> >
> > On Wed, Aug 9, 2017 at 3:11 PM, Dmitry Pavlov 
> > wrote:
> >
> > > +1 to Nikolay, this is very often question from newcomers.
> > > It is not clear that current* method may return outdated value from
> some
> > > moment in past.
> > >
> > > Nikolay, how long outdated value can be returned by method?
> > >
> > > ср, 9 авг. 2017 г. в 15:00, Николай Ижиков :
> > >
> > > > Vladimir,
> > > >
> > > > > There is nothing wrong with U.currentTimeMillis() at the
> > > > moment.
> > > >
> > > > I think we can't rely on the return value for time measurement.
> > > > Is it true? Is it OK for you?
> > > >
> > > > It very counterintuitive for me as newcomer.
> > > >
> > > >
> > > > 2017-08-09 14:55 GMT+03:00 Vladimir Ozerov :
> > > >
> > > > > You cannot check it with benchmarks, because behavior of this
> method
> > > will
> > > > > vary between different JVMs, OSes and hardware. It can be different
> > > even
> > > > > with the same OS depending on it's settings. Again - let's just
> avoid
> > > > > unnecessary work. There is nothing wrong with U.currentTimeMillis()
> > at
> > > > the
> > > > > moment.
> > > > >
> > > > > On Wed, Aug 9, 2017 at 2:52 PM, Dmitry Pavlov <
> dpavlov@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Vladimir, could we check it using benchmarks? Internet contains a
> > lot
> > > > of
> > > > > > articles about this issue. But do we know if it is still actual
> for
> > > new
> > > > > > VMs?
> > > > > >
> > > > > > ср, 9 авг. 2017 г. в 14:50, Dmitry Pavlov  >:
> > > > > >
> > > > > > > It seems System.currentTimeMillis () is now in intrinsic list.
> > This
> > > > > means
> > > > > > > on modern JVMs performance penalty will not be so significiant.
> > > > > > >
> > > > > > > Nickolay, could you please raise standalone ticket for
> > > > > > U.currentTimeMillis
> > > > > > > () ?
> > > > > > >
> > > > > > > Could you also please check if system.nanoTime /
> > > system.currentTimeMs
> > > > > can
> > > > > > > fix https://issues.apache.org/jira/browse/IGNITE-5963 When you
> > > > create
> > > > > a
> > > > > > > PR, I can start several run for Ignite Cache 6 suite to check
> if
> > > > issue
> > > > > is
> > > > > > > still reprodacible.
> > > > > > >
> > > > > > > ср, 9 авг. 2017 г. в 14:41, Yakov Zhdanov  >:
> > > > > > >
> > > > > > >> Nickolay, IgniteUtils#currentTimeMillis() is some kind of an
> > old
> > > > > > heritage.
> > > > > > >> I guess nobody remembers when this method has been
> introduced. I
> > > > agree
> > > > > > >> that
> > > > > > >> we can use System.currentTimeMillis(). I would suggest you
> file
> > a
> > > > > ticket
> > > > > > >> and replace this method calls with System.currentTimeMillis().
> > > > Sounds
> > > > > > >> good?
> > > > > > >>
> > > > > > >> As far as reliable elapsed time measurement I agree with you
> > that
> > > > > > >> nanoTime() is better here, but it is definitely not a reason
> for
> > > > > > mentioned
> > > > > > >> failure, since that test is launched in single JVM on a
> machine
> > > that
> > > > > > most
> > > > > > >> probably does not do any ntp syncs during the test to make
> > > Ignite's
> > > > > > >> timeout
> > > > > > >> machinery fail.
> > > > > > >>
> > > > > > >> Please file a ticket to switch Ignite's timeouts to nanoTime()
> > at
> > > > some
> > > > > > >> point.
> > > > > > >>
> > > > > > >> --Yakov
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > 

Re: IgniteUtils#currentTimeMillis

[Dmitriy Setrakyan]

On Wed, Aug 9, 2017 at 5:32 AM, Николай Ижиков 
wrote:

> Vladimir,
>
> As far as I can understand behaviour of U.currentTimeMillis() breaks
> transaction timeout test:
>

So, if you change the call to System.currentTimeMillis(), the test passes?

[jira] [Created] (IGNITE-6011) EntryProcessor can make data inconsistent if fails on TX cache.

[Andrew Mashenkov (JIRA)]

Andrew Mashenkov created IGNITE-6011:


 Summary: EntryProcessor can make data inconsistent if fails on TX 
cache.
 Key: IGNITE-6011
 URL: https://issues.apache.org/jira/browse/IGNITE-6011
 Project: Ignite
  Issue Type: Bug
  Components: cache
Reporter: Andrew Mashenkov


I'd expect that with FULL_SYNC write mode and TX cache data always be 
consistent.
And if EntryProcessor fails on primary (or backup) node and pass on backup (or 
primary) then whole transaction will be rolled back.

But looks like it is not true, I observe old value on node where EP has failed 
and new value on other nodes.

PFA repro attached.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2420: IGNITE-6004 SQL: add "inlineSize" to QuerySqlFiel...

[tledkov-gridgain]

GitHub user tledkov-gridgain opened a pull request:

https://github.com/apache/ignite/pull/2420

IGNITE-6004 SQL: add "inlineSize" to QuerySqlField annotation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6004

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2420.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2420


commit d70f69bc8afe0b96421fca4e2004cc70c66a8434
Author: tledkov-gridgain 
Date:   2017-08-09T12:49:18Z

IGNITE-6004 SQL: add "inlineSize" to QuerySqlField annotation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: IgniteUtils#currentTimeMillis

[Николай Ижиков]

Dmitry,

> So, if you change the call to System.currentTimeMillis(), the test passes?

Yes

> I would propose to either increase TX_TIMEOUT or sleep multiplier to make
test more reliable.

Yes, I fix test in that way.

For me the goal of this discussion is to understand reasons to keep current
method implementation.




2017-08-09 15:45 GMT+03:00 Dmitriy Setrakyan :

> On Wed, Aug 9, 2017 at 5:32 AM, Николай Ижиков 
> wrote:
>
> > Vladimir,
> >
> > As far as I can understand behaviour of U.currentTimeMillis() breaks
> > transaction timeout test:
> >
>
> So, if you change the call to System.currentTimeMillis(), the test passes?
>



-- 
Nikolay Izhikov
nizhikov@gmail.com

[GitHub] ignite pull request #2072: ignite-5097-dotnet-integration

[daradurvs]

Github user daradurvs closed the pull request at:

https://github.com/apache/ignite/pull/2072


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1533: IGNITE-3422 No way to control object initializati...

[daradurvs]

Github user daradurvs closed the pull request at:

https://github.com/apache/ignite/pull/1533


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1902: IGNITE-5097 BinaryMarshaller should write ints in...

[daradurvs]

Github user daradurvs closed the pull request at:

https://github.com/apache/ignite/pull/1902


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Hibernate dialect for Ignite

[Dmitriy Setrakyan]

Igniters,

Given that we have a very rich SQL support starting with version 2.1, does
it make sense to create an Ignite SQL dialect for Hibernate?

https://docs.jboss.org/hibernate/orm/3.6/reference/en-US/html/session-configuration.html#configuration-optional-dialects

D.

Re: Data compression in Ignite 2.0

[Vyacheslav Daradur]

Vladimir, thank you for detailed explanation.

I think I've understanded the main idea of described storage compression.

I'll join the new discussion after researching of given material and
comlpetion of varint-optimization [1].

[1] https://issues.apache.org/jira/browse/IGNITE-5097

2017-08-02 15:43 GMT+03:00 Alexey Kuznetsov :

> Vova,
>
> Finally we back to my initial idea - to look how "big databases compress"
> data :)
>
>
> Just to remind how IBM DB2 do this[1].
>
> [1] http://www.ibm.com/developerworks/data/library/techarticle/dm-
> 1205db210compression/
>
> On Tue, Aug 1, 2017 at 4:15 PM, Vladimir Ozerov 
> wrote:
>
> > Vyacheslav,
> >
> > This is not about my needs, but about the product :-) BinaryObject is a
> > central entity used for both data transfer and data storage. This is both
> > good and bad at the same time.
> >
> > Good thing is that as we optimize binary protocol, we improve both
> network
> > and storage performance at the same time. We have at least 3 things which
> > will be included into the product soon: varint encoding [1], optimized
> > string encoding [2] and null-field optimization [3]. Bad thing is that
> > binary object format is not well suited for data storage optimizations,
> > including compression. For example, one good compression technique is to
> > organize data in column-store format, or to introduce shared "dictionary"
> > with unique values on cache level. In both cases N equal values are not
> > stored N times. Instead, we store one value and N references to it, or
> so.
> > This way 2x-10x compression is possible depending on workload type.
> Binary
> > object protocol with some compression on top of it cannot give such
> > improvement, because it will compress data in individual objects, instead
> > of compressing the whole cache data in a single context.
> >
> > That said, I propose to give up adding compression to BinaryObject. This
> is
> > a dead end. Instead, we should:
> > 1) Optimize protocol itself to be more compact, as described in
> > aforementioned Ignite tickets
> > 2) Start new discussion about storage compression
> >
> > You can read papers of other vendors to get better understanding on
> > possible compression options. E.g. Oracle has a lot of compression
> > techniques, including heat maps, background compression, per-block
> > compression, data dictionaries, etc. [4].
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-5097
> > [2] https://issues.apache.org/jira/browse/IGNITE-5655
> > [3] https://issues.apache.org/jira/browse/IGNITE-3939
> > [4] http://www.oracle.com/technetwork/database/options/
> > compression/advanced-
> > compression-wp-12c-1896128.pdf
> >
> > Vladimir.
> >
> >
>
> --
> Alexey Kuznetsov
>



-- 
Best Regards, Vyacheslav D.

[GitHub] ignite pull request #2366: IGNITE-5882 Duplicated dependency in pom.xml of c...

[daradurvs]

Github user daradurvs closed the pull request at:

https://github.com/apache/ignite/pull/2366


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2342: for-dotNET-tests-ignite-5712

[daradurvs]

Github user daradurvs closed the pull request at:

https://github.com/apache/ignite/pull/2342


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1652: IGNITE-3682 GridFunc: move all inner anonymous cl...

[daradurvs]

Github user daradurvs closed the pull request at:

https://github.com/apache/ignite/pull/1652


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: release procedure

[Konstantin Boudnik]

There's also #4:
 - providing an official environment, comprised of the toolchain,
compilers, libs, etc,. The same environment (read "a container") could
be used by an individual developer, RM, and/or in CI system for
builds, tests, etc. And then you can have #3 pretty much for free!

We are doing this in Bigtop for a much more complex environment, set
of components and supported OS. I am sure it would be easy to do in
Ignite.
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Sun, Aug 6, 2017 at 5:37 AM, Oleg Ostanin  wrote:
> Hi,
>
> I'd like to start a discussion about Apache Ignite release procedure.
>
> I'm working on ticket Ignite-5249 "The release build procedure should be
> placed on the CI/CD server and available to run for the release engineer."
>
> https://issues.apache.org/jira/browse/IGNITE-5249
>
> Currently we have three options for release:
>
> 1. Release engineer can do all the necessary steps on his local machine. It
> will require installing tons of soft like maven, doxigen, candle and so on.
> Also building .net part of the project will require access to Windows OS.
> Build steps will not be transparent for community. Environment will not be
> the same for each release which can lead to the compatibility issues.
>
> 2. All the steps (including signing) can be done on the public continuous
> integration server. Environment will be the same for each release, all the
> steps will be transparent for community, but it will require uploading at
> least one private gpg certificate on the server. This is the high security
> risk and I'm mentioning this option only for the sake of completeness.
>
> 3. Building of the project can be performed on the public continuous
> integration server and then artifacts can be downloaded on the local
> machine and signed and deployed to the staging repository from that local
> machine by running maven commands. No sharing of any credentials and
> certificates will be needed, environment will be the same for each release,
> all the steps will be transparent for community, artifacts created on the
> CI server can be verified by check-sums after uploading to the repository.
>
> Please, let me know if you have any suggestion or any questions about
> anything related.

[jira] [Created] (IGNITE-6015) Test fail in Ignite Cache 2: GridCachePartitionedGetAndTransformStoreSelfTest.testGetAndTransform

[Dmitriy Govorukhin (JIRA)]

Dmitriy Govorukhin created IGNITE-6015:
--

 Summary: Test fail in Ignite Cache 2: 
GridCachePartitionedGetAndTransformStoreSelfTest.testGetAndTransform 
 Key: IGNITE-6015
 URL: https://issues.apache.org/jira/browse/IGNITE-6015
 Project: Ignite
  Issue Type: Test
Reporter: Dmitriy Govorukhin
 Fix For: 2.2


java.util.concurrent.TimeoutException: Test has been timed out 
[test=testGetAndTransform, timeout=30]
at 
org.apache.ignite.testframework.junits.GridAbstractTest.runTest(GridAbstractTest.java:1949)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at 
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208)
at 
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:156)
at 
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:82)
at 
org.apache.maven.plugin.surefire.InPluginVMSurefireStarter.runSuitesInProcess(InPluginVMSurefireStarter.java:82)
at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:951)
at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:831)
at 
org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:729)
at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:3



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Data compression in Ignite

[Ivan Rakov]

+1 for per-page approach (2).
I'm not sure if dictionary-based encoding within page is really 
efficient. Compression quality grows with increasing size of sample 
(it's pageSize in our case - not so big), and we can't encode "words" on 
the border of two entries. Also, we'll have to "garbage-collect" page 
dictionary from time to time (as set of entries may change and some 
"words" may become redundant).

However, if Oracle implemented it in profitable way, we can do it as well :)

I have a concern about (3) that it will add significant latency/network 
overhead. Accessing distributed caсhe with dictionary at every put is 
not free.


Also, I'd like to propose one more approach (let it be (4)) - local 
per-partition dictionary. We can store set of dictionary "words" in B+ 
tree or any other structure in pages - it will be automatically 
persisted. Efficiency of encoding will be higher than in per-page case. 
Thoughts?
The open question for per-partition dictionary is how to 
"garbage-collect" it. In per-page case we can do it under page write 
lock, but here we have to do it in more tricky way.


Best Regards,
Ivan Rakov


On 09.08.2017 17:48, Vladimir Ozerov wrote:

Igniters,

I had several private talks with Igniters about data compression and would
like to share the summary with ... Igniters :-)

Currently all Ignite's data is uncompressed. It leads to excessive network
traffic, GC pressure and disk IO (in case of persistence). Most modern
databases are able to compress data, what gives them 2-4x size reduction on
typical workloads. We need compression in Ignite.

There are several options I'd like to discuss. The main difference between
them - on what "level" to compress: per-entry, per-data-page or per-cache.

*1) Per-entry compression*
Apache Geode uses this approach. Every cache entry is compressed using
Snappy. This is very easy to implement, but every entry access (e.g.
reading single field) require full decompression or even re-compression,
what could lead to higher CPU consumption and worse performance.

*2) Per-data-page compression*
Oracle and DB2 use this approach. Pages are compressed with
dictionary-based approach (e.g. LZV). It is important, that they do not
compress the whole page. Instead, only actual data is compressed, while
page structure remains intact. Dictionary is placed within the page. This
way it is possible to work with individual entries and even individual
fields without full page decompression. Another important thing - it is not
necessary to re-compress the page on each write. Instead, data is stored in
uncompressed form first, and compressed even after certain threshold is
reached. So negative CPU impact is minimal. Typical compression rate would
be higher than in per-entry case, because the more data you have, the
better it can be compressed.

*3) Per-cache compression*
Suggested by Alex Goncharuk. We could have a dictionary for the whole
cache. This way we could achieve the highest compression rate possible. The
downside is complex implementation - we would have to develop an algorithm
of sharing the dictionary within the cluster. At some point the dictionary
could become too huge to fit in-memory, so we should either control it's
size or spill it to disk.

I propose to use per-data-page approach as both gives nice compression rate
and relatively easy to implement.

Please share your thoughts.

Vladimir.

[jira] [Created] (IGNITE-6016) Get rid of checking topology hash in ackTopology

[Ivan Rakov (JIRA)]

Ivan Rakov created IGNITE-6016:
--

 Summary: Get rid of checking topology hash in ackTopology
 Key: IGNITE-6016
 URL: https://issues.apache.org/jira/browse/IGNITE-6016
 Project: Ignite
  Issue Type: Improvement
  Components: general
Affects Versions: 2.1
Reporter: Ivan Rakov
Assignee: Ivan Rakov
 Fix For: 2.2


We check topologyHash in ackTopology in order to avoid printing "Topology 
snapshot" message twice. It's redundant - we can just atomically increase 
topology version with GridAtomicLong#setIfGreater.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2422: IGNITE-5963: Add additional check to Thread.sleep...

[nizhikov]

GitHub user nizhikov opened a pull request:

https://github.com/apache/ignite/pull/2422

IGNITE-5963: Add additional check to Thread.sleep to make test correc…

…t in all cases.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nizhikov/ignite IGNITE-5963

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2422.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2422


commit e12de1f4471de5ae46eb35f9dbb275403a8bf4ca
Author: Nikolay Izhikov 
Date:   2017-08-09T15:37:09Z

IGNITE-5963: Add additional check to Thread.sleep to make test correct in 
all cases.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2424: IGNITE-6016 Get rid of checking topology hash in ...

[glukos]

GitHub user glukos opened a pull request:

https://github.com/apache/ignite/pull/2424

IGNITE-6016 Get rid of checking topology hash in ackTopology



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6016

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2424.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2424


commit 2d409a7f9a52c6769e2f0b7858e49691d04313d6
Author: Ivan Rakov 
Date:   2017-08-09T16:09:06Z

IGNITE-6016 Get rid of checking topology hash in ackTopology




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-6018) Introduce WAL backward compatibility for new DataPage insert/update records

[Pavel Kovalenko (JIRA)]

Pavel Kovalenko created IGNITE-6018:
---

 Summary: Introduce WAL backward compatibility for new DataPage 
insert/update records
 Key: IGNITE-6018
 URL: https://issues.apache.org/jira/browse/IGNITE-6018
 Project: Ignite
  Issue Type: Sub-task
Reporter: Pavel Kovalenko
Assignee: Pavel Kovalenko
Priority: Blocker
 Fix For: 2.2


Once we store reference to DataRecord for DataPage insert/update records we 
should be able to read/write both versions of that records (with reference or 
with payload) for backward compatibility purposes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2423: IGNITE-5943

[EdShangGG]

GitHub user EdShangGG opened a pull request:

https://github.com/apache/ignite/pull/2423

IGNITE-5943



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite IGNITE-5943

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2423.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2423


commit 2d93a7c11871a2b23cfb2e6f9854dccb48eac4f1
Author: Andrey Novikov 
Date:   2017-08-03T03:42:08Z

IGNITE-5888 Web Console: Fixed maven project generation.

commit 36787e149693103fa22ba4093259d2fd8425689a
Author: Andrey Novikov 
Date:   2017-08-03T03:45:59Z

IGNITE-5906 Fixed race on activities merge.

commit 9e9f1d1e4cdcb3ade018f3eabcf2b2d10a7c003b
Author: tledkov-gridgain 
Date:   2017-08-04T08:16:52Z

IGNITE-5920: Fixed CacheClientBinaryQueryExample": set 
CacheKeyConfiguration explicitly to enable affinity co-location. This closes 
#2389.

commit 944056e5431b6018bb653c563db1356f0148fb9a
Author: anovikov 
Date:   2017-08-04T11:10:36Z

IGNITE-5908 Restore splash.

commit a1652b8f7748d3bf1e42437f9ad36f6f44e19bf5
Author: Sergey Chugunov 
Date:   2017-08-07T08:34:01Z

IGNITE-5950 incorrect assertion fixed

commit cfa5ac269c37718191be42d5554637b78c2a7466
Author: Eduard Shangareev 
Date:   2017-08-09T15:57:24Z

IGNITE-5943 Communication. Server node may reject client connection during 
massive clients join




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-6017) Ignite IGFS: IgfsStreamsSelfTest#testCreateFileFragmented fails

[Pavel Kovalenko (JIRA)]

Pavel Kovalenko created IGNITE-6017:
---

 Summary: Ignite IGFS: IgfsStreamsSelfTest#testCreateFileFragmented 
fails
 Key: IGNITE-6017
 URL: https://issues.apache.org/jira/browse/IGNITE-6017
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.1
Reporter: Pavel Kovalenko
Priority: Minor
 Fix For: 2.2


Failure is almost can't be reproduced locally.

Suppose it is the same problem as in IGNITE-5957 .

{noformat}
junit.framework.AssertionFailedError: expected:<2> but was:<1>
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at junit.framework.TestCase.assertEquals(TestCase.java:409)
at 
org.apache.ignite.internal.processors.igfs.IgfsStreamsSelfTest.testCreateFileFragmented(IgfsStreamsSelfTest.java:264)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:176)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2000)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:132)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1915)
at java.lang.Thread.run(Thread.java:745)
Aug 09, 2017 1:15:56 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: DataStreamer operation failed.
class org.apache.ignite.IgniteCheckedException: Data streamer has been 
cancelled: DataStreamerImpl 
[rcvr=org.apache.ignite.internal.processors.datastreamer.DataStreamerCacheUpdaters$BatchedSorted@13c54950,
 ioPlcRslvr=null, cacheName=igfs-internal-igfs-data, bufSize=512, 
parallelOps=16, timeout=-1, autoFlushFreq=0, 
bufMappings={908d1a4c-b352-4af5-b039-ded60c20=Buffer [node=TcpDiscoveryNode 
[id=908d1a4c-b352-4af5-b039-ded60c20, addrs=[127.0.0.1], 
sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, 
lastExchangeTime=1502241356486, loc=true, ver=2.2.0#19700101-sha1:, 
isClient=false], isLocNode=true, idGen=0, 
sem=java.util.concurrent.Semaphore@2bdbd7f0[Permits = 16], 
batchTopVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], entriesCnt=1, 
locFutsSize=0, reqsSize=0]}, cacheObjProc=GridProcessorAdapter [], 
cacheObjCtx=org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryContext@6e3de40e,
 cancelled=true, failCntr=0, activeFuts=GridConcurrentHashSet 
[elements=[GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, 
hash=431192362], GridFutureAdapter [ignoreInterrupts=false, state=INIT, 
res=null, hash=625896337], GridFutureAdapter [ignoreInterrupts=false, 
state=INIT, res=null, hash=1440203156]]], jobPda=null, depCls=null, 
fut=DataStreamerFuture [super=GridFutureAdapter [ignoreInterrupts=false, 
state=DONE, res=null, hash=1612913644]], publicFut=IgniteFuture 
[orig=DataStreamerFuture [super=GridFutureAdapter [ignoreInterrupts=false, 
state=DONE, res=null, hash=1612913644]]], disconnectErr=null, closed=true, 
lastFlushTime=1502241356435, skipStore=false, keepBinary=false, maxRemapCnt=32, 
remapSem=java.util.concurrent.Semaphore@194e0ba1[Permits = 2147483647], 
remapOwning=false]
at 
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$5.apply(DataStreamerImpl.java:865)
at 
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$5.apply(DataStreamerImpl.java:834)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461)
at 
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.onNodeLeft(DataStreamerImpl.java:1757)
at 
org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$6.run(DataStreamerImpl.java:952)
at 
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6687)
at 
org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:817)
at 

Re: Resurrect FairAffinityFunction

[Valentin Kulichenko]

As far as I know, all logical caches with the same affinity function and
node filter will end up in the same group. If that's the case, I like the
idea. This is exactly what I was looking for.

-Val

On Wed, Aug 9, 2017 at 8:18 AM, Evgenii Zhuravlev 
wrote:

> Dmitriy,
>
> Yes, you're right. Moreover, it looks like a good practice to combine
> caches that will be used for collocated JOINs in one group since it reduces
> overall overhead.
>
> I think it's not a problem to add this restriction to the SQL JOIN level if
> we will decide to use this solution.
>
> Evgenii
>
>
>
>
> 2017-08-09 17:07 GMT+03:00 Dmitriy Setrakyan :
>
> > On Wed, Aug 9, 2017 at 6:28 AM, ezhuravl 
> wrote:
> >
> > > Folks,
> > >
> > > I've started working on a https://issues.apache.org/
> > > jira/browse/IGNITE-5836
> > > ticket and found that the recently added feature with cacheGroups doing
> > > pretty much the same that was described in this issue. CacheGroup
> > > guarantees
> > > that all caches within a group have same assignments since they share a
> > > single underlying 'physical' cache.
> > >
> >
> > > I think we can return FairAffinityFunction and add information to its
> > > Javadoc that all caches with same AffinityFunction and NodeFilter
> should
> > be
> > > combined in cache group to avoid a problem with inconsistent previous
> > > assignments.
> > >
> > > What do you guys think?
> > >
> >
> > Are you suggesting that we can only reuse the same FairAffinityFunction
> > across the logical caches within the same group? This would mean that
> > caches from the different groups cannot participate in JOINs or
> collocated
> > compute.
> >
> > I think I like the idea, however, we need to make sure that we enforce
> this
> > restriction, at least at the SQL JOIN level.
> >
> > Alexey G, Val, would be nice to hear your thoughts on this.
> >
> >
> > >
> > > Evgenii
> > >
> > >
> > >
> > > --
> > > View this message in context: http://apache-ignite-
> > > developers.2346864.n4.nabble.com/Resurrect-FairAffinityFunction-
> > > tp19987p20669.html
> > > Sent from the Apache Ignite Developers mailing list archive at
> > Nabble.com.
> > >
> >
>

Re: Cluster auto activation design proposal

[Denis Magda]

Sergey,

That’s the only concern I have:

* 5. User takes out nodes from cluster (e.g. for maintenance purposes): no
  rebalance happens until user recreates BLT on new cluster topology.*

What if a node is crashed (or some other kind of outage) in the middle of the 
night and the user has to be sure that survived nodes will rearrange and 
rebalancing partitions? 

—
Denis


> On Aug 4, 2017, at 9:21 AM, Sergey Chugunov  wrote:
> 
> Folks,
> 
> I've summarized all results from our discussion so far on wiki page:
> https://cwiki.apache.org/confluence/display/IGNITE/Automatic+activation+design+-+draft
> 
> I hope I reflected the most important details and going to add API
> suggestions for all use cases soon.
> 
> Feel free to give feedback here or in comments under the page.
> 
> Thanks,
> Sergey.
> 
> On Thu, Aug 3, 2017 at 5:40 PM, Alexey Kuznetsov 
> wrote:
> 
>> Hi,
>> 
 1. User creates new BLT using WebConsole or other tool and "applies" it
>> to brand-new cluster.
>> 
>> Good idea, but we also should implement *command-line utility* for the same
>> use case.
>> 
>> --
>> Alexey Kuznetsov
>> 

Re: Zeppelin, MyBatis and Vert.x update to AI 2.1

[Denis Magda]

Roman, thanks!

Andrey, will you be able to complete it this week?

—
Denis

> On Aug 3, 2017, at 7:14 AM, Andrey Gura  wrote:
> 
> Hi,
> 
> I'll do it soon. But there is problem with Zeppelin because CI should
> be configured on local machine. So I need more time than usually for
> update Ignite version in Zeppelin (e.g. Ignite 2.0 still not relased
> in Zeppelin).
> 
> On Tue, Aug 1, 2017 at 9:13 AM, Roman Shtykh  
> wrote:
>> Denis,
>> I updated MyBatis integration, and will release it this week.
>> -- Roman
>> 
>> 
>>On Tuesday, August 1, 2017 5:23 AM, Denis Magda  wrote:
>> 
>> 
>> Andrey, Roman,
>> As maintainers of the projects in the subject, could you please go ahead and 
>> update Ignite version to 2.1 
>> there?https://cwiki.apache.org/confluence/display/IGNITE/External+Integrations
>> —Denis
>> 
>> 

Re: Ignite.close(), G.stop(name, true). Change flag cancel to false

[Denis Magda]

I’m totally for the replacement of ‘crashed’ with ‘stopped’. 

As for the waiting of the checkpointing completion I would NOT do it the 
default behavior and would rather check the ‘cancel’ flag to make a decision. 
If the ‘cancel’ is ‘true’ (which is default) then we’re not going to wait for 
the completion and should print a message that   ‘cancel’ has to be set to 
‘false’ explicitly if the user prefers to wait while the checkpoint is over 
before shutting down a node.

—
Denis

> On Aug 4, 2017, at 4:31 AM, Ivan Rakov  wrote:
> 
> My vote is still for making message softer (crashed -> stopped) and keeping 
> logic as is.
> 
> Example with File.close() is good, but I think it's not the case here. The 
> state on disk after node stop *will not* reflect all user actions made before 
> Ignite.close() call, independent of whether node was stopped during 
> checkpoint.
> Ignite will recover to actual state anyway, the only difference is WAL replay 
> algorithm (stopping during checkpoint will force Ignite to replay delta 
> records).
> 
> However, waiting for checkpoint on node stop brings two advantages:
> 1) Next start will be faster - less WAL records to replay.
> 2) Partition files will be locally consistent after node stop. User will be 
> able to save partition file for any kind of analysis.
> 
> Are they strong enough to force user to wait on stop?
> 
> Best Regards,
> Ivan Rakov
> 
> On 04.08.2017 13:42, Vyacheslav Daradur wrote:
>> Hi guys, I'll just add my opinion if you don't mind.
>> 
>>> May be we should implement Vladimir's suggestion to flush the pages
>> without
>>> respect to the cancel flag? Are there any thoughts on this?
>> I think It's  good suggestion.
>> But in case of unit-testing a developer usually call #stopAllGrids() at the
>> end of all tests.
>> The method GridAbstactTest#stopAllGrids() is built on top of the
>> method G.stop(name,
>> true) including.
>> IMO in that case checkpoints' flushing isn't necessary.
>> 
>> 
>> 2017-08-04 13:25 GMT+03:00 Dmitry Pavlov :
>> 
>>> Thank you all for replies.
>>> 
>>> I like idea to replace 'crashed' to 'stop'.  'crashed' word is really
>>> confusing.
>>> 
>>> But still, if I call close () on file, all data is flushed to disk. But for
>>> ignite.close () checkpoint may be not finished.
>>> 
>>> May be we should implement Vladimir's suggestion to flush the pages without
>>> respect to the cancel flag? Are there any thoughts on this?
>>> 
>>> пт, 4 авг. 2017 г. в 11:12, Vladimir Ozerov :
>>> 
 Ivan,
 
 Hanging on Ignite.close() will confuse user no more than restore on start
 after graceful shutdown. IMO correct approach here would be to:
 1) wait for checkpoint completion irrespective of "cancel" flag, because
 this flag relates to compute jobs only as per documentation
 2) print an INFO message to the log that we are saving a checkpoint due
>>> to
 node stop.
 
 On Fri, Aug 4, 2017 at 10:54 AM, Ivan Rakov 
>>> wrote:
> Dmitriy,
> 
> From my point of view, invoking stop(true) is correct behaviour.
> 
> Stopping node in the middle of checkpoint is absolutely valid case.
 That's
> how persistence works - node will restore memory state if stopped at
>>> any
> moment.
> On the other hand, checkpoint may last for a long time. Thread hanging
>>> on
> Ignite.close() may confuse user much more than "crashed in the middle
>>> of
> checkpoint" message.
> 
> Best Regards,
> Ivan Rakov
> 
> 
> On 03.08.2017 22:34, Dmitry Pavlov wrote:
> 
>> Hi Igniters,
>> 
>> I’ve created the simplest example using Ignite 2.1 and persistence
>>> (see
>> the
>> code below). I've included Ignite instance into try-with-resources (I
>> think
>> it is default approach for AutoCloseable inheritors).
>> 
>> But next time when I started this server I got message: “Ignite node
>> crashed in the middle of checkpoint. Will restore memory state and
 enforce
>> checkpoint on node start.”
>> 
>> This happens because in close() method we don’t wait checkpoint to
>>> end.
 I
>> am afraid this behaviour may confuse users on the first use of the
>> product.
>> 
>> What do you think if we change Ignite.close() functioning from
 stop(true)
>> to stop(false)? This will allow to wait checkpoints to finish by
 default.
>> Alternatively, we may improve example to show how to shutdown server
 node
>> correctly. Current PersistentStoreExample does not cover server node
>> shutdown.
>> 
>> Any concerns on close() method change?
>> 
>> Sincerely,
>> Dmitriy Pavlov
>> 
>> 
>> IgniteConfiguration cfg = new IgniteConfiguration();
>> cfg.setPersistentStoreConfiguration(new
>>> PersistentStoreConfiguration());
>> try (Ignite ignite = Ignition.start(cfg)){

[GitHub] ignite pull request #2383: IGNITE-5912: Redis EXPIRE/PEXPIRE commands.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2383


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

IGNITE-5714 Context switching for pessimistic transactions

[ALEKSEY KUZNETSOV]

Hi, Igntrs!

Currently we have context switching implemented for optimistic transactions
: https://issues.apache.org/jira/browse/IGNITE-5712.



So, the next step is to implement it for pessimistic transactions :
https://issues.apache.org/jira/browse/IGNITE-5714



The problem with them lies in *IgniteTxAdapter#threadId*. Thread id is
transferred between nodes by GridDistributedTxPrepareRequest when key is
got locked.

When we suspend and resume transaction, thread id is got changed locally,
but not on remote nodes.



After studying the code, it seemed we can eliminate thread id from
transactions completely.

For that reason, i want to start implementing additional tests, that will
cover transaction logic. Tickets would be created for them.
Later on I will provide test scenarious and send you. *Will appreciate any
ideas from you on new tests, thanks!*



It will be the first step. The next one will be refactoring and eliminating
thread id. What do you think ?

-- 

*Best Regards,*

*Kuznetsov Aleksey*

[jira] [Created] (IGNITE-6013) Web agent: refactor processing response from cluster

[Alexey Kuznetsov (JIRA)]

Alexey Kuznetsov created IGNITE-6013:


 Summary: Web agent: refactor processing response from cluster
 Key: IGNITE-6013
 URL: https://issues.apache.org/jira/browse/IGNITE-6013
 Project: Ignite
  Issue Type: Improvement
Reporter: Alexey Kuznetsov
Assignee: Alexey Kuznetsov


RestExecutor.sendRequest() contain following code:
{code}

try (Response resp = httpClient.newCall(reqBuilder.build()).execute()) {
String content = resp.body().string();

if (resp.isSuccessful()) {
JsonNode node = mapper.readTree(content);
 
{code}

Problems: 
# String content = resp.body().string(); >> Generate not needed String.
# JsonNode node = mapper.readTree(content);  >> Generates a big tree of 
JsonNodes
 
Could be fixed like:
RestResponseHolder res = MAPPER.readValue(resp.body().byteStream(), 
RestResponseHolder.class); 

Where for RestResponseHolder should also created optimized 
RestResponseHolderDeserializer that will extract response as String without 
building tree of JsonNodes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: IgniteUtils#currentTimeMillis

[Vladimir Ozerov]

Yakov,

For example:
http://pzemtsov.github.io/2017/07/23/the-slow-currenttimemillis.html

>>> We’ve learned that the slow execution of currentTimeMillis() was caused
by two factors:
>>> - JVM using gettimeofday() instead of clock_gettime()
>>> - gettimeofday() being very slow if HPET time source is used.

You mat get slow currentTimeMillis() - up to 1mcs - unless you set proper
time source.


On Wed, Aug 9, 2017 at 4:15 PM, Yakov Zhdanov  wrote:

> As Dmitry P mentioned System.currentTimeMillis() is JVM intrinsic.
> Moreover, there is a daemon thread that updates the internal value which
> will not be needed after the change.
>
> If we remove U.currentTimeMillis() code will become more clear and
> consistent. Why we think that we can implement this particular timer better
> than JVM developers?
>
> Vladmir, can you please share a benchmark that will show the problems? If
> it will then it will be a strong argument to keep the current
> implementation.
>
> --Yakov
>

[jira] [Created] (IGNITE-6014) Add transaction prepare and commit markers to WAL

[Alexey Goncharuk (JIRA)]

Alexey Goncharuk created IGNITE-6014:


 Summary: Add transaction prepare and commit markers to WAL
 Key: IGNITE-6014
 URL: https://issues.apache.org/jira/browse/IGNITE-6014
 Project: Ignite
  Issue Type: Improvement
  Components: persistence
Affects Versions: 2.1
Reporter: Alexey Goncharuk
 Fix For: 2.2


This may be useful for various recovery procedures in the future



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Data compression in Ignite

[Dmitriy Setrakyan]

Vova,

I do not understand how (2) will help us reduce the size of the serialized
object. It sounds like we need to serialize an object the way we do today,
and then compress it when it arrives to the server side and gets stored in
some page. This looks like a double whammy.

I do not understand why the approach (3) is difficult. Yes, we will have to
implement a shared compression table, but I am not sure if it will ever
grow too big. As a matter of fact, we can stop adding entries to it, after
it reaches a certain pre-defined size. Also, we already know how to persist
data, so spilling the compression table to disk should not be an issue.

If my assumption about (2) is correct, I would like to ask you to give
another thought to the (3).

D.

On Wed, Aug 9, 2017 at 7:48 AM, Vladimir Ozerov 
wrote:

> Igniters,
>
> I had several private talks with Igniters about data compression and would
> like to share the summary with ... Igniters :-)
>
> Currently all Ignite's data is uncompressed. It leads to excessive network
> traffic, GC pressure and disk IO (in case of persistence). Most modern
> databases are able to compress data, what gives them 2-4x size reduction on
> typical workloads. We need compression in Ignite.
>
> There are several options I'd like to discuss. The main difference between
> them - on what "level" to compress: per-entry, per-data-page or per-cache.
>
> *1) Per-entry compression*
> Apache Geode uses this approach. Every cache entry is compressed using
> Snappy. This is very easy to implement, but every entry access (e.g.
> reading single field) require full decompression or even re-compression,
> what could lead to higher CPU consumption and worse performance.
>
> *2) Per-data-page compression*
> Oracle and DB2 use this approach. Pages are compressed with
> dictionary-based approach (e.g. LZV). It is important, that they do not
> compress the whole page. Instead, only actual data is compressed, while
> page structure remains intact. Dictionary is placed within the page. This
> way it is possible to work with individual entries and even individual
> fields without full page decompression. Another important thing - it is not
> necessary to re-compress the page on each write. Instead, data is stored in
> uncompressed form first, and compressed even after certain threshold is
> reached. So negative CPU impact is minimal. Typical compression rate would
> be higher than in per-entry case, because the more data you have, the
> better it can be compressed.
>
> *3) Per-cache compression*
> Suggested by Alex Goncharuk. We could have a dictionary for the whole
> cache. This way we could achieve the highest compression rate possible. The
> downside is complex implementation - we would have to develop an algorithm
> of sharing the dictionary within the cluster. At some point the dictionary
> could become too huge to fit in-memory, so we should either control it's
> size or spill it to disk.
>
> I propose to use per-data-page approach as both gives nice compression rate
> and relatively easy to implement.
>
> Please share your thoughts.
>
> Vladimir.
>

Re: IgniteUtils#currentTimeMillis

[Alexey Kuznetsov]

Nikolay,

As far as I understand U.currentTimeMillis() should be used where time is
not a major value (metrics for example).

But in test with transaction (that you are mentioned) we should use
System.currentTimeMillis().

In general we should think about U.currentTimeMillis() and avoid it usage
in places where result of this function could be compared with some value.

Vova, make sense?

On Wed, Aug 9, 2017 at 7:49 PM, Николай Ижиков 
wrote:

> Dmitry,
>
> > So, if you change the call to System.currentTimeMillis(), the test
> passes?
>
> Yes
>
> > I would propose to either increase TX_TIMEOUT or sleep multiplier to make
> test more reliable.
>
> Yes, I fix test in that way.
>
> For me the goal of this discussion is to understand reasons to keep current
> method implementation.
>
>
>
>
> 2017-08-09 15:45 GMT+03:00 Dmitriy Setrakyan :
>
> > On Wed, Aug 9, 2017 at 5:32 AM, Николай Ижиков 
> > wrote:
> >
> > > Vladimir,
> > >
> > > As far as I can understand behaviour of U.currentTimeMillis() breaks
> > > transaction timeout test:
> > >
> >
> > So, if you change the call to System.currentTimeMillis(), the test
> passes?
> >
>
>
>
> --
> Nikolay Izhikov
> nizhikov@gmail.com
>



-- 
Alexey Kuznetsov

Re: IgniteUtils#currentTimeMillis

[Yakov Zhdanov]

As Dmitry P mentioned System.currentTimeMillis() is JVM intrinsic.
Moreover, there is a daemon thread that updates the internal value which
will not be needed after the change.

If we remove U.currentTimeMillis() code will become more clear and
consistent. Why we think that we can implement this particular timer better
than JVM developers?

Vladmir, can you please share a benchmark that will show the problems? If
it will then it will be a strong argument to keep the current
implementation.

--Yakov

Re: Resurrect FairAffinityFunction

[ezhuravl]

Folks,

I've started working on a https://issues.apache.org/jira/browse/IGNITE-5836
ticket and found that the recently added feature with cacheGroups doing
pretty much the same that was described in this issue. CacheGroup guarantees
that all caches within a group have same assignments since they share a
single underlying 'physical' cache.

I think we can return FairAffinityFunction and add information to its
Javadoc that all caches with same AffinityFunction and NodeFilter should be
combined in cache group to avoid a problem with inconsistent previous
assignments.

What do you guys think?

Evgenii



--
View this message in context: 
http://apache-ignite-developers.2346864.n4.nabble.com/Resurrect-FairAffinityFunction-tp19987p20669.html
Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

Re: IgniteUtils#currentTimeMillis

[Evgeniy Stanilovskiy]

I assume that Vladimir mention this mesurements:
https://shipilev.net/blog/2014/nanotrusting-nanotime/
can we simple measure with JMH x86 and arm our realization vs system call?


As Dmitry P mentioned System.currentTimeMillis() is JVM intrinsic.
Moreover, there is a daemon thread that updates the internal value which
will not be needed after the change.

If we remove U.currentTimeMillis() code will become more clear and
consistent. Why we think that we can implement this particular timer  
better

than JVM developers?

Vladmir, can you please share a benchmark that will show the problems? If
it will then it will be a strong argument to keep the current
implementation.

--Yakov

[jira] [Created] (IGNITE-6012) Improve GridJettyRestHandler.processRequest()

[Alexey Kuznetsov (JIRA)]

Alexey Kuznetsov created IGNITE-6012:


 Summary: Improve GridJettyRestHandler.processRequest()
 Key: IGNITE-6012
 URL: https://issues.apache.org/jira/browse/IGNITE-6012
 Project: Ignite
  Issue Type: Improvement
  Components: rest
Affects Versions: 2.1
Reporter: Alexey Kuznetsov
Assignee: Alexey Kuznetsov
 Fix For: 2.2


In case of large result  
{code}
private void processRequest(String act, HttpServletRequest req, 
HttpServletResponse res) {
...
json = jsonMapper.writeValueAsString(cmdRes)

{code}

Will fail with OOME, beacuse jsonMapper.writeValueAsString(cmdRes) internally 
will create a StringBuilder and will try to allocate large amount of memory.

This could be easily fixed by writing object directly to response output stream 
via. 
{code}
jsonMapper.writeValue(out, cmdRes);
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2421: IGNITE-5995: ODBC: Fix for SQLGetData

[isapego]

GitHub user isapego opened a pull request:

https://github.com/apache/ignite/pull/2421

IGNITE-5995: ODBC: Fix for SQLGetData



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5995

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2421.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2421


commit af8c1a82977504378adabf35a8d58d444f81dfe9
Author: Igor Sapego 
Date:   2017-08-08T16:55:46Z

IGNITE-5995: Added tests.

commit 89064118f2e070de7785ce3cec251273d171ab0f
Author: Igor Sapego 
Date:   2017-08-08T17:50:44Z

IGNITE-5995: Fix for tests

commit f2ce7cee88bac5796c705fab297de1fb900a1bb0
Author: Igor Sapego 
Date:   2017-08-08T17:51:19Z

IGNITE-5995: Meta queries fixed




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Add isPrimary() and isBackup() methods on CacheQueryEntryEvent

[Kozlov Maxim]

Igniters,

What do you think about the new API?


> 3 авг. 2017 г., в 15:56, Kozlov Maxim  написал(а):
> 
> Guys, 
> 
> I'm sorry for the misunderstanding, I was tired at the end of the day :-)
> In the process of working on the task, I had to add 2 methods to the public 
> interface.
> The first method is #isBackup(), it returns 'true' if cache is being updated 
> on the backup node.
> The second method is #isPrimary(), it returns 'true' if cache is being 
> updated on the primary node.
> Their main purpose is to show where a continuous query filter has been 
> invoked.
> 
> Any thoughts about such solution?
> 
>> 3 авг. 2017 г., в 14:45, Anton Vinogradov  
>> написал(а):
>> 
>> Folks,
>> 
>> As far as I see, Issue still in PatchAvailable state, what did you mean by
>> "solved"?
>> 
>> On Wed, Aug 2, 2017 at 8:01 PM, Kozlov Maxim  wrote:
>> 
>>> Sure.
>>> 
>>> CacheQueryEntryEvent:
>>> 
>>> public abstract boolean isBackup();
>>> public abstract boolean isPrimary();
>>> 
>>> 
 2 авг. 2017 г., в 19:56, Nikolai Tikhonov 
>>> написал(а):
 
 Max,
 
 Thank you for your contribution! Could you share here what exactly was
 added to interface?
 
 On Wed, Aug 2, 2017 at 7:53 PM, Kozlov Maxim 
>>> wrote:
 
> Igniters,
> 
> When you solved the 3878[1] ticket, two methods were added[2]:
>>> isPrimary()
> and isBackup() on the CacheQueryEntryEvent in a public API. Do you agree
> with this decision?
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-3878 <
> https://issues.apache.org/jira/browse/IGNITE-3878>
> [2] https://github.com/apache/ignite/pull/1393 <
>>> https://github.com/apache/
> ignite/pull/1393>
> 
> --
> Best Regards,
> Max K.
> 
> 
> 
> 
> 
>>> 
>>> --
>>> Best Regards,
>>> Max K.
>>> 
>>> 
>>> 
>>> 
>>> 
> 
> --
> Best Regards,
> Max K.
> 
> 
> 
> 

--
Best Regards,
Max K.

Data compression in Ignite

[Vladimir Ozerov]

Igniters,

I had several private talks with Igniters about data compression and would
like to share the summary with ... Igniters :-)

Currently all Ignite's data is uncompressed. It leads to excessive network
traffic, GC pressure and disk IO (in case of persistence). Most modern
databases are able to compress data, what gives them 2-4x size reduction on
typical workloads. We need compression in Ignite.

There are several options I'd like to discuss. The main difference between
them - on what "level" to compress: per-entry, per-data-page or per-cache.

*1) Per-entry compression*
Apache Geode uses this approach. Every cache entry is compressed using
Snappy. This is very easy to implement, but every entry access (e.g.
reading single field) require full decompression or even re-compression,
what could lead to higher CPU consumption and worse performance.

*2) Per-data-page compression*
Oracle and DB2 use this approach. Pages are compressed with
dictionary-based approach (e.g. LZV). It is important, that they do not
compress the whole page. Instead, only actual data is compressed, while
page structure remains intact. Dictionary is placed within the page. This
way it is possible to work with individual entries and even individual
fields without full page decompression. Another important thing - it is not
necessary to re-compress the page on each write. Instead, data is stored in
uncompressed form first, and compressed even after certain threshold is
reached. So negative CPU impact is minimal. Typical compression rate would
be higher than in per-entry case, because the more data you have, the
better it can be compressed.

*3) Per-cache compression*
Suggested by Alex Goncharuk. We could have a dictionary for the whole
cache. This way we could achieve the highest compression rate possible. The
downside is complex implementation - we would have to develop an algorithm
of sharing the dictionary within the cluster. At some point the dictionary
could become too huge to fit in-memory, so we should either control it's
size or spill it to disk.

I propose to use per-data-page approach as both gives nice compression rate
and relatively easy to implement.

Please share your thoughts.

Vladimir.

[GitHub] ignite pull request #2387: IGNITE-5736 Add test of backward-compatibility

[daradurvs]

Github user daradurvs closed the pull request at:

https://github.com/apache/ignite/pull/2387


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: IgniteUtils#currentTimeMillis

[Vladimir Ozerov]

In short, the reason is avoiding potential performance problems.

On Wed, Aug 9, 2017 at 3:49 PM, Николай Ижиков 
wrote:

> Dmitry,
>
> > So, if you change the call to System.currentTimeMillis(), the test
> passes?
>
> Yes
>
> > I would propose to either increase TX_TIMEOUT or sleep multiplier to make
> test more reliable.
>
> Yes, I fix test in that way.
>
> For me the goal of this discussion is to understand reasons to keep current
> method implementation.
>
>
>
>
> 2017-08-09 15:45 GMT+03:00 Dmitriy Setrakyan :
>
> > On Wed, Aug 9, 2017 at 5:32 AM, Николай Ижиков 
> > wrote:
> >
> > > Vladimir,
> > >
> > > As far as I can understand behaviour of U.currentTimeMillis() breaks
> > > transaction timeout test:
> > >
> >
> > So, if you change the call to System.currentTimeMillis(), the test
> passes?
> >
>
>
>
> --
> Nikolay Izhikov
> nizhikov@gmail.com
>

Re: Resurrect FairAffinityFunction

[Dmitriy Setrakyan]

On Wed, Aug 9, 2017 at 6:28 AM, ezhuravl  wrote:

> Folks,
>
> I've started working on a https://issues.apache.org/
> jira/browse/IGNITE-5836
> ticket and found that the recently added feature with cacheGroups doing
> pretty much the same that was described in this issue. CacheGroup
> guarantees
> that all caches within a group have same assignments since they share a
> single underlying 'physical' cache.
>

> I think we can return FairAffinityFunction and add information to its
> Javadoc that all caches with same AffinityFunction and NodeFilter should be
> combined in cache group to avoid a problem with inconsistent previous
> assignments.
>
> What do you guys think?
>

Are you suggesting that we can only reuse the same FairAffinityFunction
across the logical caches within the same group? This would mean that
caches from the different groups cannot participate in JOINs or collocated
compute.

I think I like the idea, however, we need to make sure that we enforce this
restriction, at least at the SQL JOIN level.

Alexey G, Val, would be nice to hear your thoughts on this.


>
> Evgenii
>
>
>
> --
> View this message in context: http://apache-ignite-
> developers.2346864.n4.nabble.com/Resurrect-FairAffinityFunction-
> tp19987p20669.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.
>

Re: Data compression in Ignite

[Yakov Zhdanov]

Agree, I like 2 best of all.

--Yakov

Re: Resurrect FairAffinityFunction

[Evgenii Zhuravlev]

Dmitriy,

Yes, you're right. Moreover, it looks like a good practice to combine
caches that will be used for collocated JOINs in one group since it reduces
overall overhead.

I think it's not a problem to add this restriction to the SQL JOIN level if
we will decide to use this solution.

Evgenii




2017-08-09 17:07 GMT+03:00 Dmitriy Setrakyan :

> On Wed, Aug 9, 2017 at 6:28 AM, ezhuravl  wrote:
>
> > Folks,
> >
> > I've started working on a https://issues.apache.org/
> > jira/browse/IGNITE-5836
> > ticket and found that the recently added feature with cacheGroups doing
> > pretty much the same that was described in this issue. CacheGroup
> > guarantees
> > that all caches within a group have same assignments since they share a
> > single underlying 'physical' cache.
> >
>
> > I think we can return FairAffinityFunction and add information to its
> > Javadoc that all caches with same AffinityFunction and NodeFilter should
> be
> > combined in cache group to avoid a problem with inconsistent previous
> > assignments.
> >
> > What do you guys think?
> >
>
> Are you suggesting that we can only reuse the same FairAffinityFunction
> across the logical caches within the same group? This would mean that
> caches from the different groups cannot participate in JOINs or collocated
> compute.
>
> I think I like the idea, however, we need to make sure that we enforce this
> restriction, at least at the SQL JOIN level.
>
> Alexey G, Val, would be nice to hear your thoughts on this.
>
>
> >
> > Evgenii
> >
> >
> >
> > --
> > View this message in context: http://apache-ignite-
> > developers.2346864.n4.nabble.com/Resurrect-FairAffinityFunction-
> > tp19987p20669.html
> > Sent from the Apache Ignite Developers mailing list archive at
> Nabble.com.
> >
>