Re: Internal classes are exposed in public API

[Anton Vinogradov]

>> . Remove @Deprecated from old API (because it strange to have one
>> deprecated API and second experimental API)
>> 2. Add @IgniteExperimetnal to new API (because... see item. 1)
-1

We should have all the options.

Deprecated -> will be removed at major release (because this way is slow or
overcomplicated) but still be used now,
Annotation is absent -> just a stable (proper) API,
Experimental -> not stable yet or partially implemented or in fact an
experimental

On Thu, Jan 30, 2020 at 2:47 PM Maxim Muzafarov  wrote:

> Igniters,
>
>
> How about to start a vote (formal) over the whole community to choose
> between these two options we have? With one caveat that veto votes are not
> applicable to this discussion. Let the whole community decide what
> direction we should move on.
>
> On Thu, 30 Jan 2020 at 14:34, Nikolay Izhikov  wrote:
>
> > Hello.
> >
> > > 2. Add @IgniteExperimetnal to new API (because... see item. 1
> >
> > +1
> >
> > > 1. Remove @Deprecated from old API (because it strange to have one
> > deprecated API and second experimental API)
> >
> > -1
> >
> > I propose to update deprecation message and provide metric name for each
> > deprecated method.
> >
> > > @deprecated Use {@link JmxMetricExporterSPI} instead. Name of the
> metric
> > «io.dataregion.pageCount»
> >
> > Andrey.
> >
> > We doesn’t come to an agreement that API should be changed.
> > We should discuss the design of the Metric API and your proposals for it
> > in another thread.
> >
> > Please, avoid arguments like «this API will be 100% changed» in this
> > discussion.
> >
> > > 30 янв. 2020 г., в 14:21, Andrey Gura  написал(а):
> > >
> > > From my point of view we still don't have consensus.
> > >
> > > I think we should do at least the following:
> > >
> > > 1. Remove @Deprecated from old API (because it strange to have one
> > > deprecated API and second experimental API)
> > > 2. Add @IgniteExperimetnal to new API (because... see item. 1)
> > > 3. Do not merge IGNITE-12553 (because it adds new public interface
> > > that 100% will be changed)
> > >
> > > ideally we should also:
> > >
> > > 4. Add metrics that available only via new API to the old API (because
> > > otherwise we force user interact with both API's)
> > >
> > > On Thu, Jan 30, 2020 at 12:35 PM Alexey Goncharuk
> > >  wrote:
> > >>
> > >> Folks,
> > >>
> > >> I tried to re-read the whole thread and honestly got lost at the end
> :)
> > Do
> > >> we have a consensus (if yes, what are the steps?) or should we have a
> > call
> > >> as Maxim suggested?
> > >>
> > >> I think it is in our best interest to get this agreed upon fast to
> > release
> > >> AI 2.8 soon.
> >
> >
>

Re: Excessive backups performance suggestion.

[Anton Vinogradov]

The statement seems to be correct for cases when we'd like to perform fast
loading and computation and not afraid of data loss.
Performance boost always means we lose some guarantee.
Let's just update the proposal with the explicit warning that reducing
backups to zero may lead to data loss on any node failure.

On Thu, Feb 6, 2020 at 1:09 PM Alexei Scherbakov <
alexey.scherbak...@gmail.com> wrote:

> Definitely wrong recommendation.
> Please file a ticket for removal.
>
> чт, 6 февр. 2020 г. в 12:26, Zhenya Stanilovsky  >:
>
> >
> > Igniters, i found that suggesting zero backup for performance
> improvements
> > sounds like malicious and further problems suggestion. May be it`s time
> to
> > remove it ?
> >
> > Example:
> > [07:59:27] Performance suggestions for grid (fix if possible)
> > [07:59:27] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
> > [07:59:27] ^-- Decrease number of backups (set 'backups' to 0)
> >
> >
>
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>

Re: [REVIEW REQUEST] IGNITE-12630 Remove developers sections from parent pom.xml

[Anton Vinogradov]

Looks good to me.

On Thu, Feb 6, 2020 at 11:45 AM Ivan Pavlukhin  wrote:

> Igniters,
>
> I raised a PR for a ticket [1] removing  section from
> parent pom.xml. I described the motivation in the ticket. Shortly,
> this section has a little meaning today and even worse is misleading.
>
> Please review.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12630
>
> Best regards,
> Ivan Pavlukhin
>

Re: Excessive backups performance suggestion.

[Anton Vinogradov]

>> most users do not want to lose data by default.
But some ready to sacrifice.

I know at least one case when "pure speed without insurance" required when
the ML model is studying.

On Thu, Feb 6, 2020 at 1:39 PM Alexei Scherbakov <
alexey.scherbak...@gmail.com> wrote:

> I'm quite sure most users do not want to lose data by default.
>
> It's not wise to recommend such thing even with explicit warning.
>
> чт, 6 февр. 2020 г. в 13:30, Anton Vinogradov :
>
> > The statement seems to be correct for cases when we'd like to perform
> fast
> > loading and computation and not afraid of data loss.
> > Performance boost always means we lose some guarantee.
> > Let's just update the proposal with the explicit warning that reducing
> > backups to zero may lead to data loss on any node failure.
> >
> > On Thu, Feb 6, 2020 at 1:09 PM Alexei Scherbakov <
> > alexey.scherbak...@gmail.com> wrote:
> >
> > > Definitely wrong recommendation.
> > > Please file a ticket for removal.
> > >
> > > чт, 6 февр. 2020 г. в 12:26, Zhenya Stanilovsky
> >  > > >:
> > >
> > > >
> > > > Igniters, i found that suggesting zero backup for performance
> > > improvements
> > > > sounds like malicious and further problems suggestion. May be it`s
> time
> > > to
> > > > remove it ?
> > > >
> > > > Example:
> > > > [07:59:27] Performance suggestions for grid (fix if possible)
> > > > [07:59:27] To disable, set
> > -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
> > > > [07:59:27] ^-- Decrease number of backups (set 'backups' to 0)
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> > >
> >
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>

Re: Forbid mixed cache groups with both atomic and transactional caches

[Anton Vinogradov]

Seems, we already started the separation by atomic operations restriction
inside the transactions [1].
See no reason to allow mixes in this case.

[1] https://issues.apache.org/jira/browse/IGNITE-2313

On Tue, Feb 4, 2020 at 2:28 PM Ivan Rakov  wrote:

> Igniters,
>
> Apparently it's possible in Ignite to configure a cache group with both
> ATOMIC and TRANSACTIONAL caches.
> Proof: IgniteCacheGroupsTest#testContinuousQueriesMultipleGroups* tests.
> In my opinion, it would be better to remove such possibility from the
> product. There are several reasons:
>
> 1) The original idea of grouping caches was optimizing storage overhead and
> PME time by joining data of similar caches into the same partitions. ATOMIC
> and TRANSACTIONAL caches provide different guarantees and are designed for
> different use cases, thus they can hardly be called "similar".
>
> 2) Diving deeper: synchronization protocols and possible reasons for
> primary-backup divergences are conceptually different for ATOMIC and
> TRANSACTIONAL cases. In TRANSACTIONAL case, transactions recovery protocol
> allows to recover consistency if any participating node will fail, but for
> ATOMIC caches there's possible scenario with failure of primary node where
> neither of backups will contain the most recent state of the data. Example:
> one backup have received updates 1, 3, 5 while another have received 2, 4
> (which is possible due to message reordering), and even tracking counters
> [1] won't restore the consistency. The problem is that we can't distinguish
> what kind of conflict we have faced in case update counters have diverged
> in a mixed group.
>
> 3) Mixed groups are poorly tested. I can't find any tests except a couple
> of smoke tests in IgniteCacheGroupsTest. We can't be sure that different
> synchronization protocols will work correctly for such configurations,
> especially under load and with a variety of dependent configuration
> parameters.
>
> 4) I have never heard of any feedback on mixed groups. I have asked
> different people on this and no one recalled any attempts to configure such
> groups. I believe that in fact no one has ever tried to do it.
>
> Please let me know if you are aware of any cases where mixed groups are
> used or reasons to keep them. Otherwise I'll create a ticket to prohibit
> mixed configurations.
>
> [1]: https://issues.apache.org/jira/browse/IGNITE-11797
>
> --
> Best Regards,
> Ivan Rakov
>

Re: [VOTE] Allow or prohibit a joint use of @deprecated and @IgniteExperimental

[Anton Vinogradov]

-1 Prohibit

On Mon, Feb 10, 2020 at 5:30 PM Alexey Kuznetsov 
wrote:

> -1 Prohibit
>
> From my point of view, we should not deprecate the old API if the new API
> is marked as experemental.
>
>
> On Mon, Feb 10, 2020 at 4:47 PM Konstantin Orlov 
> wrote:
>
> > -1 Prohibit
> >
> > We should not deprecate the old API if the new API could change in the
> > near future.
> >
> >
>
> --
> Alexey Kuznetsov
>

Partition reserve/release asymmetry

[Anton Vinogradov]

Folks,
Yardstick run (opt-serial-put-get-1-backup) failed with interesting
exception:
Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
o.a.i.i.transactions.IgniteTxHeuristicCheckedException: Committing a
transaction has produced runtime exception]]
class
org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException:
Committing a transaction has produced runtime exception
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.heuristicException(IgniteTxAdapter.java:800)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:838)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:893)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1452)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1375)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:123)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:241)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:239)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1843)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1468)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1365)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:555)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Tree is being concurrently
destroyed: tx-p-470##CacheData
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.checkDestroyed(BPlusTree.java:1011)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1831)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1696)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1679)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:441)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:4288)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:4262)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1540)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:675)
... 19 more

It seems, BPlusTree was destroyed between
GridDistributedTxRemoteAdapter.java:545 and
GridDistributedTxRemoteAdapter.java:675 while partition was reserved.

See the full log [1] for details.

During investigation weird code was found:
private void release0(int sizeChange) {
while (true) {
long state = this.state.get();

int reservations = getReservations(state);

if (reservations == 0) // How can it be zero at release attempt?
return;

I've replaced this weird code with assertion [2] and checked at TeamCity
twice, nothing failed.

So, questions
1) Any Idea why we able to have zero reservations at release attempt?
2) Any objection to merging assertion instead of weird return to the master
branch?
3) Any Idea why the exception happens?

[1]
https://gist.githubusercontent.com/anton-vinogradov

Re: Master compilation error

[Anton Vinogradov]

Also, is it possible to perform the same check on the merge attempt?

Each PR can be merged from GitHub page, can we append check to the "megre
button" flow?
Can we restrict merges bypassing this button?

On Mon, Jan 20, 2020 at 2:17 PM Anton Vinogradov  wrote:

> Should we perform an additional compilation attempt on pull/xxx/merge at
> each visa request?
>
> On Mon, Jan 20, 2020 at 1:56 PM Pavel Tupitsyn 
> wrote:
>
>> > Main question is "how this may happen in case fix got the Visa [2] ?".
>> This can happen because of other changes in master.
>> "Visa" truly works only when master is in the same state during the merge
>> as it was during TC run.
>>
>> On Mon, Jan 20, 2020 at 1:50 PM Anton Vinogradov  wrote:
>>
>> > It seems, this because of IGNITE-12227 fix [1].
>> > Main question is "how this may happen in case fix got the Visa [2] ?".
>> >
>> > [1]
>> >
>> >
>> https://ci.ignite.apache.org/viewModification.html?modId=895719=false=IgniteTests24Java8_BuildApacheIgnite=vcsModificationFiles
>> > [2]
>> >
>> >
>> https://issues.apache.org/jira/browse/IGNITE-12227?focusedCommentId=17018135=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17018135
>> >
>> > On Mon, Jan 20, 2020 at 1:43 PM Николай Ижиков 
>> > wrote:
>> >
>> > > Hello. Igniters.
>> > >
>> > > Master build fails:
>> > >
>> > >
>> > >
>> >
>> https://ci.ignite.apache.org/viewLog.html?buildId=4944107=IgniteTests24Java8_BuildApacheIgnite=buildLog_IgniteTests24Java8=pull%2F7269%2Fhead
>> > >
>> > > [13:37:02][Step 3/4] [ERROR] Failed to execute goal
>> > > org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
>> > > (default-testCompile) on project ignite-core: Compilation failure:
>> > > Compilation failure:
>> > > [13:37:02][Step 3/4] [ERROR]
>> > >
>> >
>> /opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/transactions/TxPartitionCounterStateConsistencyVolatileRebalanceTest.java:[25,1]
>> > > cannot find symbol
>> > > [13:37:02][Step 3/4] [ERROR]   symbol:   static
>> > > IGNITE_BASELINE_AUTO_ADJUST_ENABLED
>> > > [13:37:02][Step 3/4] [ERROR]   location: class
>> > > [13:37:02][Step 3/4] [ERROR]
>> > >
>> >
>> /opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/transactions/TxPartitionCounterStateAbstractTest.java:[162,28]
>> > > cannot find symbol
>> > > [13:37:02][Step 3/4] [ERROR]   symbol:   variable
>> > > IGNITE_BASELINE_AUTO_ADJUST_ENABLED
>> > > [13:37:02][Step 3/4] [ERROR]   location: class
>> > >
>> >
>> org.apache.ignite.internal.processors.cache.transactions.TxPartitionCounterStateAbstractTest
>> > > [13:37:02][Step 3/4] [ERROR]
>> > >
>> >
>> /opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/transactions/TxPartitionCounterStateConsistencyVolatileRebalanceTest.java:[40,28]
>> > > cannot find symbol
>> > > [13:37:02][Step 3/4] [ERROR]   symbol:   variable
>> > > IGNITE_BASELINE_AUTO_ADJUST_ENABLED
>> > > [13:37:02][Step 3/4] [ERROR]   location: class
>> > >
>> >
>> org.apache.ignite.internal.processors.cache.transactions.TxPartitionCounterStateConsistencyVolatileRebalanceTest
>> > > [13:37:02][Step 3/4] [ERROR]
>> > >
>> >
>> /opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/transactions/TxPartitionCounterStateConsistencyVolatileRebalanceTest.java:[47,30]
>> > > cannot find symbol
>> > > [13:37:02][Step 3/4] [ERROR]   symbol:   variable
>> > > IGNITE_BASELINE_AUTO_ADJUST_ENABLED
>> > > [13:37:02][Step 3/4] [ERROR]   location: class
>> > >
>> >
>> org.apache.ignite.internal.processors.cache.transactions.TxPartitionCounterStateConsistencyVolatileRebalanceTest
>> > >
>> > >
>> >
>>
>

Re: Master compilation error

[Anton Vinogradov]

Should we perform an additional compilation attempt on pull/xxx/merge at
each visa request?

On Mon, Jan 20, 2020 at 1:56 PM Pavel Tupitsyn  wrote:

> > Main question is "how this may happen in case fix got the Visa [2] ?".
> This can happen because of other changes in master.
> "Visa" truly works only when master is in the same state during the merge
> as it was during TC run.
>
> On Mon, Jan 20, 2020 at 1:50 PM Anton Vinogradov  wrote:
>
> > It seems, this because of IGNITE-12227 fix [1].
> > Main question is "how this may happen in case fix got the Visa [2] ?".
> >
> > [1]
> >
> >
> https://ci.ignite.apache.org/viewModification.html?modId=895719=false=IgniteTests24Java8_BuildApacheIgnite=vcsModificationFiles
> > [2]
> >
> >
> https://issues.apache.org/jira/browse/IGNITE-12227?focusedCommentId=17018135=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17018135
> >
> > On Mon, Jan 20, 2020 at 1:43 PM Николай Ижиков 
> > wrote:
> >
> > > Hello. Igniters.
> > >
> > > Master build fails:
> > >
> > >
> > >
> >
> https://ci.ignite.apache.org/viewLog.html?buildId=4944107=IgniteTests24Java8_BuildApacheIgnite=buildLog_IgniteTests24Java8=pull%2F7269%2Fhead
> > >
> > > [13:37:02][Step 3/4] [ERROR] Failed to execute goal
> > > org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
> > > (default-testCompile) on project ignite-core: Compilation failure:
> > > Compilation failure:
> > > [13:37:02][Step 3/4] [ERROR]
> > >
> >
> /opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/transactions/TxPartitionCounterStateConsistencyVolatileRebalanceTest.java:[25,1]
> > > cannot find symbol
> > > [13:37:02][Step 3/4] [ERROR]   symbol:   static
> > > IGNITE_BASELINE_AUTO_ADJUST_ENABLED
> > > [13:37:02][Step 3/4] [ERROR]   location: class
> > > [13:37:02][Step 3/4] [ERROR]
> > >
> >
> /opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/transactions/TxPartitionCounterStateAbstractTest.java:[162,28]
> > > cannot find symbol
> > > [13:37:02][Step 3/4] [ERROR]   symbol:   variable
> > > IGNITE_BASELINE_AUTO_ADJUST_ENABLED
> > > [13:37:02][Step 3/4] [ERROR]   location: class
> > >
> >
> org.apache.ignite.internal.processors.cache.transactions.TxPartitionCounterStateAbstractTest
> > > [13:37:02][Step 3/4] [ERROR]
> > >
> >
> /opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/transactions/TxPartitionCounterStateConsistencyVolatileRebalanceTest.java:[40,28]
> > > cannot find symbol
> > > [13:37:02][Step 3/4] [ERROR]   symbol:   variable
> > > IGNITE_BASELINE_AUTO_ADJUST_ENABLED
> > > [13:37:02][Step 3/4] [ERROR]   location: class
> > >
> >
> org.apache.ignite.internal.processors.cache.transactions.TxPartitionCounterStateConsistencyVolatileRebalanceTest
> > > [13:37:02][Step 3/4] [ERROR]
> > >
> >
> /opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/transactions/TxPartitionCounterStateConsistencyVolatileRebalanceTest.java:[47,30]
> > > cannot find symbol
> > > [13:37:02][Step 3/4] [ERROR]   symbol:   variable
> > > IGNITE_BASELINE_AUTO_ADJUST_ENABLED
> > > [13:37:02][Step 3/4] [ERROR]   location: class
> > >
> >
> org.apache.ignite.internal.processors.cache.transactions.TxPartitionCounterStateConsistencyVolatileRebalanceTest
> > >
> > >
> >
>

Re: Internal classes are exposed in public API

[Anton Vinogradov]

+1 to  @IgniteExperimental

On Mon, Jan 20, 2020 at 12:09 PM Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> After giving it some thought, I agree with Denis - there is nothing wrong
> with exposing the new APIs, we just need to make it clear that we are still
> going to change it.
>
> Should we Introduce something like @IgniteExperimental annotation (I
> believe it has been discussed a dozen of times on the dev-list)?
>
> пт, 17 янв. 2020 г. в 23:28, Nikolay Izhikov :
>
> > +1 to mark feature or whole release as EA.
> >
> > пт, 17 янв. 2020 г., 23:00 Denis Magda :
> >
> > > Folks, if you don't mind I'll share some thoughts/suggestions as an
> > > observer who was not involved in the feature development.
> > >
> > > It's absolutely 'ok' to deprecate an API that is replaced with a much
> > > better version. However, we should do this only when the new APIs are
> > > production-ready. If there are still many limitations or open items
> then
> > > don't deprecate anything that exists and release the new APIs labeling
> as
> > > early access. What if release the improvements labeling as EA instead
> of
> > > hiding them completely?
> > >
> > > I would also encourage us to put aside emotions, don't blame or point
> > > fingers. This IEP is a great initiative and you both have already done
> a
> > > tremendous job by developing, architecting and reviewing changes. Just
> be
> > > respectful. Nobody is trying to block the feature from being released.
> > > Everyone would be glad to tap into improvements and start using them in
> > > prod. But if more time is needed for the GA let's reiterate a bit.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Fri, Jan 17, 2020 at 12:24 PM Николай Ижиков 
> > > wrote:
> > >
> > > > > Also I agree with Alexey about introducing public IgniteMonitoring
> > > facade
> > > >
> > > > This is not an issue with the API.
> > > > Just the new feature that doesn’t affects API.
> > > > Moreover, I create a ticket to fix it, already.
> > > >
> > > > > It's right. But if you add checking of statisticsEnabling property
> > then
> > > > test will fail. It's just not good tests.
> > > >
> > > > My changes doesn’t affect any `staticticsEnabled` property.
> > > > I don’ understand your point here.
> > > >
> > > > > Redundant ReadOnlyMetricRegistry.
> > > >
> > > > It’s not redundant.
> > > > It required for exporters and provide read only access to
> > MetricRegistry
> > > > existing in the node.
> > > >
> > > >
> > > > > MetricExporterSpi that requires ReadOnlyMetricRegistry.
> > > >
> > > > > Absence of newly created metrics in old MBeans that forces user use
> > > > > exporter SPI while his code base uses old MBeans.
> > > >
> > > > Why this is issue?
> > > >
> > > > > Inconsistent MetricRegistry API.
> > > >
> > > > It’s consistent.
> > > >
> > > > > Metrics lookups from map instead of using direct reference
> > > > > (performance problem).
> > > >
> > > > 1. We(You and I) did this choice together to simplify creation of the
> > new
> > > > metrics.
> > > > 2. This is not public API issue.
> > > >
> > > >
> > > > > Ignoring of statisticsEnabled flag.
> > > >
> > > > I don’t ignore this flag.
> > > > It just doesn’t exists in new framework(because of scope).
> > > > I don’t think it’s an issue.
> > > >
> > > > > JmxExporterSpi creates beans that doesn't satisfy best MBeans
> > > practices.
> > > >
> > > > Please, clarify your statement.
> > > > What is best MBeans practices you are talking about?
> > > >
> > > > > Not finished IGNITE-11927
> > > >
> > > >
> > > > How this can be API issue?
> > > >
> > > >
> > > > > 17 янв. 2020 г., в 20:52, Andrey Gura 
> написал(а):
> > > > >
> > > > >>> All issues Alexey mentioned in starting letter are fixed with my
> PR
> > > > [1].
> > > > >>> I don’t think other issues you mentioned are blocker for the
> > release.
> > > > >
> > > > > As I mentioned already IGNITE-11927 is part of IEP-35 and
> > > > > implementation seriously affects API's. Also I agree with Alexey
> > about
> > > > > introducing public IgniteMonitoring facade. So thiss PR doesn't fix
> > > > > all issues.
> > > > >
> > > > >>> I talk about ignored existing functionality.
> > > > >> There is no existing tests that was broken by this contribution.
> > > > >
> > > > > It's right. But if you add checking of statisticsEnabling property
> > > > > then test will fail. It's just not good tests.
> > > > >
> > > > >> If you know the issues with it, feel free to create a ticket I
> will
> > > fix
> > > > it ASAP.
> > > > >
> > > > > I already fix it all in IGNITE-11927
> > > > >
> > > > >>> 1. Moving IEP-35 API's to the internal package.
> > > > >> We should move the product forward and shouldn't hide major
> > > > contribution from the user just because your second guess «I don’t
> like
> > > the
> > > > API I just reviewed and approved».
> > > > >
> > > > > We should move the product forward with with really finished
> > features,
> > > > > not pieces of features.
> > > > >
> > > > >> 

Re: Master compilation error

[Anton Vinogradov]

It seems, this because of IGNITE-12227 fix [1].
Main question is "how this may happen in case fix got the Visa [2] ?".

[1]
https://ci.ignite.apache.org/viewModification.html?modId=895719=false=IgniteTests24Java8_BuildApacheIgnite=vcsModificationFiles
[2]
https://issues.apache.org/jira/browse/IGNITE-12227?focusedCommentId=17018135=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17018135

On Mon, Jan 20, 2020 at 1:43 PM Николай Ижиков  wrote:

> Hello. Igniters.
>
> Master build fails:
>
>
> https://ci.ignite.apache.org/viewLog.html?buildId=4944107=IgniteTests24Java8_BuildApacheIgnite=buildLog_IgniteTests24Java8=pull%2F7269%2Fhead
>
> [13:37:02][Step 3/4] [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
> (default-testCompile) on project ignite-core: Compilation failure:
> Compilation failure:
> [13:37:02][Step 3/4] [ERROR]
> /opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/transactions/TxPartitionCounterStateConsistencyVolatileRebalanceTest.java:[25,1]
> cannot find symbol
> [13:37:02][Step 3/4] [ERROR]   symbol:   static
> IGNITE_BASELINE_AUTO_ADJUST_ENABLED
> [13:37:02][Step 3/4] [ERROR]   location: class
> [13:37:02][Step 3/4] [ERROR]
> /opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/transactions/TxPartitionCounterStateAbstractTest.java:[162,28]
> cannot find symbol
> [13:37:02][Step 3/4] [ERROR]   symbol:   variable
> IGNITE_BASELINE_AUTO_ADJUST_ENABLED
> [13:37:02][Step 3/4] [ERROR]   location: class
> org.apache.ignite.internal.processors.cache.transactions.TxPartitionCounterStateAbstractTest
> [13:37:02][Step 3/4] [ERROR]
> /opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/transactions/TxPartitionCounterStateConsistencyVolatileRebalanceTest.java:[40,28]
> cannot find symbol
> [13:37:02][Step 3/4] [ERROR]   symbol:   variable
> IGNITE_BASELINE_AUTO_ADJUST_ENABLED
> [13:37:02][Step 3/4] [ERROR]   location: class
> org.apache.ignite.internal.processors.cache.transactions.TxPartitionCounterStateConsistencyVolatileRebalanceTest
> [13:37:02][Step 3/4] [ERROR]
> /opt/buildagent/work/7bc1c54bc719b67c/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/transactions/TxPartitionCounterStateConsistencyVolatileRebalanceTest.java:[47,30]
> cannot find symbol
> [13:37:02][Step 3/4] [ERROR]   symbol:   variable
> IGNITE_BASELINE_AUTO_ADJUST_ENABLED
> [13:37:02][Step 3/4] [ERROR]   location: class
> org.apache.ignite.internal.processors.cache.transactions.TxPartitionCounterStateConsistencyVolatileRebalanceTest
>
>

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Anton Vinogradov]

Good Idea, this will also check that the release process is alive.

On Wed, Jan 22, 2020 at 12:04 PM Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> Folks, Maxim,
>
> Do you mind if I build the current state of ignite-2.8 branch and upload a
> maven staging as rc0 (step 4.3.2 of the release process)? I want run some
> tests for the fixes that are already included to the branch.
>
> вт, 21 янв. 2020 г. в 14:28, Maxim Muzafarov :
>
> > Folks,
> >
> >
> > I think both of these issues [1] [2] are critical to 2.8 release and
> > we must include them.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-12547
> > Excessive AtomicLong instantiations lead to GC pressure.
> >
> > [2] https://issues.apache.org/jira/browse/IGNITE-12530
> > Pages list caching can cause IgniteOOME when the checkpoint is
> > triggered by "too many dirty pages" reason.
> >
> >
> > On Mon, 20 Jan 2020 at 19:00, Alex Plehanov 
> > wrote:
> > >
> > > Guys,
> > >
> > > There is an issue [1] caused by page list caching [2], which also
> affects
> > > 2.8 release. IgniteOutOfMemoryException can be thrown in some cases
> (data
> > > region is small, a checkpoint is triggered by "too many dirty pages"
> > reason
> > > and pages list cache is rather big).
> > > The fix is ready and merged to master, I suggest to include this fix to
> > 2.8
> > > release. What do you think?
> > >
> > > [1]: https://issues.apache.org/jira/browse/IGNITE-12530
> > > [2]: https://issues.apache.org/jira/browse/IGNITE-6930
> > >
> > > пн, 20 янв. 2020 г. в 12:57, Alexey Goncharuk <
> > alexey.goncha...@gmail.com>:
> > >
> > > > Maxim,
> > > >
> > > > I took a quick look at IGNITE-12456 and I am not sure it's about data
> > > > corruption. In the attached logs blocked system threads are reported,
> > > > however, there is no enough information to investigate the issue (the
> > full
> > > > thread dump was not attached). I asked the ticket creator to attach
> > missing
> > > > pieces.
> > > >
> > > > Should we consider moving this ticket to a next release?
> > > >
> > > > пн, 20 янв. 2020 г. в 08:54, Zhenya Stanilovsky
> >  > > > >:
> > > >
> > > > >
> > > > > Maxim, performance fix issue [1] already in master, if no
> > objections, can
> > > > > u merge it into 2.8 ? Thanks !
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12547
> > > > >
> > > > > >Igniters,
> > > > > >
> > > > > >
> > > > > >Here is the actual list of BLOCKER release issues:
> > > > > >
> > > > > >IGNITE-12456 Cluster Data Store grid gets Corrupted for Load test
> > > > > >*[Unassigned]* OPEN
> > > > > >IGNITE-12489 Error during purges by expiration: Unknown page type*
> > > > > >[Unassigned]* OPEN
> > > > > >IGNITE-8641 SpringDataExample should use example-ignite.xml config
> > > > > >*[Unassigned]* OPEN
> > > > > >
> > > > > >IGNITE-12398 Apache Ignite Cluster(Amazon S3 Based Discovery)
> Nodes
> > > > > getting
> > > > > >down [Emmanouil Gkatziouras] OPEN
> > > > > >IGNITE-9184 Cluster hangs during concurrent node client and server
> > nodes
> > > > > >restart [Dmitriy Sorokin] IN PROGRESS
> > > > > >IGNITE-12553 [IEP-35] public Java metric API Improvement [Nikolay
> > > > Izhikov]
> > > > > >Blocker IN PROGRESS
> > > > > >
> > > > > >IGNITE-12227 Default auto-adjust baseline enabled flag calculated
> > > > > >incorrectly [Anton Kalashnikov] PATCH AVAILABLE
> > > > > >IGNITE-12470 Pme-free switch feature should be deactivatable
> [Sergei
> > > > > >Ryzhov] PATCH AVAILABLE
> > > > > >IGNITE-12552 [IEP-35] Expose MetricRegistry to the public API
> > > > Improvement
> > > > > >[Nikolay Izhikov] PATCH AVAILABLE
> > > > > >
> > > > > >
> > > > > >[1]  https://issues.apache.org/jira/browse/IGNITE-12456
> > > > > >[2]  https://issues.apache.org/jira/browse/IGNITE-12489
> > > > > >[3]  https://issues.apache.org/jira/browse/IGNITE-8641
> > > > > >[8]  https://issues.apache.org/jira/browse/IGNITE-12398
> > > > > >[3]  https://issues.apache.org/jira/browse/IGNITE-9184
> > > > > >[6]  https://issues.apache.org/jira/browse/IGNITE-12553
> > > > > >[7]  https://issues.apache.org/jira/browse/IGNITE-12227
> > > > > >[9]  https://issues.apache.org/jira/browse/IGNITE-12470
> > > > > >[5]  https://issues.apache.org/jira/browse/IGNITE-12552
> > > > > >
> > > > > >
> > > > > >On Sat, 18 Jan 2020 at 19:11, Sergey Antonov <
> > > > antonovserge...@gmail.com
> > > > > >
> > > > > >wrote:
> > > > > >
> > > > > >> Maxim,
> > > > > >>
> > > > > >> Conflicts in pr [1] are resolved. TC Run all is started.
> > > > > >>
> > > > > >> [1]  https://github.com/apache/ignite/pull/7238
> > > > > >>
> > > > > >> пт, 17 янв. 2020 г. в 16:04, Sergey Antonov <
> > > > antonovserge...@gmail.com
> > > > > >:
> > > > > >>
> > > > > >>> Maxim,
> > > > > >>>
> > > > > >>> I will do that on monday (20/01).
> > > > > >>>
> > > > > >>> пт, 17 янв. 2020 г. в 13:08, Maxim Muzafarov <
> mmu...@apache.org
> > >:
> > > > > >>>
> > > > >  Sergey,
> > > > > 
> > > > > 
> > > > >  Can you, please, resolve the PR 

Re: Partition reserve/release asymmetry

[Anton Vinogradov]

Everything is fine.
Merged to master branch.

On Fri, Jan 10, 2020 at 9:48 AM Anton Vinogradov  wrote:

> >> Does the issue reproduce in
> >> subsequent runs?
> Unfortunately no.
> We performed 30+ runs without "success".
>
> >> I think we can add an assertion to
> >> GridDhtLocalPartition#destroy() method to check that reservations is 0
> Ok, I will check and merge in case of success.
> Created the Issue to handle this [1].
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12524
>
> On Thu, Jan 9, 2020 at 1:46 PM Alexey Goncharuk <
> alexey.goncha...@gmail.com> wrote:
>
>> Hello Anton,
>>
>> Thanks for digging into this. The logic with checking the
>> reservations count seems fishy to me as well, so I have no objections with
>> the suggested change. This "if" statement does not answer why the
>> partition
>> was being destroyed during the commit, though. Does the issue reproduce in
>> subsequent runs?
>>
>> The logic around reserve/release seems ok to me, however, the
>> eviction/renting code looks overly complicated, perhaps, there is a bug
>> somewhere there? I think we can add an assertion to
>> GridDhtLocalPartition#destroy() method to check that reservations is 0
>> when
>> this method is called (there is a check for EVICTED state already there)
>>
>> --AG
>>
>> чт, 9 янв. 2020 г. в 09:45, Anton Vinogradov :
>>
>> > Folks,
>> > Yardstick run (opt-serial-put-get-1-backup) failed with interesting
>> > exception:
>> > Critical system error detected. Will be handled accordingly to
>> configured
>> > handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
>> > super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
>> > [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
>> > failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
>> > o.a.i.i.transactions.IgniteTxHeuristicCheckedException: Committing a
>> > transaction has produced runtime exception]]
>> > class
>> >
>> org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException:
>> > Committing a transaction has produced runtime exception
>> > at
>> >
>> >
>> org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.heuristicException(IgniteTxAdapter.java:800)
>> > at
>> >
>> >
>> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:838)
>> > at
>> >
>> >
>> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:893)
>> > at
>> >
>> >
>> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1452)
>> > at
>> >
>> >
>> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1375)
>> > at
>> >
>> >
>> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:123)
>> > at
>> >
>> >
>> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:241)
>> > at
>> >
>> >
>> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:239)
>> > at
>> >
>> >
>> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
>> > at
>> >
>> >
>> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
>> > at
>> >
>> >
>> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
>> > at
>> >
>> >
>> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
>> > at
>> >
>> >
>> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
>> > at
>> >
>> >
>> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
>> > at
>> >
>> >
>> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1843)
>> >

Re: Partition reserve/release asymmetry

[Anton Vinogradov]

>> Does the issue reproduce in
>> subsequent runs?
Unfortunately no.
We performed 30+ runs without "success".

>> I think we can add an assertion to
>> GridDhtLocalPartition#destroy() method to check that reservations is 0
Ok, I will check and merge in case of success.
Created the Issue to handle this [1].

[1] https://issues.apache.org/jira/browse/IGNITE-12524

On Thu, Jan 9, 2020 at 1:46 PM Alexey Goncharuk 
wrote:

> Hello Anton,
>
> Thanks for digging into this. The logic with checking the
> reservations count seems fishy to me as well, so I have no objections with
> the suggested change. This "if" statement does not answer why the partition
> was being destroyed during the commit, though. Does the issue reproduce in
> subsequent runs?
>
> The logic around reserve/release seems ok to me, however, the
> eviction/renting code looks overly complicated, perhaps, there is a bug
> somewhere there? I think we can add an assertion to
> GridDhtLocalPartition#destroy() method to check that reservations is 0 when
> this method is called (there is a check for EVICTED state already there)
>
> --AG
>
> чт, 9 янв. 2020 г. в 09:45, Anton Vinogradov :
>
> > Folks,
> > Yardstick run (opt-serial-put-get-1-backup) failed with interesting
> > exception:
> > Critical system error detected. Will be handled accordingly to configured
> > handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> > super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
> > [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
> > failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
> > o.a.i.i.transactions.IgniteTxHeuristicCheckedException: Committing a
> > transaction has produced runtime exception]]
> > class
> >
> org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException:
> > Committing a transaction has produced runtime exception
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.heuristicException(IgniteTxAdapter.java:800)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:838)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:893)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1452)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1375)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:123)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:241)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:239)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
> > at
> >
> >
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
> > at
> >
> >
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1843)
> > at
> >
> >
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1468)
> > at
> >
> >
> org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
> > at
> >
> >
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1365)
> > at
> >
> >
> org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:555)
> > at
> >
> org.apache.ignite.internal.util.wo

Re: PME speedup #2, TX recovery delay elimination.

[Anton Vinogradov]

Ivan,

The "magic numbers" are always the "magic numbers" :)
We must get rid of them to see problems covered by them.

>> Was there any
>> performance measurements for such multiple left nodes cases?
Since this fix made to speedup pme-free switch which prohibits the merges,
the answer is "no".

BTW, the fix was merged to master.

On Thu, Dec 26, 2019 at 2:21 PM Ivan Pavlukhin  wrote:

> Anton,
>
> Thank you for your efforts! And sorry for a late reply.
>
> I am a little bit familiar with tx recovery. I personally like the
> idea of removing such "magic" logic from the code. I think a proper
> way is either justify and sustain (tests, documentation) some behavior
> or get rid of it.
>
> Regarding a delay before tx recovery. My understanding was that it
> might be useful when multiple (client) nodes leaves almost at the same
> time (perhaps due to some network connectivity issues). With a delay
> recovering multiple failed nodes will be grouped into one recovery
> round (+PME). Correct me if my understanding is wrong. Was there any
> performance measurements for such multiple left nodes cases?
>
> вт, 24 дек. 2019 г. в 16:22, Anton Vinogradov :
> >
> > Rechecked TC two more times.
> > Going to merge to master in case no objections here.
> >
> > On Mon, Dec 23, 2019 at 1:44 PM Anton Vinogradov  wrote:
> >
> > > Igniters,
> > >
> > > One more PME optimization ready to be reviewed.
> > > I found a strange tx recovery delay caused by
> IGNITE_TX_SALVAGE_TIMEOUT.
> > > I've checked the code and tests and found no reason to delay recovery.
> > >
> > > So, the issue [1] is ready to be reviewed.
> > >
> > > Improvement checked with benchmark [2] and fix, obviously, 100 ms
> faster
> > > :)
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-12272
> > > [2]
> > >
> https://github.com/anton-vinogradov/ignite/commit/f8c27253b0ecfe7381418f505aafe942efe5a0a8
> > >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Re: PME speedup #2, TX recovery delay elimination.

[Anton Vinogradov]

Rechecked TC two more times.
Going to merge to master in case no objections here.

On Mon, Dec 23, 2019 at 1:44 PM Anton Vinogradov  wrote:

> Igniters,
>
> One more PME optimization ready to be reviewed.
> I found a strange tx recovery delay caused by IGNITE_TX_SALVAGE_TIMEOUT.
> I've checked the code and tests and found no reason to delay recovery.
>
> So, the issue [1] is ready to be reviewed.
>
> Improvement checked with benchmark [2] and fix, obviously, 100 ms faster
> :)
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12272
> [2]
> https://github.com/anton-vinogradov/ignite/commit/f8c27253b0ecfe7381418f505aafe942efe5a0a8
>

Re: Ignite 2.8 announcement plan

[Anton Vinogradov]

>> Does it mean that the PME is skipped only for cases when the native
>> persistence is used and a failed node was in the baseline topology?
The cluster should be fully rebalanced (late affinity assignment happened)
and the failed node should be from the baseline topology.
Otherwise, you'll gain regular (PME based) switch.

>> How about pure in-memory clusters and clusters with CacheStore?
Such configuration leads to partitions reassignment in case of node left.
This means we have to relocate primaries and backups once they will be
rebalanced.
This leads to subsequent LAA PME (after Left switch).
Initial PME is required to calculate and apply new affinity and perform
preparations for LAA.

On Tue, Mar 10, 2020 at 7:25 PM Denis Magda  wrote:

> Anton,
>
> Does it mean that the PME is skipped only for cases when the native
> persistence is used and a failed node was in the baseline topology? How
> about pure in-memory clusters and clusters with CacheStore?
>
> -
> Denis
>
>
> On Tue, Mar 10, 2020 at 12:40 AM Anton Vinogradov  wrote:
>
> > Denis,
> >
> > >> the blocking PME no longer happens if a node leaves the cluster
> > We should mention this should be the baseline node
> >
> > On Tue, Mar 10, 2020 at 9:40 AM Denis Magda  wrote:
> >
> > > Pavel, thank you. I referred to your article from a blog post to be
> > > published on blogs.apache.org. Please feel free to share feedback
> before
> > > it's published. You can use the comment feature of Google Docs:
> > >
> > >
> >
> https://docs.google.com/document/d/1ssTC1Jf_reqZFWgl4ayhaohiAJCpzdL4tPrHTxvfvAM/edit?usp=sharing
> > >
> > >
> > > @Alexey Zinoviev  , @Andrey Gura
> > >  , @Nikolay Izhikov , @Anton
> > > Vinogradov   could you glance at the paragraphs
> > mentioning
> > > the improvements you contributed to the release? I just want to be sure
> > my
> > > understanding is correct.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Mon, Mar 9, 2020 at 3:54 AM Pavel Tupitsyn 
> > > wrote:
> > >
> > >> Published: https://ptupitsyn.github.io/Whats-New-In-Ignite-Net-2.8/
> > >>
> > >> On Sat, Mar 7, 2020 at 1:49 AM Pavel Tupitsyn 
> > >> wrote:
> > >>
> > >> > Denis, ok, I'll publish on Monday afternoon then (UTC), weekend is
> not
> > >> the
> > >> > best time.
> > >> >
> > >> > Thanks,
> > >> > Pavel
> > >> >
> > >> > On Sat, Mar 7, 2020 at 1:25 AM Denis Magda 
> wrote:
> > >> >
> > >> >> Pavel,
> > >> >>
> > >> >> Thanks for clarifying the way partition-awareness handles topology
> > >> >> changes.
> > >> >>
> > >> >> Please publish your article as soon as you're ready. I plan to
> finish
> > >> mine
> > >> >> on Monday-Tuesday and will refer to yours.
> > >> >>
> > >> >> -
> > >> >> Denis
> > >> >>
> > >> >>
> > >> >> On Fri, Mar 6, 2020 at 1:36 AM Pavel Tupitsyn <
> ptupit...@apache.org>
> > >> >> wrote:
> > >> >>
> > >> >> > Denis, thanks for the feedback!
> > >> >> > When should I publish the post? Right after official release
> > >> >> announcement,
> > >> >> > or later?
> > >> >> >
> > >> >> > > I would use "thick client" as a term instead of the "classic
> > >> client"
> > >> >> > Good point, fixed
> > >> >> >
> > >> >> > > partition-awareness doesn't handle topology changes
> automatically
> > >> >> > (partition map won't be updated on the client-side)
> > >> >> > This is not true, we keep the partition map up to date by
> checking
> > >> >> > AffinityTopologyVersion [1]
> > >> >> >
> > >> >> > > Now I can run Ignite.NET easily on my Mac OS machine
> > >> >> > Just to clarify, you can do that since 2.4 release [2].
> > >> >> > In 2.8 we have refined build-related things (jar file handling),
> > and
> > >> >> made
> > >> >> > sure that recent .NET versions are supported.
> > >> >> >
> > >> >> > [1]
>

Re: Ignite 2.8 documentation

[Anton Vinogradov]

Artem,
I've updated the Read Repair page

On Thu, Mar 5, 2020 at 3:51 PM Artem Budnikov 
wrote:

> Anton,
>
> Could you please review the page about Read Rapair?
>
> https://apacheignite.readme.io/docs/read-repair
>
> -Artem
>
> On 05.03.2020 12:20, Artem Budnikov wrote:
> > OK, I'll recreate it.
> >
> > Nikolay, please let me know when you are finished with the Metrics and
> > system views documentation. I'm done with the list of docs we
> > identified in this thread and want to publish v. 2.8.
> >
> > -Artem
> >
> > On 05.03.2020 11:55, Maxim Muzafarov wrote:
> >> Artem,
> >>
> >> I've created that. It is not public and can be safely removed since
> >> it's a full copy of 2.7.6 (at that moment)
> >>
> >> On Thu, 5 Mar 2020 at 11:53, Artem Budnikov
> >>  wrote:
> >>> Guys,
> >>>
> >>> I see that there is already version 2.8 for Ignite docs on readme.io.
> >>> Who created it and when? I've changed some pages under 2.7.6 version
> >>> without knowing that there is a newer version.
> >>>
> >>> -Artem
> >>>
> >>> On 05.03.2020 11:45, Artem Budnikov wrote:
>  I'm confused. Igor, could you please double check?
> 
>  -Artem
> 
>  On 05.03.2020 04:15, Ivan Pavlukhin wrote:
> >> That's right, only C++ and .NET clients have partition awareness
> > Are your sure? Was not the feature implemented for java thin client
> > in [1]?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-11898
> >
> > Best regards,
> > Ivan Pavlukhin
> >
> > ср, 4 мар. 2020 г. в 18:18, Denis Magda :
> >> Maxim,
> >>
> >> Yes, it's preferable to have metrics pages fully completed even
> >> though the
> >> feature is an experimental state. We want to encourage to try it out
> >> and
> >> switch to the new APIs eventually. Without technical instructions
> >> available
> >> our users will have a hard time checking the new capabilities.
> >>
> >> Nikolay, thanks a lot!
> >>
> >> -
> >> Denis
> >>
> >>
> >> On Wed, Mar 4, 2020 at 8:52 AM Nikolay Izhikov  >
> >> wrote:
> >>
> >>> I think yes.
> >>>
> >>> I will prepare documentation shortly.
> >>>
>  4 марта 2020 г., в 17:50, Maxim Muzafarov 
> >>> написал(а):
>  Folks,
> 
> 
>  Do we need a fully complete public documentation for metrics
>  marked
>  with @ExperimentalFeature? The API can significantly be changed.
> 
>  On Wed, 4 Mar 2020 at 17:10, Artem Budnikov
>  
> >>> wrote:
> > The only feature that is left is "WAL page compression"
> >
> > The three other features are  limitations that were present in
> > Ignite
> > 2.7 and now they are removed. Since they were never mentioned
> > in the
> > docs, there is nothing to do.
> >
> > -Artem
> >
> > On 04.03.2020 17:02, Denis Magda wrote:
> >>> I'll work on the following items today and tomorrow:
> >>>
> >>>  - JDBC: Support for query cancellation
> >>>
> >>>
> >>>  - JDBC: Support for query timeout
> >>>
> >>>
> >>>  - suspend/resume for pessimistic transactions
> >>>
> >>>
> >>>  - WAL page compression
> >>>
> >>>
> >> Artem, are these the only tickets left apart from the metrics &
> >> monitoring? @Nikolay
> >> Izhikov  how soon will you be able to
> >> finish the
> >> metrics documentation pages?
> >>
> >> -
> >> Denis
> >>
> >>
> >> On Wed, Mar 4, 2020 at 2:55 AM Artem Budnikov
> >> 
> >> wrote:
> >>
> >>> Hi everyone,
> >>>
> >>> I have created the docs for the following items so far:
> >>>
> >>>  -Default Ignite work dir location
> >>>
> >>>
> >>>
> https://apacheignite.readme.io/docs/getting-started-28#section-setting-up-work-directory
> >>>
> >>>
> >>>  - Baseline auto-adjust feature
> >>>
> >>>
> >>>
> https://apacheignite.readme.io/docs/baseline-topology-28#section-baseline-topology-autoadjustment
> >>>
> >>>
> >>>  - Cluster (de)activation events documentation
> >>>
> >>>
> >>>
> https://apacheignite.readme.io/docs/baseline-topology-28#section-cluster-activationdeactivation-events
> >>>
> >>>
> >>>  - Remove SqlQuery documentation
> >>>  done
> >>>
> >>>  - Partition awareness for thin clients
> >>>
> >>>
> >>>
> https://apacheignite-net.readme.io/docs/thin-client-28#section-partition-awareness
> >>>
> >>>
> >>>
> https://apacheignite-cpp.readme.io/docs/thin-client-28#section-partition-awareness
> >>>
> 

Re: Ignite 2.8 announcement plan

[Anton Vinogradov]

Denis,

>> the blocking PME no longer happens if a node leaves the cluster
We should mention this should be the baseline node

On Tue, Mar 10, 2020 at 9:40 AM Denis Magda  wrote:

> Pavel, thank you. I referred to your article from a blog post to be
> published on blogs.apache.org. Please feel free to share feedback before
> it's published. You can use the comment feature of Google Docs:
>
> https://docs.google.com/document/d/1ssTC1Jf_reqZFWgl4ayhaohiAJCpzdL4tPrHTxvfvAM/edit?usp=sharing
>
>
> @Alexey Zinoviev  , @Andrey Gura
>  , @Nikolay Izhikov , @Anton
> Vinogradov   could you glance at the paragraphs mentioning
> the improvements you contributed to the release? I just want to be sure my
> understanding is correct.
>
> -
> Denis
>
>
> On Mon, Mar 9, 2020 at 3:54 AM Pavel Tupitsyn 
> wrote:
>
>> Published: https://ptupitsyn.github.io/Whats-New-In-Ignite-Net-2.8/
>>
>> On Sat, Mar 7, 2020 at 1:49 AM Pavel Tupitsyn 
>> wrote:
>>
>> > Denis, ok, I'll publish on Monday afternoon then (UTC), weekend is not
>> the
>> > best time.
>> >
>> > Thanks,
>> > Pavel
>> >
>> > On Sat, Mar 7, 2020 at 1:25 AM Denis Magda  wrote:
>> >
>> >> Pavel,
>> >>
>> >> Thanks for clarifying the way partition-awareness handles topology
>> >> changes.
>> >>
>> >> Please publish your article as soon as you're ready. I plan to finish
>> mine
>> >> on Monday-Tuesday and will refer to yours.
>> >>
>> >> -
>> >> Denis
>> >>
>> >>
>> >> On Fri, Mar 6, 2020 at 1:36 AM Pavel Tupitsyn 
>> >> wrote:
>> >>
>> >> > Denis, thanks for the feedback!
>> >> > When should I publish the post? Right after official release
>> >> announcement,
>> >> > or later?
>> >> >
>> >> > > I would use "thick client" as a term instead of the "classic
>> client"
>> >> > Good point, fixed
>> >> >
>> >> > > partition-awareness doesn't handle topology changes automatically
>> >> > (partition map won't be updated on the client-side)
>> >> > This is not true, we keep the partition map up to date by checking
>> >> > AffinityTopologyVersion [1]
>> >> >
>> >> > > Now I can run Ignite.NET easily on my Mac OS machine
>> >> > Just to clarify, you can do that since 2.4 release [2].
>> >> > In 2.8 we have refined build-related things (jar file handling), and
>> >> made
>> >> > sure that recent .NET versions are supported.
>> >> >
>> >> > [1]
>> >> >
>> >> >
>> >>
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
>> >> > [2] https://habr.com/ru/company/gridgain/blog/347374/ (in Russian)
>> >> >
>> >> > On Fri, Mar 6, 2020 at 2:40 AM Denis Magda 
>> wrote:
>> >> >
>> >> > > Pavel, thanks,
>> >> > >
>> >> > > I enjoyed reading the blog, crystal clear and straight to the
>> point!
>> >> > Please
>> >> > > consider these several items that might strengthen the article a
>> bit:
>> >> > >
>> >> > >- I would use "thick client" as a term instead of the "classic
>> >> client"
>> >> > >(and mention that Ignite.NET client is a thick one). The thick
>> >> client
>> >> > is
>> >> > >already a coined term that based on my observations are used a
>> lot
>> >> by
>> >> > > dev
>> >> > >and user communities. Also, you might add some differences of
>> thick
>> >> > vs.
>> >> > >thin taking from this page -
>> >> > >
>> >> > >
>> >> >
>> >>
>> https://www.gridgain.com/docs/latest/installation-guide/deployment-modes#thick-vs-thin-clients
>> >> > >- Should we mention that presently partition-awareness doesn't
>> >> handle
>> >> > >topology changes automatically (partition map won't be updated
>> on
>> >> the
>> >> > >client-side)? This might be a blocker for some users.
>> >> > >- Excited to read about the cross-platform support, th

Active nodes aliveness WatchDog

[Anton Vinogradov]

Igniters,
Do we have some feature allows to check nodes aliveness on a regular basis?

Scenario:
Precondition
  The cluster has no load but some node's JVM crashed.

Expected actual
  The user performs an operation (eg. cache put) related to this node (via
another node) and waits for some timeout to gain it's dead.
  The cluster starts the switch to relocate primary partitions to alive
nodes.
  Now user able to retry the operation.

Desired
  Some WatchDog checks nodes aliveness on a regular basis.
  Once a failure detected, the cluster starts the switch.
  Later, the user performs an operation on an already fixed cluster and
waits for nothing.

It would be good news if the "Desired" case is already Actual.
Can somebody point to the feature that performs this check?

Re: Active nodes aliveness WatchDog

[Anton Vinogradov]

Stephen,

> Nodes check on their neighbours and notify the remaining nodes if one
disappears.
Could you explain how this works in detail?
How can I set/change check frequency?

On Wed, Apr 8, 2020 at 11:13 AM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> This is one of the functions of the DiscoverySPI. Nodes check on their
> neighbours and notify the remaining nodes if one disappears. When the
> topology changes, it triggers a rebalance, which relocates primary
> partitions to live nodes. This is entirely transparent to clients.
>
> It gets more complex… like there’s the partition loss policy and
> rebalancing doesn’t always happen (configurable, persistence, etc)… but
> broadly it does as you expect.
>
> Regards,
> Stephen
>
> > On 8 Apr 2020, at 08:40, Anton Vinogradov  wrote:
> >
> > Igniters,
> > Do we have some feature allows to check nodes aliveness on a regular
> basis?
> >
> > Scenario:
> > Precondition
> >  The cluster has no load but some node's JVM crashed.
> >
> > Expected actual
> >  The user performs an operation (eg. cache put) related to this node (via
> > another node) and waits for some timeout to gain it's dead.
> >  The cluster starts the switch to relocate primary partitions to alive
> > nodes.
> >  Now user able to retry the operation.
> >
> > Desired
> >  Some WatchDog checks nodes aliveness on a regular basis.
> >  Once a failure detected, the cluster starts the switch.
> >  Later, the user performs an operation on an already fixed cluster and
> > waits for nothing.
> >
> > It would be good news if the "Desired" case is already Actual.
> > Can somebody point to the feature that performs this check?
>
>
>

Re: Active nodes aliveness WatchDog

[Anton Vinogradov]

It seems you're talking about Failure Detection (Timeouts).
Will it detect node failure on still cluster?

On Wed, Apr 8, 2020 at 11:52 AM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> The configuration parameters that I’m aware of are here:
>
>
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/spi/discovery/tcp/TcpDiscoverySpi.html
>
> Other people would be better placed to discuss the internals.
>
> Regards.
> Stephen
>
> > On 8 Apr 2020, at 09:32, Anton Vinogradov  wrote:
> >
> > Stephen,
> >
> >> Nodes check on their neighbours and notify the remaining nodes if one
> > disappears.
> > Could you explain how this works in detail?
> > How can I set/change check frequency?
> >
> > On Wed, Apr 8, 2020 at 11:13 AM Stephen Darlington <
> > stephen.darling...@gridgain.com> wrote:
> >
> >> This is one of the functions of the DiscoverySPI. Nodes check on their
> >> neighbours and notify the remaining nodes if one disappears. When the
> >> topology changes, it triggers a rebalance, which relocates primary
> >> partitions to live nodes. This is entirely transparent to clients.
> >>
> >> It gets more complex… like there’s the partition loss policy and
> >> rebalancing doesn’t always happen (configurable, persistence, etc)… but
> >> broadly it does as you expect.
> >>
> >> Regards,
> >> Stephen
> >>
> >>> On 8 Apr 2020, at 08:40, Anton Vinogradov  wrote:
> >>>
> >>> Igniters,
> >>> Do we have some feature allows to check nodes aliveness on a regular
> >> basis?
> >>>
> >>> Scenario:
> >>> Precondition
> >>> The cluster has no load but some node's JVM crashed.
> >>>
> >>> Expected actual
> >>> The user performs an operation (eg. cache put) related to this node
> (via
> >>> another node) and waits for some timeout to gain it's dead.
> >>> The cluster starts the switch to relocate primary partitions to alive
> >>> nodes.
> >>> Now user able to retry the operation.
> >>>
> >>> Desired
> >>> Some WatchDog checks nodes aliveness on a regular basis.
> >>> Once a failure detected, the cluster starts the switch.
> >>> Later, the user performs an operation on an already fixed cluster and
> >>> waits for nothing.
> >>>
> >>> It would be good news if the "Desired" case is already Actual.
> >>> Can somebody point to the feature that performs this check?
> >>
> >>
> >>
>
>
>

Re: Active nodes aliveness WatchDog

[Anton Vinogradov]

Stephen,
Thanks for the hint.

Vladimir,
Great idea! Let me know if any help needed.

On Wed, Apr 8, 2020 at 2:19 PM Vladimir Steshin  wrote:

> Hi everyone.
>
> I think we should check behavior of failure detection with tests or find
> them if already written. I’ll research this question and rise a ticket
> if a reproducer appears.
>
>
>
> 08.04.2020 12:19, Stephen Darlington пишет:
> > Yes. Nodes are always chatting to each another even if there are no
> requests coming In.
> >
> > Here’s the status message:
> https://github.com/apache/ignite/blob/e9b3c4cebaecbeec9fa51bd6ec32a879fb89948a/modules/core/src/main/java/org/apache/ignite/spi/discovery/tcp/messages/TcpDiscoveryStatusCheckMessage.java
> >
> > Regards,
> > Stephen
> >
> >> On 8 Apr 2020, at 10:04, Anton Vinogradov  wrote:
> >>
> >> It seems you're talking about Failure Detection (Timeouts).
> >> Will it detect node failure on still cluster?
> >>
> >> On Wed, Apr 8, 2020 at 11:52 AM Stephen Darlington <
> >> stephen.darling...@gridgain.com> wrote:
> >>
> >>> The configuration parameters that I’m aware of are here:
> >>>
> >>>
> >>>
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/spi/discovery/tcp/TcpDiscoverySpi.html
> >>>
> >>> Other people would be better placed to discuss the internals.
> >>>
> >>> Regards.
> >>> Stephen
> >>>
> >>>> On 8 Apr 2020, at 09:32, Anton Vinogradov  wrote:
> >>>>
> >>>> Stephen,
> >>>>
> >>>>> Nodes check on their neighbours and notify the remaining nodes if one
> >>>> disappears.
> >>>> Could you explain how this works in detail?
> >>>> How can I set/change check frequency?
> >>>>
> >>>> On Wed, Apr 8, 2020 at 11:13 AM Stephen Darlington <
> >>>> stephen.darling...@gridgain.com> wrote:
> >>>>
> >>>>> This is one of the functions of the DiscoverySPI. Nodes check on
> their
> >>>>> neighbours and notify the remaining nodes if one disappears. When the
> >>>>> topology changes, it triggers a rebalance, which relocates primary
> >>>>> partitions to live nodes. This is entirely transparent to clients.
> >>>>>
> >>>>> It gets more complex… like there’s the partition loss policy and
> >>>>> rebalancing doesn’t always happen (configurable, persistence, etc)…
> but
> >>>>> broadly it does as you expect.
> >>>>>
> >>>>> Regards,
> >>>>> Stephen
> >>>>>
> >>>>>> On 8 Apr 2020, at 08:40, Anton Vinogradov  wrote:
> >>>>>>
> >>>>>> Igniters,
> >>>>>> Do we have some feature allows to check nodes aliveness on a regular
> >>>>> basis?
> >>>>>> Scenario:
> >>>>>> Precondition
> >>>>>> The cluster has no load but some node's JVM crashed.
> >>>>>>
> >>>>>> Expected actual
> >>>>>> The user performs an operation (eg. cache put) related to this node
> >>> (via
> >>>>>> another node) and waits for some timeout to gain it's dead.
> >>>>>> The cluster starts the switch to relocate primary partitions to
> alive
> >>>>>> nodes.
> >>>>>> Now user able to retry the operation.
> >>>>>>
> >>>>>> Desired
> >>>>>> Some WatchDog checks nodes aliveness on a regular basis.
> >>>>>> Once a failure detected, the cluster starts the switch.
> >>>>>> Later, the user performs an operation on an already fixed cluster
> and
> >>>>>> waits for nothing.
> >>>>>>
> >>>>>> It would be good news if the "Desired" case is already Actual.
> >>>>>> Can somebody point to the feature that performs this check?
> >>>>>
> >>>>>
> >>>
> >>>
> >
>

Re: Inconsistent API IgniteClient and REST

[Anton Vinogradov]

Folks,

The question is not about "+1" or "-1".
The question is "why?".

This looks like some special feature to solve some special security issue
but may be just a bad/broken/obsolete/unrefactored code.
Changing this semantic we'll fix bad code or break some contract. 50% to
50%.

Let's keep REST case as is for now but start an investigation to gain
security consistent across all APIs, if possible.

On Tue, Mar 31, 2020 at 4:59 PM Andrey Kuznetsov  wrote:

> I'd prefer marking ADMIN_CACHE as deprecated, but postpone its removal from
> GridRestProcessor till next Ignte release (2.10 or 3.0?). For now we could
> just add checks for CACHE_CREATE / CACHE_DESTROY there along
> with ADMIN_CACHE.
>
> вт, 31 мар. 2020 г. в 12:30, Nikolay Izhikov :
>
> > Hello, Sergey.
> >
> >
> > I’m +1 to make this change.
> >
> > I think we should make security consistent across all APIs.
> >
> > > 31 марта 2020 г., в 12:14, Sergei Ryzhov 
> > написал(а):
> > >
> > > Hello!
> > > Now the work of permissions for API IgniteClient and REST is different.
> > > To create/delete a cache:
> > > IgniteClient authorises
> > CACHE_CREATE/CACHE_DESTROY.(GridCacheProcessor#authorizeCacheCreate <
> >
> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java#L3983
> >,
> > authorizeCacheDestroy <
> >
> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java#L3973
> > >)
> > > REST authorises ADMIN_CACHE.(GridRestProcessor#authorize <
> >
> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/rest/GridRestProcessor.java#L841
> > >)
> > > I think this is inconsistent.
> > >
> > > I suggest ADMIN_CACHE mark @Deprecated
> > > and replace it in the GridRestProcessor with CACHE_CREATE /
> > CACHE_DESTROY
> > > while maintaining backward compatibility for ADMIN_CACHE.
> > >
> > > This will allow us to remove ADMIN_CACHE in the future.
> > >
> > >
> > >
> > > Sergei Ryzhov
> > > s.vi.ryz...@gmail.com
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
>
> --
> Best regards,
>   Andrey Kuznetsov.
>

Re: Inconsistent API IgniteClient and REST

[Anton Vinogradov]

That's exactly the question I have.
Since no tests and documentation can be found - then we have no contract
and it's safe to change the semantic.
But we should recheck search results twice because we're talking about the
security.

On Wed, Apr 1, 2020 at 11:44 AM Nikolay Izhikov  wrote:

> Hello, Anton.
>
> What is «contract» for you?
> Do we have this contract written somewhere?
>
>
> > 1 апр. 2020 г., в 11:35, Anton Vinogradov  написал(а):
> >
> > Folks,
> >
> > The question is not about "+1" or "-1".
> > The question is "why?".
> >
> > This looks like some special feature to solve some special security issue
> > but may be just a bad/broken/obsolete/unrefactored code.
> > Changing this semantic we'll fix bad code or break some contract. 50% to
> > 50%.
> >
> > Let's keep REST case as is for now but start an investigation to gain
> > security consistent across all APIs, if possible.
> >
> > On Tue, Mar 31, 2020 at 4:59 PM Andrey Kuznetsov 
> wrote:
> >
> >> I'd prefer marking ADMIN_CACHE as deprecated, but postpone its removal
> from
> >> GridRestProcessor till next Ignte release (2.10 or 3.0?). For now we
> could
> >> just add checks for CACHE_CREATE / CACHE_DESTROY there along
> >> with ADMIN_CACHE.
> >>
> >> вт, 31 мар. 2020 г. в 12:30, Nikolay Izhikov :
> >>
> >>> Hello, Sergey.
> >>>
> >>>
> >>> I’m +1 to make this change.
> >>>
> >>> I think we should make security consistent across all APIs.
> >>>
> >>>> 31 марта 2020 г., в 12:14, Sergei Ryzhov 
> >>> написал(а):
> >>>>
> >>>> Hello!
> >>>> Now the work of permissions for API IgniteClient and REST is
> different.
> >>>> To create/delete a cache:
> >>>> IgniteClient authorises
> >>> CACHE_CREATE/CACHE_DESTROY.(GridCacheProcessor#authorizeCacheCreate <
> >>>
> >>
> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java#L3983
> >>> ,
> >>> authorizeCacheDestroy <
> >>>
> >>
> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java#L3973
> >>>> )
> >>>> REST authorises ADMIN_CACHE.(GridRestProcessor#authorize <
> >>>
> >>
> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/rest/GridRestProcessor.java#L841
> >>>> )
> >>>> I think this is inconsistent.
> >>>>
> >>>> I suggest ADMIN_CACHE mark @Deprecated
> >>>> and replace it in the GridRestProcessor with CACHE_CREATE /
> >>> CACHE_DESTROY
> >>>> while maintaining backward compatibility for ADMIN_CACHE.
> >>>>
> >>>> This will allow us to remove ADMIN_CACHE in the future.
> >>>>
> >>>>
> >>>>
> >>>> Sergei Ryzhov
> >>>> s.vi.ryz...@gmail.com
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >> --
> >> Best regards,
> >>  Andrey Kuznetsov.
> >>
>
>

Get rid of @Nullable and @NotNull

[Anton Vinogradov]

Folks,

Found we still use @Nullable annotation.

What's the reason for using it?
Everything is Object and Nullable :)

How about get rid of @Nullable usages and restrict its usage by checkstyle
plugin?

BTW, We already "do not use @NotNull annotation" (с) Coding Guidelines [1]
which may have some cense in contrast to @Nullable.
But I see a lot of usages at code.

How about to get rid of @NotNull too and to add such check to checkstyle
plugin too?

[1]
https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines#CodingGuidelines-@Annotations

Re: Get rid of @Nullable and @NotNull

[Anton Vinogradov]

Folks,

First of all, @NotNull already restricted.
And cool guys should use asserts and explicit checks instead of
checked-by-nobody annotations.
Some contributors use vim to write the code, how will it check correctness?
Or we should use only the IDEA?

Second,
>> I would consider every not annotated item as effectively @Nullable
that's an absolutely correct statement!

>> potential NPE places - where you forgot to check for null object marked
@Nullable
What's the difference between "@Nullable Object field" and just "Object
field"?
Correct, absolutely no difference.
Both can be null.

@Nullable annotation protects you from NPE even worse that Arbidol protects
you from flu.

On Fri, Mar 27, 2020 at 3:30 PM Dmitriy Pavlov  wrote:

> +1 for using @Nullable as it is a kind of documentation
> +1 for using @NotNull as it protects from
> wrong fields/variables assignments if modern IDE is used.
>
> пт, 27 мар. 2020 г. в 14:28, Sergey-A Kosarev :
>
> > Classification: Public
> >
> > +1 for using @Nullable.
> >
> > @Nullable is not panacea, but much better than nothing.
> >
> > It's clear indication that you should check the marked object for null
> > before using. There is advantage in using @Nullable than @NotNull - Idea
> or
> > any tool like findbugs shows you potential NPE places - where you forgot
> to
> > check for null  object marked @Nullable, otherwise if you use only
> @NotNull
> > it's much harder to see NPE problems as, I believe, always will be too
> many
> > noise - places where developers just forgot to place annotation.
> >
> > And another arguable point is statistics, in my experience there are more
> > objects that not null by nature than nullable. So if you should mark only
> > all @NotNull objects, you will need to place much more annotations than
> if
> > mark only @Nullable objects.
> >
> > Kind regards,
> > Sergey Kosarev
> >
> >
> > -Original Message-
> > From: Ivan Pavlukhin [mailto:vololo...@gmail.com]
> > Sent: 27 March 2020 13:28
> > To: dev 
> > Subject: Re: Get rid of @Nullable and @NotNull
> >
> > Here is my opinion. As we do not have a common practice to use @Nullable
> > and @NotNull annotations everywhere I would consider every not annotated
> > item as effectively @Nullable. @NotNull seems a useful annotation for me,
> > sometimes it is a really good thing to assume that something cannot be
> null
> > instead of doing explicit null check everywhere.
> >
> > Best regards,
> > Ivan Pavlukhin
> >
> > пт, 27 мар. 2020 г. в 13:05, Denis Garus :
> > >
> > > Hi!
> > > I'm not sure that @Nullable can really fix the NPE problem.
> > > Currently, we have @Nullable annotation and NPE simultaneously.
> > > The best way to avoid NPE is by using a null object pattern.
> > > I agree we shouldn't rely on @Nullable.
> > >
> > >
> > > пт, 27 мар. 2020 г. в 12:58, Sergey Antonov  >:
> > >
> > > > I disagree.
> > > >
> > > > Intellij idea IDE has a static code analysis, which uses that
> > > > annotation too. IDE highlights possible problems. It helps to make
> > > > our code more stable and bugless.
> > > >
> > > > пт, 27 мар. 2020 г. в 12:06, Pavel Tupitsyn :
> > > >
> > > > > I disagree, it would be a step back.
> > > > >
> > > > > > What's the reason for using it?
> > > > > Null was a billion dollar mistake [1].
> > > > > NullPointerExceptions happen quite a lot in Ignite, and
> > > > > annotations
> > > > provide
> > > > > some clues to avoid those.
> > > > >
> > > > > [1]
> > > > > https://en.wikipedia.org/wiki/Tony_Hoare#Apologies_and_retractions
> > > > >
> > > > > On Fri, Mar 27, 2020 at 10:58 AM Anton Vinogradov 
> > wrote:
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > > Found we still use @Nullable annotation.
> > > > > >
> > > > > > What's the reason for using it?
> > > > > > Everything is Object and Nullable :)
> > > > > >
> > > > > > How about get rid of @Nullable usages and restrict its usage by
> > > > > checkstyle
> > > > > > plugin?
> > > > > >
> > > > > > BTW, We already "do not use @NotNull annotation" (с) Coding
> > > > > > Guideline

Re: [DISCUSSION] `static final` code-style constant naming rules

[Anton Vinogradov]

>> +1 to force upper case for `static final` variables.
+1 too

On Mon, Apr 27, 2020 at 12:08 PM Nikolay Izhikov 
wrote:

> +1 to force upper case for `static final` variables.
>
> > 25 апр. 2020 г., в 07:39, Ivan Pavlukhin 
> написал(а):
> >
> > Maxim,
> >
> > Thank you for efforts in a code quality improvement!
> >
> > Unfortunately I do not agree with the proposal. Usually I like to
> > refer a following guide [1]. While a question "what is a constant" is
> > not trivial but I suppose that every static final field capitalization
> > can cause some problems. For example, a developer can make false
> > assumptions about immutability and thread safety basing on a
> > capitalized field name.
> >
> > [1]
> https://google.github.io/styleguide/javaguide.html#s5.2.4-constant-names
> >
> > Best regards,
> > Ivan Pavlukhin
> >
> > пт, 24 апр. 2020 г. в 21:58, Pavel Tupitsyn :
> >>
> >>> All static final object reference types that are never followed by "."
> >> (dot)
> >> With this way of thinking we can say that everything is a constant if we
> >> don't change it - no need for static final.
> >> "Constant" is usually something that you can't change, even you want -
> >> compiler won't let you.
> >> In Java that would be static final primitives and read-only objects like
> >> String.
> >>
> >> On Fri, Apr 24, 2020 at 8:25 PM Pavel Pereslegin 
> wrote:
> >>
> >>> Maxim,
> >>>
>  But what exactly is `constant` means?! I'd suggest applying this rule
>  to all class fields with `static final` modifiers without any clauses.
>  (check Java Naming convention [2] paragraph 3.3).
> >>>
> >>> I disagree with this and want to clarify what exactly the Java naming
> >>> convention says:
> >>>
> > The following are considered to be constants:
> > 1. All static final primitive types (Remember that all interface
> fields
> >>> are inherently static final).
> > 2. All static final object reference types that are never followed by
> >>> "." (dot).
> > 3. All static final arrays that are never followed by"[" (dot)
> >>>
> >>> I don't see that convention says "any static final field".
> >>>
> >>> пт, 24 апр. 2020 г. в 20:11, Sergey Antonov  >:
> 
>  Maxim, It's a good idea!
> 
>  Please don't forget to update out code style guidelines too.
> 
>  Thank you for keeping the code cleaner!
> 
>  пт, 24 апр. 2020 г. в 19:49, Maxim Muzafarov :
> 
> > Igniters,
> >
> >
> > It is not directly mentioned through the Apache Ignite Coding
> > Guidelines [1] about naming the `static final` class fields using
> only
> > upper-case letters. I'd like to suggest to fill this gap.
> >
> >> Constants should all be upper-case.
> > But what exactly is `constant` means?! I'd suggest applying this rule
> > to all class fields with `static final` modifiers without any
> clauses.
> > (check Java Naming convention [2] paragraph 3.3).
> >
> >
> > I've prepared PR [3] with capitalizing letters on all of the constant
> > names simultaneously with supporting the standard ConstantName
> > checkstyle [4] rule.
> >
> > Can we proceed with this change?
> > WDYT?
> >
> >
> > [1]
> >
> >>>
> https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines#CodingGuidelines-Naming
> > [2]
> >
> >>>
> https://web.archive.org/web/20120911192801/developers.sun.com/sunstudio/products/archive/whitepapers/java-style.pdf
> > [3] https://github.com/apache/ignite/pull/7662
> > [4] https://issues.apache.org/jira/browse/IGNITE-12888
> >
> 
> 
>  --
>  BR, Sergey Antonov
> >>>
>
>

Re: [DISCUSSION] Major changes in Ignite in 2020

[Anton Vinogradov]

Folks,

I keep working on crash recovery speed-up.

The main goal is to have put/get operations latency less than 500 ms on
node fail/left.
Currently, latency can be increased to seconds or even dozens of seconds.

The task is split to 2 threads

- Switch and tx recovery speed-up.
Speed-up can be gained by code refactoring, simplification, and tuning as
well as by special tricks like pme-free switch or cellular affinity usage.

- Failure detection.
Already found that some constants used at failure detection are hardcoded
and large.
Also, code responsible for this feature performs a lot of re-checks and
re-waits and you may have detection time close to failureDetectionTimeout
x2 or even x3.
Another problem is GC, and it may increase failure detection dramatically,
so, watchdog started from another JVM or from native code can help here.

On Tue, Apr 28, 2020 at 2:13 AM Denis Magda  wrote:

> Nikolay, thanks for responding,
>
> Would you prefer adding this item to the roadmap page after finishing the
> idea testing? If there are still many unknowns it can make to the roadmap
> after you share the results and propose a final solution.
>
> -
> Denis
>
>
> On Mon, Apr 27, 2020 at 2:06 PM Nikolay Izhikov 
> wrote:
>
> > Hello, Igniters.
> >
> > Right now I working on POC for some «framework» for integration testing
> of
> > Ignite.
> >
> > Feature I want to see in this framework:
> >
> > * To manage several Ignite instances inside docker or bare metal
> > servers.
> > * To manage Ignite cluster
> > * Start/stop arbitrary application to interact with Ignite.
> > * Kill nodes
> > * collect logs.
> > * implement complex real-life based scenarios
> > * etc.
> >
> > My POC are based on duck tape library [1]
> >
> > [1] https://github.com/confluentinc/ducktape
> >
> > > 25 апр. 2020 г., в 00:51, Denis Magda  написал(а):
> > >
> > > Dmitry, highlighted such a disclaimer in italic. Thanks.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Thu, Apr 23, 2020 at 3:53 AM Dmitriy Pavlov 
> > wrote:
> > >
> > >> Hi Denis,
> > >>
> > >> Thank you for driving this.
> > >>
> > >> Igniters,
> > >>
> > >> I would suggest to stress that Apache Ignite community does not
> > guarantee
> > >> these features to be available.
> > >>
> > >> Can we add some kind of disclaimer that says Ignite Roadmap does not
> > imply
> > >> any obligations regarding availability and timeline. A number of
> > >> contributions can be done on best efforts principle, it is always
> > tricky to
> > >> make a promise.
> > >>
> > >> Sincerely,
> > >> Dmitriy Pavlov
> > >>
> > >> чт, 23 апр. 2020 г. в 00:06, Denis Magda :
> > >>
> > >>> Igniters,
> > >>>
> > >>> Here is a draft of our very first roadmap. I decided to make it damp
> > >> simple
> > >>> but descriptive:
> > >>>
> > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Roadmap
> > >>>
> > >>> What we need to do next is to:
> > >>>
> > >>>   - Fill in the "Readiness Estimated Time" column with your best
> guess
> > >> of
> > >>>   when an improvement is to be ready for a release.
> > >>>   - Add references to JIRAs or IEPs to the first column.
> > >>>   - Add the names of those contributors who will be cooperating with
> > you
> > >>>   during development. Goes to the "Contributors" column.
> > >>>
> > >>> Once this step is complete, we'll see how many features converge
> around
> > >> the
> > >>> same date/months and we'll plan through 2.9, 2.10, etc. releases
> > >>> accordingly.
> > >>>
> > >>> Please don't put this aside for a while, let's move on quicker. If
> the
> > >>> roadmap misses any contributions that you are going to add, then edit
> > the
> > >>> wiki page.
> > >>>
> > >>> -
> > >>> Denis
> > >>>
> > >>>
> > >>> On Wed, Apr 15, 2020 at 3:35 AM Nikita Amelchev <
> nsamelc...@gmail.com>
> > >>> wrote:
> > >>>
> >  Hello, Igniters.
> > 
> >  I am going to contribute a new feature - profiling tool and
> >  performance report. This is part of IEP-35. [1]
> > 
> >  The tool will be able to collect performance statistics and create a
> >  human-readable report. It will help to analyze workload and to tune
> >  configuration and applications.
> > 
> >  Example of report [2, 3, 4].
> > 
> >  [1]
> > 
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> >  [2]
> > 
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647581/p1.png
> >  [3]
> > 
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647582/p2.png
> >  [4]
> > 
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool?preview=/145723859/148647583/p3.png
> > 
> >  сб, 11 апр. 2020 г. в 13:54, Alexei Scherbakov <
> >  

Re: [DISCUSSION] Major changes in Ignite in 2020

[Anton Vinogradov]

Denis,

IEP-45 [1] created.
Some tasks already released as a part of 2.8, some are in master now (2.9),
and some will be finished later (2.10).

[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up

On Wed, Apr 29, 2020 at 8:31 PM Denis Magda  wrote:

> Hi Anton,
>
> Thanks for sharing the plans with details. Could you please do us a favor
> and add this item to the roadmap page? I will also help if you link the
> item to a JIRA ticket or IEP that includes the details you shared here.
>
> -
> Denis
>
>
> On Tue, Apr 28, 2020 at 12:55 AM Anton Vinogradov  wrote:
>
>> Folks,
>>
>> I keep working on crash recovery speed-up.
>>
>> The main goal is to have put/get operations latency less than 500 ms on
>> node fail/left.
>> Currently, latency can be increased to seconds or even dozens of seconds.
>>
>> The task is split to 2 threads
>>
>> - Switch and tx recovery speed-up.
>> Speed-up can be gained by code refactoring, simplification, and tuning as
>> well as by special tricks like pme-free switch or cellular affinity usage.
>>
>> - Failure detection.
>> Already found that some constants used at failure detection are hardcoded
>> and large.
>> Also, code responsible for this feature performs a lot of re-checks and
>> re-waits and you may have detection time close to failureDetectionTimeout
>> x2 or even x3.
>> Another problem is GC, and it may increase failure detection dramatically,
>> so, watchdog started from another JVM or from native code can help here.
>>
>> On Tue, Apr 28, 2020 at 2:13 AM Denis Magda  wrote:
>>
>> > Nikolay, thanks for responding,
>> >
>> > Would you prefer adding this item to the roadmap page after finishing
>> the
>> > idea testing? If there are still many unknowns it can make to the
>> roadmap
>> > after you share the results and propose a final solution.
>> >
>> > -
>> > Denis
>> >
>> >
>> > On Mon, Apr 27, 2020 at 2:06 PM Nikolay Izhikov 
>> > wrote:
>> >
>> > > Hello, Igniters.
>> > >
>> > > Right now I working on POC for some «framework» for integration
>> testing
>> > of
>> > > Ignite.
>> > >
>> > > Feature I want to see in this framework:
>> > >
>> > > * To manage several Ignite instances inside docker or bare
>> metal
>> > > servers.
>> > > * To manage Ignite cluster
>> > > * Start/stop arbitrary application to interact with Ignite.
>> > > * Kill nodes
>> > > * collect logs.
>> > > * implement complex real-life based scenarios
>> > > * etc.
>> > >
>> > > My POC are based on duck tape library [1]
>> > >
>> > > [1] https://github.com/confluentinc/ducktape
>> > >
>> > > > 25 апр. 2020 г., в 00:51, Denis Magda 
>> написал(а):
>> > > >
>> > > > Dmitry, highlighted such a disclaimer in italic. Thanks.
>> > > >
>> > > > -
>> > > > Denis
>> > > >
>> > > >
>> > > > On Thu, Apr 23, 2020 at 3:53 AM Dmitriy Pavlov 
>> > > wrote:
>> > > >
>> > > >> Hi Denis,
>> > > >>
>> > > >> Thank you for driving this.
>> > > >>
>> > > >> Igniters,
>> > > >>
>> > > >> I would suggest to stress that Apache Ignite community does not
>> > > guarantee
>> > > >> these features to be available.
>> > > >>
>> > > >> Can we add some kind of disclaimer that says Ignite Roadmap does
>> not
>> > > imply
>> > > >> any obligations regarding availability and timeline. A number of
>> > > >> contributions can be done on best efforts principle, it is always
>> > > tricky to
>> > > >> make a promise.
>> > > >>
>> > > >> Sincerely,
>> > > >> Dmitriy Pavlov
>> > > >>
>> > > >> чт, 23 апр. 2020 г. в 00:06, Denis Magda :
>> > > >>
>> > > >>> Igniters,
>> > > >>>
>> > > >>> Here is a draft of our very first roadmap. I decided to make it
>> damp
>> > > >> simple
>> > > >>> but descriptive:
>> > > >>>

Re: [DISCUSS] Data loss handling improvements

[Anton Vinogradov]

Seems I got the vision, thanks.
There should be only 2 ways to reset lost partition: to gain an owner from
resurrected first or to remove ex-owner from baseline (partition will be
rearranged).
And we should make a decision for every lost partition before calling the
reset.

On Wed, May 6, 2020 at 8:02 PM Alexei Scherbakov <
alexey.scherbak...@gmail.com> wrote:

> ср, 6 мая 2020 г. в 12:54, Anton Vinogradov :
>
> > Alexei,
> >
> > 1,2,4,5 - looks good to me, no objections here.
> >
> > >> 3. Lost state is impossible to reset if a topology doesn't have at
> least
> > >> one owner for each lost partition.
> >
> > Do you mean that, according to your example, where
> > >> a node2 has left, soon a node3 has left. If the node2 is returned to
> > >> the topology first, it would have stale data for some keys.
> > we have to have node2 at cluster to be able to reset "lost" to node2's
> > data?
> >
>
> Not sure if I understand a question, but try to answer using an example:
> Assume 3 nodes n1, n2, n3, 1 backup, persistence enabled, partition p is
> owned by n2 and n3.
> 1. Topology is activated.
> 2. cache.put(p, 0) // n2 and n3 have p->0, updateCounter=1
> 3. n2 has failed.
> 4. cache.put(p, 1) // n3 has p->1, updateCounter=2
> 5. n3 has failed, partition loss is happened.
> 6. n2 joins a topology, it has stale data (p->0)
>
> We actually have 2 issues:
> 7. cache.put(p, 2) will success, n2 has p->2, n3 has p->0, data is diverged
> and will not be adjusted by counters rebalancing if n3 is later joins a
> topology.
> or
> 8. n3 joins a topology, it has actual data (p->1) but rebalancing will not
> work because joining node has highest counter (it can only be a demander in
> this scenario).
>
> In both cases rebalancing by counters will not work causing data divergence
> in copies.
>
>
> >
> > >> at least one owner for each lost partition.
> > What the reason to have owners for all lost partitions when we want to
> > reset only some (available)?
> >
>
> It's never were possible to reset only subset of lost partitions. The
> reason is to make guarantee of resetLostPartitions method - all cache
> operations are resumed, data is correct.
>
>
> > Will it be possible to perform operations on non-lost partitions when the
> > cluster has at least one lost partition?
> >
>
> Yes it will be.
>
>
> >
> > On Wed, May 6, 2020 at 11:45 AM Alexei Scherbakov <
> > alexey.scherbak...@gmail.com> wrote:
> >
> > > Folks,
> > >
> > > I've almost finished a patch bringing some improvements to the data
> loss
> > > handling code, and I wish to discuss proposed changes with the
> community
> > > before submitting.
> > >
> > > *The issue*
> > >
> > > During the grid's lifetime, it's possible to get into a situation when
> > some
> > > data nodes have failed or mistakenly stopped. If a number of stopped
> > nodes
> > > exceeds a certain threshold depending on configured backups, count a
> data
> > > loss will occur. For example, a grid having one backup (meaning at
> least
> > > two copies of each data partition exist at the same time) can tolerate
> > only
> > > one node loss at the time. Generally, data loss is guaranteed to occur
> if
> > > backups + 1 or more nodes have failed simultaneously using default
> > affinity
> > > function.
> > >
> > > For in-memory caches, data is lost forever. For persistent caches, data
> > is
> > > not physically lost and accessible again after failed nodes are
> returned
> > to
> > > the topology.
> > >
> > > Possible data loss should be taken into consideration while designing
> an
> > > application.
> > >
> > >
> > >
> > > *Consider an example: money is transferred from one deposit to another,
> > and
> > > all nodes holding data for one of the deposits are gone.In such a
> case, a
> > > transaction temporary cannot be completed until a cluster is recovered
> > from
> > > the data loss state. Ignoring this can cause data inconsistency.*
> > > It is necessary to have an API telling us if an operation is safe to
> > > complete from the perspective of data loss.
> > >
> > > Such an API exists for some time [1] [2] [3]. In short, a grid can be
> > > configured to switch caches to the partial availability mode if data
> loss
> > > is detected.
> > >
> > > Let's give two

Re: Crash recovery speed-up #3, Cellular Switch

[Anton Vinogradov]

Folks,

It seems, we have tacit agreement here.
Going to merge fix May 15.

On Tue, May 12, 2020 at 10:08 AM Anton Vinogradov  wrote:

> Denis,
>
> Rebalance is not expected here since this optimization works only on a
> fully rebalanced cluster with baseline.
>
> On Sat, May 9, 2020 at 12:48 AM Denis Magda  wrote:
>
>> Hi Anton,
>>
>> Generally, it means that Ignite will keep executing
>> operations/transactions
>> that are mapped into the partitions of those cells that won't be
>> rebalanced, is that correct?
>>
>> -
>> Denis
>>
>>
>> On Wed, May 6, 2020 at 3:24 AM Anton Vinogradov  wrote:
>>
>> > Igniters,
>> >
>> > PME-free switch [1] (since 2.8) skips PME on node left when possible
>> > (baseline + fully rebalanced cluster).
>> > This means we already wait for nothing (except recovery) to perform the
>> > switch.
>> > This optimization allows continuing already started operations during or
>> > after the switch if they are not affected by failed primary.
>> > But upcoming operations still can't be started until the switch is
>> finished
>> > cluster-wide.
>> >
>> > Let me propose an additional optimization - Cellular switch.
>> > Cellular Affinity [2] means that nodes combined into virtual cells
>> where,
>> > for each partition, backups located at the same cell with primaries.
>> > The simplest way to gain Cellular Affinity is to use backup filters [3].
>> >
>> > Cellular Affinity allows to finish the switch outside the affected cell
>> > instantly with the following assumptions:
>> > - Replicated caches should be recovered first since every node affected
>> (as
>> > a backup) by any failed primary.
>> >   But, it is expected that replicated caches effectively read-only (has
>> > extremely rare updates), so, nothing to wait here.
>> > - Upcoming replicated transactions (with non-failed primaries) can be
>> > started but can't be committed until switch finished cluster-wide.
>> > - Upcoming transactions related to the broken cell will wait for cell
>> > recovery (cluster-wide switch finish).
>> >
>> > ... and this means:
>> > In addition to PME-free switch, where we able to continue already
>> started
>> > operations during or after the switch, now we also able to perform most
>> of
>> > the upcoming operations during the switch.
>> >
>> > In other words, Cellular switch has little effect on the operation's
>> > latency, when operation not related to the failed cell.
>> >
>> > According to benchmark [4] which checks "how fast upcoming transactions
>> > (started after switch start) can be committed when we have thousands of
>> > prepared transactions (prepared before switch start)", we have 5326 ms
>> [5]
>> > operation's latency on master and 65 ms [6] with the proposed fix,
>> which is
>> > ~100 times faster.
>> >
>> > Fix [7] (as a part of IEP-45 [8]) ready to be reviewed.
>> > Waiting for your review!
>> >
>> >
>> > [1]
>> >
>> >
>> http://apache-ignite-developers.2346864.n4.nabble.com/Non-blocking-PME-Phase-One-Node-fail-tp43531p44586.html
>> > [2]
>> >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up#IEP-45:CrashRecoverySpeed-Up-Cellularswitch
>> > [3]
>> >
>> >
>> https://gist.github.com/anton-vinogradov/c50f9d0ce3e3e2997646f84ba7eba5f5#file-bench-java-L417
>> > [4]
>> >
>> https://gist.github.com/anton-vinogradov/c50f9d0ce3e3e2997646f84ba7eba5f5
>> > [5]
>> >
>> >
>> https://gist.github.com/anton-vinogradov/a35a3a8151b7494aa84b83f58cb75889#file-master-txt-L15
>> > [6]
>> >
>> >
>> https://gist.github.com/anton-vinogradov/a35a3a8151b7494aa84b83f58cb75889#file-fix-txt-L15
>> > [7] https://issues.apache.org/jira/browse/IGNITE-12617
>> > [8]
>> >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
>> >
>>
>

Re: Proposal: set default transaction timeout to 5 minutes

[Anton Vinogradov]

+1

On Mon, May 18, 2020 at 9:45 PM Sergey Antonov 
wrote:

> +1
>
> пн, 18 мая 2020 г. в 21:26, Andrey Mashenkov :
>
> > +1
> >
> > On Mon, May 18, 2020 at 9:19 PM Ivan Rakov 
> wrote:
> >
> > > Hi Igniters,
> > >
> > > I have a very simple proposal. Let's set default TX timeout to 5
> minutes
> > > (right now it's 0 = no timeout).
> > > Pros:
> > > 1. Deadlock detection procedure is triggered on timeout. In case user
> > will
> > > get into key-level deadlock, he'll be able to discover root cause from
> > the
> > > logs (even though load will hang for a while) and skip step with
> googling
> > > and debugging.
> > > 2. Almost every system with transactions has timeout enabled by
> default.
> > >
> > > WDYT?
> > >
> > > --
> > > Best Regards,
> > > Ivan Rakov
> > >
> >
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
> >
>
>
> --
> BR, Sergey Antonov
>

Re: [DISCUSS] Best way to re-encrypt existing data (TDE cache key rotation).

[Anton Vinogradov]

+1 to "In place re-encryption".

- It has a simple design.
- Clusters under load may require just load to re-encrypt the data.
(Friendly to load).
- Easy to throttle.
- Easy to continue.
- Design compatible with the multi-key architecture.
- It can be optimized to use own WAL buffer and to re-encrypt pages without
restoring them to on-heap.

On Thu, May 14, 2020 at 1:54 AM Pavel Pereslegin  wrote:

> Hello Igniters.
>
> Recently, master key rotation for Apache Ignite Transparent Data
> Encryption was implemented [1], but some security standards (PCI DSS
> at least) require rotation of all encryption keys [2]. Currently,
> encryption occurs when reading/writing pages to disk, cache encryption
> keys are stored in metastore.
>
> I'm going to contribute cache encryption key rotation and want to
> consult what is the best way to re-encrypting existing data, I see two
> different strategies.
>
> 1. In place re-encryption:
> Using the old key, sequentially read all the pages from the datastore,
> mark as dirty and log them into the WAL. After checkpoint pages will
> be stored to disk encrypted with the new key (as usual, along with
> updates). This strategy requires store the identifier (number) of the
> encryption key into the encrypted page.
> pros:
>   - can work in the background with minimal performance impact (this
> impact can be managed).
> cons:
>   - page duplication in the WAL may affect performance and historical
> rebalance.
>
> 2. Copy partition with re-encryption.
> This strategy is similar to partition snapshotting [3] - create
> partition copy encrypted with the new key and then replace the
> original partition file with the new one (see details [4]).
> pros:
>   - should work faster than "in place" re-encryption.
> cons:
>   - re-encryption in active cluster (and on unstable topology) can be
> difficult to implement.
>
> (See more detailed comparison [5])
>
> Re-encryption of existing data is a long and rare procedure (It is
> recommended to change the key every 6 months, but at least once every
> 2 years). Thus, re-encryption can be implemented for maintenance mode
> (for example, on a stable topology in a read-only cluster) and in such
> case the approach with partition copying seems simpler and faster.
>
> So, what do you think - do we need "online" re-encryption and which of
> the proposed options is best suited for this?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12186
> [2] https://www.pcisecuritystandards.org/documents/PCI_DSS_v3-2-1.pdf
> [3]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-43%3A+Cluster+snapshots#IEP-43:Clustersnapshots-Partitionscopystrategy
> [4]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652384#TDE.Phase-3.Cachekeyrotation.-Copywithre-encryptiondesign
> .
> [5]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652384#TDE.Phase-3.Cachekeyrotation.-Comparison
>

Re: Extensions for control.sh

[Anton Vinogradov]

Denis,

In addition to extending the features list it's also important to find some
way to allow customization of control.sh connection configuration/code.
For example, it may be useful to set some attributes to binary rest client.

On Thu, May 14, 2020 at 2:09 AM Denis Magda  wrote:

> Perfect idea to use this the tool for configuration and addition of
> extensions!
>
> -
> Denis
>
>
> On Wed, May 13, 2020 at 11:43 AM Denis Mekhanikov 
> wrote:
>
> > Hi everyone!
> >
> > Control.sh is a command-line management tool that you can use to manage
> > your grid and check its vital parameters like topology version or
> > availability of baseline nodes. It has is good set of commands which are
> > suitable to work with vanilla Ignite.
> >
> > There is also a way to extend functionality of Ignite by implementing a
> > 3rd-party plugin or a module. Any plugin or external module should have
> > some kind of API to manage and monitor its activity.
> > If a command-line management command needs to be added, then the only
> > way to achieve that is to provide an additional script, separate from
> > control.sh. If you use multiple such plugins, then the set of required
> > tools may grow and lead to confusion, which script should be used to
> > configure which extension. Instead of doing that it would be convenient
> > for users to have ability to use the same script, but with an extended
> > set of options. It should make lifes of 3rd-party vendors easier.
> >
> > Currently many integrations and community-supported modules are being
> > moved outside of the core product:
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization
> > I think it makes sense to provide a possibility to configure extensions
> > using control.sh, since their number will grow over time, and some of
> > them will require some runtime configuration.
> >
> > What do you think?
> >
> > Denis
> >
> >
>

Re: Extensions for control.sh

[Anton Vinogradov]

Denis,

> Do you mean, that external plugins should be able to configure the
> connection that is used to communicate with a cluster?
Yes

> Could you give an example ...
The same case I mentioned before.
We may need binary rest attributes to be set.
Some code should get them somewhere (eg. from file, system property, JMV
option, from usb-devce or keyboard) and set then via
clientCfg.setUserAttributes(userAttr); inside control.sh code.

On Thu, May 14, 2020 at 1:20 PM Denis Mekhanikov 
wrote:

> Anton,
>
> Do you mean, that external plugins should be able to configure the
> connection that is used to communicate with a cluster? Could you give an
> example, what kind of plugin would benefit from it?
>
> If there are some connection-specific properties that can change the way
> how control.sh communicates with a cluster, then it makes sense to
> donate such configuration to Ignite. Or am I missing something?
>
> Denis
>
> On 14.05.2020 11:43, Anton Vinogradov wrote:
> > Denis,
> >
> > In addition to extending the features list it's also important to find
> some
> > way to allow customization of control.sh connection configuration/code.
> > For example, it may be useful to set some attributes to binary rest
> client.
> >
> > On Thu, May 14, 2020 at 2:09 AM Denis Magda  wrote:
> >
> >> Perfect idea to use this the tool for configuration and addition of
> >> extensions!
> >>
> >> -
> >> Denis
> >>
> >>
> >> On Wed, May 13, 2020 at 11:43 AM Denis Mekhanikov <
> dmekhani...@gmail.com>
> >> wrote:
> >>
> >>> Hi everyone!
> >>>
> >>> Control.sh is a command-line management tool that you can use to manage
> >>> your grid and check its vital parameters like topology version or
> >>> availability of baseline nodes. It has is good set of commands which
> are
> >>> suitable to work with vanilla Ignite.
> >>>
> >>> There is also a way to extend functionality of Ignite by implementing a
> >>> 3rd-party plugin or a module. Any plugin or external module should have
> >>> some kind of API to manage and monitor its activity.
> >>> If a command-line management command needs to be added, then the only
> >>> way to achieve that is to provide an additional script, separate from
> >>> control.sh. If you use multiple such plugins, then the set of required
> >>> tools may grow and lead to confusion, which script should be used to
> >>> configure which extension. Instead of doing that it would be convenient
> >>> for users to have ability to use the same script, but with an extended
> >>> set of options. It should make lifes of 3rd-party vendors easier.
> >>>
> >>> Currently many integrations and community-supported modules are being
> >>> moved outside of the core product:
> >>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization
> >>> I think it makes sense to provide a possibility to configure extensions
> >>> using control.sh, since their number will grow over time, and some of
> >>> them will require some runtime configuration.
> >>>
> >>> What do you think?
> >>>
> >>> Denis
> >>>
> >>>
>

Re: [DISCUSS] Data loss handling improvements

[Anton Vinogradov]

Alexei,

1,2,4,5 - looks good to me, no objections here.

>> 3. Lost state is impossible to reset if a topology doesn't have at least
>> one owner for each lost partition.

Do you mean that, according to your example, where
>> a node2 has left, soon a node3 has left. If the node2 is returned to
>> the topology first, it would have stale data for some keys.
we have to have node2 at cluster to be able to reset "lost" to node2's data?

>> at least one owner for each lost partition.
What the reason to have owners for all lost partitions when we want to
reset only some (available)?
Will it be possible to perform operations on non-lost partitions when the
cluster has at least one lost partition?

On Wed, May 6, 2020 at 11:45 AM Alexei Scherbakov <
alexey.scherbak...@gmail.com> wrote:

> Folks,
>
> I've almost finished a patch bringing some improvements to the data loss
> handling code, and I wish to discuss proposed changes with the community
> before submitting.
>
> *The issue*
>
> During the grid's lifetime, it's possible to get into a situation when some
> data nodes have failed or mistakenly stopped. If a number of stopped nodes
> exceeds a certain threshold depending on configured backups, count a data
> loss will occur. For example, a grid having one backup (meaning at least
> two copies of each data partition exist at the same time) can tolerate only
> one node loss at the time. Generally, data loss is guaranteed to occur if
> backups + 1 or more nodes have failed simultaneously using default affinity
> function.
>
> For in-memory caches, data is lost forever. For persistent caches, data is
> not physically lost and accessible again after failed nodes are returned to
> the topology.
>
> Possible data loss should be taken into consideration while designing an
> application.
>
>
>
> *Consider an example: money is transferred from one deposit to another, and
> all nodes holding data for one of the deposits are gone.In such a case, a
> transaction temporary cannot be completed until a cluster is recovered from
> the data loss state. Ignoring this can cause data inconsistency.*
> It is necessary to have an API telling us if an operation is safe to
> complete from the perspective of data loss.
>
> Such an API exists for some time [1] [2] [3]. In short, a grid can be
> configured to switch caches to the partial availability mode if data loss
> is detected.
>
> Let's give two definitions according to the Javadoc for
> *PartitionLossPolicy*:
>
> ·   *Safe* (data loss handling) *policy* - cache operations are only
> available for non-lost partitions (PartitionLossPolicy != IGNORE).
>
> ·   *Unsafe policy* - cache operations are always possible
> (PartitionLossPolicy = IGNORE). If the unsafe policy is configured, lost
> partitions automatically re-created on the remaining nodes if needed or
> immediately owned if a last supplier has left during rebalancing.
>
> *That needs to be fixed*
>
> 1. The default loss policy is unsafe, even for persistent caches in the
> current implementation. It can result in unintentional data loss and
> business invariants' failure.
>
> 2. Node restarts in the persistent grid with detected data loss will cause
> automatic resetting of LOST state after the restart, even if the safe
> policy is configured. It can result in data loss or partition desync if not
> all nodes are returned to the topology or returned in the wrong order.
>
>
> *An example: a grid has three nodes, one backup. The grid is under load.
> First, a node2 has left, soon a node3 has left. If the node2 is returned to
> the topology first, it would have stale data for some keys. Most recent
> data are on node3, which is not in the topology yet. Because a lost state
> was reset, all caches are fully available, and most probably will become
> inconsistent even in safe mode.*
> 3. Configured loss policy doesn't provide guarantees described in the
> Javadoc depending on the cluster configuration[4]. In particular, unsafe
> policy (IGNORE) cannot be guaranteed if a baseline is fixed (not
> automatically readjusted on node left), because partitions are not
> automatically get reassigned on topology change, and no nodes are existing
> to fulfill a read/write request. Same for READ_ONLY_ALL and READ_WRITE_ALL.
>
> 4. Calling resetLostPartitions doesn't provide a guarantee for full cache
> operations availability if a topology doesn't have at least one owner for
> each lost partition.
>
> The ultimate goal of the patch is to fix API inconsistencies and fix the
> most crucial bugs related to data loss handling.
>
> *The planned changes are:*
>
> 1. The safe policy is used by default, except for in-memory grids with
> enabled baseline auto-adjust [5] with zero timeout [6]. In the latter case,
> the unsafe policy is used by default. It protects from unintentional data
> loss.
>
> 2. Lost state is never reset in the case of grid nodes restart (despite
> full restart). It makes real data loss impossible in persistent grids if
> 

Crash recovery speed-up #3, Cellular Switch

[Anton Vinogradov]

Igniters,

PME-free switch [1] (since 2.8) skips PME on node left when possible
(baseline + fully rebalanced cluster).
This means we already wait for nothing (except recovery) to perform the
switch.
This optimization allows continuing already started operations during or
after the switch if they are not affected by failed primary.
But upcoming operations still can't be started until the switch is finished
cluster-wide.

Let me propose an additional optimization - Cellular switch.
Cellular Affinity [2] means that nodes combined into virtual cells where,
for each partition, backups located at the same cell with primaries.
The simplest way to gain Cellular Affinity is to use backup filters [3].

Cellular Affinity allows to finish the switch outside the affected cell
instantly with the following assumptions:
- Replicated caches should be recovered first since every node affected (as
a backup) by any failed primary.
  But, it is expected that replicated caches effectively read-only (has
extremely rare updates), so, nothing to wait here.
- Upcoming replicated transactions (with non-failed primaries) can be
started but can't be committed until switch finished cluster-wide.
- Upcoming transactions related to the broken cell will wait for cell
recovery (cluster-wide switch finish).

... and this means:
In addition to PME-free switch, where we able to continue already started
operations during or after the switch, now we also able to perform most of
the upcoming operations during the switch.

In other words, Cellular switch has little effect on the operation's
latency, when operation not related to the failed cell.

According to benchmark [4] which checks "how fast upcoming transactions
(started after switch start) can be committed when we have thousands of
prepared transactions (prepared before switch start)", we have 5326 ms [5]
operation's latency on master and 65 ms [6] with the proposed fix, which is
~100 times faster.

Fix [7] (as a part of IEP-45 [8]) ready to be reviewed.
Waiting for your review!


[1]
http://apache-ignite-developers.2346864.n4.nabble.com/Non-blocking-PME-Phase-One-Node-fail-tp43531p44586.html
[2]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up#IEP-45:CrashRecoverySpeed-Up-Cellularswitch
[3]
https://gist.github.com/anton-vinogradov/c50f9d0ce3e3e2997646f84ba7eba5f5#file-bench-java-L417
[4]
https://gist.github.com/anton-vinogradov/c50f9d0ce3e3e2997646f84ba7eba5f5
[5]
https://gist.github.com/anton-vinogradov/a35a3a8151b7494aa84b83f58cb75889#file-master-txt-L15
[6]
https://gist.github.com/anton-vinogradov/a35a3a8151b7494aa84b83f58cb75889#file-fix-txt-L15
[7] https://issues.apache.org/jira/browse/IGNITE-12617
[8]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up

Re: Crash recovery speed-up #3, Cellular Switch

[Anton Vinogradov]

Denis,

Rebalance is not expected here since this optimization works only on a
fully rebalanced cluster with baseline.

On Sat, May 9, 2020 at 12:48 AM Denis Magda  wrote:

> Hi Anton,
>
> Generally, it means that Ignite will keep executing operations/transactions
> that are mapped into the partitions of those cells that won't be
> rebalanced, is that correct?
>
> -
> Denis
>
>
> On Wed, May 6, 2020 at 3:24 AM Anton Vinogradov  wrote:
>
> > Igniters,
> >
> > PME-free switch [1] (since 2.8) skips PME on node left when possible
> > (baseline + fully rebalanced cluster).
> > This means we already wait for nothing (except recovery) to perform the
> > switch.
> > This optimization allows continuing already started operations during or
> > after the switch if they are not affected by failed primary.
> > But upcoming operations still can't be started until the switch is
> finished
> > cluster-wide.
> >
> > Let me propose an additional optimization - Cellular switch.
> > Cellular Affinity [2] means that nodes combined into virtual cells where,
> > for each partition, backups located at the same cell with primaries.
> > The simplest way to gain Cellular Affinity is to use backup filters [3].
> >
> > Cellular Affinity allows to finish the switch outside the affected cell
> > instantly with the following assumptions:
> > - Replicated caches should be recovered first since every node affected
> (as
> > a backup) by any failed primary.
> >   But, it is expected that replicated caches effectively read-only (has
> > extremely rare updates), so, nothing to wait here.
> > - Upcoming replicated transactions (with non-failed primaries) can be
> > started but can't be committed until switch finished cluster-wide.
> > - Upcoming transactions related to the broken cell will wait for cell
> > recovery (cluster-wide switch finish).
> >
> > ... and this means:
> > In addition to PME-free switch, where we able to continue already started
> > operations during or after the switch, now we also able to perform most
> of
> > the upcoming operations during the switch.
> >
> > In other words, Cellular switch has little effect on the operation's
> > latency, when operation not related to the failed cell.
> >
> > According to benchmark [4] which checks "how fast upcoming transactions
> > (started after switch start) can be committed when we have thousands of
> > prepared transactions (prepared before switch start)", we have 5326 ms
> [5]
> > operation's latency on master and 65 ms [6] with the proposed fix, which
> is
> > ~100 times faster.
> >
> > Fix [7] (as a part of IEP-45 [8]) ready to be reviewed.
> > Waiting for your review!
> >
> >
> > [1]
> >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Non-blocking-PME-Phase-One-Node-fail-tp43531p44586.html
> > [2]
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up#IEP-45:CrashRecoverySpeed-Up-Cellularswitch
> > [3]
> >
> >
> https://gist.github.com/anton-vinogradov/c50f9d0ce3e3e2997646f84ba7eba5f5#file-bench-java-L417
> > [4]
> >
> https://gist.github.com/anton-vinogradov/c50f9d0ce3e3e2997646f84ba7eba5f5
> > [5]
> >
> >
> https://gist.github.com/anton-vinogradov/a35a3a8151b7494aa84b83f58cb75889#file-master-txt-L15
> > [6]
> >
> >
> https://gist.github.com/anton-vinogradov/a35a3a8151b7494aa84b83f58cb75889#file-fix-txt-L15
> > [7] https://issues.apache.org/jira/browse/IGNITE-12617
> > [8]
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
> >
>

Re: [DISCUSSION] Ignite integration testing framework.

[Anton Vinogradov]

Nikolay,

Great proposal!

Could you please explain how to perform testing on the different
environments?
For Example, you provided a rebalance test.

Is it possible to perform this test with TDE enabled?
Is it possible to perform it on custom OS (eg. Windows)?
Is it possible to perform in on bare metal?

I hope you'll answer yes to all, and the question mostly about details.

On Thu, May 21, 2020 at 10:27 AM Nikolay Izhikov 
wrote:

> Hello, Igniters.
>
> I created a PoC [1] for the integration tests of Ignite.
>
> Let me briefly explain the gap I want to cover:
>
> 1. For now, we don’t have a solution for automated testing of Ignite on
> «real cluster».
> By «real cluster» I mean cluster «like a production»:
> * client and server nodes deployed on different hosts.
> * thin clients perform queries from some other hosts
> * etc.
>
> 2. We don’t have a solution for automated benchmarks of some internal
> Ignite process
> * PME
> * rebalance.
> This means we don’t know - Do we perform rebalance(or PME) in 2.7.0 faster
> or slower than in 2.8.0 for the same cluster?
>
> 3. We don’t have a solution for automated testing of Ignite integration in
> a real-world environment:
> Ignite-Spark integration can be taken as an example.
> I think some ML solutions also should be tested in real-world deployments.
>
> Solution:
>
> I propose to use duck tape library from confluent (apache 2.0 license)
> I tested it both on the real cluster(Yandex Cloud) and on the local
> environment(docker) and it works just fine.
>
> PoC contains following services:
>
> * Simple rebalance test:
> Start 2 server nodes,
> Create some data with Ignite client,
> Start one more server node,
> Wait for rebalance finish
> * Simple Ignite-Spark integration test:
> Start 1 Spark master, start 1 Spark worker,
> Start 1 Ignite server node
> Create some data with Ignite client,
> Check data in application that queries it from Spark.
>
> All tests are fully automated.
> Logs collection works just fine.
> You can see an example of the tests report - [4].
>
> Pros:
>
> * Ability to test local changes(no need to public changes to some remote
> repository or similar).
> * Ability to parametrize test environment(run the same tests on different
> JDK, JVM params, config, etc.)
> * Isolation by default so system tests are as reliable as possible.
> * Utilities for pulling up and tearing down services easily in clusters in
> different environments (e.g. local, custom cluster, Vagrant, K8s, Mesos,
> Docker, cloud providers, etc.)
> * Easy to write unit tests for distributed systems
> * Adopted and successfully used by other distributed open source project -
> Apache Kafka.
> * Collect results (e.g. logs, console output)
> * Report results (e.g. expected conditions met, performance results, etc.)
>
> WDYT?
>
> [1] https://github.com/nizhikov/ignite/pull/15
> [2] https://github.com/confluentinc/ducktape
> [3] https://ducktape-docs.readthedocs.io/en/latest/run_tests.html
> [4] https://yadi.sk/d/JC8ciJZjrkdndg

Re: 2.9.1 release proposal

[Anton Vinogradov]

+1 to start the 2.9.1 release process once as soon as 2.9 released.

On Mon, Oct 19, 2020 at 8:49 PM Nikolay Izhikov  wrote:

> Hello, Yaroslav.
>
> I support the idea.
> But, we should carefully work with the release scope.
>
> IGNITE-13477 - fix for a SQL tracing that will be available only in 2.10
> IGNITE-13427 - fix for a new system view that exists in master only, we
> need to include IGNITE-13409
>
> Other tickets should be checked, also.
> Is this a real fix of a released bug or we fix some issue we bring with
> the brand new contribution.
>
>
> I propose to include the following tickets, also:
>
> CMD tools improvements:
>
> IGNITE-13488 - Command to print metric value
> IGNITE-13426 - Command to print system view content
> IGNITE-13422 - Parameter to explicitly enable experimental commands
>
> IGNITE-13380 - Output IgniteSystemProperties via ignite.sh
>
> New system views:
>
> IGNITE-13409 Metastorage and DistributedMetastorage viewы.
> IGNITE-13408 BinaryMetadata view.
>
>
> > 19 окт. 2020 г., в 18:20, Yaroslav Molochkov 
> написал(а):
> >
> > Hello, Igniters!
> >
> > I've compiled a list of tickets that, i think, deserve to be released in
> a
> > minor Ignite release (meaning in 2.9.1) after 2.9. Here they are:
> >
> > IGNITE-13569
> > disable archiving + walCompactionEnabled probably broke reading from wal
> on
> > server restart
> >
> > IGNITE-13418
> > Deadlock on multiple cache delete
> >
> > IGNITE-13563
> > Deserializing IBinaryObject containing an IBinaryObject field fails
> >
> > IGNITE-13575
> > Invalid blocking section in GridNioWorker and GridNioClientWorker leads
> to
> > false positive blocking thread detection
> >
> > IGNITE-13458
> > RebalancingPartitionsTotal metrics
> >
> > IGNITE-13536
> > .NET: Child processes become zombies when persistence is used with
> > direct-io on Linux
> >
> > IGNITE-13500
> > Checkpoint read lock fail if it is taking under write lock during the
> > stopping node
> >
> > IGNITE-13431
> > NPE during Cassandra Store initialization with PRIMITIVE strategy
> >
> > IGNITE-13417
> > Cache Interceptors deserialization on client nodes
> >
> > IGNITE-13495
> > ZookeeperDiscoverySpiMBeanImpl#getCoordinator can return invalid node as
> > coordinator
> >
> > IGNITE-13479
> > Both ignite.sh and control.sh use the same JVM_OPTS. Control.sh doesn't
> > start if JMX port was set
> >
> > IGNITE-11312
> > JDBC: Thin driver reports incorrect property names
> >
> > IGNITE-13462
> > .NET: Thin client Dispose hangs when continuous query is active on .NET
> > Core 3.x
> >
> > IGNITE-13484
> > C++ odbc-example losing some values if run with 1 additional node
> >
> > IGNITE-13477
> > Fix NPE in SQL tracing implementation.
> >
> > IGNITE-13435
> > Fixing some unrecorded issues command warm-up control.sh
> >
> > IGNITE-13403
> > Update JDBC metadata to match actual capabilities
> >
> > IGNITE-13427
> > The local metastorage system view fails if unmarshallable values present
> >
> > IGNITE-13401
> > Unsupported protocol version exception when getting cache configuration
> > from Java thin client
> >
> > IGNITE-13388
> > apache-ignite deb package depends on a non-existent package and can't be
> > installed on Debian 10
> >
> > IGNITE-13397
> > NPE in logSupplierDone(UUID nodeId)
> >
> > IGNITE-13296
> > .NET: TransactionImpl finalizer can crash the process
> >
> > IGNITE-13382
> > DurableBackgroundTask can abandon incomplete task
> >
> > IGNITE-12509
> > CACHE_REBALANCE_STOPPED event raises for wrong caches in case of
> specified
> > RebalanceDelay
> >
> > IGNITE-13072
> > Synchronization problems when different classloaders are used for
> > deployment of same class
> >
> > IGNITE-13379
> > Exception occur on SQL caches when client reconnect
> >
> > IGNITE-13363
> > GridDhtCacheEntry::toString locks
> >
> > IGNITE-13373
> > WAL segmentns do not released on releaseHistoryForPreloading()
> >
> > IGNITE-13540
> > Exchange worker, waiting for new task from queue, considered as blocked.
> >
> > IGNITE-13491
> > Fix incorrect topology snapshot logger output about coordinator change.
> >
> > IGNITE-13439
> > Printing detailed classpath slowdowns node initialization
> >
> > WDYT?
>
>

Re: [DISCUSSION] IEP-47 Native persistence defragmentation

[Anton Vinogradov]

Folks,

Modern OS never ask you to schedule defragmentation and turn your PC off,
it performs it while you're browsing.
Why we're going to implement distributed system defragmentation in the old
(offline) way?

All you need is to implement free/reuse-list sorting. They should provide
pages closest to the file beginning.
So, every insert/update will automatically defragment the entry.
Also, a special process should iterate over the partitions in a reverse way
just performing dummy updates.
The partition file may be safely truncated after the iterator.

Props:
- Your system still operating (no downtime)
- Defragmentation can be performed partially
- Defragmentation can be scheduled to periods of inactivity or performed on
a regular basis
- SQL will not be broken (no reason to recalculate the whole index, it will
be recalculated in a regular way on every entry update)
- Topology changes allowed
- Easy to implement/understand

Cons:
- Performance degradation (solvable by throttling)

On Mon, Jun 1, 2020 at 4:04 PM Sergey Chugunov 
wrote:

> Hi Ivan,
>
> I have an idea about suggested maintenance mode.
>
> First of all, I agree with your ideas about discovery restrictions: node
> should not join topology when performing defragmentation.
>
> At the same time I haven't heard about requests for this mode from users,
> so we don't know much about possible requirements.
> So I suggest to implement it in a pragmatical way: instead of inventing
> (unknown in reality) user scenarios lets develop minimal but yet
> well-designed functionality that suites our case. If we constrain our
> implementation with reasonable set of restrictions that's OK.
>
> So my idea is the following: to transit a node to maintenance user has to
> send special command to the node (e.g. with new command in control.sh
> utility or via JMX interface). Node saves maintenance request in local
> metastorage and waits for restart. User has to manually restart that node
> in order to finish moving it to maintenance mode.
>
> When node restarts and finds maintenance request it creates special type of
> discovery SPI that will not try to join topology at all yet node is able to
> start all necessary components and APIs like REST processor or JMX
> interface.
>
> When in maintenance, we'll be able to do defragmentation safely and remove
> maintenance request from metastorage only when it is completed (with all
> fault-tolerance logic in mind).
>
> As we don't have a mechanism (like watcher) to perform a "safe restart" (by
> safe I mean Ignite restart without OS-level process restart) we cannot
> finish maintenance mode without another manual restart but I think it is a
> reasonable restriction as maintenance mode shouldn't be an every-day
> routine and will be used quite rare.
>
> What do you think about it?
>
> On Tue, May 26, 2020 at 5:58 PM Ivan Bessonov 
> wrote:
>
> > Hello Igniters,
> >
> > I'd like to discuss this new IEP with you: [1]. The main idea is to have
> a
> > procedure that relocates
> > pages to the top of the file as compact as possible which allows us to
> > trim the file and increase its
> > fill-factor. It will be configured manually and executed after the
> restart,
> > but before node joins
> > topology (it means any load would be impossible during defragmentation).
> It
> > is described in detail
> > in the IEP itself, please read it. This topic was also previously
> discussed
> > here on dev-list in [2].
> >
> > Here I would like to list a few moments that are not as clear and require
> > your opinion.
> >
> >  - what are your overall thoughts on the IEP? Any concerns?
> >
> >  - maintenance mode - how do we communicate with the node that's not in
> > topology? What are
> >the options? As far as I know, we have no current tools like this.
> >
> >  - checkpointer refactoring - these changes will involve intensive
> writing
> > of pages to the storage.
> >If we're going to reuse the offheap page model to perform
> > defragmentation then the
> >checkpointing mechanism will have to be adapted in some form.
> >Are you fine with this? Or we need a separate discussion?
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
> > [2]
> >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html
> >
> >
> > --
> > Sincerely yours,
> > Ivan Bessonov
> >
>

Re: [DISCUSSION] IEP-47 Native persistence defragmentation

[Anton Vinogradov]

Ivan,

Thanks for hints.
Invalidated memory cache was restored :)

On Tue, Jun 2, 2020 at 2:55 PM Ivan Bessonov  wrote:

> Hello Anton,
>
> I'd like to address your last message. First of all, it was already
> partially discussed
> in this thread: [1] To reiterate - expected performance degradation will be
> significant.
> There's no way that we can throttle it because free/reuse lists have to be
> maintained
> sorted all the time. And these are very optimized data structures.
>
> More then that, "dummy" updates clash with data access, this is a very
> dangerous
> thing to do. And these updates don't save you from the situation when last
> pages in
> the file are not data pages, but tree pages, for example. They are much
> harder to
> move. Not only you should update all links to it but also do it
> effectively, without
> blocking the tree too much. I can think of many other examples.
>
> *Easy to implement/understand*
>  - no, it's not easy at all, defragmentation under the load is a very
> challenging thing to
>implement.
>
> *Why we're going to implement distributed system defragmentation in the old
> (offline) way?*
>  - because it's easier and safer, and it won't introduce any performance
> degradation.
>
> [1]
>
> http://apache-ignite-developers.2346864.n4.nabble.com/How-to-free-up-space-on-disc-after-removing-entries-from-IgniteCache-with-enabled-PDS-td39839.html
>
> вт, 2 июн. 2020 г. в 14:17, Anton Vinogradov :
>
> > Folks,
> >
> > Modern OS never ask you to schedule defragmentation and turn your PC off,
> > it performs it while you're browsing.
> > Why we're going to implement distributed system defragmentation in the
> old
> > (offline) way?
> >
> > All you need is to implement free/reuse-list sorting. They should provide
> > pages closest to the file beginning.
> > So, every insert/update will automatically defragment the entry.
> > Also, a special process should iterate over the partitions in a reverse
> way
> > just performing dummy updates.
> > The partition file may be safely truncated after the iterator.
> >
> > Props:
> > - Your system still operating (no downtime)
> > - Defragmentation can be performed partially
> > - Defragmentation can be scheduled to periods of inactivity or performed
> on
> > a regular basis
> > - SQL will not be broken (no reason to recalculate the whole index, it
> will
> > be recalculated in a regular way on every entry update)
> > - Topology changes allowed
> > - Easy to implement/understand
> >
> > Cons:
> > - Performance degradation (solvable by throttling)
> >
> > On Mon, Jun 1, 2020 at 4:04 PM Sergey Chugunov <
> sergey.chugu...@gmail.com>
> > wrote:
> >
> > > Hi Ivan,
> > >
> > > I have an idea about suggested maintenance mode.
> > >
> > > First of all, I agree with your ideas about discovery restrictions:
> node
> > > should not join topology when performing defragmentation.
> > >
> > > At the same time I haven't heard about requests for this mode from
> users,
> > > so we don't know much about possible requirements.
> > > So I suggest to implement it in a pragmatical way: instead of inventing
> > > (unknown in reality) user scenarios lets develop minimal but yet
> > > well-designed functionality that suites our case. If we constrain our
> > > implementation with reasonable set of restrictions that's OK.
> > >
> > > So my idea is the following: to transit a node to maintenance user has
> to
> > > send special command to the node (e.g. with new command in control.sh
> > > utility or via JMX interface). Node saves maintenance request in local
> > > metastorage and waits for restart. User has to manually restart that
> node
> > > in order to finish moving it to maintenance mode.
> > >
> > > When node restarts and finds maintenance request it creates special
> type
> > of
> > > discovery SPI that will not try to join topology at all yet node is
> able
> > to
> > > start all necessary components and APIs like REST processor or JMX
> > > interface.
> > >
> > > When in maintenance, we'll be able to do defragmentation safely and
> > remove
> > > maintenance request from metastorage only when it is completed (with
> all
> > > fault-tolerance logic in mind).
> > >
> > > As we don't have a mechanism (like watcher) to perform a "safe restart"
> > (by
> > > safe I mean Ignite restart without OS-level process restart) 

Re: [DISCUSSION] Ignite integration testing framework.

[Anton Vinogradov]

Folks,
First, I've created PR [1] with ducktests improvements

PR contains the following changes
- Pme-free switch proof-benchmark (2.7.6 vs master)
- Ability to check (compare with) previous releases (eg. 2.7.6 & 2.8)
- Global refactoring
-- benchmarks javacode simplification
-- services python and java classes code deduplication
-- fail-fast checks for java and python (eg. application should explicitly
write it finished with success)
-- simple results extraction from tests and benchmarks
-- javacode now configurable from tests/benchmarks
-- proper SIGTERM handling at javacode (eg. it may finish last operation
and log results)
-- docker volume now marked as delegated to increase execution speed for
mac & win users
-- Ignite cluster now start in parallel (start speed-up)
-- Ignite can be configured at test/benchmark
- full and module assembly scripts added

Second, I'd like to propose to accept ducktests [2] (ducktape integration)
as a target "PoC check & real topology benchmarking tool".

Ducktape pros
- Developed for distributed system by distributed system developers.
- Developed since 2014, stable.
- Proven usability by usage at Kafka.
- Dozens of dozens tests and benchmarks at Kafka as a great example pack.
- Built-in Docker support for rapid development and checks.
- Great for CI automation.

As an additional motivation, at least 3 teams
- IEP-45 team (to check crash-recovery speed-up (discovery and Zabbix
speed-up))
- Ignite SE Plugins team (to check plugin's features does not slow-down or
broke AI features)
- Ignite SE QA team (to append already developed smoke/load/failover tests
to AI codebase)
now, wait for ducktest merge to start checking cases they working on in AI
way.

Thoughts?

[1] https://github.com/apache/ignite/pull/7967
[2] https://github.com/apache/ignite/tree/ignite-ducktape

On Tue, Jun 16, 2020 at 12:22 PM Nikolay Izhikov 
wrote:

> Hello, Maxim.
>
> Thank you for so detailed explanation.
>
> Can we put the content of this discussion somewhere on the wiki?
> So It doesn’t get lost.
>
> I divide the answer in several parts. From the requirements to the
> implementation.
> So, if we agreed on the requirements we can proceed with the discussion of
> the implementation.
>
> 1. Requirements:
>
> The main goal I want to achieve is *reproducibility* of the tests.
> I’m sick and tired with the zillions of flaky, rarely failed, and almost
> never failed tests in Ignite codebase.
> We should start with the simplest scenarios that will be as reliable as
> steel :)
>
> I want to know for sure:
>   - Is this PR makes rebalance quicker or not?
>   - Is this PR makes PME quicker or not?
>
> So, your description of the complex test scenario looks as a next step to
> me.
>
> Anyway, It’s cool we already have one.
>
> The second goal is to have a strict test lifecycle as we have in JUnit and
> similar frameworks.
>
> > It covers production-like deployment and running a scenarios over a
> single database instance.
>
> Do you mean «single cluster» or «single host»?
>
> 2. Existing tests:
>
> > A Combinator suite allows to run set of operations concurrently over
> given database instance.
> > A Consumption suite allows to run a set production-like actions over
> given set of Ignite/GridGain versions and compare test metrics across
> versions
> > A Yardstick suite
> > A Stress suite that simulates hardware environment degradation
> > An Ultimate, DR and Compatibility suites that performs functional
> regression testing
> > Regression
>
> Great news that we already have so many choices for testing!
> Mature test base is a big +1 for Tiden.
>
> 3. Comparison:
>
> > Criteria: Test configuration
> > Ducktape: single JSON string for all tests
> > Tiden: any number of YaML config files, command line option for
> fine-grained test configuration, ability to select/modify tests behavior
> based on Ignite version.
>
> 1. Many YAML files can be hard to maintain.
> 2. In ducktape, you can set parameters via «—parameters» option. Please,
> take a look at the doc [1]
>
> > Criteria: Cluster control
> > Tiden: additionally can address cluster as a whole and execute remote
> commands in parallel.
>
> It seems we implement this ability in the PoC, already.
>
> > Criteria: Test assertions
> > Tiden: simple asserts, also few customized assertion helpers.
> > Ducktape: simple asserts.
>
> Can you, please, be more specific.
> What helpers do you have in mind?
> Ducktape has an asserts that waits for logfile messages or some process
> finish.
>
> > Criteria: Test reporting
> > Ducktape: limited to its own text/HTML format
>
> Ducktape have
> 1. Text reporter
> 2. Customizable HTML reporter
> 3. JSON reporter.
>
> We can show JSON with the any template or tool.
>
> > Criteria: Provisioning and deployment
> > Ducktape: can provision subset of hosts from cluster for test needs.
> However, that means, that test can’t be scaled without test code changes.
> Does not do any deploy, relies on external means, e.g. pre-packaged in
> 

Re: [DISCUSSION] Ignite integration testing framework.

[Anton Vinogradov]

> Have you had a chance to deploy ducktests in bare metal?
Working on servers obtaining.

On Thu, Jul 9, 2020 at 10:11 AM Max Shonichev  wrote:

> Anton,
>
> well, strange thing, but clean up and rerun helped.
>
>
> Ubuntu 18.04
>
>
> 
> SESSION REPORT (ALL TESTS)
> ducktape version: 0.7.7
> session_id:   2020-07-06--003
> run time: 4 minutes 44.835 seconds
> tests run:5
> passed:   5
> failed:   0
> ignored:  0
>
> 
> test_id:
>
> ignitetest.tests.benchmarks.add_node_rebalance_test.AddNodeRebalanceTest.test_add_node.version=2.8.1
> status: PASS
> run time:   41.927 seconds
> {"Rebalanced in (sec)": 1.02205491065979}
>
> 
> test_id:
>
> ignitetest.tests.benchmarks.add_node_rebalance_test.AddNodeRebalanceTest.test_add_node.version=dev
> status: PASS
> run time:   51.985 seconds
> {"Rebalanced in (sec)": 0.0760810375213623}
>
> 
> test_id:
>
> ignitetest.tests.benchmarks.pme_free_switch_test.PmeFreeSwitchTest.test.version=2.7.6
> status: PASS
> run time:   1 minute 4.283 seconds
> {"Streamed txs": "1900", "Measure duration (ms)": "34818", "Worst
> latency (ms)": "31035"}
>
> 
> test_id:
>
> ignitetest.tests.benchmarks.pme_free_switch_test.PmeFreeSwitchTest.test.version=dev
> status: PASS
> run time:   1 minute 13.089 seconds
> {"Streamed txs": "73134", "Measure duration (ms)": "35843", "Worst
> latency (ms)": "139"}
>
> 
> test_id:
>
> ignitetest.tests.spark_integration_test.SparkIntegrationTest.test_spark_client
> status: PASS
> run time:   53.332 seconds
>
> 
>
>
> MacBook
>
> 
> SESSION REPORT (ALL TESTS)
> ducktape version: 0.7.7
> session_id:   2020-07-06--001
> run time: 6 minutes 58.612 seconds
> tests run:5
> passed:   5
> failed:   0
> ignored:  0
>
> 
> test_id:
>
> ignitetest.tests.benchmarks.add_node_rebalance_test.AddNodeRebalanceTest.test_add_node.version=2.8.1
> status: PASS
> run time:   48.724 seconds
> {"Rebalanced in (sec)": 3.2574470043182373}
>
> 
> test_id:
>
> ignitetest.tests.benchmarks.add_node_rebalance_test.AddNodeRebalanceTest.test_add_node.version=dev
> status: PASS
> run time:   1 minute 23.210 seconds
> {"Rebalanced in (sec)": 2.165921211242676}
>
> 
> test_id:
>
> ignitetest.tests.benchmarks.pme_free_switch_test.PmeFreeSwitchTest.test.version=2.7.6
> status: PASS
> run time:   1 minute 12.659 seconds
> {"Streamed txs": "642", "Measure duration (ms)": "33177", "Worst latency
> (ms)": "31063"}
>
> 
> test_id:
>
> ignitetest.tests.benchmarks.pme_free_switch_test.PmeFreeSwitchTest.test.version=dev
> status: PASS
> run time:   1 minute 57.257 seconds
> {"Streamed txs": "32924", "Measure duration (ms)": "48252", "Worst
> latency (ms)": "1010"}
>
> 
> test_id:
>
> ignitetest.tests.spark_integration_test.SparkIntegrationTest.test_spark_client
> status: PASS
> run time:   1 minute 36.317 seconds
>
> =
>
> while relative numbers proportion remains the same for different Ignite
> versions, absolute number for mac/linux differ more then twice.
>
> I'm finalizing code with 'local Tiden' appliance for your tests.  PR
> would be ready soon.
>
> Have y

Re: [DISCUSSION] Ignite integration testing framework.

[Anton Vinogradov]

Max,

Thanks for the check!

> Is it OK for those tests to fail?
No.
I see really strange things at logs.
Looks like you have concurrent ducktests run started not expected services,
and this broke the tests.
Could you please clean up the docker (use clean-up script [1]).
Compile sources (use script [2]) and rerun the tests.

[1]
https://github.com/anton-vinogradov/ignite/blob/dc98ee9df90b25eb5d928090b0e78b48cae2392e/modules/ducktests/tests/docker/clean_up.sh
[2]
https://github.com/anton-vinogradov/ignite/blob/3c39983005bd9eaf8cb458950d942fb592fff85c/scripts/build.sh

On Mon, Jul 6, 2020 at 12:03 PM Nikolay Izhikov  wrote:

> Hello, Maxim.
>
> Thanks for writing down the minutes.
>
> There is no such thing as «Nikolay team» on the dev-list.
> I propose to focus on product requirements and what we want to gain from
> the framework instead of taking into account the needs of some team.
>
> Can you, please, write down your version of requirements so we can reach a
> consensus on that and therefore move to the discussion of the
> implementation?
>
> > 6 июля 2020 г., в 11:18, Max Shonichev  написал(а):
> >
> > Yes, Denis,
> >
> > common ground seems to be as follows:
> > Anton Vinogradov and Nikolay Izhikov would try to prepare and run PoC
> over physical hosts and share benchmark results. In the meantime, while I
> strongly believe that dockerized approach to benchmarking is a road to
> misleading and false positives, I'll prepare a PoC of Tiden in dockerized
> environment to support 'fast development prototyping' usecase Nikolay team
> insist on. It should be a matter of few days.
> >
> > As a side note, I've run Anton PoC locally and would like to have some
> comments about results:
> >
> > Test system: Ubuntu 18.04, docker 19.03.6
> > Test commands:
> >
> >
> > git clone -b ignite-ducktape g...@github.com:anton-vinogradov/ignite.git
> > cd ignite
> > mvn clean install -DskipTests -Dmaven.javadoc.skip=true
> -Pall-java,licenses,lgpl,examples,!spark-2.4,!spark,!scala
> > cd modules/ducktests/tests/docker
> > ./run_tests.sh
> >
> > Test results:
> >
> 
> > SESSION REPORT (ALL TESTS)
> > ducktape version: 0.7.7
> > session_id:   2020-07-05--004
> > run time: 7 minutes 36.360 seconds
> > tests run:5
> > passed:   3
> > failed:   2
> > ignored:  0
> >
> 
> > test_id:
> ignitetest.tests.benchmarks.add_node_rebalance_test.AddNodeRebalanceTest.test_add_node.version=2.8.1
> > status: FAIL
> > run time:   3 minutes 12.232 seconds
> >
> 
> > test_id:
> ignitetest.tests.benchmarks.pme_free_switch_test.PmeFreeSwitchTest.test.version=2.7.6
> > status: FAIL
> > run time:   1 minute 33.076 seconds
> >
> >
> > Is it OK for those tests to fail? Attached is full test report
> >
> >
> > On 02.07.2020 17:46, Denis Magda wrote:
> >> Folks,
> >> Please share the summary of that Slack conversation here for records
> once
> >> you find common ground.
> >> -
> >> Denis
> >> On Thu, Jul 2, 2020 at 3:22 AM Nikolay Izhikov 
> wrote:
> >>> Igniters.
> >>>
> >>> All who are interested in integration testing framework discussion are
> >>> welcome into slack channel -
> >>>
> https://join.slack.com/share/zt-fk2ovehf-TcomEAwiXaPzLyNKZbmfzw?cdn_fallback=2
> >>>
> >>>
> >>>
> >>>> 2 июля 2020 г., в 13:06, Anton Vinogradov  написал(а):
> >>>>
> >>>> Max,
> >>>> Thanks for joining us.
> >>>>
> >>>>> 1. tiden can deploy artifacts by itself, while ducktape relies on
> >>>>> dependencies being deployed by external scripts.
> >>>> No. It is important to distinguish development, deploy, and
> >>> orchestration.
> >>>> All-in-one solutions have extremely limited usability.
> >>>> As to Ducktests:
> >>>> Docker is responsible for deployments during development.
> >>>> CI/CD is responsible for deployments during release and nightly
> checks.
> >>> It's up to the team to chose AWS, VM, BareMetal, and even OS.
> >>>> Ducktape is responsible for orchestration.
> >>>>
> >>>&

Re: [DISCUSSION] Ignite integration testing framework.

[Anton Vinogradov]

g at Sberbank site we used the same
> tiden scripts that we use in our lab, the only change was putting a list of
> hosts into config.
> >
> > If we used ducktape solution we would have to instead prepare some
> deployment scripts to pre-initialize Sberbank hosts, for example, with
> Ansible or Chef.
> >
> >
> > You have solved this deficiency with docker by putting all dependencies
> into one uber-image and that looks like simple and elegant solution,
> > however, that effectively limits you to single-host testing.
> >
> > I guess we all know about docker hyped ability to run over distributed
> virtual networks. We used to go that way, but quickly found that it is more
> of the hype than real work. In real environments, there are problems with
> routing, DNS, multicast and broadcast traffic, and many others, that turn
> docker-based distributed solution into a fragile hard-to-maintain monster.
> >
> > Please, if you believe otherwise, perform a run of your PoC over at
> least two physical hosts and share results with us.
> >
> > If you consider that one physical docker host is enough, please, don't
> overlook that we want to run real scale scenarios, with 50-100 cache
> groups, persistence enabled and a millions of keys loaded.
> >
> > Practical limit for such configurations is 4-6 nodes per single physical
> host. Otherwise, tests become flaky due to resource starvation.
> >
> > Please, if you believe otherwise, perform at least a 10 of runs of your
> PoC with other tests running at TC (we're targeting TeamCity, right?) and
> share results so we could check if the numbers are reproducible.
> >
> > I stress this once more: functional integration tests are OK to run in
> Docker and CI, but running benchmarks in Docker is a big NO GO.
> >
> >
> > Second property let us write tests that require real-parallel actions
> over hosts.
> >
> > For example, agreed scenario for PME benchmarkduring "PME optimization
> stream" was as follows:
> >
> >  - 10 server nodes, preloaded with 1M of keys
> >  - 4 client nodes perform transactional load  (client nodes physically
> separated from server nodes)
> >  - during load:
> >  -- 5 server nodes stopped in parallel
> >  -- after 1 minute, all 5 nodes are started in parallel
> >  - load stopped, logs are analysed for exchange times.
> >
> > If we had stopped and started 5 nodes one-by-one, as ducktape does, then
> partition map exchange merge would not happen and we could not have
> measured PME optimizations for that case.
> >
> >
> > These are limitations of ducktape that we believe as a more important
> > argument "against" than you provide "for".
> >
> >
> >
> >
> > On 30.06.2020 14:58, Anton Vinogradov wrote:
> >> Folks,
> >> First, I've created PR [1] with ducktests improvements
> >> PR contains the following changes
> >> - Pme-free switch proof-benchmark (2.7.6 vs master)
> >> - Ability to check (compare with) previous releases (eg. 2.7.6 & 2.8)
> >> - Global refactoring
> >> -- benchmarks javacode simplification
> >> -- services python and java classes code deduplication
> >> -- fail-fast checks for java and python (eg. application should
> explicitly write it finished with success)
> >> -- simple results extraction from tests and benchmarks
> >> -- javacode now configurable from tests/benchmarks
> >> -- proper SIGTERM handling at javacode (eg. it may finish last
> operation and log results)
> >> -- docker volume now marked as delegated to increase execution speed
> for mac & win users
> >> -- Ignite cluster now start in parallel (start speed-up)
> >> -- Ignite can be configured at test/benchmark
> >> - full and module assembly scripts added
> > Great job done! But let me remind one of Apache Ignite principles:
> > week of thinking save months of development.
> >
> >
> >> Second, I'd like to propose to accept ducktests [2] (ducktape
> integration) as a target "PoC check & real topology benchmarking tool".
> >> Ducktape pros
> >> - Developed for distributed system by distributed system developers.
> > So does Tiden
> >
> >> - Developed since 2014, stable.
> > Tiden is also pretty stable, and development start date is not a good
> argument, for example pytest is since 2004, pytest-xdist (plugin for
> distributed testing) is since 2010, but we don't see it as a alternative at
> all.
> >
> >> - Proven usability by usage at Kafka.
> > Tiden is proven usable 

Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]

[Anton Vinogradov]

You're now at the "Ignite Release Managers" group.
Please check you gain access.

On Fri, Jun 26, 2020 at 9:38 PM Alex Plehanov 
wrote:

> Guys,
>
> I've created the 2.9 release confluence page [1].
> Also, I found that I don't have permission to Team City release tasks. Can
> anyone give me such permissions?
>
> [1]: https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.9
>
> пт, 26 июн. 2020 г. в 13:38, Alexey Zinoviev :
>
> > Igniters
> >
> > Unfortunately, the ML model export/import feature
> >  is under development
> > yet.
> > I need to delay it before the 2.10 release.
> >
> >
> >
> > пт, 26 июн. 2020 г. в 06:50, Alex Plehanov :
> >
> > > Denis,
> > >
> > > Yes, I still ready to manage it.
> > > Today I will prepare a release page on wiki and will try to go over
> > tickets
> > > list.
> > > Also, I have plans to cut the branch by the end of next week if there
> are
> > > no objections.
> > >
> > >
> > > пт, 26 июн. 2020 г. в 03:48, Denis Magda :
> > >
> > > > Igniters,
> > > >
> > > > Are we moving forward with this release? Alex Plehanov, are you still
> > > ready
> > > > to manage it? It seems like everyone agreed with the timeline you
> > > proposed
> > > > in the very beginning.
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Tue, Jun 16, 2020 at 8:52 AM Denis Magda 
> wrote:
> > > >
> > > > > Sergey, Ivan,
> > > > >
> > > > > Could you please check the questions below? If it's time-consuming
> to
> > > > > rework continuous queries, then the new mode can become available
> in
> > > the
> > > > > experimental state and should not let register continuous queries
> to
> > > > avoid
> > > > > potential deadlocks. Overall, this design gap in continuous queries
> > was
> > > > > like a bomb that has just detonated [1]. Anyway, this new
> > connectivity
> > > > mode
> > > > > will be priceless even if you can't use continuous queries with
> them
> > > > > because right now we cannot even start a thick client inside of a
> > > > > serverless function.
> > > > >
> > > > > Alexey Plehanov,
> > > > >
> > > > > It looks like we can proceed with the release taking your
> timelines.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-13156
> > > > >
> > > > > -
> > > > > Denis
> > > > >
> > > > >
> > > > > On Wed, Jun 10, 2020 at 4:16 PM Denis Magda 
> > wrote:
> > > > >
> > > > >> Ivan, Sergey,
> > > > >>
> > > > >> How much effort should we put to resolve the issue with
> > > > >> continuous queries? Are you working on that issue actively? Let's
> > try
> > > to
> > > > >> guess what would be the ETA.
> > > > >>
> > > > >> -
> > > > >> Denis
> > > > >>
> > > > >>
> > > > >> On Wed, Jun 10, 2020 at 3:55 AM Ivan Bessonov <
> > bessonov...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Hello,
> > > > >>>
> > > > >>> Sorry for the delay. Sergey Chugunov (sergey.chugu...@gmail.com)
> > > just
> > > > >>> replied
> > > > >>> to the main conversation about "communication via discovery" [1].
> > We
> > > > >>> work on it
> > > > >>> together and recently have found one hard-to-fix scenario,
> detailed
> > > > >>> description is
> > > > >>> provided in Sergey's reply.
> > > > >>>
> > > > >>> In short, July 10th looks realistic only if we introduce new
> > behavior
> > > > in
> > > > >>> its current
> > > > >>> implementation, with new setting and IgniteExperimental status.
> > > Blocker
> > > > >>> here is
> > > > >>> current implementation of Continuos Query protocol that in some
> > cases
> > > > >>> (described
> > > > >>> at the end) initiates TCP connection right from discovery thread
> > > which
> > > > >>> obviously
> > > > >>> leads to deadlock. We haven't estimated efforts needed to
> redesign
> > of
> > > > CQ
> > > > >>> protocol
> > > > >>> but it is definitely a risk and fixing it isn't feasible with a
> > code
> > > > >>> freeze at 10th of July.
> > > > >>> So my verdict: we can include this new feature in 2.9 scope as
> > > > >>> experimental and with
> > > > >>> highlighted limitation on CQ usage. Is that OK?
> > > > >>>
> > > > >>> CQ limitation: server needs to open a communication connection to
> > the
> > > > >>> client if during
> > > > >>> CQ registration client tries to p2p deploy new class not
> available
> > on
> > > > >>> server classpath.
> > > > >>> In other cases registration of CQ should be fine.
> > > > >>>
> > > > >>> [1]
> > > > >>>
> > > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-New-Ignite-settings-for-IGNITE-12438-and-IGNITE-13013-td47586.html
> > > > >>>
> > > > >>> вт, 9 июн. 2020 г. в 19:36, Ivan Rakov :
> > > > >>>
> > > >  Hi,
> > > > 
> > > >  Indeed, the tracing feature is almost ready. Discovery,
> > > communication
> > > >  and
> > > >  transactions tracing will be introduced, as well as an option to
> > > >  configure
> > > >  tracing in runtime. Right now we are working on final
> performance
> > > >  

Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]

[Anton Vinogradov]

Cellular switch

On Wed, Jun 3, 2020 at 4:10 PM Nikolay Izhikov  wrote:

> Snapshots.
>
> > 3 июня 2020 г., в 16:10, Ilya Kasnacheev 
> написал(а):
> >
> > Hello!
> >
> > Can you please clarify what is the scope of 2.9 release?
> >
> > I have a feeling that we don't really have any big features in the
> current
> > 2.9 code base. No Calcite, etc.
> >
> > So I'm asking whether it is worth it.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 3 июн. 2020 г. в 14:45, Alex Plehanov :
> >
> >> Hello Igniters,
> >>
> >> AI 2.8.1 is finally released and as we discussed here [1] its time to
> start
> >> the discussion about 2.9 release.
> >>
> >> I want to propose myself to be the release manager of the 2.9 release.
> >>
> >> What about release time, I agree with Maxim that we should deliver
> features
> >> as frequently as possible. If some feature doesn't fit into release
> dates
> >> we should better include it into the next release and schedule the next
> >> release earlier then postpone the current release.
> >>
> >> I propose the following dates for 2.9 release:
> >>
> >> Scope Freeze: June 26, 2020
> >> Code Freeze: July 10, 2020
> >> Voting Date: July 31, 2020
> >> Release Date: August 7, 2019
> >>
> >> WDYT?
> >>
> >> [1] :
> >>
> >>
> http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-Releases-Plan-td47360.html#a47575
> >>
>
>

Re: Move TcpDiscoveryZookeeperIpFinder to ignite-extensions as part of IEP-36

[Anton Vinogradov]

Ivan,

Sound like an obsolete compromise partial solution kept just because of the
"Zookeeper" word at its name, not because of some profit.

>> So I suggest to move this single class with tests to separate module in
>> ignite-extensions.
+1

On Fri, Jul 24, 2020 at 11:27 AM Ivan Daschinsky 
wrote:

> Hello, Igniters.
>
> Currently, in module ignite-zookeeper, that contains full implementation of
> ZookeeperDiscoverySpi, also presents one class that looks like a little
> snippet and I have some concerns about presence of this class in the
> module.
>
> 1) This class is a simple snippet-like implementation of
> TcpDiscoverIpFinder
> 2) It creates EPHEMERAL_SEQUENTIAL ZNode in root directory with json,
> containing IP addresses of joining node.
> 3) It reads registered children znodes from root directory  and use these
> addresses to participate in common TcpDiscovery process.
>
> 1) This class has nothing in common with ZookeeperDiscovery, but
> 2) It brings to module additional dependencies (apache-curator framework,
> jackson and so on)
> 3) All of these dependencies are not required to ZookeeperDiscoverySpi.
> 4) The usefulness of it is doubtful, because we have already fully
> functional ZookeeperDiscovery and use Zookeeper quorum as just simple store
> for IP addresses without utilizing full zookeeper power is questionable
> decision.
>
> So I suggest to move this single class with tests to separate module in
> ignite-extensions.
>
> What do you think?
>
> --
> Sincerely yours, Ivan Daschinskiy
>

Re: [DISCUSSION] Ignite integration testing framework.

[Anton Vinogradov]

t; >
> >
> > Ubuntu 18.04
> >
> >
> 
>
> >
> > SESSION REPORT (ALL TESTS)
> > ducktape version: 0.7.7
> > session_id:   2020-07-06--003
> > run time: 4 minutes 44.835 seconds
> > tests run:5
> > passed:   5
> > failed:   0
> > ignored:  0
> >
> 
>
> >
> > test_id:
> >
> ignitetest.tests.benchmarks.add_node_rebalance_test.AddNodeRebalanceTest.test_add_node.version=2.8.1
>
> >
> > status: PASS
> > run time:   41.927 seconds
> > {"Rebalanced in (sec)": 1.02205491065979}
> >
> 
>
> >
> > test_id:
> >
> ignitetest.tests.benchmarks.add_node_rebalance_test.AddNodeRebalanceTest.test_add_node.version=dev
>
> >
> > status: PASS
> > run time:   51.985 seconds
> > {"Rebalanced in (sec)": 0.0760810375213623}
> >
> 
>
> >
> > test_id:
> >
> ignitetest.tests.benchmarks.pme_free_switch_test.PmeFreeSwitchTest.test.version=2.7.6
>
> >
> > status: PASS
> > run time:   1 minute 4.283 seconds
> > {"Streamed txs": "1900", "Measure duration (ms)": "34818", "Worst
> > latency (ms)": "31035"}
> >
> 
>
> >
> > test_id:
> >
> ignitetest.tests.benchmarks.pme_free_switch_test.PmeFreeSwitchTest.test.version=dev
>
> >
> > status: PASS
> > run time:   1 minute 13.089 seconds
> > {"Streamed txs": "73134", "Measure duration (ms)": "35843", "Worst
> > latency (ms)": "139"}
> >
> 
>
> >
> > test_id:
> >
> ignitetest.tests.spark_integration_test.SparkIntegrationTest.test_spark_client
>
> >
> > status: PASS
> > run time:   53.332 seconds
> >
> 
>
> >
> >
> >
> > MacBook
> >
> 
>
> >
> > SESSION REPORT (ALL TESTS)
> > ducktape version: 0.7.7
> > session_id:   2020-07-06--001
> > run time: 6 minutes 58.612 seconds
> > tests run:5
> > passed:   5
> > failed:   0
> > ignored:  0
> >
> 
>
> >
> > test_id:
> >
> ignitetest.tests.benchmarks.add_node_rebalance_test.AddNodeRebalanceTest.test_add_node.version=2.8.1
>
> >
> > status: PASS
> > run time:   48.724 seconds
> > {"Rebalanced in (sec)": 3.2574470043182373}
> >
> 
>
> >
> > test_id:
> >
> ignitetest.tests.benchmarks.add_node_rebalance_test.AddNodeRebalanceTest.test_add_node.version=dev
>
> >
> > status: PASS
> > run time:   1 minute 23.210 seconds
> > {"Rebalanced in (sec)": 2.165921211242676}
> >
> 
>
> >
> > test_id:
> >
> ignitetest.tests.benchmarks.pme_free_switch_test.PmeFreeSwitchTest.test.version=2.7.6
>
> >
> > status:     PASS
> > run time:   1 minute 12.659 seconds
> > {"Streamed txs": "642", "Measure duration (ms)": "33177", "Worst latency
> > (ms)": "31063"}
> >
> 
>
> >
> > test_id:
> >
> ignitetest.tests.benchmarks.pme_free_switch_test.PmeFreeSwitchTest.test.version=dev
>
> >
> > status: PASS
> > run time:   1 minute 57.257 seconds
> > {"Streamed txs": "32924", "Measure duration (ms)": "48252", "Worst
> > latency (ms)": "1010"}
> >
> --

Re: [VOTE] Define Apache Ignite as a Distributed Database

[Anton Vinogradov]

+1

On Tue, Nov 24, 2020 at 10:24 AM Pavel Tupitsyn 
wrote:

> +1
>
> On Tue, Nov 24, 2020 at 3:25 AM Saikat Maitra 
> wrote:
>
> > +1
> >
> > On Mon, Nov 23, 2020 at 4:55 PM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > +1
> > >
> > > On Mon, Nov 23, 2020 at 2:44 PM Denis Magda  wrote:
> > >
> > > > Igniters,
> > > >
> > > > With this vote, I'd like to formally wrap up our discussion on
> defining
> > > > Ignite as a "distributed database" instead of an "in-memory
> computing"
> > > > platform. See the following discussion for the rationale and context:
> > > >
> > > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Renaming-Ignite-s-product-category-td49246.html
> > > >
> > > > If the vote passes, the website will define Ignite as "a distributed
> > > > database for in-memory speed at petabyte scale" to underscore our
> > > in-memory
> > > > origins (and the primary reason why the project is selected) but not
> > > > diminishing our persistence capabilities. All prominent use cases
> such
> > as
> > > > caching, high-performance computing, etc. will remain visible on the
> > > > website. There is nothing wrong if a distributed database is used as
> a
> > > > cache or high-performance compute cluster; it's up to an application
> > > > developer to decide.
> > > >
> > > > Overall, please cast your vote for defining Ignite as a "distributed
> > > > database":
> > > > +1 - support the change
> > > > -1 - disagree with the change, explain why
> > > > 0 - neutral
> > > >
> > > > This is a majority vote that is open for the next 7 days and to be
> > closed
> > > > on Monday, Nov 30th:
> > > > https://www.timeanddate.com/countdown/to?year=2020=11=30
> > > >
> > > > -
> > > > Denis
> > > >
> > >
> >
>

Re: [DISCUSS] Missed (non-suited) tests

[Anton Vinogradov]

>> WDYT?
LGTM

On Wed, Nov 25, 2020 at 12:23 AM Max Timonin 
wrote:

> Ilya, Anton, Ivan, hi!
>
> I fix some comments you leave in the PR. Also I checked some test suites
> and found that some tests are written in the core module but depend on the
> indexing module (or other modules). Some of such test classes contain tests
> that are related to the core functionality, but some to indexing. I'm not
> sure if it is correct to move a whole suite with all tests from the
> indexing module to the core, as it will hide some core tests from the core
> module.
>
> I believe that the correct solution is to investigate every such test and
> move it to the right module. But I think this work will take a lot of time
> and should be performed in a separate ticket, I will do it in the
> background.
>
> I think currently we should proceed with a way I introduced in PR:
> 1. Create fake suites for all such tests (written in core, suited in other
> modules: indexing/spring/zookeeper/etc) in the core module. I named such
> suites with prefix Core*.
> 2. Replace tests in modules with links to fake suites.
> 3. Create an umbrella Jira ticket to discover every fake suite and replace
> it with a new one in the right module.
> 4. Merge this PR for introducing a new travis check to avoid losing
> new tests.
>
> WDYT?
>
> List of such mixed suites:
>
> 1. suite IgniteBinaryCacheQueryTestSuite
>
> test GridCacheQueryIndexingDisabledSelfTest
> test IgniteCacheBinaryObjectsScanSelfTest
> test IgniteCacheBinaryObjectsScanWithEventsSelfTest)
>
>
> 2. suite IgniteCacheQuerySelfTestSuite3
>
> test GridCacheContinuousQueryNodesFilteringTest
>
>
> 3. suite IgniteCacheQuerySelfTestSuite5
>
> test ContinuousQueryRemoteFilterMissingInClassPathSelfTest
>
> 4. suite IgniteCacheQuerySelfTestSuite6
>
> test CacheContinuousQueryOperationP2PTest
>
> test CacheContinuousQueryFilterDeploymentFailedTest
>
>
> 5. all tests in suite IgnitePdsWithIndexingCoreTestSuite
>
>
> 6. and some others.
>
> On Wed, Nov 18, 2020 at 12:38 PM Max Timonin 
> wrote:
>
> > Hi Ilya! Thank you for up the topic. I will come back with fixes and
> > comments in a couple of days.
> >
> > On Tue, Nov 17, 2020 at 4:26 PM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com>
> > wrote:
> >
> >> Hello!
> >>
> >> I have left some comments and there's also more discussion there. Can
> you
> >> please look?
> >>
> >> Thanks,
> >> --
> >> Ilya Kasnacheev
> >>
> >>
> >> вт, 3 нояб. 2020 г. в 00:03, Max Timonin :
> >>
> >> > Hi!
> >> >
> >> > I've updated PR: https://github.com/apache/ignite/pull/8367. Anton,
> >> Ivan,
> >> > Ivan could you please review it?
> >> >
> >> > Some moments to mention:
> >> > 1. I've added new suites: SerializerSuite
> >> (ignite-cassandra-serializers),
> >> > DistanceTestSuite, NaiveBayesTestSuite (ignite-ml). Should we
> configure
> >> a
> >> > TeamCity to run them?
> >> >
> >> > 2. Some tests marked as failed, I'll create corresponding tickets for
> >> them
> >> > after PR approved:
> >> > - IgnitePKIndexesMigrationToUnwrapPkTest
> >> > - P2PGridifySelfTest
> >> > - GridCacheMultithreadedFailoverAbstractTest
> >> > - WalCompactionAfterRestartTest
> >> > - GridTcpCommunicationSpiLogTest
> >> > - ComplexSecondaryKeyUnwrapSelfTest
> >> > - SqlTransactionsSelfTest
> >> >
> >> > 3. Add docs to DEVNOTES.txt
> >> >
> >> > On Mon, Nov 2, 2020 at 11:44 AM Anton Vinogradov 
> wrote:
> >> >
> >> > > > As I understand we
> >> > > > can't just move suites between modules, as TeamCity may depend on
> >> the
> >> > > path
> >> > > > to them.
> >> > > See no problem to update TC as well.
> >> > >
> >> > > On Fri, Oct 30, 2020 at 4:32 PM Ivan Daschinsky <
> ivanda...@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > I suggests to mark these tests with @Ignore and file tickets to
> fix
> >> > them.
> >> > > >
> >> > > > пт, 30 окт. 2020 г. в 16:26, Ivan Daschinsky  >:
> >> > > >
> >> > > > > Hi
> >> > > > >
> >> > > > > WalCompactionAfterRestartTest -- yes we need it. This te

TcpDiscovery detects NONE_LEFT faster than ZookeeperDiscovery in SIGKILL case

[Anton Vinogradov]

Folks,

I've found an interesting thing about delays on node failed detection
depending on discovery type.

For example, when we have 33 nodes cluster (33 vms in the cloud in my
case), we have the following failure detection delays:
~ 10 seconds on TcpDiscovery
~ 8.5 seconds on ZookeeperDiscovery
when we dropping connections via iptables (emulating network issues, GC,
Hangs, etc),

But when we stop node via SIGKILL we have the unexpected following:
~ 0.5 seconds on TcpDiscovery
~ 7.5!!! seconds on ZookeeperDiscovery

TcpDiscovery handles failure faster on SIGKILL because it periodically (~2
times per second) checks that socket is alive and when it sees it closed
there is no reason to wait for anything else.
ZookeeperDiscovery waits up to the 10 seconds specified at default
failureDetectionTimeout and has no socket-alive checks.

Should we change default failureDetectionTimeout/zookeeperSessionDuration
when ZookeeperDiscovery is used to have at least the same failure detection
performance as at TcpDiscovery?

Are there any other ways to tune ZookeeperDiscovery defaults?

Are there any chances to have socket-alive checks when ZookeeperDiscovery
used?

Re: [DISCUSS] Disable socket linger by default in TCP discovery SPI.

[Anton Vinogradov]

PR's merged.
Please make sure that users who use SSL will be notified to set linger at
2.10 migration doc.

On Fri, Nov 6, 2020 at 1:01 PM Steshin Vladimir  wrote:

>  The tickets are: [1] disables linger by default and [2] is the doc.
>
>
> [1] https://issues.apache.org/jira/browse/IGNITE-13643
>
> [2] https://issues.apache.org/jira/browse/IGNITE-13662
>
> 05.11.2020 11:00, Anton Vinogradov пишет:
> > Folks,
> > Seems, we've got an agreement that the fix is necessary.
> > Do we need to do except the following?
> >>> zero linger as default + warning on SSL enabled on JVM before the fix +
> > warning at documentation + migration notes
> >
> > On Tue, Nov 3, 2020 at 2:38 PM Steshin Vladimir 
> wrote:
> >
> >>   Ilya, hi.
> >>
> >>
> >>   Of course: /TcpDiscoverySpi.setSoLinger(int)/ property. Always
> been.
> >>
> >>
> >> 02.11.2020 20:14, Ilya Kasnacheev пишет:
> >>> Hello!
> >>>
> >>> Is there any option to re-enable linger on SSL sockets?
> >>>
> >>> Telling people to re-configure does not help if they can't.
> >>>
> >>> Regards,
>

Re: [DISCUSSION] Apache Ignite Release 2.10 (time, scope, manager)

[Anton Vinogradov]

> - Distributed environment tests [2] (Nikolay Izhikov, Anton
> Vinogradov, Ivan Daschinskiy)
> Will all related issues be included in the 2.10?
We're at the final preparations phase. Going to start the merge discussion
soon.

On Wed, Nov 11, 2020 at 4:48 PM Alexey Zinoviev 
wrote:

> Yes, Model Export/Import for ML will be finished, now it's ready, but under
> testing on my side.
>
> ср, 11 нояб. 2020 г. в 16:08, Nikolay Izhikov :
>
> > I suggest to include CDC feature to the 2.10 scope.
> >
> > > 11 нояб. 2020 г., в 16:06, Maxim Muzafarov 
> > написал(а):
> > >
> > > Igniters,
> > >
> > >
> > > Can you clarify the issue statuses according to the Apache Ignite
> > > Roadmap [1] and the 2.10 Apache Ignite Release? Here are some major
> > > issues which are expected to be included in 2.10 with the unknown
> > > status:
> > >
> > > - Distributed environment tests [2] (Nikolay Izhikov, Anton
> > > Vinogradov, Ivan Daschinskiy)
> > > Will all related issues be included in the 2.10?
> > >
> > > - Native persistence defragmentation [6][7] (Anton Kalashnikov, Sergey
> > Chugunov)
> > > Are we on the last mile with this for 2.10?
> > >
> > > - Consistency-related improvements for atomic caches [3] (Alexey
> > Scherbakov)
> > > Do we have the JIRA issue? Will the issue be included in 2.10?
> > >
> > > - Tombstone objects during rebalancing [4] (Alexey Scherbakov)
> > > The issue [4] is in progress state. Will it be finished prior to code
> > freeze?
> > >
> > > - SQL runtime statistics (Konstantin Orlov)
> > > Do we have an IEP for this or JIRA issue?
> > >
> > > - ML Model export/import [5] (Alexey Zinoviev)
> > > - Will the issue be finished?
> > >
> > >
> > > [1]
> > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Roadmap
> > > [2]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-56%3A+Distributed+environment+tests
> > > [3]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-12+Make+ATOMIC+Caches+Consistent+Again
> > > [4] https://issues.apache.org/jira/browse/IGNITE-11704
> > > [5] https://issues.apache.org/jira/browse/IGNITE-6642
> > > [6] https://issues.apache.org/jira/browse/IGNITE-13143
> > > [7]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation
> > >
> > > On Tue, 10 Nov 2020 at 15:58, Maxim Muzafarov 
> wrote:
> > >>
> > >> Folks,
> > >>
> > >> I've created the release page on the Confluence. Feel free to mail in
> > >> case of any errors.
> > >>
> > >> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.10
> > >>
> > >> On Fri, 6 Nov 2020 at 17:56, Maxim Muzafarov 
> wrote:
> > >>>
> > >>> Igniters,
> > >>>
> > >>>
> > >>> Let's finalize the discussion [1] about the next upcoming major
> Apache
> > >>> Ignite 2.10  release. The major improvements related to the proposed
> > >>> release:
> > >>> - Improvements for partition clearing related parts
> > >>> - Add tracing of SQL queries.
> > >>> - CPP: Implement Cluster API
> > >>> - .NET: Thin client: Transactions
> > >>> - .NET: Thin Client: Continuous Query
> > >>> - Java Thin client Kubernetes discovery
> > >>>
> > >>> etc.
> > >>> Total: 166 RESOLVED issues [2].
> > >>>
> > >>>
> > >>> Let's start the discussion about Time and Scope, and also I propose
> > >>> myself as the release manager of the Apache Ignite 2.10. If you'd
> like
> > >>> to lead this release, please, let us know, I see no problems to chose
> > >>> a better candidate.
> > >>>
> > >>>
> > >>> Proposed release timeline:
> > >>>
> > >>> Scope Freeze: December 10, 2020
> > >>> Code Freeze: December 24, 2020
> > >>> Voting Date: January 18, 2021
> > >>> Release Date: January 25, 2021
> > >>>
> > >>>
> > >>> Proposed release scope:
> > >>> [2]
> > >>>
> > >>>
> > >>> WDYT?
> > >>>
> > >>>
> > >>> [1]
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/2-9-1-release-proposal-tp49769p49867.html
> > >>> [2]
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.10%20and%20status%20in%20(Resolved%2C%20Closed)%20and%20resolution%20%3D%20Fixed%20order%20by%20priority%20
> >
> >
>

Re: [DISCUSS] Missed (non-suited) tests

[Anton Vinogradov]

Folks,
What's the current state of this thread?
AFAIU, we found unused and wrongly located tests and developed some
checker, could we split this to some PRs?
Let's merge tests usage fix and location fixes first, this will provide us
an ability to automate check using Travis.

On Tue, Oct 20, 2020 at 12:06 PM Ivan Pavlukhin  wrote:

> Max, Ivan,
>
> Using explicit @Ignore and the automated check sounds good to me. If
> nobody has arguments against it I think we should do it.
>
> 2020-10-19 19:30 GMT+03:00, Max Timonin :
> > Hi Ivan,
> >
> > I've checked the ticket you provide. It contains subtasks to uncomment or
> > to remove some unused tests. It definitely describes some cases I've
> found.
> > So what do you think if I uncomment them in suites, add @Ignore
> annotation
> > for those tests while the tickets are open? This will help to find out
> > tests that were forgiven in a recent time.
> >
> > Also I believe that this check must be automated. I didn't find a way how
> > uncomment / unused tests are found in the ticket. If there is no any - I
> > propose mine PR for this purpose.
> >
> >
> >
> > On Mon, Oct 19, 2020 at 5:24 PM Ivan Daschinsky 
> > wrote:
> >
> >> Ivan, as far as I understand, Max also created verification check for
> not
> >> included test and found a few tests, that have never been included in
> any
> >> testsuites.
> >>
> >> Also, I suppose, that even if we cannot run some tests, these tests
> >> should
> >> be ignored using annotation, but not commented.
> >>
> >> пн, 19 окт. 2020 г. в 16:33, Ivan Pavlukhin :
> >>
> >> > Hi Max,
> >> >
> >> > There is an existing effort about "abandoned" tests
> >> > https://issues.apache.org/jira/browse/IGNITE-9210
> >> >
> >> > 2020-10-19 16:25 GMT+03:00, Max Timonin :
> >> > > Hi Igniters!
> >> > >
> >> > > I made a research into tests that aren't included in any test suite.
> >> > > As
> >> > > TeamCity runs tests by suites so there could be tests that never run
> >> > > on
> >> > TC.
> >> > > So I tried implementing a simple check for such tests and include it
> >> > > in
> >> > > Ignite's travis config.
> >> > >
> >> > > The check runs while "mvn test" command and piggy-backs on the maven
> >> > > surefire plugin. I replaced the junit provider with a custom one
> that
> >> > > checks if a class is a test or a suite (there are some Ignite
> >> > > specific
> >> > > stuff), marks tests that are in suites and raises an exception if
> >> > > there
> >> > are
> >> > > non-suited tests. It's implemented as a part of maven command so it
> >> runs
> >> > > for every module separately.
> >> > >
> >> > > I've prepared draft PR with this check:
> >> > > https://github.com/apache/ignite/pull/8367
> >> > > Travis check report is here:
> >> > > https://travis-ci.org/github/apache/ignite/jobs/737046387
> >> > > As It's a draft, so I skip some maven configuration steps for a
> >> > > while.
> >> > Also
> >> > > I run the check only for the core module.
> >> > >
> >> > > But I have some results that want to discuss before continue the
> >> > > work:
> >> > > 1. Currently in the core module there are 53 tests that aren't part
> >> > > of
> >> > any
> >> > > test suite. I'm not sure about the reason for every test. So I just
> >> > > put
> >> > > below a list of the tests and last contributor to a file that
> >> > > contains
> >> a
> >> > > test.
> >> > >
> >> > > 2. Some tests are located in the core module, but suites are in a
> >> > > different, for example ignite-indexing suite
> >> > > IgniteCacheQuerySelfTestSuite3 contains
> >> > > only tests written in the core module, and none from the indexing
> >> module.
> >> > > Also there are suites in spring, uri-deploy, zookeeper modules. In
> my
> >> PR
> >> > > I've just copied the test suites to the core module.
> >> > >
> >> > > 3. Some test classes are named with the "Abstract" suffix but don't
> >> have
> >> > > the corresponding modifier (for example,
> >> > > IgniteTxTimeoutAbstractTest).
> >> > So,
> >> > > I add the modifier for every such file if it's not a part of any
> >> > > suite.
> >> > >
> >> > > What do you think about this check? If Ignite needs it, let's
> discuss
> >> > next
> >> > > things:
> >> > > 1. Mark tests that should never be in any suite by some reason;
> >> > > 2. Fix the missed tests;
> >> > > 3. How to declare suites that contains tests from a different
> module;
> >> > > 4. How to check if TC runs all suites.
> >> > >
> >> > > List of non-suited tests in the core module:
> >> > >
> >> > > maksim.stepac...@gmail.com:
> >> > > GridTcpCommunicationSpiLogTest
> >> > >
> >> > > nizhi...@apache.org:
> >> > > IgniteCacheClientMultiNodeUpdateTopologyLockTest
> >> > > CacheClientsConcurrentStartTest
> >> > > IgniteOutOfMemoryPropagationTest
> >> > > GridCacheP2PUndeploySelfTest
> >> > > GridCacheRebalancingOrderingTest
> >> > > IgniteMassLoadSandboxTest
> >> > > PageLockTrackerMXBeanImplTest
> >> > > 

Re: [DISCUSS] Disable socket linger by default in TCP discovery SPI.

[Anton Vinogradov]

> When TLS 1.3 is introduced, whole sun.security.ssl.SSLSocketImpl was
> rewritten.
Correct, I meant rewritten TLSv1.3, the good news that 1.2- also were fixed.
so,
-- brand new TLS with any linger
-- plain old TLS with linger>0

On Fri, Oct 30, 2020 at 3:10 PM Ivan Daschinsky  wrote:

> Ilya, Anton.
> It means that not if TLS 1.3 is worked ok and with TLS < 1.2 is not ok.
>
> When TLS 1.3 is introduced, whole sun.security.ssl.SSLSocketImpl was
> rewritten.
> There is not any code anymore that could cause a deadlock.
> Therefore, in JDK, that supports TLS 1.3, this option is unnecessary, even
> if you use TLS 1.2
>
>
> пт, 30 окт. 2020 г. в 14:46, Anton Vinogradov :
>
> > Ilya
> > > I think we should still keep setting linger if SSL is enabled
> > Modern (updated) JVMs do not require this.
> > AFAIK, Problem caused this workaround already fixed everywhere, including
> > JDK 8.
> >
> > > If SSL only works with TLSv1.3 and no linger
> > SSL works if
> > -- TLSv1.3 with any linger
> > -- TLSv1.2- with linger>0
> >
> > > we should make TLSv1.3 a
> > > default. If JVM does not support it, users will have to reconfigure
> > > explicitly.
> > I don't think it's a good idea to reconfigure production environments
> this
> > way.
> >
> > P.s.
> > My +1 to zero linger as default + warning on SSL enabled on JVM before
> the
> > fix + warning at documentation + migration notes
> >
> > On Fri, Oct 30, 2020 at 2:19 PM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> > >
> > wrote:
> >
> > > Hello!
> > >
> > > I think we should still keep setting linger if SSL is enabled, and not
> > > expect user to enable it (or face consequences).
> > >
> > > If SSL only works with TLSv1.3 and no linger, we should make TLSv1.3 a
> > > default. If JVM does not support it, user will have to reconfigure
> > > explicitly.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > пт, 30 окт. 2020 г. в 14:05, Steshin Vladimir :
> > >
> > > > *
> > > >
> > > > Hi, Igniters.
> > > >
> > > > We’ve found that enabled by default socket linger causes unexpected
> > > > delay in detection of node failure.
> > > >
> > > >
> > > > Moreover, long closing of socket works as Thread.sleep() within
> > > > algorithms of failure detection and connection recovery in TCP
> > > > discovery. These time gaps lead to hardly predictable behavior of the
> > > > discovery. When the socket linger is enabled, it’s hard or even
> > > > impossible to figure out what time is taken to detect node failure
> and
> > > > restore connections with the provided settings.
> > > >
> > > > Socket linger was enabled only as a workaround for SSL bugs (i.e.
> [2],
> > > > [3]). It was enabled without including in failure processing routines
> > in
> > > > TCP discovery SPI as described above. SSL bugs, mentioned above, were
> > > > fixed and backported to various JDK, supporting TLS 1.3 ([4] and
> [5]).
> > > >
> > > >
> > > > I’d suggest to disable socket linger by default, because enabled
> socket
> > > > linger prolongs detection of node failure. The ticket is [1]. In case
> > of
> > > > SSL issues the linger could be enabled. Or one may just update JDK.
> > > > We'll provide the documentation.
> > > >
> > > > WDYT?
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-13643
> > > >
> > > > [2] https://bugs.openjdk.java.net/browse/JDK-8219658
> > > >
> > > > [3]https://issues.apache.org/jira/browse/IGNITE-12818
> > > >
> > > > [4]https://bugs.openjdk.java.net/browse/JDK-8245468
> > > >
> > > > [5]
> > https://www.oracle.com/java/technologies/javase/8u261-relnotes.html
> > > >
> > > > *
> > > >
> > >
> >
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>

Re: [DISCUSS] Disable socket linger by default in TCP discovery SPI.

[Anton Vinogradov]

Ilya
> I think we should still keep setting linger if SSL is enabled
Modern (updated) JVMs do not require this.
AFAIK, Problem caused this workaround already fixed everywhere, including
JDK 8.

> If SSL only works with TLSv1.3 and no linger
SSL works if
-- TLSv1.3 with any linger
-- TLSv1.2- with linger>0

> we should make TLSv1.3 a
> default. If JVM does not support it, users will have to reconfigure
> explicitly.
I don't think it's a good idea to reconfigure production environments this
way.

P.s.
My +1 to zero linger as default + warning on SSL enabled on JVM before the
fix + warning at documentation + migration notes

On Fri, Oct 30, 2020 at 2:19 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> I think we should still keep setting linger if SSL is enabled, and not
> expect user to enable it (or face consequences).
>
> If SSL only works with TLSv1.3 and no linger, we should make TLSv1.3 a
> default. If JVM does not support it, user will have to reconfigure
> explicitly.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 30 окт. 2020 г. в 14:05, Steshin Vladimir :
>
> > *
> >
> > Hi, Igniters.
> >
> > We’ve found that enabled by default socket linger causes unexpected
> > delay in detection of node failure.
> >
> >
> > Moreover, long closing of socket works as Thread.sleep() within
> > algorithms of failure detection and connection recovery in TCP
> > discovery. These time gaps lead to hardly predictable behavior of the
> > discovery. When the socket linger is enabled, it’s hard or even
> > impossible to figure out what time is taken to detect node failure and
> > restore connections with the provided settings.
> >
> > Socket linger was enabled only as a workaround for SSL bugs (i.e. [2],
> > [3]). It was enabled without including in failure processing routines in
> > TCP discovery SPI as described above. SSL bugs, mentioned above, were
> > fixed and backported to various JDK, supporting TLS 1.3 ([4] and [5]).
> >
> >
> > I’d suggest to disable socket linger by default, because enabled socket
> > linger prolongs detection of node failure. The ticket is [1]. In case of
> > SSL issues the linger could be enabled. Or one may just update JDK.
> > We'll provide the documentation.
> >
> > WDYT?
> >
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-13643
> >
> > [2] https://bugs.openjdk.java.net/browse/JDK-8219658
> >
> > [3]https://issues.apache.org/jira/browse/IGNITE-12818
> >
> > [4]https://bugs.openjdk.java.net/browse/JDK-8245468
> >
> > [5] https://www.oracle.com/java/technologies/javase/8u261-relnotes.html
> >
> > *
> >
>

Re: [DISCUSS] Ignite 3.0 development approach

[Anton Vinogradov]

Alexey,

Do we have any estimates of how fast we'll be able to gain production-ready
AI 3.0 in case of a "new repo" choice?

On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk 
wrote:

> Nikolay,
>
> What new features are we planning to implement for Ignite 2.x? I think once
> we commence working on Ignite 3.0, we should gradually cease the activity
> on Ignite 2.x to mere bugfixes because such parallel development will be
> overwhelming regardless of how we choose to proceed.
>
> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov :
>
> > To be clear:
> >
> > > I would suggest creating a new repository for Ignite 3.0 (perhaps, a
> new
> > clean branch, but a new repo looks nicer to me) and a new Ignite 3.0
> > TeamCity project.
> >
> > +1 for new Team City project.
> > +1 for new branch for Ignite3.
> > -1 for new repo.
> >
> >
> > > 2 нояб. 2020 г., в 13:35, Nikolay Izhikov 
> > написал(а):
> > >
> > > Hello, Alexey.
> > >
> > > I think it will hurt our project more than help.
> > > Developing new features for 2 separate branches with the different APIs
> > and internal structure is overwhelming
> > >
> > > Maybe we should relax a bit requirements for Ignite3?
> > > Maybe we should move step by step and make Ignite3 with new
> > configuration than Ignite4 with new transactions, etc?
> > >
> > >> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk <
> alexey.goncha...@gmail.com>
> > написал(а):
> > >>
> > >> Igniters,
> > >>
> > >> I wanted to pitch a rather radical idea regarding the Ignite 3.0
> > >> development which has occurred to me some time ago.
> > >>
> > >> We already have several IEPs targeted to Ignite 3.0 which imply major
> > >> changes to the codebase (the change in replication protocol and thus
> > >> transactions, change in binary format, updated metastorage, etc). We
> > also
> > >> planned significant changes in public APIs: configuration format
> change,
> > >> improvements in cache APIs, SQL APIs, transaction mode rework. The
> > wishlist
> > >> of changes for Ignite 3.0 is huge.
> > >>
> > >> So, I was wondering whether it makes sense to try to change the old
> > >> codebase, or start with a new baseline and move old pieces of code
> that
> > do
> > >> not require significant rework. Personally, I would go with the second
> > >> option for the following reasons:
> > >>
> > >>  - We have a chance to shift the development paradigm in the project
> and
> > >>  introduce the practice of true unit-tests. In the new baseline at the
> > >>  beginning there will be no ability to run an end-to-end scenario,
> thus
> > we
> > >>  will be forced to write unit-tests. So far, such practice was hard to
> > >>  implement because of tight coupling between Ignite components and
> > inability
> > >>  to instantiate components without an instance of KernalContext. For
> > >>  example, we should be able to thoroughly test internal primitives,
> > such as
> > >>  replication protocol (without actual communication), distributed
> > >>  metastorage contracts, persistence layer, etc.
> > >>  - We will significantly reduce the development cycle in the beginning
> > >>  (right now the RunAll takes two hours of astronomical time with empty
> > TC;
> > >>  in the new approach developer will be able to run ALL tests locally
> in
> > a
> > >>  matter of minutes)
> > >>  - We can get rid of TC bot and enforce green TC by integrating TC
> build
> > >>  results with GitHub PRs (the same way Travis is currently integrated
> > to PR
> > >>  check). We should restrict PR merge without a TC check
> > >>  - We will still have to re-write all tests, but only once. If we try
> to
> > >>  modify the old codebase, we would need to modify all the tests for
> > every
> > >>  major change (public API change, configuration change)
> > >>  - We will have fewer conflicts when working together. For example, I
> > >>  cannot imagine how one would merge two changes of getting rid of
> > >>  IgniteFuture and changes in replication protocol, for example
> > >>
> > >> Technically, I would suggest creating a new repository for Ignite 3.0
> > >> (perhaps, a new clean branch, but a new repo looks nicer to me) and a
> > new
> > >> Ignite 3.0 TeamCity project.
> > >>
> > >> While it may seem quite radical, I do believe that this approach will
> > give
> > >> us more benefits than trying to make such major changes in the
> existing
> > >> codebase. If needed, let's schedule a discord chat like before to
> > discuss
> > >> this.
> > >>
> > >> WDYT?
> > >
> >
> >
>

Re: [DISCUSS] Missed (non-suited) tests

[Anton Vinogradov]

> As I understand we
> can't just move suites between modules, as TeamCity may depend on the path
> to them.
See no problem to update TC as well.

On Fri, Oct 30, 2020 at 4:32 PM Ivan Daschinsky  wrote:

> I suggests to mark these tests with @Ignore and file tickets to fix them.
>
> пт, 30 окт. 2020 г. в 16:26, Ivan Daschinsky :
>
> > Hi
> >
> > WalCompactionAfterRestartTest -- yes we need it. This test failed because
> > of race (test shold be rewritten a little bit)
> >
> > пт, 30 окт. 2020 г. в 16:15, Max Timonin :
> >
> >> Hi!
> >>
> >> Yes, you're correct. I've developed the check and started to clean tests
> >> (move them to suites, mark some tests with Ignore, etc.). I finish work
> on
> >> the core module. I hope it was the biggest one, and others are less. If
> >> so,
> >> I think I will finish the work on other modules in 1 or 2 weeks, as I do
> >> this activity in the background (~10% of my work time). Actually I've
> >> found
> >> 3 failed tests in the core module that aren't in any suite, so I need
> time
> >> to discover reason of failures and if we actually need those tests:
> >>
> >> GridCacheMultithreadedFailoverAbstractTest
> >> WalCompactionAfterRestartTest
> >> GridTcpCommunicationSpiLogTest
> >>
> >> Also we should decide how to be with wrongly located es. As I understand
> >> we
> >> can't just move suites between modules, as TeamCity may depend on the
> path
> >> to them. So, for such cases I've just created suites in the right
> module,
> >> and replaced the test list with the new class suite. It does not look
> >> pretty enough, but I think It's a path of least resistance. WDYT?
> >>
> >> BEFORE:
> >> Module A -> SuiteA -> testA1, testA2, testB1, testB2
> >> Module B -> testB1, testB2
> >>
> >> AFTER:
> >> Module A -> SuiteA, SuiteB
> >> Module B -> SuiteB -> testB1, testB2
> >>
> >> On Fri, Oct 30, 2020 at 3:38 PM Anton Vinogradov  wrote:
> >>
> >> > Folks,
> >> > What's the current state of this thread?
> >> > AFAIU, we found unused and wrongly located tests and developed some
> >> > checker, could we split this to some PRs?
> >> > Let's merge tests usage fix and location fixes first, this will
> provide
> >> us
> >> > an ability to automate check using Travis.
> >> >
> >> > On Tue, Oct 20, 2020 at 12:06 PM Ivan Pavlukhin 
> >> > wrote:
> >> >
> >> > > Max, Ivan,
> >> > >
> >> > > Using explicit @Ignore and the automated check sounds good to me. If
> >> > > nobody has arguments against it I think we should do it.
> >> > >
> >> > > 2020-10-19 19:30 GMT+03:00, Max Timonin :
> >> > > > Hi Ivan,
> >> > > >
> >> > > > I've checked the ticket you provide. It contains subtasks to
> >> uncomment
> >> > or
> >> > > > to remove some unused tests. It definitely describes some cases
> I've
> >> > > found.
> >> > > > So what do you think if I uncomment them in suites, add @Ignore
> >> > > annotation
> >> > > > for those tests while the tickets are open? This will help to find
> >> out
> >> > > > tests that were forgiven in a recent time.
> >> > > >
> >> > > > Also I believe that this check must be automated. I didn't find a
> >> way
> >> > how
> >> > > > uncomment / unused tests are found in the ticket. If there is no
> >> any -
> >> > I
> >> > > > propose mine PR for this purpose.
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, Oct 19, 2020 at 5:24 PM Ivan Daschinsky <
> >> ivanda...@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > >> Ivan, as far as I understand, Max also created verification check
> >> for
> >> > > not
> >> > > >> included test and found a few tests, that have never been
> included
> >> in
> >> > > any
> >> > > >> testsuites.
> >> > > >>
> >> > > >> Also, I suppose, that even if we cannot run some tests, these
> tests
> >> > > >> should
> >> > > >> be

Re: [DISCUSS] Disable socket linger by default in TCP discovery SPI.

[Anton Vinogradov]

Folks,
Seems, we've got an agreement that the fix is necessary.
Do we need to do except the following?
>> zero linger as default + warning on SSL enabled on JVM before the fix +
warning at documentation + migration notes

On Tue, Nov 3, 2020 at 2:38 PM Steshin Vladimir  wrote:

>  Ilya, hi.
>
>
>  Of course: /TcpDiscoverySpi.setSoLinger(int)/ property. Always been.
>
>
> 02.11.2020 20:14, Ilya Kasnacheev пишет:
> > Hello!
> >
> > Is there any option to re-enable linger on SSL sockets?
> >
> > Telling people to re-configure does not help if they can't.
> >
> > Regards,
>

Re: [DISCUSS] Ignite 3.0 development approach

[Anton Vinogradov]

reaking changes in the persistence layer, configuration and
> > > > > >> behavior.
> > > > > >>>>>  Additionally, these components are now tightly coupled, so
> > there
> > > > is
> > > > > >> no
> > > > > >>>>> way
> > > > > >>>>>  these two changes can be implemented in parallel and then
> > merged
> > > > > >>>>> together
> > > > > >>>>>  easily. So what we will end up with is having to implement
> > these
> > > > > >>>> changes
> > > > > >>>>>  sequentially, fixing all existing tests twice, and
> essentially
> > > > > >>>> throwing
> > > > > >>>>>  away half of the work done because the other part of the
> > change
> > > is
> > > > > >>>>>  re-implemented
> > > > > >>>>>  - Similar example goes with getting rid of
> > IgniteInternalFuture
> > > > and
> > > > > >>>>>  replacing it with CompletableFuture, and any other change
> that
> > > > > >> touches
> > > > > >>>>> the
> > > > > >>>>>  asynchronous part of the code.
> > > > > >>>>>
> > > > > >>>>> Third, I do not see how this choice affects the UX of Ignite.
> > The
> > > > end
> > > > > >>>> user
> > > > > >>>>> experience must be fixed in the IEP regardless of the
> > development
> > > > > >> process
> > > > > >>>>> and the fact that we have gaps in this area in Ignite 2.x
> just
> > > > > confirms
> > > > > >>>>> that.
> > > > > >>>>>
> > > > > >>>>> Pavel, agree that a repo/branch is a technicality, I guess if
> > > > > >>>> reformulate,
> > > > > >>>>> my point is that we might agree to have a single development
> > > master
> > > > > >>>> branch
> > > > > >>>>> with 'disassembled' end-to-end functionality for some period
> of
> > > > time
> > > > > to
> > > > > >>>>> speed up development, and re-assemble the core features after
> > > > having
> > > > > >>>>> submodules tested independently.
> > > > > >>>>>
> > > > > >>>>> Nikolay,
> > > > > >>>>>> We have many features that have to evolve.
> > > > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support,
> > etc.
> > > > > >>>>> This is not very specific. In the end, resources are limited
> > and
> > > we
> > > > > >> will
> > > > > >>>>> not be able to drive both tracks simultaneously, especially
> > > after a
> > > > > >>>> couple
> > > > > >>>>> of features having been implemented for Ignite 3.0. If there
> > are
> > > > > indeed
> > > > > >>>>> some major changes that we want to do in Ignite 2.x instead
> of
> > > > > putting
> > > > > >>>>> effort into 3.0 - let's discuss them. I am just not aware of
> > any,
> > > > > >> that's
> > > > > >>>>> why I am eager to move forward with Ignite 3.0.
> > > > > >>>>>
> > > > > >>>>>> We have bugs and issues that can be fixed in 2.x without
> > > breaking
> > > > > >>>> backward
> > > > > >>>>> compatibility.
> > > > > >>>>>> We have many users who are happy with the 2.x with all it’s
> > > > issues.
> > > > > >>>>> These changes will be covered by end-to-end tests and
> migrated
> > to
> > > > > >> Ignite
> > > > > >>>>> 3.0, so I see no issues here.
> > > > > >>>>>
> > > > > >>>>> Finally, Anton & Nikolay
> > > > > >>>>> I do not have an estimate for this simply because the
> activity

Re: [DISCUSSION] Code style. Variable abbrevations

[Anton Vinogradov]

>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I understand that this rule seems too strict or may be weird. But
> >>>>>>> removing the rule could lead to review comments like "use future
> >>>>>>> instead of fut". So my proposal is to change rule from "required"
> to
> >>>>>>> "recommended".
> >>>>>>>
> >>>>>>> On Wed, Jun 16, 2021 at 2:49 PM Valentin Kulichenko
> >>>>>>>  wrote:
> >>>>>>>>
> >>>>>>>> Konstantin, thanks for chiming in.
> >>>>>>>>
> >>>>>>>> That's exactly my concern. Overformalization is generally not a
> good
> >>>>>>>> thing.
> >>>>>>>> Someone used "mess" to abbreviate "message"? That is surely not a
> >> good
> >>>>>>>> coding style, but that's what code reviews are for. I believe that
> >> our
> >>>>>>>> committers are more than capable of identifying such issues.
> >>>>>>>>
> >>>>>>>> At the same time, with the current rules, we are *forced* to use
> >>>>>>>> abbreviations like "locAddrGrpMgr", whether we like it or not. All
> >> it
> >>>>>>>> does
> >>>>>>>> is makes it harder to contribute to Ignite, without providing any
> >>>>>>>> clear
> >>>>>>>> value.
> >>>>>>>>
> >>>>>>>> -Val
> >>>>>>>>
> >>>>>>>> On Thu, Jun 10, 2021 at 9:46 AM Konstantin Orlov <
> >> kor...@gridgain.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> +1 for making this optional
> >>>>>>>>>
> >>>>>>>>> Common abbreviation rules is good for sure, but sometimes I
> getting
> >>>>>>>>> sick
> >>>>>>>>> of all those locAddrGrpMgr.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Regards,
> >>>>>>>>> Konstantin Orlov
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On 7 Jun 2021, at 14:31, Nikolay Izhikov 
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hello, Anton, Alexei.
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the feedback.
> >>>>>>>>>>
> >>>>>>>>>> Personally, I’m pretty happy current abbreviation rules too.
> >>>>>>>>>> Let see what we can do to make our codebase even more
> consistent.
> >>>>>>>>>>
> >>>>>>>>>>> 7 июня 2021 г., в 13:23, Alexei Scherbakov <
> >>>>>>>>> alexey.scherbak...@gmail.com> написал(а):
> >>>>>>>>>>>
> >>>>>>>>>>> -1
> >>>>>>>>>>> Common abbrevs add quality to the code.
> >>>>>>>>>>>
> >>>>>>>>>>> пн, 7 июн. 2021 г. в 12:38, Anton Vinogradov :
> >>>>>>>>>>>
> >>>>>>>>>>>> -1 here.
> >>>>>>>>>>>> We can fix the code and set up the rule.
> >>>>>>>>>>>>
> >>>>>>>>>>>> This will help to prevent having a weird abbreviation like
> >> "mess"
> >>>>>>>>>>>> (from
> >>>>>>>>>>>> "message") or "ign" (from "Ignite").
> >>>>>>>>>>>> Also, the abbreviations list (hardcoded at IDEA plugin) allows
> >> to
> >>>>>>>>>>>> have
> >>>>>>>>> same
>

Re: [DISCUSSION] Brand new distributed environment testing framework ready to be reviewed/merged.

[Anton Vinogradov]

Folks,

Here's the video [1] that explains the proposal in detail.
Feel free to ask questions here.

[1] https://www.youtube.com/watch?v=f-i9COU5uAQ (in Russian)

On Tue, Jun 8, 2021 at 2:51 PM Anton Vinogradov  wrote:

> Igniters,
> Let me present a framework, we developed, that allows automating Apache
> Ignite testing on a real cluster.
> The framework was initially presented at Ignite Summit.
>
> In brief,
> The framework allows automating operations with any applications on a real
> cluster using ssh in a form of a python test.
>
> Features:
> - Ignite nodes can be started/stopped on a Docker or a real cluster with
> any custom configuration
> - Any AI version is supported (released or compiled from sources)
> - Ignite forks are also supported «out of the box»
> - Any other application execution is also possible, eg. we implemented
> starters for Spark and Zookeeper
> - Cluster can be managed using the control.sh, we made this a part of the
> test API
> - Custom Java applications can be executed remotely with/without a
> built-in Ignite node or a Thin client
> - Any ssh command can be executed remotely, and the result will be
> available locally (at the python test)
> - Network can be broken by iptables change to check communication issues
> - Tests can be executed in parallel when cluster size bigger than tests
> requirements
>
> As a result, seems, we may automate any testing scenario we can even
> imagine.
>
> Framework based on Ducktape [1] library from Kafka team, that's why we
> called it Ducktests.
>
> The Ducktests were developed during work on IEP-56 [2] and already were
> used during the work on IEP-45 [3].
> IEP-45 measurement results examples [4.1] [4.2] were demonstrated at the
> HighLoad conference last month.
>
> Code available at PR-9117 [5] and ready to be reviewed/merged.
> Feel free to ask questions or make proposals.
>
> [1] https://ducktape-docs.readthedocs.io/en/latest/index.html
> [2]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-56%3A+Distributed+environment+tests
> [3]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
> [4.1] https://youtu.be/UZsvCNjbkww?t=1065 (in Russian)
> [4.2] https://youtu.be/UZsvCNjbkww?t=1609 (in Russian)
> [5] https://github.com/apache/ignite/pull/9117
>

[DISCUSSION] Brand new distributed environment testing framework ready to be reviewed/merged.

[Anton Vinogradov]

Igniters,
Let me present a framework, we developed, that allows automating Apache
Ignite testing on a real cluster.
The framework was initially presented at Ignite Summit.

In brief,
The framework allows automating operations with any applications on a real
cluster using ssh in a form of a python test.

Features:
- Ignite nodes can be started/stopped on a Docker or a real cluster with
any custom configuration
- Any AI version is supported (released or compiled from sources)
- Ignite forks are also supported «out of the box»
- Any other application execution is also possible, eg. we implemented
starters for Spark and Zookeeper
- Cluster can be managed using the control.sh, we made this a part of the
test API
- Custom Java applications can be executed remotely with/without a built-in
Ignite node or a Thin client
- Any ssh command can be executed remotely, and the result will be
available locally (at the python test)
- Network can be broken by iptables change to check communication issues
- Tests can be executed in parallel when cluster size bigger than tests
requirements

As a result, seems, we may automate any testing scenario we can even
imagine.

Framework based on Ducktape [1] library from Kafka team, that's why we
called it Ducktests.

The Ducktests were developed during work on IEP-56 [2] and already were
used during the work on IEP-45 [3].
IEP-45 measurement results examples [4.1] [4.2] were demonstrated at the
HighLoad conference last month.

Code available at PR-9117 [5] and ready to be reviewed/merged.
Feel free to ask questions or make proposals.

[1] https://ducktape-docs.readthedocs.io/en/latest/index.html
[2]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-56%3A+Distributed+environment+tests
[3]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
[4.1] https://youtu.be/UZsvCNjbkww?t=1065 (in Russian)
[4.2] https://youtu.be/UZsvCNjbkww?t=1609 (in Russian)
[5] https://github.com/apache/ignite/pull/9117

Re: [DISCUSSION] Code style. Variable abbrevations

[Anton Vinogradov]

-1 here.
We can fix the code and set up the rule.

This will help to prevent having a weird abbreviation like "mess" (from
"message") or "ign" (from "Ignite").
Also, the abbreviations list (hardcoded at IDEA plugin) allows to have same
names across the whole code, this should simplify the reading.

On Sat, Jun 5, 2021 at 10:49 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> I also support removing this requirement. It’s not the first time someone
> brings this up, and so far we haven’t been able to fix it. Not worth it in
> my view.
>
> -Val
>
> On Sat, Jun 5, 2021 at 11:54 AM Nikolay Izhikov 
> wrote:
>
> > Hello, guys.
> >
> > Thanks for the feedback.
> >
> > Dmitry,
> >
> > > Manual rule enforcement saves a couple of symbols in code
> >
> > I’m talking about automatic check.
> > We can implement it easily.
> > So, if we decide to keep this rule all we need is to fix current
> > violations (several thousand!).
> > After it, checkstyle will automatically enforce this rule.
> > As you may know, currently, any PR checked according to our checkstyle
> > rules.
> > Please, take a look at little green check sign after PR name.
> >
> >
> > > 5 июня 2021 г., в 00:57, Dmitry Pavlov 
> написал(а):
> > >
> > > +1 for removal. Cnt and count both seem to be fine.
> > >
> > > Manual rule enforcement saves a couple of symbols in code, but requires
> > some time from both contributor and reviewer.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > On 2021/06/04 19:18:37, Pavel Tupitsyn  wrote:
> > >> In my opinion, we should remove this rule.
> > >> Looks like a waste of time. freq or frequency, cnt or count, it is
> fine
> > >> either way.
> > >>
> > >> On Fri, Jun 4, 2021 at 7:39 PM Nikolay Izhikov 
> > wrote:
> > >>
> > >>> Hello, Igniters.
> > >>>
> > >>> Right now, we have the rule to use some predefined list of
> abbrevation
> > for
> > >>> variable names [1].
> > >>> Some of the reviewers ask to follow this rule strictly.
> > >>>
> >  It is required to use abbreviated form for code consistency.
> > >>>
> > >>> I tried to implement this rule in form of checkstyle check [2] and it
> > >>> seems like many of use doesn’t follow this requirement.
> > >>> My check found 4124 violation in core module.
> > >>>
> > >>> Should we make this rule optional in documentation or should we
> remove
> > it
> > >>> completely?
> > >>> Or should someone proceed and fix all the violations?
> > >>>
> > >>> WDYT?
> > >>>
> > >>>
> > >>> Example of output:
> > >>> ```
> > >>> [ERROR]
> > >>>
> >
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:94:
> > >>> Abbrevation should be used for CACHE_NAME_LOCAL_ATOMIC! Please, use
> > loc,
> > >>> instead of LOCAL [IgniteAbbrevationsRule]
> > >>> [ERROR]
> > >>>
> >
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:97:
> > >>> Abbrevation should be used for CACHE_NAME_LOCAL_TX! Please, use loc,
> > >>> instead of LOCAL [IgniteAbbrevationsRule]
> > >>> [ERROR]
> > >>>
> >
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:264:
> > >>> Abbrevation should be used for checkpointManager! Please, use mgr,
> > instead
> > >>> of Manager [IgniteAbbrevationsRule]
> > >>> [ERROR]
> > >>>
> >
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsPartitionPreloadTest.java:63:
> > >>> Abbrevation should be used for DEFAULT_REGION! Please, use dflt,
> > instead of
> > >>> DEFAULT [IgniteAbbrevationsRule]
> > >>> [ERROR]
> > >>>
> >
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWholeClusterRestartTest.java:47:
> > >>> Abbrevation should be used for ENTRIES_COUNT! Please, use cnt,
> instead
> > of
> > >>> COUNT [IgniteAbbrevationsRule]
> > >>> [ERROR]
> > >>>
> >
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsRebalancingOnNotStableTopologyTest.java:49:
> > >>> Abbrevation should be used for CHECKPOINT_FREQUENCY! Please, use
> freq,
> > >>> instead of FREQUENCY [IgniteAbbrevationsRule]
> > >>> [ERROR]
> > >>>
> >
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsTransactionsHangTest.java:75:
> > >>> Abbrevation should be used for MAX_KEY_COUNT! Please, use cnt,
> instead
> > of
> > >>> COUNT [IgniteAbbrevationsRule]
> > >>> [ERROR]
> > >>>
> >
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsTransactionsHangTest.java:78:
> > >>> Abbrevation should be used for CHECKPOINT_FREQUENCY! Please, use
> 

Re: [DISCUSSION] Brand new distributed environment testing framework ready to be reviewed/merged.

[Anton Vinogradov]

Denis,

Unfortunately, we had some issues with this recording (low slides recording
resolution, missed slides and random slides switch delay), which makes it
slightly unwatchable :(
Anyway, if you brave enough, you may open the slides [1] and the recording
[2] simultaneously to solve this :)

[1]
https://go.gridgain.com/rs/491-TWR-806/images/11_IgniteSummit2021-Deep-dive-into-testing.pdf
[2] https://www.youtube.com/watch?v=uRRlGrSA3NY

On Wed, Jun 23, 2021 at 5:40 PM Denis Magda  wrote:

> Congrats, Anton, that's a valuable contribution! I attended your session at
> the Ignite Summit and wonder if you should share that recording with an
> English-speaking part of the community?
>
> -
> Denis
>
> On Wed, Jun 23, 2021 at 7:37 AM Anton Vinogradov  wrote:
>
> > Folks,
> >
> > Here's the video [1] that explains the proposal in detail.
> > Feel free to ask questions here.
> >
> > [1] https://www.youtube.com/watch?v=f-i9COU5uAQ (in Russian)
> >
> > On Tue, Jun 8, 2021 at 2:51 PM Anton Vinogradov  wrote:
> >
> > > Igniters,
> > > Let me present a framework, we developed, that allows automating Apache
> > > Ignite testing on a real cluster.
> > > The framework was initially presented at Ignite Summit.
> > >
> > > In brief,
> > > The framework allows automating operations with any applications on a
> > real
> > > cluster using ssh in a form of a python test.
> > >
> > > Features:
> > > - Ignite nodes can be started/stopped on a Docker or a real cluster
> with
> > > any custom configuration
> > > - Any AI version is supported (released or compiled from sources)
> > > - Ignite forks are also supported «out of the box»
> > > - Any other application execution is also possible, eg. we implemented
> > > starters for Spark and Zookeeper
> > > - Cluster can be managed using the control.sh, we made this a part of
> the
> > > test API
> > > - Custom Java applications can be executed remotely with/without a
> > > built-in Ignite node or a Thin client
> > > - Any ssh command can be executed remotely, and the result will be
> > > available locally (at the python test)
> > > - Network can be broken by iptables change to check communication
> issues
> > > - Tests can be executed in parallel when cluster size bigger than tests
> > > requirements
> > >
> > > As a result, seems, we may automate any testing scenario we can even
> > > imagine.
> > >
> > > Framework based on Ducktape [1] library from Kafka team, that's why we
> > > called it Ducktests.
> > >
> > > The Ducktests were developed during work on IEP-56 [2] and already were
> > > used during the work on IEP-45 [3].
> > > IEP-45 measurement results examples [4.1] [4.2] were demonstrated at
> the
> > > HighLoad conference last month.
> > >
> > > Code available at PR-9117 [5] and ready to be reviewed/merged.
> > > Feel free to ask questions or make proposals.
> > >
> > > [1] https://ducktape-docs.readthedocs.io/en/latest/index.html
> > > [2]
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-56%3A+Distributed+environment+tests
> > > [3]
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
> > > [4.1] https://youtu.be/UZsvCNjbkww?t=1065 (in Russian)
> > > [4.2] https://youtu.be/UZsvCNjbkww?t=1609 (in Russian)
> > > [5] https://github.com/apache/ignite/pull/9117
> > >
> >
>

Re: [DISCUSSION] Brand new distributed environment testing framework ready to be reviewed/merged.

[Anton Vinogradov]

With regard to the request of Ivan Daschinskiy checked that we are able to
release code from the PR.
Looks like everything [1] is fine, ignite-ducketests are not a part of the
zip file.

[1]
https://ci.ignite.apache.org/buildConfiguration/Releases_ApacheIgniteMain_ReleaseBuild/6064064?buildTab=artifacts#%2Frelease-2.11.0-rc0.zip!%2Fmaven;%2Frelease-2.11.0-rc0.zip!%2Fpackages
(you may need special permission to open this page)

On Mon, Jun 28, 2021 at 11:01 AM Nikolay Izhikov 
wrote:

> Hello.
>
> As one of the authors of PR fully support merge of this module.
> Have plans to use it during development of other features.
> Guys, please, share your feedback.
>
> Do we need to improve something prior merge?
>
>
> > 24 июня 2021 г., в 14:41, Denis Magda  написал(а):
> >
> > Anyway, it's much more community-friendly. Thanks, Anton!
> >
> > --
> > Denis
> >
> > -
> > Denis
> >
> > On Thu, Jun 24, 2021 at 3:43 AM Anton Vinogradov  wrote:
> >
> >> Denis,
> >>
> >> Unfortunately, we had some issues with this recording (low slides
> recording
> >> resolution, missed slides and random slides switch delay), which makes
> it
> >> slightly unwatchable :(
> >> Anyway, if you brave enough, you may open the slides [1] and the
> recording
> >> [2] simultaneously to solve this :)
> >>
> >> [1]
> >>
> >>
> https://go.gridgain.com/rs/491-TWR-806/images/11_IgniteSummit2021-Deep-dive-into-testing.pdf
> >> [2] https://www.youtube.com/watch?v=uRRlGrSA3NY
> >>
> >> On Wed, Jun 23, 2021 at 5:40 PM Denis Magda  wrote:
> >>
> >>> Congrats, Anton, that's a valuable contribution! I attended your
> session
> >> at
> >>> the Ignite Summit and wonder if you should share that recording with an
> >>> English-speaking part of the community?
> >>>
> >>> -
> >>> Denis
> >>>
> >>> On Wed, Jun 23, 2021 at 7:37 AM Anton Vinogradov 
> wrote:
> >>>
> >>>> Folks,
> >>>>
> >>>> Here's the video [1] that explains the proposal in detail.
> >>>> Feel free to ask questions here.
> >>>>
> >>>> [1] https://www.youtube.com/watch?v=f-i9COU5uAQ (in Russian)
> >>>>
> >>>> On Tue, Jun 8, 2021 at 2:51 PM Anton Vinogradov 
> wrote:
> >>>>
> >>>>> Igniters,
> >>>>> Let me present a framework, we developed, that allows automating
> >> Apache
> >>>>> Ignite testing on a real cluster.
> >>>>> The framework was initially presented at Ignite Summit.
> >>>>>
> >>>>> In brief,
> >>>>> The framework allows automating operations with any applications on a
> >>>> real
> >>>>> cluster using ssh in a form of a python test.
> >>>>>
> >>>>> Features:
> >>>>> - Ignite nodes can be started/stopped on a Docker or a real cluster
> >>> with
> >>>>> any custom configuration
> >>>>> - Any AI version is supported (released or compiled from sources)
> >>>>> - Ignite forks are also supported «out of the box»
> >>>>> - Any other application execution is also possible, eg. we
> >> implemented
> >>>>> starters for Spark and Zookeeper
> >>>>> - Cluster can be managed using the control.sh, we made this a part of
> >>> the
> >>>>> test API
> >>>>> - Custom Java applications can be executed remotely with/without a
> >>>>> built-in Ignite node or a Thin client
> >>>>> - Any ssh command can be executed remotely, and the result will be
> >>>>> available locally (at the python test)
> >>>>> - Network can be broken by iptables change to check communication
> >>> issues
> >>>>> - Tests can be executed in parallel when cluster size bigger than
> >> tests
> >>>>> requirements
> >>>>>
> >>>>> As a result, seems, we may automate any testing scenario we can even
> >>>>> imagine.
> >>>>>
> >>>>> Framework based on Ducktape [1] library from Kafka team, that's why
> >> we
> >>>>> called it Ducktests.
> >>>>>
> >>>>> The Ducktests were developed during work on IEP-56 [2] and already
> >> were
> >>>>> used during the work on IEP-45 [3].
> >>>>> IEP-45 measurement results examples [4.1] [4.2] were demonstrated at
> >>> the
> >>>>> HighLoad conference last month.
> >>>>>
> >>>>> Code available at PR-9117 [5] and ready to be reviewed/merged.
> >>>>> Feel free to ask questions or make proposals.
> >>>>>
> >>>>> [1] https://ducktape-docs.readthedocs.io/en/latest/index.html
> >>>>> [2]
> >>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-56%3A+Distributed+environment+tests
> >>>>> [3]
> >>>>>
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
> >>>>> [4.1] https://youtu.be/UZsvCNjbkww?t=1065 (in Russian)
> >>>>> [4.2] https://youtu.be/UZsvCNjbkww?t=1609 (in Russian)
> >>>>> [5] https://github.com/apache/ignite/pull/9117
> >>>>>
> >>>>
> >>>
> >>
>
>

Re: [VOTE][EXTENSION] Release Apache Ignite spring-data-all-ext extensions 1.0.0 RC4

[Anton Vinogradov]

Folks, we already have Ignite release semi-automated at TeamCity.
We able to assembly, publish as RC and check published.
Seems, we must do the same at extensions.
This way we'll check issues like this even before starting the vote.

On Mon, Jun 28, 2021 at 2:14 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> -1 (binding)
>
> The root element of the source zip file should be
> ignite-spring-data-all-ext-1.0.0-src/ directory. No other directories/files
> should be at the root of the source zip file.
>
> This is a *blocker* but should be very straightforward to fix.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 25 июн. 2021 г. в 17:46, Sergei Ryzhov :
>
> > +1
> > Building modules from source and running tests successfully on Java 8
> >
> > пт, 25 июн. 2021 г. в 16:52, Alex Plehanov :
> >
> > > +1
> > >
> > > Check sha512, sign of archives and versions of released artifacts (BTW,
> > > version of parent artifact is 1.0.0-SNAPSHOT, I don't think it's a
> > blocker
> > > now, but it's better to be fixed in the next releases).
> > > Check licenses with "mvn validate -DskipTests -P check-licenses"
> > > Build modules from the source (Note: build successfully on Java 8, but
> > > build on Java 11 has failed due to maven-javadoc-plugin. To build on
> Java
> > > 11 I have used this command: "mvn clean install -DskipTests
> > > -Dmaven.javadoc.skip=true")
> > > Run examples (create new maven project and copy sources from examples
> > > directories):
> > > from modules/spring-data-2.0-ext/examples with artifacts: Ignite
> > > v2.10.0/ignite-spring-data-2.0-ext v1.0.0/spring-data v2.0
> > > from modules/spring-data-2.2-ext/examples with artifacts: Ignite
> > > v2.10.0/ignite-spring-data-2.2-ext v1.0.0/spring-data v2.2
> > >
> > > пт, 25 июн. 2021 г. в 16:26, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> > >:
> > >
> > > > Hello!
> > > >
> > > > The contents of the source package are much more reasonable now.
> > However,
> > > > there's still an issue that root element of zip archive is not a
> > > directory:
> > > > ~/Downloads/spring% unzip ignite-spring-data-all-ext-1.0.0-src.zip
> > > > Archive:  ignite-spring-data-all-ext-1.0.0-src.zip
> > > >   creating: checkstyle/
> > > >  inflating: checkstyle/checkstyle-suppressions.xml
> > > >  inflating: checkstyle/checkstyle.xml
> > > >   creating: config/
> > > >  inflating: config/example-ignite.xml
> > > >   creating: modules/
> > > >   creating: modules/spring-data-2.0-ext/
> > > >  inflating: modules/spring-data-2.0-ext/README.txt
> > > >
> > > >
> > > > whereas
> > > > ~/Downloads% unzip -l apache-ignite-2.8.1-src.zip| head
> > > > Archive:  apache-ignite-2.8.1-src.zip
> > > > 864220966caa4157c4fee8a1bc85171623963604
> > > >  Length  DateTimeName
> > > > -  -- -   
> > > >0  2020-05-21 16:42   apache-ignite-2.8.1-src/
> > > > 2029  2020-05-21 16:42   apache-ignite-2.8.1-src/.gitignore
> > > >0  2020-05-21 16:42   apache-ignite-2.8.1-src/.idea/
> > > >0  2020-05-21 16:42
> > > >   apache-ignite-2.8.1-src/.idea/inspectionProfiles/
> > > >80905  2020-05-21 16:42
> > > >
>  apache-ignite-2.8.1-src/.idea/inspectionProfiles/Project_Default.xml
> > > > 5119  2020-05-21 16:42   apache-ignite-2.8.1-src/CONTRIBUTING.md
> > > >
> > > >
> > > > The root element of source sip should
> > > > be ignite-spring-data-all-ext-1.0.0-src/ directory.
> > > >
> > > > This is a small issue but a *blocker*, maybe you can do a quick
> > rebuild?
> > > >
> > > > Another issue is that I don't understand how to run examples. If I
> add
> > > the
> > > > entire directory as maven project, the examples/main directory is not
> > > > considered source root and will not be built/run
> > > > Maybe I'm missing something.
> > > >
> > > > Can you add is as an additional sources root, perhaps? You can even
> > > build a
> > > > jar with classifier 'examples' if that's OK.
> > > >
> > > > Regards,
> > > >
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > пт, 25 июн. 2021 г. в 14:17, Mikhail Petrov :
> > > >
> > > > > +1
> > > > >
> > > > > Checked on Ubuntu 20.04. Created separate Spring Applications for
> > each
> > > > > version of Spring Data Integrations using the Maven RC repository
> and
> > > > > following the documentation. Tested common use cases.
> > > > >
> > > > > On 25.06.2021 12:43, Nikita Amelchev wrote:
> > > > > > Dear Ignite Community,
> > > > > >
> > > > > > I have uploaded a release candidate of the following extension
> > > modules:
> > > > > >
> > > > > > spring-data-commons
> > > > > > spring-data-ext
> > > > > > spring-data-2.0-ext
> > > > > > spring-data-2.2-ext
> > > > > >
> > > > > > The release candidate of the extensions:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-spring-data-all-ext/1.0.0-rc4/
> > > > > >
> > > > > > The instruction for building from the source package is placed in
> > the
> > > > > > DEVNOTES file.
> > > > > >
> > 

Re: [DISCUSSION] Brand new distributed environment testing framework ready to be reviewed/merged.

[Anton Vinogradov]

Seems, we have a silent agreement here :)
We also discussed the proposal with the Ignite QA Meetup participants and
found this contribution useful.

Going to merge tomorrow, if there are no objections.

On Mon, Jun 28, 2021 at 3:21 PM Anton Vinogradov  wrote:

> With regard to the request of Ivan Daschinskiy checked that we are able to
> release code from the PR.
> Looks like everything [1] is fine, ignite-ducketests are not a part of the
> zip file.
>
> [1]
> https://ci.ignite.apache.org/buildConfiguration/Releases_ApacheIgniteMain_ReleaseBuild/6064064?buildTab=artifacts#%2Frelease-2.11.0-rc0.zip!%2Fmaven;%2Frelease-2.11.0-rc0.zip!%2Fpackages
> (you may need special permission to open this page)
>
> On Mon, Jun 28, 2021 at 11:01 AM Nikolay Izhikov 
> wrote:
>
>> Hello.
>>
>> As one of the authors of PR fully support merge of this module.
>> Have plans to use it during development of other features.
>> Guys, please, share your feedback.
>>
>> Do we need to improve something prior merge?
>>
>>
>> > 24 июня 2021 г., в 14:41, Denis Magda  написал(а):
>> >
>> > Anyway, it's much more community-friendly. Thanks, Anton!
>> >
>> > --
>> > Denis
>> >
>> > -
>> > Denis
>> >
>> > On Thu, Jun 24, 2021 at 3:43 AM Anton Vinogradov  wrote:
>> >
>> >> Denis,
>> >>
>> >> Unfortunately, we had some issues with this recording (low slides
>> recording
>> >> resolution, missed slides and random slides switch delay), which makes
>> it
>> >> slightly unwatchable :(
>> >> Anyway, if you brave enough, you may open the slides [1] and the
>> recording
>> >> [2] simultaneously to solve this :)
>> >>
>> >> [1]
>> >>
>> >>
>> https://go.gridgain.com/rs/491-TWR-806/images/11_IgniteSummit2021-Deep-dive-into-testing.pdf
>> >> [2] https://www.youtube.com/watch?v=uRRlGrSA3NY
>> >>
>> >> On Wed, Jun 23, 2021 at 5:40 PM Denis Magda  wrote:
>> >>
>> >>> Congrats, Anton, that's a valuable contribution! I attended your
>> session
>> >> at
>> >>> the Ignite Summit and wonder if you should share that recording with
>> an
>> >>> English-speaking part of the community?
>> >>>
>> >>> -
>> >>> Denis
>> >>>
>> >>> On Wed, Jun 23, 2021 at 7:37 AM Anton Vinogradov 
>> wrote:
>> >>>
>> >>>> Folks,
>> >>>>
>> >>>> Here's the video [1] that explains the proposal in detail.
>> >>>> Feel free to ask questions here.
>> >>>>
>> >>>> [1] https://www.youtube.com/watch?v=f-i9COU5uAQ (in Russian)
>> >>>>
>> >>>> On Tue, Jun 8, 2021 at 2:51 PM Anton Vinogradov 
>> wrote:
>> >>>>
>> >>>>> Igniters,
>> >>>>> Let me present a framework, we developed, that allows automating
>> >> Apache
>> >>>>> Ignite testing on a real cluster.
>> >>>>> The framework was initially presented at Ignite Summit.
>> >>>>>
>> >>>>> In brief,
>> >>>>> The framework allows automating operations with any applications on
>> a
>> >>>> real
>> >>>>> cluster using ssh in a form of a python test.
>> >>>>>
>> >>>>> Features:
>> >>>>> - Ignite nodes can be started/stopped on a Docker or a real cluster
>> >>> with
>> >>>>> any custom configuration
>> >>>>> - Any AI version is supported (released or compiled from sources)
>> >>>>> - Ignite forks are also supported «out of the box»
>> >>>>> - Any other application execution is also possible, eg. we
>> >> implemented
>> >>>>> starters for Spark and Zookeeper
>> >>>>> - Cluster can be managed using the control.sh, we made this a part
>> of
>> >>> the
>> >>>>> test API
>> >>>>> - Custom Java applications can be executed remotely with/without a
>> >>>>> built-in Ignite node or a Thin client
>> >>>>> - Any ssh command can be executed remotely, and the result will be
>> >>>>> available locally (at the python test)
>> >>>>> - Network can be broken by iptables change to check communication
>> >>> issues
>> >>>>> - Tests can be executed in parallel when cluster size bigger than
>> >> tests
>> >>>>> requirements
>> >>>>>
>> >>>>> As a result, seems, we may automate any testing scenario we can even
>> >>>>> imagine.
>> >>>>>
>> >>>>> Framework based on Ducktape [1] library from Kafka team, that's why
>> >> we
>> >>>>> called it Ducktests.
>> >>>>>
>> >>>>> The Ducktests were developed during work on IEP-56 [2] and already
>> >> were
>> >>>>> used during the work on IEP-45 [3].
>> >>>>> IEP-45 measurement results examples [4.1] [4.2] were demonstrated at
>> >>> the
>> >>>>> HighLoad conference last month.
>> >>>>>
>> >>>>> Code available at PR-9117 [5] and ready to be reviewed/merged.
>> >>>>> Feel free to ask questions or make proposals.
>> >>>>>
>> >>>>> [1] https://ducktape-docs.readthedocs.io/en/latest/index.html
>> >>>>> [2]
>> >>>>>
>> >>>>
>> >>>
>> >>
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-56%3A+Distributed+environment+tests
>> >>>>> [3]
>> >>>>>
>> >>>>
>> >>>
>> >>
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up
>> >>>>> [4.1] https://youtu.be/UZsvCNjbkww?t=1065 (in Russian)
>> >>>>> [4.2] https://youtu.be/UZsvCNjbkww?t=1609 (in Russian)
>> >>>>> [5] https://github.com/apache/ignite/pull/9117
>> >>>>>
>> >>>>
>> >>>
>> >>
>>
>>

Re: [DISCUSSION] Brand new distributed environment testing framework ready to be reviewed/merged.

[Anton Vinogradov]

Ducktests were merged, thanks to everyone involved!

Co-authored-by: Nikolay Izhikov
Co-authored-by: Ivan Daschinskiy
Co-authored-by: Maksim Timonin
Co-authored-by: Vladsz83
Co-authored-by: oleg-ostanin
Co-authored-by: SwirMix
Co-authored-by: Sergei Ryzhov
Co-authored-by: Mikhail Filatov
Co-authored-by: Dmitriy Sorokin
Co-authored-by: emvdovets

On Mon, Jun 28, 2021 at 6:01 PM Anton Vinogradov  wrote:

> Seems, we have a silent agreement here :)
> We also discussed the proposal with the Ignite QA Meetup participants and
> found this contribution useful.
>
> Going to merge tomorrow, if there are no objections.
>
> On Mon, Jun 28, 2021 at 3:21 PM Anton Vinogradov  wrote:
>
>> With regard to the request of Ivan Daschinskiy checked that we are able
>> to release code from the PR.
>> Looks like everything [1] is fine, ignite-ducketests are not a part of
>> the zip file.
>>
>> [1]
>> https://ci.ignite.apache.org/buildConfiguration/Releases_ApacheIgniteMain_ReleaseBuild/6064064?buildTab=artifacts#%2Frelease-2.11.0-rc0.zip!%2Fmaven;%2Frelease-2.11.0-rc0.zip!%2Fpackages
>> (you may need special permission to open this page)
>>
>> On Mon, Jun 28, 2021 at 11:01 AM Nikolay Izhikov 
>> wrote:
>>
>>> Hello.
>>>
>>> As one of the authors of PR fully support merge of this module.
>>> Have plans to use it during development of other features.
>>> Guys, please, share your feedback.
>>>
>>> Do we need to improve something prior merge?
>>>
>>>
>>> > 24 июня 2021 г., в 14:41, Denis Magda  написал(а):
>>> >
>>> > Anyway, it's much more community-friendly. Thanks, Anton!
>>> >
>>> > --
>>> > Denis
>>> >
>>> > -
>>> > Denis
>>> >
>>> > On Thu, Jun 24, 2021 at 3:43 AM Anton Vinogradov 
>>> wrote:
>>> >
>>> >> Denis,
>>> >>
>>> >> Unfortunately, we had some issues with this recording (low slides
>>> recording
>>> >> resolution, missed slides and random slides switch delay), which
>>> makes it
>>> >> slightly unwatchable :(
>>> >> Anyway, if you brave enough, you may open the slides [1] and the
>>> recording
>>> >> [2] simultaneously to solve this :)
>>> >>
>>> >> [1]
>>> >>
>>> >>
>>> https://go.gridgain.com/rs/491-TWR-806/images/11_IgniteSummit2021-Deep-dive-into-testing.pdf
>>> >> [2] https://www.youtube.com/watch?v=uRRlGrSA3NY
>>> >>
>>> >> On Wed, Jun 23, 2021 at 5:40 PM Denis Magda 
>>> wrote:
>>> >>
>>> >>> Congrats, Anton, that's a valuable contribution! I attended your
>>> session
>>> >> at
>>> >>> the Ignite Summit and wonder if you should share that recording with
>>> an
>>> >>> English-speaking part of the community?
>>> >>>
>>> >>> -
>>> >>> Denis
>>> >>>
>>> >>> On Wed, Jun 23, 2021 at 7:37 AM Anton Vinogradov 
>>> wrote:
>>> >>>
>>> >>>> Folks,
>>> >>>>
>>> >>>> Here's the video [1] that explains the proposal in detail.
>>> >>>> Feel free to ask questions here.
>>> >>>>
>>> >>>> [1] https://www.youtube.com/watch?v=f-i9COU5uAQ (in Russian)
>>> >>>>
>>> >>>> On Tue, Jun 8, 2021 at 2:51 PM Anton Vinogradov 
>>> wrote:
>>> >>>>
>>> >>>>> Igniters,
>>> >>>>> Let me present a framework, we developed, that allows automating
>>> >> Apache
>>> >>>>> Ignite testing on a real cluster.
>>> >>>>> The framework was initially presented at Ignite Summit.
>>> >>>>>
>>> >>>>> In brief,
>>> >>>>> The framework allows automating operations with any applications
>>> on a
>>> >>>> real
>>> >>>>> cluster using ssh in a form of a python test.
>>> >>>>>
>>> >>>>> Features:
>>> >>>>> - Ignite nodes can be started/stopped on a Docker or a real cluster
>>> >>> with
>>> >>>>> any custom configuration
>>> >>>>> - Any AI version is supported (released or compiled from sources)
>>> >

Re: [VOTE][EXTENSION] Release Apache Ignite performance-statistics-ext extension 1.0.0 RC4

[Anton Vinogradov]

Folks, seems we MUST automate such a check, as well as the whole
extension release process before the next release attempt?
We may (must?) use AI release automation as a booster.
Please let me know if any help required.

On Tue, Jun 29, 2021 at 11:59 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> -1 (binding)
>
> I did not check the rest of the package, but both of zip deliverables
> suffer from having a lot of files at the root of zip file.
>
>- ignite-performance-statistics-ext-1.0.0-bin.zip
><
> https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-performance-statistics-ext/1.0.0-rc4/ignite-performance-statistics-ext-1.0.0-bin.zip
> >
> should
>have ignite-performance-statistics-ext-1.0.0-bin/ as a root directory
>and nothing else on the 1st level
>- ignite-performance-statistics-ext-1.0.0-src.zip
><
> https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-performance-statistics-ext/1.0.0-rc4/ignite-performance-statistics-ext-1.0.0-src.zip
> >
> should
>have ignite-performance-statistics-ext-1.0.0-src/ as a root directory
> and
>nothing else on the 1st level.
>
>
> While we are waiting for rebuild, I hope other users will try the actual
> functionality.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 29 июн. 2021 г. в 01:51, Saikat Maitra :
>
> > +1
> >
> > On Fri, Jun 25, 2021 at 6:34 AM Sergei Ryzhov 
> > wrote:
> >
> > > +1
> > >
> > > --
> > > Best regards,
> > > Sergei Ryzhov
> > >
> >
>

Re: Re[2]: [DISCUSSION] Apache Ignite Release 2.10 (time, scope, manager)

[Anton Vinogradov]

Maxim,

https://issues.apache.org/jira/browse/IGNITE-13873
Is ready for review, is it possible to wait for it?

On Wed, Feb 24, 2021 at 12:01 AM Maxim Muzafarov  wrote:

> Folks,
>
>
> I'd like to cherry-pick to the release branch the following fixes:
>
> Fix security context for JDBC bulk-load operations
> https://issues.apache.org/jira/browse/IGNITE-13472
>
> Fixed an issue that caused a deadlock when user cache was created in
> parallel with TTL worker was in progress.
> https://issues.apache.org/jira/browse/IGNITE-14078
>
> On Thu, 18 Feb 2021 at 20:26, Maxim Muzafarov  wrote:
> >
> > Maxim, Igor,
> >
> > Thanks for sharing details.
> > Let's wait for these fixes.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-14206
> > [2] https://issues.apache.org/jira/browse/IGNITE-14204
> >
> > On Thu, 18 Feb 2021 at 18:35, Igor Sapego  wrote:
> > >
> > > Maxim,
> > >
> > > I believe I could fix the ticket [1] by the end of the next week.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-14204
> > >
> > > Best Regards,
> > > Igor
> > >
> > >
> > > On Thu, Feb 18, 2021 at 6:30 PM Max Timonin 
> wrote:
> > >
> > > > Hi! I've today found an issue [1], there is wrong handling of
> inlined POJO.
> > > > This bug appeared in 2.9.0 and makes it impossible to work with
> > > > multi-column indexes created on old AI versions that contain inlined
> POJO
> > > > keys in the middle.
> > > >
> > > > I'm working on the fix and asked Konstantin Orlov to review it. I
> think
> > > > that it will take only a couple of days to fix this issue. From my
> side, it
> > > > looks like a release blocker, but we've been living with that since
> 2.9.0.
> > > > WDYT if the release can wait for the fix?
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-14206
> > > >
> > > > On Thu, Feb 18, 2021 at 5:38 PM Maxim Muzafarov 
> wrote:
> > > >
> > > > > Folks,
> > > > >
> > > > >
> > > > > Do we have any estimations of how long does it take to fix the
> issue
> > > > > [1] in C++ thin client? The bug must be fixed for sure, however, I
> > > > > tend to continue with the release (if fixing the bug require a few
> > > > > weeks) and prepare a batch of fixes for the 2.10.1.
> > > > >
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14204
> > > > >
> > > > > On Thu, 18 Feb 2021 at 13:22, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> > > > >
> > > > > wrote:
> > > > > >
> > > > > > Hello!
> > > > > >
> > > > > > There was a ticket filed about the new feature, transactions in
> C++
> > > > thin
> > > > > > client, making this feature unusable if there's more than one
> > > > connection
> > > > > > endpoint:
> > > > > > https://issues.apache.org/jira/browse/IGNITE-14204
> > > > > >
> > > > > > I don't think this is a genuine blocker for 2.10, since there is
> a
> > > > > > workaround (use just one endpoint), nevertheless it is a critical
> > > > crasher
> > > > > > bug, so I wonder if Igor could take a look at it promptly,
> include the
> > > > > fix
> > > > > > in 2.10.
> > > > > >
> > > > > > Regards,
> > > > > > --
> > > > > > Ilya Kasnacheev
> > > > > >
> > > > > >
> > > > > > чт, 18 февр. 2021 г. в 09:03, Zhenya Stanilovsky
> > > > >  > > > > > >:
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I fill the ticket with drop problem, plz take a look [1]
> > > > > > >
> > > > > > > [1]  https://issues.apache.org/jira/browse/IGNITE-14205
> > > > > > >
> > > > > > > >Ilya,
> > > > > > > >
> > > > > > > >Thanks!
> > > > > > > >I've added this step to the Release Process wiki page also
> [1].
> > > > > > > >
> > > > > > > >[1]
> > > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P1.2ImplementationandScopeFinalization
> > > > > > > >
> > > > > > > >On Wed, 17 Feb 2021 at 18:05, Ilya Kasnacheev <
> > > > > ilya.kasnach...@gmail.com
> > > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> Hello!
> > > > > > > >>
> > > > > > > >> I have added ignite-2.10 to Nightly build triggering. I
> hope we
> > > > will
> > > > > > > have a
> > > > > > > >> 2.10 nightly tomorrow. Shame that I didn't do it earlier.
> > > > > > > >>
> > > > > > >
> > > > >
> > > >
> https://ci.ignite.apache.org/buildConfiguration/Releases_NightlyRelease_RunApacheIgniteNightlyRelease?branch=ignite-2.10=overview=builds
> > > > > > > >>
> > > > > > > >> Regards,
> > > > > > > >> --
> > > > > > > >> Ilya Kasnacheev
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> ср, 17 февр. 2021 г. в 17:37, Maxim Muzafarov <
> mmu...@apache.org
> > > > > >:
> > > > > > > >>
> > > > > > > >> > Folks,
> > > > > > > >> >
> > > > > > > >> > There is a possible issue with the performance for 2.9.1
> vs
> > > > 2.10.
> > > > > I'm
> > > > > > > >> > trying to reproduce on a stress-testing environment.
> > > > > > > >> > [1]
> > > > > > > >> >
> > > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/download/attachments/165224767/atomicPutRandomValueFullSync.jpg?api=v2
> > > > 

Re: [VOTE] Release pyignite 0.4.0-rc1

[Anton Vinogradov]

Checked the installation from sources.
+1

On Sat, Apr 17, 2021 at 7:47 PM Ivan Daschinsky 
wrote:

> Another reason why whe should host docs on readthedocs.io only is the
> fact,
> that
> pyignite has a separate release cycle. We will release and add more
> features frequently. It's strange to wait for new major ignite version to
> update documentation, as for me.
>
> сб, 17 апр. 2021 г. в 19:36, Ivan Daschinsky :
>
> > Denis, I don't think so. It's maybe sound strange, but python community
> > rely on readthedocs.io and it already has all neede doc's updated.
> > Python package already has comprehensive documentation in the rst format
> > and it is hosted and automatically built on every push.
> > Here is the docs of rc1, that is built on git tag
> >
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.4.0.rc1/
> > When we release, I simply just add tag on commit and fresh brand new docs
> > will be available.
> > Link to this doc will be in pypi.org index.
> >
> > I suppose, that the best option is just add a link to readthedocs.io on
> > https://ignite.apache.org/doc and just completely remove all other
> python
> > stuff from it.
> > It is strange to support and edit both versions, as for me.
> >
> > сб, 17 апр. 2021 г. в 19:14, Denis Magda :
> >
> >> +1
> >>
> >> Does it require documentation updates?
> >>
> >> -
> >> Denis
> >>
> >>
> >> On Fri, Apr 16, 2021 at 10:05 AM Ivan Daschinsky 
> >> wrote:
> >>
> >> > Ivan Daschinsky 
> >> > чт, 15 апр., 21:37 (19 часов назад)
> >> > кому: dev
> >> > Dear Igniters!
> >> >
> >> > Release candidate binaries are at least uploaded and ready for vote
> >> > You can find them here:
> >> > https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.4.0-rc1
> >> >
> >> > If you follow the link above, you will find source package (*.tar.gz
> and
> >> > *.zip)
> >> > and binary packages (wheels) for windows (x86, amd64) and linux (x86
> and
> >> > x86_64)
> >> > for pythons 36, 37, 38, 39. Also, there are sha512 and gpg signatures.
> >> >
> >> > You can install binary package for specific version of python using
> pip
> >> > For example do this on linux for python 3.8
> >> > >> pip install pyignite-0.4.0-cp38-cp38-manylinux1_x86_64.whl
> >> >
> >> > You can build and install package from source using this command:
> >> > >> pip install pyignite-0.4.0.tar.gz
> >> > You can build wheel on your platform using this command:
> >> > >> pip wheel --no-deps pyignite-0.4.0.tar.gz
> >> >
> >> > For building C module, you should have python headers and C compiler
> >> > installed.
> >> > (i.e. for ubuntu sudo apt install build-essential python3-dev)
> >> > In Mac OS X xcode-tools and python from homebrew are the best option.
> >> >
> >> > In order to check whether C module works, use following:
> >> > >> from pyignite import _cutils
> >> > >> print(_cutils.hashcode('test'))
> >> > >> 3556498
> >> >
> >> > You can find documentation here:
> >> >
> >>
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.4.0.rc1
> >> >
> >> > You can find examples here (to check them, you should start ignite
> >> > locally):
> >> >
> >> >
> >>
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.4.0.rc1/examples.html
> >> > Also, examples can be found in source archive in examples subfolder.
> >> > docker-compose.yml is supplied in order to start ignite quickly.
> >> >
> >> > Release notes:
> >> >
> >> >
> >>
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=9fee8ead3f70718767f6e11f93d1c7e77c61657b;hb=466b54527e6e42bc585c594d840a959d0b8626ef
> >> >
> >> > Git release tag was created:
> >> >
> >> >
> >>
> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.4.0.rc1
> >> >
> >> > The vote is formal, see voting guidelines
> >> > https://www.apache.org/foundation/voting.html
> >> >
> >> > +1 - to accept pyignite-0.4.0-rc1
> >> > 0 - don't care either way
> >> > -1 - DO NOT accept pyignite-0.4.0-rc1
> >> >
> >> > The vote will end at April, 21 15:00 UTC.
> >> >
> >>
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
> >
>

Re: [VOTE] Release pyignite 0.4.0-rc1

[Anton Vinogradov]

Also checked the hashcode generation
>> from pyignite import _cutils
>> print(_cutils.hashcode('test'))
>> 3556498

On Mon, Apr 19, 2021 at 11:01 AM Anton Vinogradov  wrote:

> Checked the installation from sources.
> +1
>
> On Sat, Apr 17, 2021 at 7:47 PM Ivan Daschinsky 
> wrote:
>
>> Another reason why whe should host docs on readthedocs.io only is the
>> fact,
>> that
>> pyignite has a separate release cycle. We will release and add more
>> features frequently. It's strange to wait for new major ignite version to
>> update documentation, as for me.
>>
>> сб, 17 апр. 2021 г. в 19:36, Ivan Daschinsky :
>>
>> > Denis, I don't think so. It's maybe sound strange, but python community
>> > rely on readthedocs.io and it already has all neede doc's updated.
>> > Python package already has comprehensive documentation in the rst format
>> > and it is hosted and automatically built on every push.
>> > Here is the docs of rc1, that is built on git tag
>> >
>> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.4.0.rc1/
>> > When we release, I simply just add tag on commit and fresh brand new
>> docs
>> > will be available.
>> > Link to this doc will be in pypi.org index.
>> >
>> > I suppose, that the best option is just add a link to readthedocs.io on
>> > https://ignite.apache.org/doc and just completely remove all other
>> python
>> > stuff from it.
>> > It is strange to support and edit both versions, as for me.
>> >
>> > сб, 17 апр. 2021 г. в 19:14, Denis Magda :
>> >
>> >> +1
>> >>
>> >> Does it require documentation updates?
>> >>
>> >> -
>> >> Denis
>> >>
>> >>
>> >> On Fri, Apr 16, 2021 at 10:05 AM Ivan Daschinsky > >
>> >> wrote:
>> >>
>> >> > Ivan Daschinsky 
>> >> > чт, 15 апр., 21:37 (19 часов назад)
>> >> > кому: dev
>> >> > Dear Igniters!
>> >> >
>> >> > Release candidate binaries are at least uploaded and ready for vote
>> >> > You can find them here:
>> >> > https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.4.0-rc1
>> >> >
>> >> > If you follow the link above, you will find source package (*.tar.gz
>> and
>> >> > *.zip)
>> >> > and binary packages (wheels) for windows (x86, amd64) and linux (x86
>> and
>> >> > x86_64)
>> >> > for pythons 36, 37, 38, 39. Also, there are sha512 and gpg
>> signatures.
>> >> >
>> >> > You can install binary package for specific version of python using
>> pip
>> >> > For example do this on linux for python 3.8
>> >> > >> pip install pyignite-0.4.0-cp38-cp38-manylinux1_x86_64.whl
>> >> >
>> >> > You can build and install package from source using this command:
>> >> > >> pip install pyignite-0.4.0.tar.gz
>> >> > You can build wheel on your platform using this command:
>> >> > >> pip wheel --no-deps pyignite-0.4.0.tar.gz
>> >> >
>> >> > For building C module, you should have python headers and C compiler
>> >> > installed.
>> >> > (i.e. for ubuntu sudo apt install build-essential python3-dev)
>> >> > In Mac OS X xcode-tools and python from homebrew are the best option.
>> >> >
>> >> > In order to check whether C module works, use following:
>> >> > >> from pyignite import _cutils
>> >> > >> print(_cutils.hashcode('test'))
>> >> > >> 3556498
>> >> >
>> >> > You can find documentation here:
>> >> >
>> >>
>> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.4.0.rc1
>> >> >
>> >> > You can find examples here (to check them, you should start ignite
>> >> > locally):
>> >> >
>> >> >
>> >>
>> https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.4.0.rc1/examples.html
>> >> > Also, examples can be found in source archive in examples subfolder.
>> >> > docker-compose.yml is supplied in order to start ignite quickly.
>> >> >
>> >> > Release notes:
>> >> >
>> >> >
>> >>
>> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob;f=RELEASE_NOTES.txt;h=9fee8ead3f70718767f6e11f93d1c7e77c61657b;hb=466b54527e6e42bc585c594d840a959d0b8626ef
>> >> >
>> >> > Git release tag was created:
>> >> >
>> >> >
>> >>
>> https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=refs/tags/0.4.0.rc1
>> >> >
>> >> > The vote is formal, see voting guidelines
>> >> > https://www.apache.org/foundation/voting.html
>> >> >
>> >> > +1 - to accept pyignite-0.4.0-rc1
>> >> > 0 - don't care either way
>> >> > -1 - DO NOT accept pyignite-0.4.0-rc1
>> >> >
>> >> > The vote will end at April, 21 15:00 UTC.
>> >> >
>> >>
>> >
>> >
>> > --
>> > Sincerely yours, Ivan Daschinskiy
>> >
>>
>

Re: Storing Teamcity projects settings in Version Control

[Anton Vinogradov]

> I think TC config should be stored in the same repo as the corresponding
> code (2.x config in 2.x repo, 3.x in 3.x, etc).
This will kill repo history.
You'll see dozens of TC config updates vs a single Ignite fix

> it would be great to be able to test them by simply creating a new branch
> in a single repo.
Where are you going to apply configs, do you have your own TC? ;)

On Tue, Aug 17, 2021 at 10:27 AM Pavel Tupitsyn 
wrote:

> Dmitry, Petr,
>
> I think TC config should be stored in the same repo as the corresponding
> code (2.x config in 2.x repo, 3.x in 3.x, etc).
>
> Changes and updates to build scripts and project structure often come
> together with changes to TC configuration,
> it would be great to be able to test them by simply creating a new branch
> in a single repo.
>
> On Mon, Aug 16, 2021 at 10:17 PM Petr Ivanov  wrote:
>
> > Hi, Dmitry.
> >
> >
> > I think we should start with adding current projects as-is in form of
> > autogenerated Kotlin code, but continue to use UI for editing.
> > Later at some point we should replace autogenerated code with valid one
> > (this can be done configuration by configuration), that will allow use
> the
> > power of DSL and programming language to create dynamic configuration and
> > use other benefits.
> >
> > Also, while I was thinking about it, I wondered — should we add whole TC
> > config to single repository, or divide per project (AI 2.x, AI 3.x,
> > Extensions, Thin Clients, etc.) and add corresponding TeamCity
> > configuration as part of the project into the same repository.
> > TeamCity configuration is added in form of maven project in .teamcity
> > directory in the project's root and will not interfere with the project
> > itself. Also this scheme is preferable due to linear development cycle
> (no
> > parallel versions in development for each project).
> >
> > I think I am ready to drive this initiative when and if we agree on this
> > matter and will discuss all the details of such migration.
> >
> >
> > > On 16 Aug 2021, at 19:36, Dmitry Pavlov  wrote:
> > >
> > > Hi Igniters,
> > >
> > > Once there was a discussion about placing our build configurations (TC)
> > settings in a VSC repository. This idea was suggested because we wanted
> to
> > validate internals of configs using IDE/text editor.
> > >
> > >
> >
> https://www.jetbrains.com/help/teamcity/storing-project-settings-in-version-control.html
> > >
> > > Now, with introduction of a second instance this idea migth be useful
> > twice, we'll be able to
> > > - view, track history, track authorship (? I'm not 100% sure, it is to
> > be checked)
> > > - use VSC as a signle source of thurth
> > > - we can remove TC Bot's code for validating all configs using REST
> > >
> > > How does that sound to you? Is it better to look at Kotlin DSL? Or
> could
> > XML be enought?
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> >
> >
>

Re: [ANNOUNCE] New committer: Petr Ivanov

[Anton Vinogradov]

My regards!

On Thu, Aug 19, 2021 at 1:11 PM andrei  wrote:

> Congrats, Petr!
>
> 8/19/2021 1:00 PM, Andrey Mashenkov пишет:
> > Congrats, Petr!
> >
> > On Thu, Aug 19, 2021 at 12:58 PM Dmitry Pavlov 
> wrote:
> >
> >> Hello Ignite community,
> >>
> >> The Project Management Committee (PMC) for Apache Ignite
> >> has invited Petr Ivanov (vveider) to become a committer and we are
> pleased
> >> to announce that he has accepted.
> >>
> >> Petr is community veteran, who supports TeamCity instance, as well as TC
> >> Bot, and other services. He contributes also to supporting our
> >> checkstyle/PMD/maven scripts, docker images, DEB, and RPM packages,
> Ignite
> >> start scripts, and many more things, which are crucial for Apache Ignite
> >> development process and running in producition.
> >>
> >> Please join me in congratulating Petr with his new role in the
> community.
> >>
> >> Being a committer enables easier contribution to the
> >> project since there is no need to go via the patch
> >> submission process. This should enable better productivity.
> >>
> >> Best Regards,
> >> Dmitriy Pavlov
> >> on behalf of Apache Ignite PMC
> >>
> >
>

Re: Deprecating LOCAL cache

[Anton Vinogradov]

> 1. What is expected behaviour if an old thin client requests creation of
> LOCAL cache on the newest ignite cluster?
Unsupported operation exception.

> 2. Should we completely remove LOCAL caches support in thin clients (i.e.
pyignite) before 2.13 release?
Removal should happen at 2.13.

On Tue, Sep 14, 2021 at 10:30 AM Ivan Daschinsky 
wrote:

> >> 2. 2.13 - complete removal LOCAL caches from codebase.
> Let's discuss this step with more details.
> 1. What is expected behaviour if an old thin client requests creation of
> LOCAL cache on the newest ignite cluster?
> 2. Should we completely remove LOCAL caches support in thin clients (i.e.
> pyignite) before 2.13 release?
>
> вт, 14 сент. 2021 г. в 10:11, Nikolay Izhikov :
>
> > I proposed the following plan:
> >
> > 1. 2.12 - deprecation of LOCAL caches.
> > 2. 2.13 - complete removal LOCAL caches from codebase.
> >
> > > 13 сент. 2021 г., в 13:30, Ivan Daschinsky 
> > написал(а):
> > >
> > > I personally support deprecation, but we should at least have a plan.
> > > I suppose that putting annotations and removing documentation are not
> > > enough.
> > >
> > >
> > > пн, 13 сент. 2021 г. в 13:22, Maxim Muzafarov :
> > >
> > >> Ivan,
> > >>
> > >> I don't think we can remove LOCAL caches at the nearest time, so there
> > >> is no plan for that. I can only imagine a single release that will
> > >> contain all the breaking changes we want to apply in 2.x version.
> > >>
> > >> My point here is only about deprecation:
> > >> - there are a lot of motivation points to remove written in this
> thread;
> > >> - I always hear from the support team that they do not recommend using
> > >> local caches;
> > >> - I haven't seen any bugs fixed for a long time for local caches
> > >> (suppose that we are not maintaining them);
> > >>
> > >> I just want to make sure that all these points are reflected in the
> > >> code base, so propose to mark them as deprecated.
> > >>
> > >> On Mon, 13 Sept 2021 at 11:29, Ivan Daschinsky 
> > >> wrote:
> > >>>
> > >>> Hi, Maxim. And what is the plan of removing this functionality? And I
> > >> also
> > >>> have some questions regarding deprecation in binary protocol
> > >>>
> > >>> Currently thin client binary protocol
> > >>> 1. Does support LOCAL caches
> > >>> 2. Does not support node filters.
> > >>>
> > >>> I can hardly imagine the usefulness of this feature on thin clients,
> > >>> especially with partition awareness, but nevertheless.
> > >>> What is expected behaviour if this feature is removed from newest
> > version
> > >>> of Apache Ignite server and and and old client is requesting
> > >>> creation of LOCAL cache?
> > >>>
> > >>> вс, 12 сент. 2021 г. в 15:10, Maxim Muzafarov :
> > >>>
> >  Folks,
> > 
> >  Let's get back to the discussion of obsolete LOCAL caches since a
> lot
> >  of time has passed since the last discussion.
> >  I've created an issue [1] for deprecation. Let's deprecate them at
> >  least at the next 2.12 release.
> > 
> >  WDYT?
> > 
> > 
> >  [1] https://issues.apache.org/jira/browse/IGNITE-15499
> > 
> >  On Fri, 27 Jul 2018 at 20:59, Valentin Kulichenko
> >   wrote:
> > >
> > > Guys,
> > >
> > > Use cases for local caches are rare, but they definitely exist. I
> > >> don't
> > > think it's a very good idea to deprecate this functionality at this
> >  point.
> > >
> > > At the same point, it's obviously not the most critical part of the
> > > product, so maintaining the whole separate implementation for it is
> > > probably an overkill. We had exact same story with replicated
> caches
> > >> btw
> >  -
> > > they were implemented separately which caused maintainability
> > >> issues, and
> > > we ended up removing this separate implementation. If we have the
> > >> same
> > > situation here, let's use the same solution.
> > >
> > > -Val
> > >
> > > On Fri, Jul 27, 2018 at 3:05 AM Dmitry Pavlov <
> dpavlov@gmail.com
> > >>>
> >  wrote:
> > >
> > >> Hi Dmitriy,
> > >>
> > >> I would like to stress this: I'm not saying local cache it
> > >> useless. I'm
> > >> supposing it is not used widely. I want to figure out if I'm
> > >> mistaking.
> > >>
> > >> All folks involved into user list says it is not used, so why not
> > >> to
> > >> deprecate? If we make a mistake, somebody will come to user list
> > >> and
> >  say,
> > >> 'Hey, why did you deprecate this, it is used for... in my project'
> > >>
> > >> Being very experienced Igniter you probably know real life usage
> >  examples.
> > >> And I appreciate if you or somebody else in community could share
> > >> it.
> > >>
> > >> Sincerely,
> > >> Dmitriy Pavlov
> > >>
> > >> пт, 27 июл. 2018 г. в 1:04, Dmitriy Setrakyan <
> > >> dsetrak...@apache.org>:
> > >>
> > >>> Guys,
> > >>>
> > >>> I just want to make sure we are all on 

Re: [DISCUSSION] Send documentation feedback notifications to dev list

[Anton Vinogradov]

+1 to keeping the dev list clean.

On Mon, Aug 9, 2021 at 11:00 AM Pavel Tupitsyn  wrote:

> I agree, let's keep the dev list clean.
> Automated notifications of any kind should not be sent to dev@i.a.o.
>
> PS Ivan, links 2 and 3 are the same.
>
> On Mon, Aug 9, 2021 at 8:54 AM Ivan Pavlukhin  wrote:
>
> > Igniters,
> >
> > Recently documentation feedback notifications were set up. And
> > currently destination for such notifications is dev list
> > dev@ignite.apache.org. It was announced in [1]. A notification message
> > itself looks as [2].
> >
> > Let's discuss if we all are fine with this notifications on dev list
> > as much effort was spent for keeping dev list clean, e.g. [3]. For me
> > personally these notifications are some kind of "noice" as I read the
> > list on spare time.
> >
> > Please share your opinion! Formal vote will be started if needed.
> >
> > [1]
> >
> https://lists.apache.org/thread.html/rf6b0ee140239ae119c97fe4022316b4e2f5c9f06a677bace5033e313%40%3Cdev.ignite.apache.org%3E
> > [2]
> >
> https://lists.apache.org/thread.html/rbd7c06f76559f46bc2872484529bc4fa1c503d705b6cc379156f0bb0%40%3Cdev.ignite.apache.org%3E
> > [3]
> >
> https://lists.apache.org/thread.html/rbd7c06f76559f46bc2872484529bc4fa1c503d705b6cc379156f0bb0%40%3Cdev.ignite.apache.org%3E
> >
> > --
> >
> > Best regards,
> > Ivan Pavlukhin
> >
>

Re: [ANNOUNCE] Welcome Zhenya Stanilovsky as a new committer

[Anton Vinogradov]

Congrats!

On Fri, Jul 30, 2021 at 10:19 AM ткаленко кирилл 
wrote:

> Zhenya, congratulations!
>
>  Пересылаемое сообщение 
> 30.07.2021, 09:50, "Ivan Daschinsky" :
>
>
> Zhenya, congrats, well deserved!
>
> пт, 30 июл. 2021 г. в 00:44, Andrey Mashenkov  >:
>
> >  Congratulations Zhenya!
> >
> >  On Fri, Jul 30, 2021 at 12:06 AM Maxim Muzafarov 
> >  wrote:
> >
> >  > The Project Management Committee (PMC) for Apache Ignite has invited
> >  > Zhenya Stanilovsky to become a committer and we are pleased to
> announce
> >  > that
> >  > he has accepted.
> >  >
> >  > For the last few years, Zhenya made a lot of performance fixes
> >  > especially for the core modules and important contributions to the
> >  > Apache Ignite codebase. He is actively involved in integrating the
> >  > Calcite framework with Apache Ignite. Moreover, he has been a great
> >  > help in the preparation of several Apache Ignite major releases by
> >  > carrying out stress-load tests. Besides the code contributions, Zhenya
> >  > is also an active community member and help users on dev and users
> >  > lists.
> >  >
> >  > Being a committer enables easier contribution to the project since
> there
> >  is
> >  > no need to go via the patch submission process. This should enable
> better
> >  > productivity.
> >  >
> >  > Please join me in welcoming Ivan, and congratulating him on the new
> role
> >  in
> >  > the Apache Ignite Community.
> >  >
> >  > Best Regards,
> >  > Maxim Muzafarov
> >  >
> >
> >  --
> >  Best regards,
> >  Andrey V. Mashenkov
>
>  Конец пересылаемого сообщения 
>

Re: Secondary TeamCity instance: ci2.ignite.apache.org

[Anton Vinogradov]

Great news!
Wish you good luck with this undertaking!
P.s. ci2.i.a.o sounds good to me.

On Tue, Aug 3, 2021 at 7:36 PM Petr Ivanov  wrote:

> Seems that preserving and syncing users will be difficult to achieve —
> that info is stored in DB, and partial DB replication is tricky.
>
> Also ci2.ignite.apache.org is good name, but currently it is targeted the
> same IP as original one:
> ; ANSWER SECTION:
> ci2.ignite.apache.org.  1726IN  A   195.144.253.150
>
>
>
> > On 3 Aug 2021, at 18:33, Dmitry Pavlov  wrote:
> >
> > Hi Igniters,
> >
> > I'm glad to announce that SberTech made an internal aggreement to
> sponsor a computing power to provide backup TeamCity instance. This
> instance is intended to be a geo-independent, secondary instance with
> availablility for community members.
> >
> > Thanks to JetBrains for providing license keys for TeamCity as their
> part of opensource sponsoring program.
> >
> > Most likely, we'll setup some domain name as tc.i.a.o or ci2.i.a.o.
> Please share your vision what would be most obvious.
> >
> > After a private discussions with Vitaliy Osipov and Petr Ivanov, I do
> believe that it would be possible to preserve current registered users.
> Build configurations are to be the same for the secondary instance.
> Technical details on how is could be achieved will be available later (most
> likely we'll start a sepearate thread to dicuss that).
> >
> > You are more than welcome to be an early adopters.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
>
>

Re: [DISCUSSION][IEP-80] Breaking changes in Ignite-2.x

[Anton Vinogradov]

+1

On Fri, Oct 15, 2021 at 3:41 PM Nikita Amelchev 
wrote:

> +1 for deprecation in the 2.12 release
>
> пт, 15 окт. 2021 г. в 15:35, Nikolay Izhikov :
> >
> > Hello, Igniters.
> >
> > I’ve prepared IEP-80 [1] to track breaking changes that should be done
> in Ignite 2.x.
> >
> > We agreed on the following algorithm to deal with breaking changes:
> >
> > - Ignite 2.[x] - deprecate the API + notification of the users.
> > - Ignite 2.[x+1] - API removal.
> >
> >
> > I propose to start these improvements with the d (depuration &
> removal) of the following features:
> >
> > - LOCAL caches
> > - MVCC caches
> > - legacy service grid implementation
> > - legacy JMX metrics beans.
> >
> > Deprecation in 2.12
> > Removal in 2.13
> >
> > Please, share your opinion.
> >
> > [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191334475
>
>
>
> --
> Best wishes,
> Amelchev Nikita
>

Re: NUMA aware allocator, PR review request

[Anton Vinogradov]

Let's move all GCC-related parts to ignite-extensions, release, and use
them as a maven dependency.


On Fri, Dec 3, 2021 at 1:08 PM Ivan Daschinsky  wrote:

> Ok, TC suite is ready [1].
> If there is no objections, I will merge it soon.
>
> Possible concerns -- now it is required to install build-essentials and
> libnuma-dev in order to build ignite on 64 bit linux.
> I suppose that this is not a big deal, but maybe someone will contradict?
>
>
> [1] --
>
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_NumaAllocator/?mode=builds
>
> чт, 2 дек. 2021 г. в 12:03, Ivan Daschinsky :
>
> > >> Our runs show about 7-10 speedup,
> > Sorry, typo 7-10%  speedup
> >
> > чт, 2 дек. 2021 г. в 12:01, Ivan Daschinsky :
> >
> >> Andrey, thanks!
> >>
> >> This allocator can be tested on every NUMA system.
> >> Our runs show about 7-10 speedup, if we use allocattor with interleaved
> >> strategy + -XX:+UseNUMA.
> >> But unfortunately our yardstick benches doesn't use offheap a lot,
> >> usually above one Gb.
> >> We trying to do more benches with real data and share them, possibly in
> >> meetup.
> >>
> >> AFAIK, GG lab servers are two-sockets machines, aren't they? So it is
> >> worth to run benches with a lot data on them, using
> >> allocator with interleaved strategy (you can skip specifying numa nodes,
> >> by default it will use all available) and use -XX:+UseNUMA jvm
> >> flag.
> >>
> >>
> >>
> >> чт, 2 дек. 2021 г. в 11:48, Andrey Mashenkov <
> andrey.mashen...@gmail.com
> >> >:
> >>
> >>> Ivan,
> >>>
> >>> Great job. PR looks good.
> >>>
> >>> This allocator in interleaved mode and passing `-XX:+UseNUMA` flag to
> jvm
> >>> > show promising results on yardstick benches. Technically, G1 is not a
> >>> numa
> >>> > aware collector for java versions less than 14, but allocation of
> heap
> >>> in
> >>> > interleaved mode shows good results even on java 11.
> >>>
> >>> Can you share benchmark results?
> >>> I'm not sure I'll have an Optane on my notebook in a reasonable time ;)
> >>>
> >>>
> >>> On Thu, Dec 2, 2021 at 10:41 AM Ivan Daschinsky 
> >>> wrote:
> >>>
> >>> > Semyon D. and Maks T. -- thanks a lot for review.
> >>> >
> >>> > BTW, Igniters, I will appreciate all opinions and feedback.
> >>> >
> >>> > пн, 29 нояб. 2021 г. в 10:13, Ivan Daschinsky  >:
> >>> >
> >>> > > Hi, igniters!
> >>> > >
> >>> > > There is not a big secret that nowadays NUMA is quite common in
> >>> > > multiprocessor systems.
> >>> > > And this memory architecture should be treated in specific ways.
> >>> > >
> >>> > > Support for NUMA is present in many commercial and open-source
> >>> products.
> >>> > >
> >>> > > I've implemented a NUMA aware allocator for Apache Ignite [1]
> >>> > > It is a JNI wrapper around `libnuma` and supports different
> >>> allocation
> >>> > > options.
> >>> > > I.e. interleaved, local, interleved_mask and so on. For more
> >>> information,
> >>> > > see
> >>> > > [2], [3].
> >>> > > This allocator in interleaved mode and passing `-XX:+UseNUMA` flag
> >>> to jvm
> >>> > > show promising results on yardstick benches. Technically, G1 is
> not a
> >>> > numa
> >>> > > aware collector for java versions less than 14, but allocation of
> >>> heap in
> >>> > > interleaved mode shows good results even on java 11.
> >>> > >
> >>> > > Currently, all needed libraries and tools for building this module
> >>> are
> >>> > > available on TC agents
> >>> > > setup of specific test suite is in progress [4]
> >>> > >
> >>> > > So I am asking for a review of my patch.
> >>> > >
> >>> > > [1] --  https://issues.apache.org/jira/browse/IGNITE-15922
> >>> > > [2] -- https://man7.org/linux/man-pages/man3/numa.3.html
> >>> > > [3] -- https://man7.org/linux/man-pages/man2/mbind.2.html
> >>> > > [4] -- https://issues.apache.org/jira/browse/IGNITE-15994
> >>> > >
> >>> >
> >>> >
> >>> > --
> >>> > Sincerely yours, Ivan Daschinskiy
> >>> >
> >>>
> >>>
> >>> --
> >>> Best regards,
> >>> Andrey V. Mashenkov
> >>>
> >>
> >>
> >> --
> >> Sincerely yours, Ivan Daschinskiy
> >>
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
> >
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>

Re: [ANNOUNCE] Welcome Maxim Timonin as a new committer

[Anton Vinogradov]

Welcome aboard!

On Wed, Dec 1, 2021 at 12:30 PM Вячеслав Коптилин 
wrote:

> Hi,
>
> Congrats, Maxim!
>
> Thanks,
> S.
>
> вт, 30 нояб. 2021 г. в 20:13, Maksim Timonin :
>
> > Hi, guys!
> >
> > Thanks everyone :)
> >
> > On Tue, Nov 30, 2021 at 1:16 PM Petr Ivanov  wrote:
> >
> > > Great news! Congratulations, Maksim!
> > >
> > >
> > > > On 29 Nov 2021, at 19:13, Kseniya Romanova 
> > > wrote:
> > > >
> > > > Igniters,
> > > >
> > > > The Project Management Committee (PMC) for Apache Ignite has invited
> > > Maxim
> > > > Timonin to become a committer and we are pleased to announce that he
> > has
> > > > accepted.
> > > >
> > > > Maxim makes valuable contributions to the Apache Ignite code, helps
> > > > actively to applications developers on the user list, and made a good
> > > start
> > > > in Project awareness contributing by presenting at Ignite Summit:
> Cloud
> > > > Edition.
> > > >
> > > > Being a committer enables easier contribution to the project since
> > there
> > > is
> > > > no need to go via the patch submission process. This should enable
> > better
> > > > productivity.
> > > >
> > > > Please join me in welcoming Maxim, and congratulating him on the new
> > role
> > > > in the Apache Ignite Community!
> > > >
> > > > Kseniya Romanova
> > > > on behalf of Apache Ignite PMC
> > >
> > >
> >
>

Re: Re[2]: NUMA aware allocator, PR review request

[Anton Vinogradov]

Any reason to release the same cpp sources for each release?
Any reason to increase the requirements amount for each new release?
Any reason to increase release complexity and duration?
All answers are "definitely no"

What we should do is to release cpp part once and use it as a dependency.
Extensions are a good location.

On Mon, Dec 6, 2021 at 3:11 PM Zhenya Stanilovsky
 wrote:

>
>
> +1 with Ivan, let`s store it in core product just because it looks
> like inalienable functionality and release cycle of extensions a little bit
> different.
>
>
>
> >Anton, I disagree.
> >
> >1. This should be released with main distro.
> >2. This should not be abandoned.
> >3. There is not any release process in ignite-extensions.
> >4. Everything is working now and working good.
> >
> >
> >So lets do not do this :)
> >
> >пн, 6 дек. 2021 г. в 14:49, Anton Vinogradov < a...@apache.org >:
> >
> >> Let's move all GCC-related parts to ignite-extensions, release, and use
> >> them as a maven dependency.
> >>
> >>
> >> On Fri, Dec 3, 2021 at 1:08 PM Ivan Daschinsky < ivanda...@gmail.com >
> >> wrote:
> >>
> >> > Ok, TC suite is ready [1].
> >> > If there is no objections, I will merge it soon.
> >> >
> >> > Possible concerns -- now it is required to install build-essentials
> and
> >> > libnuma-dev in order to build ignite on 64 bit linux.
> >> > I suppose that this is not a big deal, but maybe someone will
> contradict?
> >> >
> >> >
> >> > [1] --
> >> >
> >> >
> >>
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_NumaAllocator/?mode=builds
> >> >
> >> > чт, 2 дек. 2021 г. в 12:03, Ivan Daschinsky < ivanda...@gmail.com >:
> >> >
> >> > > >> Our runs show about 7-10 speedup,
> >> > > Sorry, typo 7-10% speedup
> >> > >
> >> > > чт, 2 дек. 2021 г. в 12:01, Ivan Daschinsky < ivanda...@gmail.com
> >:
> >> > >
> >> > >> Andrey, thanks!
> >> > >>
> >> > >> This allocator can be tested on every NUMA system.
> >> > >> Our runs show about 7-10 speedup, if we use allocattor with
> >> interleaved
> >> > >> strategy + -XX:+UseNUMA.
> >> > >> But unfortunately our yardstick benches doesn't use offheap a lot,
> >> > >> usually above one Gb.
> >> > >> We trying to do more benches with real data and share them,
> possibly
> >> in
> >> > >> meetup.
> >> > >>
> >> > >> AFAIK, GG lab servers are two-sockets machines, aren't they? So it
> is
> >> > >> worth to run benches with a lot data on them, using
> >> > >> allocator with interleaved strategy (you can skip specifying numa
> >> nodes,
> >> > >> by default it will use all available) and use -XX:+UseNUMA jvm
> >> > >> flag.
> >> > >>
> >> > >>
> >> > >>
> >> > >> чт, 2 дек. 2021 г. в 11:48, Andrey Mashenkov <
> >> >  andrey.mashen...@gmail.com
> >> > >> >:
> >> > >>
> >> > >>> Ivan,
> >> > >>>
> >> > >>> Great job. PR looks good.
> >> > >>>
> >> > >>> This allocator in interleaved mode and passing `-XX:+UseNUMA`
> flag to
> >> > jvm
> >> > >>> > show promising results on yardstick benches. Technically, G1 is
> >> not a
> >> > >>> numa
> >> > >>> > aware collector for java versions less than 14, but allocation
> of
> >> > heap
> >> > >>> in
> >> > >>> > interleaved mode shows good results even on java 11.
> >> > >>>
> >> > >>> Can you share benchmark results?
> >> > >>> I'm not sure I'll have an Optane on my notebook in a reasonable
> time
> >> ;)
> >> > >>>
> >> > >>>
> >> > >>> On Thu, Dec 2, 2021 at 10:41 AM Ivan Daschinsky <
> ivanda...@gmail.com
> >> >
> >> > >>> wrote:
> >> > >>>
> >> > >>> > Semyon D. and Maks T. -- thanks a lot for review.
> >> > >>> >
> &

Re: The conception of using two TeamCity servers

[Anton Vinogradov]

Petr,

> However, ROOT project will still be not under VCS and some major settings
like VCS roots, Clean-Up rules, custom step runners and so much more will
stay out of Git-based sync.
VCS roots are project-related and should not be stored somewhere outside
the project.
Also, having different roots at different TCs looks like a proper
configuration.
For example, TC2 may not contain the Ignite-3 project.

On Fri, Dec 17, 2021 at 4:13 PM Petr Ivanov  wrote:

> Separate JIRA project will ruin the concept of introducing changes for
> both code and build settings in single branch of single project.
>
>
> > On 17 Dec 2021, at 15:14, Anton Vinogradov  wrote:
> >
> > Petr,
> >
> >> I strongly suggest avoiding a separate repository for project settings.
> >> Let's store them in https://github.com/apache/ignite
> >
> > Sounds good, but we must avoid dozens of additional commits in this case.
> > Each commit should be properly formalized and related to the issue.
> > We may create a special Jira project for CI issues to separate commits.
> >
> > On Fri, Dec 17, 2021 at 2:46 PM Pavel Tupitsyn 
> wrote:
> >
> >> Petr,
> >>
> >>> why should I edit code in Apache Ignite 2.x repo to introduce new
> changes
> >> in Apache Ignite 3.x build settings
> >>
> >> I thought we can store 3.x settings in 3.x repo and so on.
> >>
> >> Looks like it does not work as I hoped it would.
> >> Thanks for your answers.
> >>
> >> On Fri, Dec 17, 2021 at 2:35 PM Petr Ivanov 
> wrote:
> >>
> >>> Pavel,
> >>>
> >>>
> >>> If you are referring to this paragraph
> >>>
> >>>• If you are using TeamCity feature branches, you can define a
> >>> branch specification when creating a project from URL (Git only) or in
> >> the
> >>> VCS root used for versioned settings. TeamCity will run a build in a
> >> branch
> >>> using the settings from this branch.
> >>>
> >>> then it is only for new projects.
> >>> After setting current one all settings will be acquired from the chosen
> >>> branch and it will not be able to change (only create new project from
> >> new
> >>> branch).
> >>>
> >>>
> >>> And it still can be done from separate repository.
> >>>
> >>>
> >>> Choosing one of the project's repo to host settings for every other
> >>> project seems non-logical — why should I edit code in Apache Ignite 2.x
> >>> repo to introduce new changes in Apache Ignite 3.x build settings?
> >>>
> >>>> On 17 Dec 2021, at 14:28, Pavel Tupitsyn 
> wrote:
> >>>>
> >>>> Petr,
> >>>>
> >>>>> you cannot run new suite added in branch because it just won't be
> >>> visible
> >>>> from UI
> >>>>
> >>>> That's unfortunate.
> >>>> However, a more important scenario is "make changes to the existing
> >>> project
> >>>> in a branch", which is supported, as I understand [1]
> >>>>
> >>>> [1]
> >>>>
> >>>
> >>
> https://www.jetbrains.com/help/teamcity/storing-project-settings-in-version-control.html#Defining+Settings+to+Apply+to+Builds
> >>>>
> >>>> On Fri, Dec 17, 2021 at 2:19 PM Petr Ivanov 
> >> wrote:
> >>>>
> >>>>> TeamCity DSL just does not work as you wish it should.
> >>>>>
> >>>>>
> >>>>> Changes from branches are not visible: you cannot run new suite added
> >> in
> >>>>> branch because it just won't be visible from UI because project UI is
> >>>>> rendered from default branch only),
> >>>>> and at least snapshot dependencies are taken from default branch
> only:
> >>>>> even if you will add dependency on already visible Run All
> >>> configuration,
> >>>>> it will be ignored.
> >>>>>
> >>>>>
> >>>>> Also about storing in already existing apache/ignite repository —
> that
> >>>>> will break the model about current separation on ignite and ignite-3,
> >>>>> ignite-extensions and so-on.
> >>>>> As we have different repositories for different parts of our project
> >> and
> >>>>> TeamCity unites them all — repository should be sepa

Re: The conception of using two TeamCity servers

[Anton Vinogradov]

Petr,

> I strongly suggest avoiding a separate repository for project settings.
> Let's store them in https://github.com/apache/ignite

Sounds good, but we must avoid dozens of additional commits in this case.
Each commit should be properly formalized and related to the issue.
We may create a special Jira project for CI issues to separate commits.

On Fri, Dec 17, 2021 at 2:46 PM Pavel Tupitsyn  wrote:

> Petr,
>
> > why should I edit code in Apache Ignite 2.x repo to introduce new changes
> in Apache Ignite 3.x build settings
>
> I thought we can store 3.x settings in 3.x repo and so on.
>
> Looks like it does not work as I hoped it would.
> Thanks for your answers.
>
> On Fri, Dec 17, 2021 at 2:35 PM Petr Ivanov  wrote:
>
> > Pavel,
> >
> >
> > If you are referring to this paragraph
> >
> > • If you are using TeamCity feature branches, you can define a
> > branch specification when creating a project from URL (Git only) or in
> the
> > VCS root used for versioned settings. TeamCity will run a build in a
> branch
> > using the settings from this branch.
> >
> > then it is only for new projects.
> > After setting current one all settings will be acquired from the chosen
> > branch and it will not be able to change (only create new project from
> new
> > branch).
> >
> >
> > And it still can be done from separate repository.
> >
> >
> > Choosing one of the project's repo to host settings for every other
> > project seems non-logical — why should I edit code in Apache Ignite 2.x
> > repo to introduce new changes in Apache Ignite 3.x build settings?
> >
> > > On 17 Dec 2021, at 14:28, Pavel Tupitsyn  wrote:
> > >
> > > Petr,
> > >
> > >> you cannot run new suite added in branch because it just won't be
> > visible
> > > from UI
> > >
> > > That's unfortunate.
> > > However, a more important scenario is "make changes to the existing
> > project
> > > in a branch", which is supported, as I understand [1]
> > >
> > > [1]
> > >
> >
> https://www.jetbrains.com/help/teamcity/storing-project-settings-in-version-control.html#Defining+Settings+to+Apply+to+Builds
> > >
> > > On Fri, Dec 17, 2021 at 2:19 PM Petr Ivanov 
> wrote:
> > >
> > >> TeamCity DSL just does not work as you wish it should.
> > >>
> > >>
> > >> Changes from branches are not visible: you cannot run new suite added
> in
> > >> branch because it just won't be visible from UI because project UI is
> > >> rendered from default branch only),
> > >> and at least snapshot dependencies are taken from default branch only:
> > >> even if you will add dependency on already visible Run All
> > configuration,
> > >> it will be ignored.
> > >>
> > >>
> > >> Also about storing in already existing apache/ignite repository — that
> > >> will break the model about current separation on ignite and ignite-3,
> > >> ignite-extensions and so-on.
> > >> As we have different repositories for different parts of our project
> and
> > >> TeamCity unites them all — repository should be separate.
> > >> And when JB will fix the branches issues — we still will be able to
> use
> > it
> > >> with creating branches with the same name on both project and TC
> > repository.
> > >>
> > >>> On 17 Dec 2021, at 14:06, Pavel Tupitsyn 
> wrote:
> > >>>
> > >>> Witaliy,
> > >>>
> >  repository is created
> >  only in the master branch
> > >>>
> > >>> I strongly suggest avoiding a separate repository for project
> settings.
> > >>> Let's store them in https://github.com/apache/ignite
> > >>>
> > >>> *1. We should be able to test code changes together with CI/CD
> > changes.*
> > >>> *2. We should be able to have different project settings in different
> > >>> branches.*
> > >>>
> > >>> For example, I may remove or rename a module in ignite/master branch
> > and
> > >>> update TC project accordingly,
> > >>> while it continues to work as before in ignite-2.12 branch where we
> > >> prepare
> > >>> the release with older code.
> > >>>
> > >>> This is super important and this is how most other projects do it
> (from
> > >> my
> > >>> experience).
> > >>>
> > >>>
> > >>>
> > >>> On Fri, Dec 17, 2021 at 11:12 AM Виталий Осилов <
> > >> osipov.wita...@gmail.com>
> > >>> wrote:
> > >>>
> >  Dear Ignite Community!
> > 
> >  I propose for discussion the conception of using two TeamCity
> servers
> > >> with
> >  a roadmap.
> >  https://ci.ignite.apache.org/
> >  https://ci2.ignite.apache.org/
> > 
> >  Storing project settings.
> >  Servers synchronize configurations between themselves using the
> > version
> >  control-storing DSL (repository at https://github.com/apache).
> > TeamCity
> >  allows to store the settings in the DSL based on the kotlin
> language.
> >  You can read more about the version control-storing DSL  here
> > 
> > 
> > >>
> >
> https://www.jetbrains.com/help/teamcity/2021.2/kotlin-dsl.html#Getting+Started+with+Kotlin+DSL
> >  This scheme will allow to maintain a single configuration for both
> TC
> > 

Re: Updating of configuration TC2: ci2.ignite.apache.org

[Anton Vinogradov]

Thanks for the update!

On Thu, Dec 16, 2021 at 9:07 PM Виталий Осилов 
wrote:

> Hi!
> TeamCity server https://ci2.ignite.apache.org/ will be unavailable on
> Friday 17.12 from 22:00 MSK to 00:00 MSK
> Works will be done to synchronize the TeamCity configuration
> https://ci2.ignite.apache.org/ with TeamCity https://ci.ignite.apache.org/
>
>
> Best regards,
> Osipov Vitaliy
> osipov.wita...@gmail.com
>

Re: [CANCEL] [VOTE] Release Apache Ignite 2.11.1 RC1

[Anton Vinogradov]

The ability to kill applications is less important than gaining access.
We may release 2.11.2 if necessary.
But now we must release 2.11.1 asap because it fixes a critical security
issue.

On Mon, Dec 20, 2021 at 2:01 PM Ivan Daschinsky  wrote:

> What do you mean, Anton? This is quite dangerous vulnerability and it is
> recommended to update log4j asap.
>
> пн, 20 дек. 2021 г. в 14:00, Anton Vinogradov :
>
> > Maxim,
> > Look like an issue is not related to security problems we fix.
> > Any reason to cancel the vote (delay release) to include this bugfix?
> >
> > On Mon, Dec 20, 2021 at 1:28 PM Maxim Muzafarov 
> wrote:
> >
> > > Cancelling the vote.
> > >
> > > I'll cherry-pick the following [1] to the release branch and prepare a
> > > new RC today.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-16153
> > >
> >
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>

Re: [CANCEL] [VOTE] Release Apache Ignite 2.11.1 RC1

[Anton Vinogradov]

Maxim,
Look like an issue is not related to security problems we fix.
Any reason to cancel the vote (delay release) to include this bugfix?

On Mon, Dec 20, 2021 at 1:28 PM Maxim Muzafarov  wrote:

> Cancelling the vote.
>
> I'll cherry-pick the following [1] to the release branch and prepare a
> new RC today.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-16153
>

Re: [ANNOUNCE] Welcome Vladislav Pyatkov as a new committer

[Anton Vinogradov]

Welcome aboard!

On Thu, Dec 23, 2021 at 10:24 AM Ivan Pavlukhin  wrote:

> Congrats, Vlad!
>
> 2021-12-22 20:48 GMT+03:00, Ivan Daschinsky :
> > Vlad, congrats! You have definitely deserved it!
> >
> > ср, 22 дек. 2021 г. в 20:40, Andrey Mashenkov <
> andrey.mashen...@gmail.com>:
> >
> >> Congrats, Vlad!
> >>
> >> ср, 22 дек. 2021 г., 20:23 Valentin Kulichenko <
> >> valentin.kuliche...@gmail.com>:
> >>
> >> > The Apache Ignite Project Management Committee (PMC) has invited
> >> Vladislav
> >> > Pyatkov to become a new committer and are happy to announce that he
> has
> >> > accepted.
> >> >
> >> > Vladislav is a veteran of the Apache Ignite project. He joined the
> >> > community in 2016.
> >> > He participated in the development of Ignite 1.x, 2.x and he is
> >> > actively
> >> > working on a new Apache Ignite 3.0 release.
> >> >
> >> > Being a committer enables easier contribution to the project since
> >> > there
> >> > is no need to go via the patch submission process. This should enable
> >> > better productivity.
> >> >
> >> > Please join me in welcoming Vlad, and congratulating him on the new
> >> > role
> >> > in the Apache Ignite Community.
> >> >
> >> > -Val
> >> >
> >>
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
> >
>
>
> --
>
> Best regards,
> Ivan Pavlukhin
>

Re: [PROPOSAL] Release Calcite-based SQL engine as an experimental feature

[Anton Vinogradov]

> it would be great to release a new SQL engine in 2.13 as an
experimental feature.
++1

On Thu, Dec 30, 2021 at 12:55 PM Alex Plehanov 
wrote:

> Andrey,
>
> >  Is this [1] a full scope of the tickets that MUST be resolved before the
> engine could be merged?
> Yes, we must resolve at least these tickets before merging. If you see any
> other release blockers fill free to attach them to this ticket.
>
> > I think we have to add instructions to the readme file on how to turn a
> new SQL engine on.
> Sure, I think it should be the part of documentation ticket.
>
> > Also, I don't like the module name "ignite-calcite", because Calcite is
> an independent project.
> Personally, I see no problems here (but it's discussable). We have a lot of
> modules where the name is an independent project: "ignite-kafka",
> "ignite-spring", "ignite-kubernetes", "ignite-log4j", "ignite-zookeeper",
> etc.
>
> > So, would you mind renaming the module to e.g. "ignite-sql-engine" or
> "ignite-sql"?
> Module "ignite-indexing" also contains SQL engine, so names like
> "ignite-sql-engine" or "ignite-sql" will be ambiguous.
>
> чт, 30 дек. 2021 г. в 13:54, Andrey Mashenkov  >:
>
> > Alex,
> > it would be great to release a new SQL engine in 2.13 as an
> > experimental feature.
> >
> > Is this [1] a full scope of the tickets that MUST be resolved before the
> > engine could be merged?
> > I think we have to add instructions to the readme file on how to turn a
> new
> > SQL engine on.
> >
> > Also, I don't like the module name "ignite-calcite", because Calcite is
> an
> > independent project.
> > and Ignite just uses it.
> > So, would you mind renaming the module to e.g. "ignite-sql-engine" or
> > "ignite-sql"?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-15436
> >
> > On Thu, Dec 30, 2021 at 11:10 AM Zhenya Stanilovsky
> >  wrote:
> >
> > >
> > > Alex, great !
> > > If someone wants to touch codebase somehow plz use this branch [1]
> > > Test passed can be found here [2] [3]
> > >
> > > [1]  https://github.com/apache/ignite/tree/sql-calcite/modules/calcite
> > > [2]
> > >
> >
> https://github.com/apache/ignite/tree/sql-calcite/modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite
> > > [3]
> > >
> >
> https://github.com/apache/ignite/tree/sql-calcite/modules/calcite/src/test/sql
> > >
> > > >
> > > >>
> > > >>>Hello, Igniters!
> > > >>>
> > > >>>As you may already know there is the new Ignite SQL engine based on
> > > Apache
> > > >>>Calcite currently under development.
> > > >>>
> > > >>>Reasons to move from H2-based engine and motivation for creating the
> > new
> > > >>>one in details described in IEP-37 [1].
> > > >>>
> > > >>>You can find all related to the new engine source code changes in
> the
> > > >>>"sql-calcite" branch [2].
> > > >>>
> > > >>>Calcite-based SQL engine is not production-ready yet and has a lot
> of
> > > known
> > > >>>issues. In the future, the new engine should be fully independent of
> > > >>>"ignite-indexing" and H2, but now it relies on schema management and
> > > >>>indexes implemented in the "ignite-indexing" module and can't work
> > > without
> > > >>>the old engine. Despite all of the above mentioned, in the current
> > > state,
> > > >>>it has its own parsing, planning and execution flow and is almost as
> > > >>>functional as the H2-based SQL engine.
> > > >>>
> > > >>>Some users are already interested in the Calcite-based engine and
> > asking
> > > >>>about the development status and release dates. Calcite-based SQL
> > engine
> > > >>>will be the only SQL engine in Ignite 3.0. Perhaps even in 2.x we
> can
> > > get
> > > >>>rid of the H2-based engine at some time in the future. There is some
> > > syntax
> > > >>>difference between Calcite and H2 (Calcite is closer to SQL
> standards
> > > than
> > > >>>H2) and a totally new execution flow. After the release of this
> > feature,
> > > >>>users can try their queries and determine if any adaptation for them
> > is
> > > >>>required. With the new planning and execution flow, perhaps, some
> > > queries
> > > >>>will be executed more effectively, users can redirect such queries
> to
> > > the
> > > >>>new engine.
> > > >>>
> > > >>>I think we can provide an opportunity to users to try the new engine
> > and
> > > >>>release it as an experimental feature with the next Apache Ignite
> > > version
> > > >>>(2.13).
> > > >>>
> > > >>>What do you think?
> > > >>
> > > >>
> > > >>
> > > >>
> >
> >
> >
> > --
> > Best regards,
> > Andrey V. Mashenkov
> >
>

Re: Move the Zookeeper IP-finder to the ignite-extensions

[Anton Vinogradov]

Sergei,

Please make sure this will not break the Ducktests.
AFAIK, we have zookeeper tests there.

On Wed, Dec 22, 2021 at 12:09 PM Sergei Ryzhov 
wrote:

> Hi, igniters,
>
> I've created an issue [1] to move the zookeeper IP-finder to the
> ignite-extensions. The motivation is the same as with migration of
> cloud-based IP-finders - to remove integration dependency of the
> release cycle on Ignite releases. Also, the log4j dependency with
> vulnerabilities (needed for Apache Curator) will be removed from the Ignite
> release.
>
> Any objections?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-16182
> --
> Best regards,
> Sergei Ryzhov
>

Re: [VOTE][EXTENSION] Release Apache Ignite spring-tx-ext, spring-cache-ext extensions 1.0.0 RC1

[Anton Vinogradov]

My -0 here, since the is no automated RC preparation and testing at Apache
TeamCity again.
Next time, I will vote with -1 if a new vote is not covered by public
automated preparation/testing as the AI release does.

On Wed, Oct 27, 2021 at 11:48 AM Nikita Amelchev 
wrote:

> Dear Ignite Community,
>
> I have uploaded a release candidate of extensions modules:
>  - spring-tx-ext 1.0.0
>  - spring-cache-ext 1.0.0
>  - spring-data-commons 1.1.0
>
> The following staging can be used for testing:
> https://repository.apache.org/content/repositories/orgapacheignite-1533
>
> The release candidate of the spring-tx-ext 1.0.0 extension:
>
> https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-spring-tx-ext/1.0.0-rc1/
>
> The release candidate of the spring-cache-ext 1.0.0 extension:
>
> https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-spring-cache-ext/1.0.0-rc1/
>
> The release candidate of the spring-data-common 1.1.0 dependent module
> is in the packages mentioned above.
>
> Tags were created:
> ignite-spring-tx-ext-1.0.0-rc1
> ignite-spring-cache-ext-1.0.0-rc1
> ignite-spring-data-commons-1.1.0-rc1
>
>
> https://gitbox.apache.org/repos/asf?p=ignite-extensions.git;a=commit;h=refs/tags/ignite-spring-tx-ext-1.0.0-rc1
>
> RELEASE NOTES:
>
> https://gitbox.apache.org/repos/asf?p=ignite-extensions.git;a=blob;f=RELEASE_NOTES.txt;h=c6b2a393e042ce956bfefd5d7d92d5ab4ff04cda;hb=HEAD
>
> DOCUMENTATION:
> Documentation of the spring-tx-ext extension:
>
> https://github.com/apache/ignite/blob/master/docs/_docs/extensions-and-integrations/spring/spring-tx.adoc
>
> Documentation of the spring-cache-ext extension:
>
> https://github.com/apache/ignite/blob/master/docs/_docs/extensions-and-integrations/spring/spring-caching.adoc
>
> The vote is formal, see voting guidelines
> https://www.apache.org/foundation/voting.html
>
> +1 - to accept the extensions modules listed in the thread
> 0 - don't care either way
> -1 - DO NOT accept either of the extensions modules listed in the
> thread (explain why)
>
> The vote will hold for 5 days and will end on November 1, 2021, 10:00 UTC
> https://www.timeanddate.com/countdown/generic?iso=20211101T10=0
>
> --
> Best wishes,
> Amelchev Nikita
>

Re: Ci and ci2 settings synchronization

[Anton Vinogradov]

Folks,

Could we have a kick-off call to determine the roadmap?

On Thu, Oct 21, 2021 at 6:52 PM Denis Magda  wrote:

> Folks, you don't need me on that call. I'm of a little value/help here.
>
> -
> Denis
>
>
> On Thu, Oct 21, 2021 at 9:51 AM Nikolay Izhikov 
> wrote:
>
> > Hello, Igniters.
> >
> > As you may know, recently, Sberbank sponsored servers capacity to setup
> > new instance of Team City [1]  to run Ignite tests.
> >
> > Right now, ci2 is fully ready for daily usage.
> > We have exported and prepared Kotlin scripts [2] to keep TC settings as a
> > code [3].
> >
> > I think we should adjust both TC instances to use the same settings and
> > run tests in the same way.
> >
> > ci.ignite.apache.org is sponsored and run by Grid Gain so
> >
> > Denis, Valentin, Petr, let's have a call to discuss how and when we can
> > implement these changes?
> >
> > [1] https://ci2.ignite.apache.org
> > [2] https://github.com/willy983/ci_tc_configuration
> > [3] https://www.jetbrains.com/help/teamcity/kotlin-dsl.html
> >
> >
> > > 3 авг. 2021 г., в 18:33, Dmitry Pavlov 
> написал(а):
> > >
> > > Hi Igniters,
> > >
> > > I'm glad to announce that SberTech made an internal aggreement to
> > sponsor a computing power to provide backup TeamCity instance. This
> > instance is intended to be a geo-independent, secondary instance with
> > availablility for community members.
> > >
> > > Thanks to JetBrains for providing license keys for TeamCity as their
> > part of opensource sponsoring program.
> > >
> > > Most likely, we'll setup some domain name as tc.i.a.o or ci2.i.a.o.
> > Please share your vision what would be most obvious.
> > >
> > > After a private discussions with Vitaliy Osipov and Petr Ivanov, I do
> > believe that it would be possible to preserve current registered users.
> > Build configurations are to be the same for the secondary instance.
> > Technical details on how is could be achieved will be available later
> (most
> > likely we'll start a sepearate thread to dicuss that).
> > >
> > > You are more than welcome to be an early adopters.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> >
> >
>

Re: 0-day CVE in log4j

[Anton Vinogradov]

Folks,

My 200 rubles here,
> I want to include it to the 2.12 scope.
Why not 2.11.1 as well?
We should provide a fixed version for current customers asap.
2.12 require migration, while 2.11.1 can be applied as-is.


On Mon, Dec 13, 2021 at 12:18 PM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:

> Another workaround appears to be using the
> -Dlog4j2.formatMsgNoLookups=true option. Also, “Java versions greater than
> 6u211, 7u201, 8u191, and 11.0.1 are less affected by this attack vector, at
> least in theory, because the JNDI can't load remote code using LDAP.”
>
> (
> https://arstechnica.com/information-technology/2021/12/minecraft-and-other-apps-face-serious-threat-from-new-code-execution-bug/
> )
>
> > On 12 Dec 2021, at 10:56, Dmitriy Pavlov  wrote:
> >
> > Hi Igniters,
> >
> > Preliminary: change of the log4j version does not affect any tests
> > (Alexander Nikolaev, correct me if I'm wrong).
> >
> > If you're using embedded Ignite, it's perfectly possible to enforce
> jog4j2
> > dependency to be 2.15.0 in your project final pom.xml or build.gradle or
> > any other build system properties.
> >
> > https://issues.apache.org/jira/browse/IGNITE-16101 ticket seems to be
> > a blocker for 2.12. But for now, as a workaround, it's possible to select
> > the latest version manually.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > сб, 11 дек. 2021 г. в 09:47, Nikita Amelchev :
> >
> >> Hello.
> >>
> >> The issue to update dependency was created:
> >> https://issues.apache.org/jira/browse/IGNITE-16101
> >>
> >> I want to include it to the 2.12 scope.
> >>
> >> сб, 11 дек. 2021 г., 09:19 Raymond Wilson :
> >>
> >>> All
> >>>
> >>> This blew up today: CVE-2021-44228 (
> >>>
> >>>
> >>
> https://www.bleepingcomputer.com/news/security/new-zero-day-exploit-for-log4j-java-library-is-an-enterprise-nightmare/
> >>> )
> >>>
> >>> Will there be a risk assessment with respect to Ignite for this CVE?
> >>>
> >>> Thanks,
> >>> Raymond.
> >>>
> >>> --
> >>> 
> >>> Raymond Wilson
> >>> Trimble Distinguished Engineer, Civil Construction Software (CCS)
> >>> 11 Birmingham Drive | Christchurch, New Zealand
> >>> raymond_wil...@trimble.com
> >>>
> >>> <
> >>>
> >>
> https://worksos.trimble.com/?utm_source=Trimble_medium=emailsign_campaign=Launch
> 
> >>>
> >>
>
>
>

Re: [DISCUSSION] Remove outdated ignite-sys-cache

[Anton Vinogradov]

> What is the issue if third-party plugins will create «ignite-sys-cache»
from the code?
Great idea!

On Fri, Dec 3, 2021 at 2:15 PM Nikolay Izhikov  wrote:

> Vyacheslav, Val, can you please clarify - What is the issue if third-party
> plugins will create «ignite-sys-cache» from the code?
>
> Like just replacing `Ignite#cache` with the `Ignite#getOrCreateCache`.
>
> > 2 дек. 2021 г., в 16:13, Вячеслав Коптилин 
> написал(а):
> >
> > Hello Maxim,
> >
> >> I don't understand why you are using *backwards compatibility* for
> > completely internal things.
> >> Why you are thinking in terms of users usage when are talking about
> > ignite-sys-cache but not thinking when refactoring
> > First of all, we are talking about all plugin developers. It does not
> > matter it is open-source or proprietary plugins.
> > Honestly, I don't have a list of all possible plugins that have already
> > been developed and available under the ASF license, for instance.
> > Do you have such a list? Can you be sure that this change will not affect
> > anyone?
> >
> >> I don't understand why you are using *backwards compatibility* for
> > completely internal things.
> >> Why you are thinking in terms of users usage when are talking about
> > ignite-sys-cache but not thinking when refactoring
> > The system cache was the crucial thing in the architecture of Apache
> Ignite
> > 1.x and 2.x (at least 2.1 - 2.11?)
> >
> >> All the internals have been reworked and nobody even mentioned that it
> > may affect some of the plugin developers.
> > Yep, perhaps, some internal parts of Apache Ignite were reworked in order
> > to avoid using the system cache.
> > However, it is still a part of Ignite and it works and can be used in
> > plugins.
> > Honestly, the mentioned alternative, I mean the distributed metastorage,
> is
> > the INTERNAL thing as well.
> > It means that plugin developer depends on INTERNAL entities. (it does not
> > matter system-cache/metastorage)
> > IMHO, such questions should be CAREFULLY discussed with no rush.
> >
> >> I do not propose to rush with the ignite-sys-cache removal. I'd like to
> > collect all the technical stuff that we depend on, fix all of them and
> > after everything will be ready do a final removal.
> > Good! We are on the same page!
> >
> >> 1. Add deprecation for the 2.12 release. I hope it is still possible.
> > If I am not mistaken, the code freeze was October 29. I think it is too
> > late to introduce this change. We can add deprecation for the 2.13
> release.
> >
> >> Apache Ignite core still have some dependencies on ignite-sys-cache that
> > should fix for 2.13
> > Ok, I agree. We need to try to find out all edge cases and add new
> > abilities to the metastorage in order to cover all known
> > issues/requirements.
> >
> >> Remove the system cache in 2.14.
> > It depends on our progress with improving the distributed metastorage. In
> > general, I hope it will be possible.
> >
> > Thanks,
> > S.
> >
> > чт, 2 дек. 2021 г. в 13:36, Maxim Muzafarov :
> >
> >> Pavel,
> >>
> >>
> >> I don't understand why you are using *backwards compatibility* for
> >> completely internal things. Why you are thinking in terms of users
> >> usage when are talking about ignite-sys-cache but not thinking when
> >> refactoring, for instance, all the checkpoint classes? Take a look at
> >> the [1] issue. All the internals have been reworked and nobody even
> >> mentioned that it may affect some of the plugin developers. This is
> >> really strange, so I can't agree with you.
> >>
> >>
> >> I do not propose to rush with the ignite-sys-cache removal. I'd like
> >> to collect all the technical stuff that we depend on, fix all of them
> >> and after everything will be ready do a final removal. Slava already
> >> mentioned some crucial cases, so I hope you and Val also help with the
> >> rest of them. Let's be more precise in terms *what we need to fix*.
> >>
> >>
> >> I've made some investigations related to the ignite-sys-cache and here
> >> is my proposal:
> >>
> >> 1. Add deprecation for the 2.12 release. I hope it is still possible.
> >>
> >> 2. Apache Ignite core still have some dependencies on ignite-sys-cache
> >> that should fix for 2.13:
> >>
> >> 2.1. CLUSTER_NAME property: when it's not set the `deploymentId` of
> >> system cache is used. Let's move it to metastorage.
> >> 2.2. When the Security is enabled each compute task name (hash to be
> >> more precise) is written to the ignite-sys-cache [2]. From my point of
> >> view, we can remove it in 2.13. Can anyone check?
> >> 2.3. Slava mentioned some issues with the distributed metastorage that
> >> might be fixed. I think this is doable.
> >>
> >> 3. Remove the system cache in 2.14.
> >>
> >>
> >>
> >> [1] https://issues.apache.org/jira/browse/IGNITE-13143
> >> [2]
> >>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/task/GridTaskProcessor.java#L201
> >>
> >> On Thu, 2 Dec 2021 at 12:56, Pavel 

Re: [VOTE] @Nullable/@NotNull annotation usage in Ignite 3

[Anton Vinogradov]

+1 to option 4

On Thu, Jan 13, 2022 at 5:15 PM Pavel Tupitsyn  wrote:

> +1 to option 2
>
> On Thu, Jan 13, 2022 at 3:52 PM Roman Puchkovskiy <
> roman.puchkovs...@gmail.com> wrote:
>
> > +1 to option 2
> >
> > чт, 13 янв. 2022 г. в 15:58, Sergey :
> > >
> > > +1 for option2 (only @Nullable)
> > >
> > > Best regards,
> > > Sergey Kosarev.
> > >
> > >
> > > чт, 13 янв. 2022 г. в 13:25, Alexander Polovtcev <
> > alexpolovt...@gmail.com>:
> > >
> > > > Dear Igniters,
> > > >
> > > > In this thread
> > > > 
> > we've
> > > > discussed possible approaches to using null-related annotations. As
> the
> > > > result, the following approaches were proposed:
> > > >
> > > >1. Use both @Nullable and @NotNull annotations everywhere;
> > > >2. Use only @Nullable;
> > > >3. Use only @NotNull;
> > > >4. Do not use* @*Nullable nor @NotNull.
> > > >
> > > > I would like to propose a vote: please post the corresponding number
> > of the
> > > > option you like. The voting will commence on Thursday, January 20 at
> > 11:00
> > > > UTC.
> > > >
> > > > --
> > > > With regards,
> > > > Aleksandr Polovtcev
> > > >
> >
>

Re: [VOTE] Release Apache Ignite 2.12.0 RC2

[Anton Vinogradov]

+1

On Tue, Jan 11, 2022 at 11:35 AM Maxim Muzafarov  wrote:

> +1
>
> On Tue, 11 Jan 2022 at 11:31, Nikolay Izhikov  wrote:
> >
> > +1
> >
> > > 11 янв. 2022 г., в 02:38, Vladimir Steshin 
> написал(а):
> > >
> > > +1
> > >
> > > 10.01.2022 15:52, Nikita Amelchev пишет:
> > >> Dear Community,
> > >>
> > >> The release candidate (2.12.0-rc2) is ready.
> > >>
> > >> I have uploaded a release candidate to:
> > >> https://dist.apache.org/repos/dist/dev/ignite/2.12.0-rc2/
> > >> https://dist.apache.org/repos/dist/dev/ignite/packages_2.12.0-rc2/
> > >>
> > >> The following staging can be used for testing:
> > >>
> https://repository.apache.org/content/repositories/orgapacheignite-1539
> > >>
> > >> Tag name is 2.12.0-rc2:
> > >>
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=commit;h=refs/tags/2.12.0-rc2
> > >>
> > >> RELEASE_NOTES:
> > >>
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.12
> > >>
> > >> Complete list of resolved issues:
> > >>
> https://issues.apache.org/jira/issues/?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.12%27))%20and%20status%20in%20(%27CLOSED%27%2C%20%27RESOLVED%27)%20ORDER%20BY%20priority
> > >>
> > >> DEVNOTES:
> > >>
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.12
> > >>
> > >> Additional checks have been performed (available for users included
> > >> into the release group on TeamCity).
> > >>
> > >> TC [Check RC: Licenses, compile, chksum]
> > >>
> https://ci2.ignite.apache.org/buildConfiguration/ignite2_Release_ApacheIgniteReleaseJava8_PrepareVote4CheckRcLicensesChecksum/6266150?showRootCauses=false=true
> > >>
> > >> TC [2] Compare w/ Previous Release
> > >>
> https://ci2.ignite.apache.org/buildConfiguration/ignite2_Release_ApacheIgniteReleaseJava8_IgniteRelease72CheckFileConsistency/6266148?showRootCauses=false=true
> > >>
> > >> The vote is formal, see voting guidelines
> > >> https://www.apache.org/foundation/voting.html
> > >>
> > >> +1 - to accept Apache Ignite 2.12.0-rc2
> > >> 0 - don't care either way
> > >> -1 - DO NOT accept Apache Ignite Ignite 2.12.0-rc2 (explain why)
> > >>
> > >> See notes on how to verify release here
> > >> https://www.apache.org/info/verification.html
> > >> and
> > >>
> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification
> > >>
> > >> This vote will be open until Thu Jan 13, 2022, 16:00 UTC. Please,
> > >> write me down the thread if you need additional time to check the
> > >> release.
> > >>
> https://www.timeanddate.com/countdown/vote?iso=20220113T16=0=VOTE+on+the+Apache+Ignite+Release+2.12.0+RC2=sanserif
> > >>
> >
>