Request for contributor permissions

[Ivan Daschinsky]

Hello Ignite Community!

My name is Ivan Daschinskiy. I want to contribute to Apache Ignite and want
to start with this issue - IGNITE-6860
, my JIRA username
*ivandasch*. Any help on this will be appreciated.

Thanks a lot!
-- 
Sincerely yours, Ivan Daschinskiy

Re: IGNITE-6827 - Review needed.

[Ivan Daschinsky]

Hi all, I've implemented corresponded .NET api.
Pavel, could you review my PR, please?


https://issues.apache.org/jira/browse/IGNITE-8075

2018-04-10 21:06 GMT+03:00 Dmitry Pavlov :

> Hi Pavel,
>
>  thank you for bring up test questions. It seems my previous comments were
> not taken into account.
>
> Igniters,
>
>  let me remind we should get passing TC suites before merge,
> https://cwiki.apache.org/confluence/display/IGNITE/How+
> to+Contribute#HowtoContribute-ReviewProcessandMaintainers
> (highlighted
> note).
>
> For disabling parity test checks please consider steps describled in
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Tests+How+To#
> IgniteTestsHowTo-Testof.NETAPIparitywithJavaAPI
>
> Sincerely,
> Dmitriy Pavlov
>
>
> пн, 9 апр. 2018 г. в 21:18, Pavel Tupitsyn :
>
> > > Pavel Tupitsyn, what about .NET stuff ?
> >
> > 1) Thank you for filing the ticket, personally I have no plans to work on
> > it in the near future.
> >
> > 2) .NET tests fail, please make sure they are fixed before merging:
> > https://ci.ignite.apache.org/viewLog.html?buildId=1175956
> >
> > TransactionsParityTest should be fixed by adding new properties to ignore
> > list with a reference to IGNITE-8075, this is simple.
> >
> > But I have concerns about
> > *CachePartitionedTest.TestTransactionScopeMultiCache, *
> > seems like something is broken with multi-cache transactions. Please
> > investigate this one.
> >
> > Thanks,
> > Pavel
> >
> >
> > On Mon, Apr 9, 2018 at 6:24 PM, Alexei Scherbakov <
> > alexey.scherbak...@gmail.com> wrote:
> >
> > > Guys,
> > >
> > > I've slightly modified public API javadoc as Denis Magda has suggested
> in
> > > PR review.
> > >
> > > Please take a look.
> > >
> > > Pavel Tupitsyn, what about .NET stuff ?
> > >
> > > I provided all necessary information in ticket [2]
> > >
> > > Upsource link [1]
> > >
> > > [1] https://reviews.ignite.apache.org/ignite/branch/PR%203624
> > >
> > > [2] https://issues.apache.org/jira/browse/IGNITE-8075
> > >
> > >
> > >
> > > пн, 9 апр. 2018 г. в 16:57, Alexey Goncharuk <
> alexey.goncha...@gmail.com
> > >:
> > >
> > > > I am not aware of any additional timeouts that we are willing to add
> in
> > > the
> > > > nearest future.
> > > >
> > > > 2018-04-09 16:01 GMT+03:00 Dmitriy Setrakyan  >:
> > > >
> > > > > On Mon, Apr 9, 2018 at 5:42 AM, Alexey Goncharuk <
> > > > > alexey.goncha...@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > Guys,
> > > > > >
> > > > > > After the review in Upsource the configuration parameter was
> > renamed
> > > > > > to txTimeoutOnPartMapSync, and it makes sense to me because PME
> is
> > an
> > > > > > implementation detail and it may change in future, partition map
> > sync
> > > > is
> > > > > a
> > > > > > more abstract term. For the same reason I like this parameter
> being
> > > > > placed
> > > > > > on transactions configuration - we do not have any parameters for
> > > PME,
> > > > so
> > > > > > the configuration property goes to an object which affects a
> > > > user-exposed
> > > > > > API.
> > > > > >
> > > > >
> > > > > AG, are we going to have any other timeouts on PME, like locks? If
> > yes,
> > > > > then I would still vote of adding PmeTimeout property.
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> > >
> >
>



-- 
Sincerely yours, Ivan Daschinskiy

PR#3697 IGNITE-8021: Delete cache config files when destroyed

[Ivan Daschinsky]

 Hi Team,

Could you review this PR https://github.com/apache/ignite/pull/3697 please?

Tests result: https://ci.ignite.apache.org/viewLog.html?buildId=1159660

Jira ticket: https://issues.apache.org/jira/browse/IGNITE-8021

-- 
Sincerely yours, Ivan Daschinskiy

Re: Apache Ignite 2.7. Last Mile

[Ivan Daschinsky]

Hi! Is it possible to merge IGNITE-9854? Fix is pretty simple, but quite
important.

ср, 17 окт. 2018 г. в 17:49, Andrey Gura :

> JFYI
>
> IGNITE-9737 and IGNITE-9710 are merged to release branch.
> On Wed, Oct 17, 2018 at 5:41 PM Pavel Tupitsyn 
> wrote:
> >
> > Thank you. Fix has been merged to master and cherry-picked to ignite-2.7.
> >
> > On Wed, Oct 17, 2018 at 1:26 PM Nikolay Izhikov 
> wrote:
> >
> > > Pavel.
> > >
> > > Ok, I agree to include this ticket into 2.7
> > > Let's do it.
> > >
> > > В Ср, 17/10/2018 в 13:20 +0300, Pavel Tupitsyn пишет:
> > > > Nikolay,
> > > >
> > > > It completely breaks a major feature under certain conditions. I
> would
> > > > consider it a blocker.
> > > >
> > > > On Wed, Oct 17, 2018 at 1:00 PM Nikolay Izhikov  >
> > > wrote:
> > > >
> > > > > Hello, Pavel.
> > > > >
> > > > > Is it a blocker?
> > > > >
> > > > > В Ср, 17/10/2018 в 12:58 +0300, Pavel Tupitsyn пишет:
> > > > > > Hi Igniters,
> > > > > >
> > > > > > I'd like to include IGNITE-9877 in 2.7, can we do that?
> > > > > > The fix is ready, I'm waiting for TC run.
> > > > > >
> > > > > > Pavel
> > > > > >
> > > > > > On Wed, Oct 17, 2018 at 11:45 AM Павлухин Иван <
> vololo...@gmail.com>
> > > > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi NIkolay,
> > > > > > >
> > > > > > > Thank you for keeping everybody focused! Regarding to my ticket
> > > > > > > IGNITE-5935.
> > > > > > > It is in final stage now. Tests look good. I believe that it
> will
> > > be
> > > > >
> > > > > merged
> > > > > > > in couple of days (at most).
> > > > > > >
> > > > > > > ср, 17 окт. 2018 г. в 11:39, Nikolay Izhikov <
> nizhi...@apache.org
> > > >:
> > > > > > >
> > > > > > > > Hello, Igniters.
> > > > > > > >
> > > > > > > > 9 tickets to go!
> > > > > > > >
> > > > > > > > Alexey Goncharuk - IGNITE-9784
> > > > > > > > Dmitriy Govorukhin - IGNITE-9898
> > > > > > > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > > > > > > Taras Ledkov - IGNITE-9882
> > > > > > > > Petr Ivanov - IGNITE-9852
> > > > > > > > Ivan Pavlukhin - IGNITE-5935
> > > > > > > > Roman Kondakov - IGNITE-9663
> > > > > > > > Alexey Stelmak - IGNITE-9776
> > > > > > > >
> > > > > > > > В Вт, 16/10/2018 в 16:20 +0300, Andrey Gura пишет:
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I've found that IGNITE-9723 was resolved but didn't cherry
> > > picked
> > > > >
> > > > > to
> > > > > > > > > ignite-2.7 branch. So I'll do it.
> > > > > > > > > On Tue, Oct 16, 2018 at 2:30 PM Nikolay Izhikov <
> > > > >
> > > > > nizhi...@apache.org>
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hello, Igniters.
> > > > > > > > > >
> > > > > > > > > > We have 13 tickets mapped to 2.7.
> > > > > > > > > > All tickets assigned to some contributor.
> > > > > > > > > >
> > > > > > > > > > Alexey Gonchruk - IGNITE-9784, IGNITE-9895
> > > > > > > > > > Vladimir Ozerov - IGNITE-9887
> > > > > > > > > > Dmitriy Govorukhin - IGNITE-9898
> > > > > > > > > > Andrey Kuznetsov   - IGNITE-9737, IGNITE-9710
> > > > > > > > > > Petr Ivanov - IGNITE-9852
> > > > > > > > > > Ivan Pavlukhin - IGNITE-5935
> > > > > > > > > > Alexey Stelmak - IGNITE-9776
> > > > > > > > > > Roman Kondakov - IGNITE-9663
> > > > > > > > > > Taras Ledkov - IGNITE-9864
> > > > > > > > > > Igor Seliverstov   - IGNITE-9292
> > > > > > > > > >
> > > > > > > > > > В Пн, 15/10/2018 в 17:49 +0300, Andrey Gura пишет:
> > > > > > > > > > > Nikolay,
> > > > > > > > > > >
> > > > > > > > > > > I'm looking at IGNITE-9737 and IGNITE-9710 which are
> > > critical
> > > > > > >
> > > > > > > issues
> > > > > > > > > > > from my point of view.
> > > > > > > > > > > I need some time for review, possible fixes and merge.
> > > > > > > > > > > I will keep you informed.
> > > > > > > > > > > On Mon, Oct 15, 2018 at 1:46 PM Igor Sapego <
> > > > >
> > > > > isap...@apache.org>
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Guys, Python client is in the master and ignite-2.7
> > > already.
> > > > > > > > > > > >
> > > > > > > > > > > > Best Regards,
> > > > > > > > > > > > Igor
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Oct 15, 2018 at 11:33 AM Vladimir Ozerov <
> > > > > > > >
> > > > > > > > voze...@gridgain.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Nikolay,
> > > > > > > > > > > > >
> > > > > > > > > > > > > AI 2.7 will include Python thin client. TC suite is
> > > crucial
> > > > > > >
> > > > > > > part
> > > > > > > > of this
> > > > > > > > > > > > > feature, so we should keep the ticket in AI 2.7
> scope.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Oct 15, 2018 at 10:57 AM Nikolay Izhikov <
> > > > > > > >
> > > > > > > > nizhi...@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hello, Igniters.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > There is no progress till 

Re: Possible concurrency bug in GridCacheLockImpl

[Ivan Daschinsky]

Sorry, I meant not GridCacheLockImpl but CacheLockImpl.

пт, 3 апр. 2020 г. в 17:04, Ivan Daschinskiy :

> Folks,
>
> Lurking through code, it seems I found some concurrency issue in subj.
>
> * This class contains two fields, volatile Thread lockedThread and
> non-volatile int cntr (used for reentrancy)
>
> * private method incrementLockingCount() is called inside lock(). In this
> method we perform volatile read in assert
> (assert (lockedThread == null && cntr == 0) || (lockedThread ==
> Thread.currentThread() && cntr > 0) then increment cntr;
> * In method unlock(), we firstly decrement cntr and after that if cntr
> equals to 0, performs volatile write to lockedThread.
>
> Suppose execution when asserts are enabled.
>
> T1  |   T2
>
> ---
> cntr = cntr - 1   (cntr == 0) |
> lockedThread = null  |
>  |lockedThread == null
> && cntr == 0
>  |   cntr = cntr + 1 (cntr
> == 1)
>
>
> There is a happens-before edge and we have strong guarantee that
> reentrancy will works and cntr will definitely equals to 1;
>
> But if assertions are disabled, something can go wrong.
>
> Moreover, if assertions are disabled, we can allow other thread do obtain
> lock even if our thread holds it.
>
> I think that this should be fixed, for example we can throw
> IllegalStateException, as in unlock() method.
>
> WDYT?
>
>
>

-- 
Sincerely yours, Ivan Daschinskiy

Re: Why OPTIMIZE_REUSE_RESULT is set to 0 explicitly in o.a.i.i.p.query.h2.ConnectionManager#DEFAULT_DB_OPTIONS

[Ivan Daschinsky]

Thank you for your response, Konstantin

I think we cannot just turn this optimization
>
Of course not, that is the reason why I started this thread.

org.apache.ignite.internal.processors.query.h2.opt.GridH2Table don't change
> it neither on remove or update


 Yes, but we can increment this counter and find possible workarounds.

 It should be benchmarked first.
>
Absolutelly agree.

I started this thread to discuss possible caveats and traps and decide
whether to implement this optimization or not.
If there is too many traps or moreover this is impossible or SQL
maintainers are against this, then it's probably not worth to do it.



ср, 22 апр. 2020 г. в 09:52, Konstantin Orlov :

> Hello, Ivan
>
> I think we cannot just turn this optimization on because a result
> invalidation counts on org.h2.engine.Database#modificationDataId (see
> org.h2.command.dml.Query#sameResultAsLast(Session, Value[], Value[],
> long)), but org.apache.ignite.internal.processors.query.h2.opt.GridH2Table
> don't change it neither on remove or update. And incrementing
> org.h2.engine.Database#modificationDataId on each data update may lead to
> some performance issues. It should be benchmarked first.
>
> --
> Regards,
> Konstantin Orlov
>
>
> > On 21 Apr 2020, at 14:51, Ivan Daschinskiy  wrote:
> >
> > Hello folks.
> >
> > Recently I trying to investigate why when the query, i.e. «SELECT * FROM
> T1 WHERE ID IN (SELECT T1_ID FROM T2), is executed locally,
> > subquery evaluated, even if result is the same. I noticed, that in
> mentioned in subj (.ConnectionManager#DEFAULT_DB_OPTIONS),
> > parameter OPTIMIZE_REUSE_RESULT explicitly set to 0. This value disable
> internal H2 optimization (see details here
> org.h2.command.dml.Query#query(int, org.h2.result.ResultTarget) and lead
> the mentioned above query to be ineffective.
> >
> > I cannot understand why this decision was made. Also I don’t find any
> evidence that setting OPTIMIZE_REUSE_RESULT to 1 could break something.
> Unfortunatelly, it is impossible to change this behavior per Session
> without forking H2, because this flag is set in org.h2.engine.Database.
> >
> > Let’s discuss possible caveats in enabling this optimization by default
> in DEFAULT_DB_OPTIONS.
> >
>
>

-- 
Sincerely yours, Ivan Daschinskiy

Re: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]

[Ivan Daschinsky]

Artem, in ignite 2.9 a way to build C++ for linux/mac os x was changed
(autotools to cmake). As an author of this change, I want to contribute in
documentation.
As far as I understand, now it should be done through PR to specific
repository. Could you please help me with this?

пт, 28 авг. 2020 г. в 16:33, Anton Kalashnikov :

> Hi Guys,
>
> As I understand we will be merging some tickets to release. May I suggest
> also add ticket [1] to 2.9 release.
>
> There are not a lot of changes in code but It's a critical fix for the
> ability to launch ignite in lamba on Azure(There are not any workaround).
>
> So if nobody minds let's merge it to 2.9.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-13013
>
> --
> Best regards,
> Anton Kalashnikov
>
>
>
> 28.08.2020, 11:16, "Alex Plehanov" :
> > Guys,
> >
> > We have benchmarked 2.9 without IGNITE-13060 and IGNITE-12568 (reverted
> it
> > locally) and got the same performance as on 2.8.1
> >
> > IGNITE-13060 (Tracing) - some code was added to hot paths, to trace these
> > hot paths, it's clear why we have performance drop here.
> >
> > IGNITE-12568 (MessageFactory refactoring) - switch/case block was
> > refactored to an array of message suppliers. The message factory is on
> the
> > hot path, which explains why this commit has an impact on total
> > performance.
> > I've checked JIT assembly output, done some JMH microbenchmarks, and
> found
> > that old implementation of MessageFactory.create() about 30-35% faster
> than
> > the new one. The reason - approach with switch/case can effectively
> inline
> > message creation code, but with an array of suppliers relatively heavy
> > "invokeinterface" cannot be skipped. I've tried to rewrite the code using
> > an abstract class for suppliers instead of an interface (to
> > replace "invokeinterface" with the "invokevirtual"), but it gives back
> only
> > 10% of method performance and in this case, code looks ugly (lambdas
> can't
> > be used). Currently, I can't find any more ways to optimize the current
> > approach (except return to the switch/case block). Andrey Gura, as the
> > author of IGNITE-12568, maybe you have some ideas about optimization?
> >
> > Perhaps we should revert IGNITE-12568, but there are some metrics already
> > created, which can't be rewritten using old message factory
> implementation
> > (IGNITE-12756). Guys, WDYT?
> >
> > пт, 28 авг. 2020 г. в 01:52, Denis Magda :
> >
> >>  Looks beautiful and easy to use, thanks, Artem! Could you please add
> the
> >>  following copyright to the footer of the pages?
> >>
> >>  *© 2020 The Apache Software Foundation.*
> >>  *Apache, Apache Ignite, the Apache feather and the Apache Ignite logo
> are
> >>  either registered trademarks or trademarks of The Apache Software
> >>  Foundation. *
> >>  *Privacy Policy*
> >>
> >>  -
> >>  Denis
> >>
> >>  On Thu, Aug 27, 2020 at 5:20 AM Artem Budnikov <
> >>  a.budnikov.ign...@gmail.com> wrote:
> >>
> >>>  Hi everyone,
> >>>
> >>>  We published the draft of Ignite 2.9 documentation on the Apache
> Ignite
> >>>  web-site. The docs are available via the following link:
> >>>
> >>>
> https://ignite.apache.org/docs/2.9.0/installation/installing-using-docker
> >>>
> >>>  Alex,
> >>>
> >>>  Is there an estimate for the release date?
> >>>
> >>>  -Artem
> >>>
> >>>  On 26.08.2020 17:47, Alex Plehanov wrote:
> >>>  > Denis,
> >>>  >
> >>>  > Currently, we are running mostly IgnitePutTxImplicitBenchmark
> without
> >>>  > persistence. For other benchmarks drop is lower and it's harder to
> find
> >>>  > problematic commit.
> >>>  >
> >>>  > ср, 26 авг. 2020 г. в 17:34, Denis Magda :
> >>>  >
> >>>  >> Alex,
> >>>  >>
> >>>  >> Thanks for sending an update. The drop is quite big. What are the
> >>>  types of
> >>>  >> benchmarks you are observing the degradation for (atomic puts,
> >>>  >> transactions, sql, etc.)?
> >>>  >>
> >>>  >> Let us know if any help by particular committers is required.
> >>>  >>
> >>>  >> -
> >>>  >> Denis
> >>>  >>
> >>>  >>
> >>>  >> On Wed, Aug 26, 2020 at 12:26 AM Alex Plehanov <
> >>>  plehanov.a...@gmail.com>
> >>>  >> wrote:
> >>>  >>
> >>>  >>> Hello, guys!
> >>>  >>>
> >>>  >>> We finally have some benchmark results. Looks like there is more
> than
> >>>  one
> >>>  >>> commit with a performance drop. Detected drops for those commits
> only
> >>>  >>> slightly higher than measurement error, so it was hard to find
> them
> >>>  and
> >>>  >> we
> >>>  >>> are not completely sure we found them all and found them right.
> >>>  >>>
> >>>  >>> Drops detected:
> >>>  >>> 2-3% drop on commit 99b0e0143e0 (IGNITE-13060 Tracing: initial
> >>>  >>> implementation)
> >>>  >>> 2-3% drop on commit 65c30ec6947 (IGNITE-12568 MessageFactory is
> >>>  >> refactored
> >>>  >>> in order to detect registration of message with the same direct
> type)
> >>>  >>>
> >>>  >>> The total drop we have on our environment - 7-8% and perhaps
> there is
> >>>  >>> something else here (benchmarks still in progress, I 

Re: [MTCGA]: new failures in builds [5665538] needs to be handled

[Ivan Daschinsky]

It seems, that in this test we use MulticastIpFinder. I suppose, that it
should be fixed.
Excerpts from a failed test's log:

[15-Oct-2020 02:47:10][WARN ][main][TcpDiscoveryMulticastIpFinder]
TcpDiscoveryMulticastIpFinder has no pre-configured addresses (it is
recommended in production to specify at least one address in
TcpDiscoveryMulticastIpFinder.getAddresses() configuration property)
[15-Oct-2020 02:47:26][ERROR][main][IgniteKernal] Failed to start manager:
GridManagerAdapter [enabled=true,
name=o.a.i.i.managers.discovery.GridDiscoveryManager]
class org.apache.ignite.IgniteCheckedException: Failed to start SPI:
TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000,
marsh=JdkMarshaller
[clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@18fc665b],
reconCnt=10, reconDelay=2000, maxAckTimeout=60, soLinger=5,
forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null,
skipAddrsRandomization=false]
  at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:281)
  at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:974)
  at
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1933)
  at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1289)
  at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2096)
  at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1748)
  at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1143)
  at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:641)
  at
org.apache.ignite.internal.processors.platform.PlatformAbstractBootstrap.start(PlatformAbstractBootstrap.java:43)
  at
org.apache.ignite.internal.processors.platform.PlatformIgnition.start(PlatformIgnition.java:74)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Impossible to
continue join, check if local discovery and communication ports are not
blocked with firewall [addr=lab11.gridgain.local/172.25.1.11:47500,
req=TcpDiscoveryJoinRequestMessage [node=TcpDiscoveryNode
[id=8d1076b1-b131-4c80-abd7-90fda1daba64,
consistentId=0:0:0:0:0:0:0:1,127.0.0.1,172.25.1.162,172.31.176.1:47500,
addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 172.25.1.162, 172.31.176.1],
sockAddrs=HashSet [/0:0:0:0:0:0:0:1:47500,
publicagent01_02.mshome.net/172.31.176.1:47500, /127.0.0.1:47500, /
172.25.1.162:47500], discPort=47500, order=0, intOrder=0,
lastExchangeTime=1602730030490, loc=true,
ver=2.10.0#20201015-sha1:, isClient=false],
dataPacket=org.apache.ignite.spi.discovery.tcp.internal.DiscoveryDataPacket@4ee43b4f,
super=TcpDiscoveryAbstractMessage [sndNodeId=null,
id=370afaa2571-8d1076b1-b131-4c80-abd7-90fda1daba64, verifierNodeId=null,
topVer=0, pendingIdx=0, failedNodes=null, isClient=false]],
discoLocalPort=47500, discoLocalPortRange=100]

чт, 15 окт. 2020 г. в 06:11, :

> Hi Igniters,
>
>  I've detected some new issue on TeamCity to be handled. You are more than
> welcomed to help.
>
>  If your changes can lead to this failure(s): We're grateful that you were
> a volunteer to make the contribution to this project, but things change and
> you may no longer be able to finalize your contribution.
>  Could you respond to this email and indicate if you wish to continue and
> fix test failures or step down and some committer may revert you commit.
>
>  *New Critical Failure in master Platform .NET (Long Running)
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PlatformNetLongRunning?branch=%3Cdefault%3E
>  Changes may lead to failure were done by
>  - bojidar marinov 
> https://ci.ignite.apache.org/viewModification.html?modId=908629
>  - igor sapego 
> https://ci.ignite.apache.org/viewModification.html?modId=908581
>  - ivan daschinskiy 
> https://ci.ignite.apache.org/viewModification.html?modId=908625
>  - ivan daschinskiy 
> https://ci.ignite.apache.org/viewModification.html?modId=908576
>  - maxim muzafarov 
> https://ci.ignite.apache.org/viewModification.html?modId=908579
>
>  - Here's a reminder of what contributors were agreed to do
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
>  - Should you have any questions please contact
> dev@ignite.apache.org
>
> Best Regards,
> Apache Ignite TeamCity Bot
> https://github.com/apache/ignite-teamcity-bot
> Notification generated at 06:11:16 15-10-2020
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: [MTCGA]: new failures in builds [5665538] needs to be handled

[Ivan Daschinsky]

>> Do you think this can be caused by the recent changes in GridNioServer
Of course no. Discovery has nothing common with GridNioServer at all.
>> but it was fine for 5 years, and now got broken.
It seems that this is an infrastructure issue. Do you
see lab11.gridgain.local in log? I suppose that this host should not be a
part of test topology.
So, if we want to have a stable test, we should not use MulticastIpFinder
except in some specific cases. At least, default multicast group should be
changed.


чт, 15 окт. 2020 г. в 10:14, Pavel Tupitsyn :

> Ivan, yes, it does use MulticastIpFinder, which can be problematic in
> tests,
> but it was fine for 5 years, and now got broken.
>
> Do you think this can be caused by the recent changes in GridNioServer?
>
> Pavel
>
> On Thu, Oct 15, 2020 at 9:41 AM Ivan Daschinsky 
> wrote:
>
> > It seems, that in this test we use MulticastIpFinder. I suppose, that it
> > should be fixed.
> > Excerpts from a failed test's log:
> >
> > [15-Oct-2020 02:47:10][WARN ][main][TcpDiscoveryMulticastIpFinder]
> > TcpDiscoveryMulticastIpFinder has no pre-configured addresses (it is
> > recommended in production to specify at least one address in
> > TcpDiscoveryMulticastIpFinder.getAddresses() configuration property)
> > [15-Oct-2020 02:47:26][ERROR][main][IgniteKernal] Failed to start
> manager:
> > GridManagerAdapter [enabled=true,
> > name=o.a.i.i.managers.discovery.GridDiscoveryManager]
> > class org.apache.ignite.IgniteCheckedException: Failed to start SPI:
> > TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000,
> > marsh=JdkMarshaller
> > [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@18fc665b],
> > reconCnt=10, reconDelay=2000, maxAckTimeout=60, soLinger=5,
> > forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null,
> > skipAddrsRandomization=false]
> >   at
> >
> >
> org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:281)
> >   at
> >
> >
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:974)
> >   at
> >
> >
> org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1933)
> >   at
> org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1289)
> >   at
> >
> >
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2096)
> >   at
> >
> >
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1748)
> >   at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1143)
> >   at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:641)
> >   at
> >
> >
> org.apache.ignite.internal.processors.platform.PlatformAbstractBootstrap.start(PlatformAbstractBootstrap.java:43)
> >   at
> >
> >
> org.apache.ignite.internal.processors.platform.PlatformIgnition.start(PlatformIgnition.java:74)
> > Caused by: class org.apache.ignite.spi.IgniteSpiException: Impossible to
> > continue join, check if local discovery and communication ports are not
> > blocked with firewall [addr=lab11.gridgain.local/172.25.1.11:47500,
> > req=TcpDiscoveryJoinRequestMessage [node=TcpDiscoveryNode
> > [id=8d1076b1-b131-4c80-abd7-90fda1daba64,
> > consistentId=0:0:0:0:0:0:0:1,127.0.0.1,172.25.1.162,172.31.176.1:47500,
> > addrs=ArrayList [0:0:0:0:0:0:0:1, 127.0.0.1, 172.25.1.162, 172.31.176.1],
> > sockAddrs=HashSet [/0:0:0:0:0:0:0:1:47500,
> > publicagent01_02.mshome.net/172.31.176.1:47500, /127.0.0.1:47500, /
> > 172.25.1.162:47500], discPort=47500, order=0, intOrder=0,
> > lastExchangeTime=1602730030490, loc=true,
> > ver=2.10.0#20201015-sha1:, isClient=false],
> >
> >
> dataPacket=org.apache.ignite.spi.discovery.tcp.internal.DiscoveryDataPacket@4ee43b4f
> > ,
> > super=TcpDiscoveryAbstractMessage [sndNodeId=null,
> > id=370afaa2571-8d1076b1-b131-4c80-abd7-90fda1daba64, verifierNodeId=null,
> > topVer=0, pendingIdx=0, failedNodes=null, isClient=false]],
> > discoLocalPort=47500, discoLocalPortRange=100]
> >
> > чт, 15 окт. 2020 г. в 06:11, :
> >
> > > Hi Igniters,
> > >
> > >  I've detected some new issue on TeamCity to be handled. You are more
> > than
> > > welcomed to help.
> > >
> > >  If your changes can lead to this failure(s): We're grateful that you
> > were
> > > a volunteer to make the contribution to this project, but things change
> > and
> > > you may no longer be able to finalize your contribution.
> > >  Could you respo

Sql consistency when partition evicts.

[Ivan Daschinsky]

Hi!
I found recently quite surprising on the first sight behaviour.

Scenario:
1. Start 2 node with indexed atomic partitioned cache with 0 backups.
2. Load sufficient amout of data (or emulate slow removal from idx)
4. Start another node.
4. Perform SELECT * FROM .

Due to partition eviction, number of returned rows from SQL are
sufficiently bigger than
expected. Reproducer is attached to jira issue [1]

Is this known issue? Is it expected behaviour due to our SQL indices design?

-
[1] -- https://issues.apache.org/jira/browse/IGNITE-13572

-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSS] Use Netty for Java thin client

[Ivan Daschinsky]

Hi.
>>  Potentially reduced resource usage - share EventLoopGroop across all
connections within one IgniteClient.
Not potentially, definitely. Current approach (one receiver thread per
TcpClientChannel and shared FJP for continuation) requires too many threads.
When TcpClientChannel is the only one, it's ok. But if we use multiple
addresses, things become worse.

>> The obvious downside is an extra dependency in the core module.
There is another downside -- we should rework our transaction's API a
little bit (Actually,  in netty socket write is performed in other thread
(channel.write is async) and
current tx logic will not work
(org.apache.ignite.internal.client.thin.TcpClientCache#writeCacheInfo))

A little bit of offtopic.
I suppose, that the java thin client (and other thin clients) should be
separated from the main ignite repo and have a separate release cycle.
For example, java thin client depends on default binary
protocol's implementation, that is notorious for heavy usage of internal
JDK api and this for example.
prevents usage of our thin client in graalvm native image.


пт, 16 окт. 2020 г. в 20:00, Pavel Tupitsyn :

> Igniters,
>
> I'm working on IEP-51 [1] to make Java thin client truly async
> and make sure user threads are never blocked
> (right now socket writes are performed from user threads).
>
> I've investigated potential approaches and came to the conclusion
> that Netty [2] is our best bet.
> - Nice Future-based async API => will greatly reduce our code complexity
>   and remove manual thread management
> - Potentially reduced resource usage - share EventLoopGroop across all
> connections within one IgniteClient
> - SSL is easy to use
> - Proven performance and reliability
>
> Other approaches, like AsynchronousSocketChannel or selectors, seem to be
> too complicated,
> especially when SSL comes into play.
> We should focus on Ignite-specific work instead of spending time on
> reinventing async IO.
>
> The obvious downside is an extra dependency in the core module.
> However, I heard some discussions about using Netty for GridNioServer in
> future.
>
> Let me know your thoughts.
>
> Pavel
>
> [1]
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-51%3A+Java+Thin+Client+Async+API
> [2] https://netty.io
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSS] Use Netty for Java thin client

[Ivan Daschinsky]

>> Why can't we use GridNioServer for java thin clients?
Yes, we can. Despite the naming, it can be used as client (set port as -1),
But doesn't have the same set of advantages as Netty. Netty has a way
better support (performance) for native transports and SSL, that
default java NIO.

But API is much, much worse.

If our goal is to keep thin client in core module in any circumstances,
that this is the only choice.

But lets see, for example, at Lettuce (netty based async redis client) - [1]
1. It supports reactive streams (additional module)
2. It supports kotlin coroutines (additional module)
I hardly believe, that we could support this in our core module.

Why not to consider separation? Why user of our thin client should have in
his classpath megabytes of unnecessary bytecode?


[1] -- https://lettuce.io/core/release/reference/index.html

пн, 19 окт. 2020 г. в 10:06, Alex Plehanov :

> Pavel,
>
> Why can't we use GridNioServer for java thin clients?
> It has the same advantages as Netty (future based async API, SSL, etc) but
> without extra dependency.
> GridClient (control.sh), for example, uses GridNioServer for communication.
>
> сб, 17 окт. 2020 г. в 11:21, Ivan Daschinsky :
>
> > Hi.
> > >>  Potentially reduced resource usage - share EventLoopGroop across all
> > connections within one IgniteClient.
> > Not potentially, definitely. Current approach (one receiver thread per
> > TcpClientChannel and shared FJP for continuation) requires too many
> > threads.
> > When TcpClientChannel is the only one, it's ok. But if we use multiple
> > addresses, things become worse.
> >
> > >> The obvious downside is an extra dependency in the core module.
> > There is another downside -- we should rework our transaction's API a
> > little bit (Actually,  in netty socket write is performed in other thread
> > (channel.write is async) and
> > current tx logic will not work
> > (org.apache.ignite.internal.client.thin.TcpClientCache#writeCacheInfo))
> >
> > A little bit of offtopic.
> > I suppose, that the java thin client (and other thin clients) should be
> > separated from the main ignite repo and have a separate release cycle.
> > For example, java thin client depends on default binary
> > protocol's implementation, that is notorious for heavy usage of internal
> > JDK api and this for example.
> > prevents usage of our thin client in graalvm native image.
> >
> >
> > пт, 16 окт. 2020 г. в 20:00, Pavel Tupitsyn :
> >
> > > Igniters,
> > >
> > > I'm working on IEP-51 [1] to make Java thin client truly async
> > > and make sure user threads are never blocked
> > > (right now socket writes are performed from user threads).
> > >
> > > I've investigated potential approaches and came to the conclusion
> > > that Netty [2] is our best bet.
> > > - Nice Future-based async API => will greatly reduce our code
> complexity
> > >   and remove manual thread management
> > > - Potentially reduced resource usage - share EventLoopGroop across all
> > > connections within one IgniteClient
> > > - SSL is easy to use
> > > - Proven performance and reliability
> > >
> > > Other approaches, like AsynchronousSocketChannel or selectors, seem to
> be
> > > too complicated,
> > > especially when SSL comes into play.
> > > We should focus on Ignite-specific work instead of spending time on
> > > reinventing async IO.
> > >
> > > The obvious downside is an extra dependency in the core module.
> > > However, I heard some discussions about using Netty for GridNioServer
> in
> > > future.
> > >
> > > Let me know your thoughts.
> > >
> > > Pavel
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-51%3A+Java+Thin+Client+Async+API
> > > [2] https://netty.io
> > >
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSS] Missed (non-suited) tests

[Ivan Daschinsky]

Ivan, as far as I understand, Max also created verification check for not
included test and found a few tests, that have never been included in any
testsuites.

Also, I suppose, that even if we cannot run some tests, these tests should
be ignored using annotation, but not commented.

пн, 19 окт. 2020 г. в 16:33, Ivan Pavlukhin :

> Hi Max,
>
> There is an existing effort about "abandoned" tests
> https://issues.apache.org/jira/browse/IGNITE-9210
>
> 2020-10-19 16:25 GMT+03:00, Max Timonin :
> > Hi Igniters!
> >
> > I made a research into tests that aren't included in any test suite. As
> > TeamCity runs tests by suites so there could be tests that never run on
> TC.
> > So I tried implementing a simple check for such tests and include it in
> > Ignite's travis config.
> >
> > The check runs while "mvn test" command and piggy-backs on the maven
> > surefire plugin. I replaced the junit provider with a custom one that
> > checks if a class is a test or a suite (there are some Ignite specific
> > stuff), marks tests that are in suites and raises an exception if there
> are
> > non-suited tests. It's implemented as a part of maven command so it runs
> > for every module separately.
> >
> > I've prepared draft PR with this check:
> > https://github.com/apache/ignite/pull/8367
> > Travis check report is here:
> > https://travis-ci.org/github/apache/ignite/jobs/737046387
> > As It's a draft, so I skip some maven configuration steps for a while.
> Also
> > I run the check only for the core module.
> >
> > But I have some results that want to discuss before continue the work:
> > 1. Currently in the core module there are 53 tests that aren't part of
> any
> > test suite. I'm not sure about the reason for every test. So I just put
> > below a list of the tests and last contributor to a file that contains a
> > test.
> >
> > 2. Some tests are located in the core module, but suites are in a
> > different, for example ignite-indexing suite
> > IgniteCacheQuerySelfTestSuite3 contains
> > only tests written in the core module, and none from the indexing module.
> > Also there are suites in spring, uri-deploy, zookeeper modules. In my PR
> > I've just copied the test suites to the core module.
> >
> > 3. Some test classes are named with the "Abstract" suffix but don't have
> > the corresponding modifier (for example, IgniteTxTimeoutAbstractTest).
> So,
> > I add the modifier for every such file if it's not a part of any suite.
> >
> > What do you think about this check? If Ignite needs it, let's discuss
> next
> > things:
> > 1. Mark tests that should never be in any suite by some reason;
> > 2. Fix the missed tests;
> > 3. How to declare suites that contains tests from a different module;
> > 4. How to check if TC runs all suites.
> >
> > List of non-suited tests in the core module:
> >
> > maksim.stepac...@gmail.com:
> > GridTcpCommunicationSpiLogTest
> >
> > nizhi...@apache.org:
> > IgniteCacheClientMultiNodeUpdateTopologyLockTest
> > CacheClientsConcurrentStartTest
> > IgniteOutOfMemoryPropagationTest
> > GridCacheP2PUndeploySelfTest
> > GridCacheRebalancingOrderingTest
> > IgniteMassLoadSandboxTest
> > PageLockTrackerMXBeanImplTest
> > IgniteBinaryMetadataUpdateNodeRestartTest
> > CacheLockCandidatesThreadTest
> > GridMBeanBaselineTest
> > RendezvousAffinityFunctionSimpleBenchmark
> >
> > samvi...@yandex.ru:
> > IgnitePdsNoSpaceLeftOnDeviceTest
> >
> > maxmu...@gmail.com:
> > GridCacheOnCopyFlagReplicatedSelfTest
> > GridCacheOnCopyFlagLocalSelfTest
> > GridCacheReplicatedAtomicReferenceMultiNodeTest
> > GridCacheReplicatedMarshallerTxTest
> > GridCacheReplicatedTxConcurrentGetTest
> > GridCacheOnCopyFlagTxPartitionedSelfTest
> > GridCacheReplicatedTxReadTest
> > GridCachePartitionedAtomicReferenceMultiNodeTest
> > GridCacheOnCopyFlagAtomicSelfTest
> >
> > mmu...@apache.org:
> > GridActivateExtensionTest
> > IgniteChangeGlobalStateCacheTest
> > IgniteChangeGlobalStateTest
> > IgniteChangeGlobalStateServiceTest
> > IgniteChangeGlobalStateDataStructureTest
> >
> > oignate...@gridgain.com:
> > CacheEntryProcessorCopySelfTest
> > MemoryLeaksOnRestartNodeTest
> > GridCacheAtomicPreloadSelfTest
> > WalCompactionAfterRestartTest
> > IgniteCacheConcurrentPutGetRemove
> > GridIoManagerBenchmark0
> >
> > nsamelc...@gmail.com:
> > GridLongRunningInitNewCrdFutureDiagnosticsTest
> > GridCacheMultithreadedFailoverAbstractTest
> >
> > alexey.goncha...@gmail.com:
> > GridCacheBinaryObjectsAtomicOnheapSelfTest
> > GridCacheBinaryObjectsAtomicNearDisabledOnheapSelfTest
> > GridCacheBinaryObjectsPartitionedOnheapSelfTest
> > GridCacheBinaryObjectsPartitionedNearDisabledOnheapSelfTest
> >
> > vladis...@gmail.com:
> > 

Re: IEP-52: Binary Delivery & Upgradability Enhancements

[Ivan Daschinsky]

Hi! I recently found that in slim binary release (at least in nightly build
of 2.9.0) there is no ignite-zookeeper module.
But, for example, ignite-kubernetes is in.
Petr, could you please clarify why this decision was made?
We currently have only two discovery implementation in project and leave
only one alternative looks not obvious for me.

ср, 16 сент. 2020 г. в 17:04, Alexei Scherbakov <
alexey.scherbak...@gmail.com>:

> Downloading additional jars will not 100% work in private networks, so we
> must provide full distro anyway.
>
> Also, I do not see any problems in downloading all dependencies, because
> their summary size is not too big.
>
> I suggest, in addition to tgz archive for manual deployment, provide Ignite
> distribution as RPM (or other package manager(s)) for Linux (MSI for
> Windows) to automate distribution upgrades, instead of reinventing the
> wheel.
>
> пн, 31 авг. 2020 г. в 10:09, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
>
> > Hi Nikolay,
> >
> > Thanks for your feedback! Absolutely, the IEP is currently in the draft
> > mode - we will be adding more details going forward. Also, Petr mentioned
> > that he wants to create a prototype which I think will make things
> clearer
> > for all of us.
> >
> > As for "download jars from the Internet on a production server", that's
> not
> > necessarily what we propose. In case of Maven, artifacts are downloaded
> > from a specified repository, which doesn't have to be on the Internet.
> > Companies that have security concerns typically have a mirror/proxy
> > containing approved artifacts only, already checked for viruses, etc.
> > Production servers use the proxy rather than public repositories.
> >
> > The concerns that you brought up are certainly valid. However, I believe
> > that using Maven actually addresses them, because Maven (as any other
> > package/dependency manager for that matter) provides the corresponding
> > functionality out of the box.
> >
> > On the contrary, users who currently download our ZIP package are
> *forced*
> > to download modules and dependencies that they do not even need and will
> > not use. In my experience, this is actually a bigger source of concern.
> >
> > Anyway, let us add more details to the IEP, and we will go from there.
> >
> > P.S. I'm also not sure I agree with your assessment of the embedded mode
> > and thick clients, but that's probably a different discussion. Please
> feel
> > free to create a new thread if you would like to discuss this.
> >
> > -Val
> >
> > On Sat, Aug 29, 2020 at 1:46 AM Nikolay Izhikov 
> > wrote:
> >
> > > Also, guys.
> > >
> > > Can we make one step back and add some requirements to the IEP.
> > > How we want to install and upgrade Ignite?
> > > What specific issues we want to solve?
> > >
> > > With this input we can be on the same page during solution discussion.
> > >
> > > > 29 авг. 2020 г., в 11:41, Nikolay Izhikov 
> > > написал(а):
> > > >
> > > > Hello, Val.
> > > >
> > > > Sorry, but I sill don’t understand how «download jars from the
> internet
> > > on production server» approach helps us to solve the issues you
> describe.
> > > >
> > > > Moreover, with this enhancement you want to add more dependencies to
> > > Ignite and create some security vulnerabilities:
> > > >   * Maven tool - new dependency. I doubt that and of the
> production
> > > server have it
> > > >   * Internet maven repository - many production servers restrict
> > > outgoing internet connection to any addresses.
> > > >   It’s hard to understand why distributed DB should have
> > the
> > > ability to connect to some kind of address on the Internet.
> > > >   * All Ignite distributions can be compromised with the hack of
> > > some random maven repo or with the malicious proxy.
> > > >   * Note, that Ignite installation or upgrade procedure can
> become
> > > dead just because of some Maven repo down.
> > > >
> > > >
> > > > I have 2 concerns:
> > > >
> > > > 1. We shouldn’t download any jars from the internet during production
> > > deployment or upgrade procedure.
> > > > 2. The IEP description, for now, is quite small for such important
> > > things as a way we distribute and upgrade Ignite.
> > > >   Petr, can you, please, make IEP more specific.
> > > >   I think we should add the following details:
> > > >   * Description of the typical Ignite deployment
> procedure
> > > including folder structure(config, persistence, .m2 and other folders)
> > > >   * This also should include examples of the bash
> > > commands if any required.
> > > >   * Description of the typical upgrade(downgrade?)
> > procedure
> > > >   * This also should include examples of the bash
> > > commands if any required.
> > > >   * Description of «single tool responsible for all
> > > operations».
> > > >   * all commands
> > > >   * all operations

Re: IEP-52: Binary Delivery & Upgradability Enhancements

[Ivan Daschinsky]

I suggested to move TcpDiscoveryZookeeperIpFinder to ignite-extension.
This tiny recipe-like class brings tons of dependency and has nothing
common with ZookeeperDiscovery, except it name.

ZookeeperDiscoveryImpl depends only on zookeeper-3.5.5.jar and
zookeeper-jute-3.5.5.jar.




чт, 17 сент. 2020 г. в 15:07, Ilya Kasnacheev :

> Hello!
>
> The situation as follows:
> libs/optional/ignite-kubernetes:
> ignite-kubernetes-2.8.1.jar  jackson-annotations-2.9.10.jar
>  jackson-core-2.9.10.jar  jackson-databind-2.9.10.jar
>
> libs/optional/ignite-zookeeper:
> curator-client-4.2.0.jar curator-x-discovery-4.2.0.jar
>  jackson-core-asl-1.9.13.jarlog4j-1.2.17.jar
> slf4j-log4j12-1.7.7.jar
> curator-framework-4.2.0.jar  guava-16.0.1.jar
>   jackson-mapper-asl-1.9.13.jar  zookeeper-3.5.5.jar
> curator-recipes-4.2.0.jarignite-zookeeper-2.8.1.jar
>  slf4j-api-1.7.7.jar
>  zookeeper-jute-3.5.5.jar
>
> ignite-zookeeper just has way more dependencies than ignite-kubernetes, and
> their size is 5-fold.
>
> These dependencies are not just weight but also vulnerability surface.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 17 сент. 2020 г. в 13:55, Ivan Daschinsky :
>
> > Hi! I recently found that in slim binary release (at least in nightly
> build
> > of 2.9.0) there is no ignite-zookeeper module.
> > But, for example, ignite-kubernetes is in.
> > Petr, could you please clarify why this decision was made?
> > We currently have only two discovery implementation in project and leave
> > only one alternative looks not obvious for me.
> >
> > ср, 16 сент. 2020 г. в 17:04, Alexei Scherbakov <
> > alexey.scherbak...@gmail.com>:
> >
> > > Downloading additional jars will not 100% work in private networks, so
> we
> > > must provide full distro anyway.
> > >
> > > Also, I do not see any problems in downloading all dependencies,
> because
> > > their summary size is not too big.
> > >
> > > I suggest, in addition to tgz archive for manual deployment, provide
> > Ignite
> > > distribution as RPM (or other package manager(s)) for Linux (MSI for
> > > Windows) to automate distribution upgrades, instead of reinventing the
> > > wheel.
> > >
> > > пн, 31 авг. 2020 г. в 10:09, Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com>:
> > >
> > > > Hi Nikolay,
> > > >
> > > > Thanks for your feedback! Absolutely, the IEP is currently in the
> draft
> > > > mode - we will be adding more details going forward. Also, Petr
> > mentioned
> > > > that he wants to create a prototype which I think will make things
> > > clearer
> > > > for all of us.
> > > >
> > > > As for "download jars from the Internet on a production server",
> that's
> > > not
> > > > necessarily what we propose. In case of Maven, artifacts are
> downloaded
> > > > from a specified repository, which doesn't have to be on the
> Internet.
> > > > Companies that have security concerns typically have a mirror/proxy
> > > > containing approved artifacts only, already checked for viruses, etc.
> > > > Production servers use the proxy rather than public repositories.
> > > >
> > > > The concerns that you brought up are certainly valid. However, I
> > believe
> > > > that using Maven actually addresses them, because Maven (as any other
> > > > package/dependency manager for that matter) provides the
> corresponding
> > > > functionality out of the box.
> > > >
> > > > On the contrary, users who currently download our ZIP package are
> > > *forced*
> > > > to download modules and dependencies that they do not even need and
> > will
> > > > not use. In my experience, this is actually a bigger source of
> concern.
> > > >
> > > > Anyway, let us add more details to the IEP, and we will go from
> there.
> > > >
> > > > P.S. I'm also not sure I agree with your assessment of the embedded
> > mode
> > > > and thick clients, but that's probably a different discussion. Please
> > > feel
> > > > free to create a new thread if you would like to discuss this.
> > > >
> > > > -Val
> > > >
> > > > On Sat, Aug 29, 2020 at 1:46 AM Nikolay Izhikov  >
> > > > wrote:
> > > >
> > > > > Also, guys.
> > > > >
> > > > > Can we make one step ba

Re: IEP-52: Binary Delivery & Upgradability Enhancements

[Ivan Daschinsky]

I mean, that we should fix this issue, not just throw away zookeeper
discovery.

чт, 17 сент. 2020 г. в 16:44, Ivan Daschinsky :

> I suggested to move TcpDiscoveryZookeeperIpFinder to ignite-extension.
> This tiny recipe-like class brings tons of dependency and has nothing
> common with ZookeeperDiscovery, except it name.
>
> ZookeeperDiscoveryImpl depends only on zookeeper-3.5.5.jar and
> zookeeper-jute-3.5.5.jar.
>
>
>
>
> чт, 17 сент. 2020 г. в 15:07, Ilya Kasnacheev :
>
>> Hello!
>>
>> The situation as follows:
>> libs/optional/ignite-kubernetes:
>> ignite-kubernetes-2.8.1.jar  jackson-annotations-2.9.10.jar
>>  jackson-core-2.9.10.jar  jackson-databind-2.9.10.jar
>>
>> libs/optional/ignite-zookeeper:
>> curator-client-4.2.0.jar curator-x-discovery-4.2.0.jar
>>  jackson-core-asl-1.9.13.jarlog4j-1.2.17.jar
>> slf4j-log4j12-1.7.7.jar
>> curator-framework-4.2.0.jar  guava-16.0.1.jar
>>   jackson-mapper-asl-1.9.13.jar  zookeeper-3.5.5.jar
>> curator-recipes-4.2.0.jarignite-zookeeper-2.8.1.jar
>>  slf4j-api-1.7.7.jar
>>  zookeeper-jute-3.5.5.jar
>>
>> ignite-zookeeper just has way more dependencies than ignite-kubernetes,
>> and
>> their size is 5-fold.
>>
>> These dependencies are not just weight but also vulnerability surface.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> чт, 17 сент. 2020 г. в 13:55, Ivan Daschinsky :
>>
>> > Hi! I recently found that in slim binary release (at least in nightly
>> build
>> > of 2.9.0) there is no ignite-zookeeper module.
>> > But, for example, ignite-kubernetes is in.
>> > Petr, could you please clarify why this decision was made?
>> > We currently have only two discovery implementation in project and leave
>> > only one alternative looks not obvious for me.
>> >
>> > ср, 16 сент. 2020 г. в 17:04, Alexei Scherbakov <
>> > alexey.scherbak...@gmail.com>:
>> >
>> > > Downloading additional jars will not 100% work in private networks,
>> so we
>> > > must provide full distro anyway.
>> > >
>> > > Also, I do not see any problems in downloading all dependencies,
>> because
>> > > their summary size is not too big.
>> > >
>> > > I suggest, in addition to tgz archive for manual deployment, provide
>> > Ignite
>> > > distribution as RPM (or other package manager(s)) for Linux (MSI for
>> > > Windows) to automate distribution upgrades, instead of reinventing the
>> > > wheel.
>> > >
>> > > пн, 31 авг. 2020 г. в 10:09, Valentin Kulichenko <
>> > > valentin.kuliche...@gmail.com>:
>> > >
>> > > > Hi Nikolay,
>> > > >
>> > > > Thanks for your feedback! Absolutely, the IEP is currently in the
>> draft
>> > > > mode - we will be adding more details going forward. Also, Petr
>> > mentioned
>> > > > that he wants to create a prototype which I think will make things
>> > > clearer
>> > > > for all of us.
>> > > >
>> > > > As for "download jars from the Internet on a production server",
>> that's
>> > > not
>> > > > necessarily what we propose. In case of Maven, artifacts are
>> downloaded
>> > > > from a specified repository, which doesn't have to be on the
>> Internet.
>> > > > Companies that have security concerns typically have a mirror/proxy
>> > > > containing approved artifacts only, already checked for viruses,
>> etc.
>> > > > Production servers use the proxy rather than public repositories.
>> > > >
>> > > > The concerns that you brought up are certainly valid. However, I
>> > believe
>> > > > that using Maven actually addresses them, because Maven (as any
>> other
>> > > > package/dependency manager for that matter) provides the
>> corresponding
>> > > > functionality out of the box.
>> > > >
>> > > > On the contrary, users who currently download our ZIP package are
>> > > *forced*
>> > > > to download modules and dependencies that they do not even need and
>> > will
>> > > > not use. In my experience, this is actually a bigger source of
>> concern.
>> > > >
>> > > > Anyway, let us add more details to the IEP, and we will go from
>> there.
>> > > >

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

I appreciate any help, thank you, Ilya.

Currently I have a small PR without ticket (link in first post),but I
decided not to file a jira issue before discussion.
Now I see, that this feature are of great interest to community. So I file
a ticket, test myself on my home laptop (ubuntu 20.04)
 and add detailed instructions to DEVNOTES.txt in a few days.

I would be happy if my someone can follow the instruction and write
possible issues.

I will notify about status update in this thread in next few days.

Thank you all very much for support!


вт, 26 мая 2020 г. в 17:50, Ilya Kasnacheev :

> Hello!
>
> I will assist with checking on Linux if you would contribute a patch.
> Please start with a ticket (or even an IEP maybe?)
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 26 мая 2020 г. в 16:47, Ivan Daschinsky :
>
> > Guys, I will certainly thoroughly test my fix not only unices, but on
> > windows too.
> > And I will describe it very thoroughly.
> >
> > When I was C++ developer (more than 10 years ago), I have not any trouble
> > at all with CMake and Visual Studio 2005.
> > Everything works and works good. Moreover, you can build with NMake,
> > msbuild and generate solutions for development.
> >
> > I suppose, for CI purposes, using NMake is a way better, than use vs
> > solutions.
> >
> > вт, 26 мая 2020 г. в 16:42, Nikolay Izhikov :
> >
> > > Hello, Igor.
> > >
> > > > Nikolay, removing support for a certain build system is a breaking
> > > change.
> > >
> > > No, it’s not.
> > > Why do you think so?
> > >
> > > Development environment and build system is a subject to change in any
> > > project.
> > > We can drop or add support of any build system any time we want.
> > >
> > > > 26 мая 2020 г., в 16:35, Ilya Kasnacheev 
> > > написал(а):
> > > >
> > > > Hello!
> > > >
> > > > I don't see why we can't get rid of autotools in a minor release,
> > > provided
> > > > that cmake actually works. Removing native VS support may be a
> > different
> > > > thing.
> > > >
> > > > Build system and precise set of dependencies is not a part of public
> > API
> > > in
> > > > my opinion.
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > вт, 26 мая 2020 г. в 16:02, Igor Sapego :
> > > >
> > > >> Great!
> > > >>
> > > >> Let's start with creating a TC suite for it.
> > > >>
> > > >> The only concern I have is that it is one more build system
> > > >> to support. Should we get rid of autotools in 3.0?
> > > >>
> > > >> Best Regards,
> > > >> Igor
> > > >>
> > > >>
> > > >> On Tue, May 26, 2020 at 2:44 PM Alexey Kukushkin <
> > > >> kukushkinale...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> +1. I recently completed a cross-IDE (MS Visual Studio & GCC/GDB)
> > > Ignite
> > > >>> C++ project and CMake in Ignite C++ would save me a day of effort.
> > > >>>
> > > >>> вт, 26 мая 2020 г. в 12:09, Pavel Tupitsyn :
> > > >>>
> > > >>>> +1
> > > >>>>
> > > >>>> On Tue, May 26, 2020 at 12:02 PM Zhenya Stanilovsky
> > > >>>>  wrote:
> > > >>>>
> > > >>>>>
> > > >>>>> Ivan huge +1 with your proposal.
> > > >>>>> I had some problems with odbc tests building too, looks like
> cmake
> > > >> will
> > > >>>>> make it more easy.
> > > >>>>>> Hello Igniters.
> > > >>>>>>
> > > >>>>>> I’d like to discuss porting build process of Ignite.C++. I think
> > > >> that
> > > >>>>> there is time to change it.
> > > >>>>>>
> > > >>>>>> *Motivation*
> > > >>>>>> Currently, it is hard to build Ignite.C++. Different build
> process
> > > >> for
> > > >>>>> windows and linux, lack of building support on Mac OS X (quite
> > > >> popular
> > > >>> OS
> > > >>>>> among developers), absolutely not IDE sup

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Guys, I will certainly thoroughly test my fix not only unices, but on
windows too.
And I will describe it very thoroughly.

When I was C++ developer (more than 10 years ago), I have not any trouble
at all with CMake and Visual Studio 2005.
Everything works and works good. Moreover, you can build with NMake,
msbuild and generate solutions for development.

I suppose, for CI purposes, using NMake is a way better, than use vs
solutions.

вт, 26 мая 2020 г. в 16:42, Nikolay Izhikov :

> Hello, Igor.
>
> > Nikolay, removing support for a certain build system is a breaking
> change.
>
> No, it’s not.
> Why do you think so?
>
> Development environment and build system is a subject to change in any
> project.
> We can drop or add support of any build system any time we want.
>
> > 26 мая 2020 г., в 16:35, Ilya Kasnacheev 
> написал(а):
> >
> > Hello!
> >
> > I don't see why we can't get rid of autotools in a minor release,
> provided
> > that cmake actually works. Removing native VS support may be a different
> > thing.
> >
> > Build system and precise set of dependencies is not a part of public API
> in
> > my opinion.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > вт, 26 мая 2020 г. в 16:02, Igor Sapego :
> >
> >> Great!
> >>
> >> Let's start with creating a TC suite for it.
> >>
> >> The only concern I have is that it is one more build system
> >> to support. Should we get rid of autotools in 3.0?
> >>
> >> Best Regards,
> >> Igor
> >>
> >>
> >> On Tue, May 26, 2020 at 2:44 PM Alexey Kukushkin <
> >> kukushkinale...@gmail.com>
> >> wrote:
> >>
> >>> +1. I recently completed a cross-IDE (MS Visual Studio & GCC/GDB)
> Ignite
> >>> C++ project and CMake in Ignite C++ would save me a day of effort.
> >>>
> >>> вт, 26 мая 2020 г. в 12:09, Pavel Tupitsyn :
> >>>
>  +1
> 
>  On Tue, May 26, 2020 at 12:02 PM Zhenya Stanilovsky
>   wrote:
> 
> >
> > Ivan huge +1 with your proposal.
> > I had some problems with odbc tests building too, looks like cmake
> >> will
> > make it more easy.
> >> Hello Igniters.
> >>
> >> I’d like to discuss porting build process of Ignite.C++. I think
> >> that
> > there is time to change it.
> >>
> >> *Motivation*
> >> Currently, it is hard to build Ignite.C++. Different build process
> >> for
> > windows and linux, lack of building support on Mac OS X (quite
> >> popular
> >>> OS
> > among developers), absolutely not IDE support, except windows and
> >> only
> > Visual Studio is supported.
> >>
> >> *Suggestion*
> >> I’d suggest to migrate to CMake build system. It is very popular
> >> among
> > open source projects, and in Apache Software Foundation too. Notable
>  user:
> > Apache Mesos, Apache Zookeeper (C client offers CMake as an
> >> alternative
>  to
> > autoconf and only option on windows), Apache Kafka (librdkafka -
> >> C/C++
> > client), Apache Thrift. Popular column-oriented database ClickHouse
> >>> also
> > uses CMake.
> >>
> >> CMake is widely supported in many IDE’s on various platforms,
> >> notably
> > Visual Studio, CLion, Xcode, QtCreator, KDevelop.
> >>
> >> *Current status*
> >>
> >> Currently, the most of work is done (see [1]) and tested on Mac OS X
> > 10.15 (some C++ porting). All tests are run without any flaws,
> >>> including
> > odbc (unixodbc), ssl, thin and thick client, installation, IDE
>  integration
> > (CLion). Next steps is to test linux and windows.
> >>
> >> But full migration isn’t possible without agreement and help of
> > community. Even if most of all you agree, migration requires
> >> additional
> > efforts in TC agents tuning and so on (event though test running
> >> fully
> > automated by CMake CTest).
> >>
> >> Lets discuss my proposition and idea.
> >>
> >> [1] -  https://github.com/apache/ignite/pull/7845
> >>
> >>
> >>
> >
> >
> >
> >
> 
> >>>
> >>>
> >>> --
> >>> Best regards,
> >>> Alexey
> >>>
> >>
>
>

-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Igor, I just said about native support from Visual Studio, not from cmake
itself.

Modern cmake can generate projects even for Visual Studio 2008 [1]

[1] --
https://cmake.org/cmake/help/v3.16/generator/Visual%20Studio%209%202008.html

вт, 26 мая 2020 г. в 16:16, Igor Sapego :

> Nikolay, removing support for a certain build system is a breaking change.
>
> Ivan, in C++ world there are companies that still use VS 2012 or even 2010.
>
> Best Regards,
> Igor
>
>
> On Tue, May 26, 2020 at 4:08 PM Ivan Daschinskiy 
> wrote:
>
> > Guys, thank you all for support
> >
> >  >> The only concern I have is that it is one more build system
> > to support. Should we get rid of autotools in 3.0?
> >
> > I'd suggest leave CMake as an only build system. Cmake could generate VS
> > projects more than 12 years ago flawlessly, moreover, VS since 2017 has
> > native support of CMake, so even this step (generation of project) is no
> > more necessary.
> >
> >
> > On 26.05.2020 16:02, Igor Sapego wrote:
> > > Great!
> > >
> > > Let's start with creating a TC suite for it.
> > >
> > > The only concern I have is that it is one more build system
> > > to support. Should we get rid of autotools in 3.0?
> > >
> > > Best Regards,
> > > Igor
> > >
> > >
> > > On Tue, May 26, 2020 at 2:44 PM Alexey Kukushkin <
> > kukushkinale...@gmail.com>
> > > wrote:
> > >
> > >> +1. I recently completed a cross-IDE (MS Visual Studio & GCC/GDB)
> Ignite
> > >> C++ project and CMake in Ignite C++ would save me a day of effort.
> > >>
> > >> вт, 26 мая 2020 г. в 12:09, Pavel Tupitsyn :
> > >>
> > >>> +1
> > >>>
> > >>> On Tue, May 26, 2020 at 12:02 PM Zhenya Stanilovsky
> > >>>  wrote:
> > >>>
> >  Ivan huge +1 with your proposal.
> >  I had some problems with odbc tests building too, looks like cmake
> > will
> >  make it more easy.
> > > Hello Igniters.
> > >
> > > I’d like to discuss porting build process of Ignite.C++. I think
> that
> >  there is time to change it.
> > > *Motivation*
> > > Currently, it is hard to build Ignite.C++. Different build process
> > for
> >  windows and linux, lack of building support on Mac OS X (quite
> popular
> > >> OS
> >  among developers), absolutely not IDE support, except windows and
> only
> >  Visual Studio is supported.
> > > *Suggestion*
> > > I’d suggest to migrate to CMake build system. It is very popular
> > among
> >  open source projects, and in Apache Software Foundation too. Notable
> > >>> user:
> >  Apache Mesos, Apache Zookeeper (C client offers CMake as an
> > alternative
> > >>> to
> >  autoconf and only option on windows), Apache Kafka (librdkafka -
> C/C++
> >  client), Apache Thrift. Popular column-oriented database ClickHouse
> > >> also
> >  uses CMake.
> > > CMake is widely supported in many IDE’s on various platforms,
> notably
> >  Visual Studio, CLion, Xcode, QtCreator, KDevelop.
> > > *Current status*
> > >
> > > Currently, the most of work is done (see [1]) and tested on Mac OS
> X
> >  10.15 (some C++ porting). All tests are run without any flaws,
> > >> including
> >  odbc (unixodbc), ssl, thin and thick client, installation, IDE
> > >>> integration
> >  (CLion). Next steps is to test linux and windows.
> > > But full migration isn’t possible without agreement and help of
> >  community. Even if most of all you agree, migration requires
> > additional
> >  efforts in TC agents tuning and so on (event though test running
> fully
> >  automated by CMake CTest).
> > > Lets discuss my proposition and idea.
> > >
> > > [1] -  https://github.com/apache/ignite/pull/7845
> > >
> > >
> > >
> > 
> > 
> > 
> > >>
> > >> --
> > >> Best regards,
> > >> Alexey
> > >>
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: Prevent insertion of cache entry if the binary field type and the type of the query entity do not match.

[Ivan Daschinsky]

I think this feature quite easy to implement. Before put in cache tree, we
can iterate over query descriptor fields and check whether
this field presents in binary object and if typeId is the same.
But I think there can l be a quite noticeable performance penalty when this
feature is enabled.
It should be thoroughly benchmarked.



чт, 28 мая 2020 г. в 10:22, Alex Plehanov :

> Pavel, Ilya,
>
> I think we should implement such a check on insert. It can be optional (for
> example enabled by property in SqlConfiguration). The sooner we found the
> problem - the better.
>
> вт, 26 мая 2020 г. в 17:56, Ilya Kasnacheev :
>
> > Hello!
> >
> > I'm not aware about any mechanism like this one built in Apache Ignite.
> >
> > I advise you to wrap Ignite's APIs into ones of your own and avoid using
> > raw Ignite API. This way you will make sure to do these checks on your
> own.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пн, 25 мая 2020 г. в 15:22, Pavel Pereslegin :
> >
> > > Hello Igniters.
> > >
> > > If type of binary field does not match query entity field type we
> > > still able to insert such entry into cache, but can't query it.
> > > In the following example we have query entity with the UUID field
> > > "name", but we insert String field "name" using binary object.
> > >
> > > IgniteCache cache = grid(0).createCache(
> > > new CacheConfiguration<>("testCache").setQueryEntities(
> > > Collections.singletonList(
> > > new QueryEntity()
> > > .setKeyFieldName("id")
> > > .setValueType("Person")
> > > .setFields(new LinkedHashMap<>(
> > > F.asMap("id", "java.lang.Integer",
> > > "name", "java.util.UUID"));
> > >
> > > BinaryObject obj = grid(0).binary().builder("Person")
> > > .setField("id", 1)
> > > .setField("name", UUID.randomUUID().toString())
> > > .build();
> > >
> > > cache.put(1, obj);
> > > assertEquals(obj, cache.withKeepBinary().get(1));
> > >
> > > String sql = "select id, name from Person where id=1";
> > >
> > > grid(0).context().query()
> > > .querySqlFields(new
> > > SqlFieldsQuery(sql).setSchema("testCache"), true)
> > > .getAll(); // java.lang.ClassCastException:
> > > java.lang.String cannot be cast to java.util.UUID
> > >
> > > The object was successfully inserted, but the "name"-field cannot be
> > > read using sql.
> > >
> > > Should it be better to prevent insertion of cache entry if the binary
> > > field type and the type of the query entity field do not match?
> > >
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Ok, PR is ready
https://issues.apache.org/jira/browse/IGNITE-13078

Build tested on Mac OS X 10.15 and Ubuntu 20.04 with CMake 3.17.2 and 3.6.1
Unfortunately, I was not able to test on Windows, but principally it should
works, but minor issues are probable.

Instruction is attached in PR.
Any use reports are welcomed!

вт, 26 мая 2020 г. в 18:51, Ivan Daschinsky :

> Stephen, looks great! I do mostly the same things in C++ code. Thank you!
>
> вт, 26 мая 2020 г. в 18:33, Stephen Darlington <
> stephen.darling...@gridgain.com>:
>
>> Not sure if it’ll help, but I made some changes to get it working on a
>> Mac with the current built system. There may be some code worth taking.
>>
>> https://github.com/apache/ignite/pull/4872 <
>> https://github.com/apache/ignite/pull/4872>
>>
>> Regards,
>> Stephen
>>
>> > On 26 May 2020, at 16:02, Ivan Daschinsky  wrote:
>> >
>> > I appreciate any help, thank you, Ilya.
>> >
>> > Currently I have a small PR without ticket (link in first post),but I
>> > decided not to file a jira issue before discussion.
>> > Now I see, that this feature are of great interest to community. So I
>> file
>> > a ticket, test myself on my home laptop (ubuntu 20.04)
>> > and add detailed instructions to DEVNOTES.txt in a few days.
>> >
>> > I would be happy if my someone can follow the instruction and write
>> > possible issues.
>> >
>> > I will notify about status update in this thread in next few days.
>> >
>> > Thank you all very much for support!
>> >
>> >
>> > вт, 26 мая 2020 г. в 17:50, Ilya Kasnacheev > >:
>> >
>> >> Hello!
>> >>
>> >> I will assist with checking on Linux if you would contribute a patch.
>> >> Please start with a ticket (or even an IEP maybe?)
>> >>
>> >> Regards,
>> >> --
>> >> Ilya Kasnacheev
>> >>
>> >>
>> >> вт, 26 мая 2020 г. в 16:47, Ivan Daschinsky :
>> >>
>> >>> Guys, I will certainly thoroughly test my fix not only unices, but on
>> >>> windows too.
>> >>> And I will describe it very thoroughly.
>> >>>
>> >>> When I was C++ developer (more than 10 years ago), I have not any
>> trouble
>> >>> at all with CMake and Visual Studio 2005.
>> >>> Everything works and works good. Moreover, you can build with NMake,
>> >>> msbuild and generate solutions for development.
>> >>>
>> >>> I suppose, for CI purposes, using NMake is a way better, than use vs
>> >>> solutions.
>> >>>
>> >>> вт, 26 мая 2020 г. в 16:42, Nikolay Izhikov :
>> >>>
>> >>>> Hello, Igor.
>> >>>>
>> >>>>> Nikolay, removing support for a certain build system is a breaking
>> >>>> change.
>> >>>>
>> >>>> No, it’s not.
>> >>>> Why do you think so?
>> >>>>
>> >>>> Development environment and build system is a subject to change in
>> any
>> >>>> project.
>> >>>> We can drop or add support of any build system any time we want.
>> >>>>
>> >>>>> 26 мая 2020 г., в 16:35, Ilya Kasnacheev > >
>> >>>> написал(а):
>> >>>>>
>> >>>>> Hello!
>> >>>>>
>> >>>>> I don't see why we can't get rid of autotools in a minor release,
>> >>>> provided
>> >>>>> that cmake actually works. Removing native VS support may be a
>> >>> different
>> >>>>> thing.
>> >>>>>
>> >>>>> Build system and precise set of dependencies is not a part of public
>> >>> API
>> >>>> in
>> >>>>> my opinion.
>> >>>>>
>> >>>>> Regards,
>> >>>>> --
>> >>>>> Ilya Kasnacheev
>> >>>>>
>> >>>>>
>> >>>>> вт, 26 мая 2020 г. в 16:02, Igor Sapego :
>> >>>>>
>> >>>>>> Great!
>> >>>>>>
>> >>>>>> Let's start with creating a TC suite for it.
>> >>>>>>
>> >>>>>> The only concern I have is that it is one more build system
>> >>>>>&g

Re: Re[2]: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Thanks you all. Run patch (I've changed some code also) on TC -- all CPP
suites are green (GCC, CLang, Win64)

пт, 29 мая 2020 г. в 15:02, Zhenya Stanilovsky :

>
>
> Ivan besides documentation [1]
> -s no —  no works
> -- catch_system_errors =no — works properly well, tests are passed.
>
> boost 1.65
>
> [1]
> https://www.boost.org/doc/libs/1_65_0/libs/test/doc/html/boost_test/utf_reference/rt_param_reference/catch_system.html
>
> >Hello!
> >
> >I didn't check tests since I don't develop AI C++, merely use it as user.
> >That's where we should wait for Igor Sapego to check.
> >
> >Regards,
> >--
> >Ilya Kasnacheev
> >
> >
> >пт, 29 мая 2020 г. в 12:20, Ivan Daschinsky < ivanda...@gmail.com >:
> >
> >> Ilya, thanks a lot! What about tests? I found one flag that must be
> >> supplied to boost.tests.
> >> This flag should fix JVM crash of C++ suites on Linux.
> >>
> >> Zhenya Stanilovsky and me have checked, that without this flag tests
> failed
> >> with SIGSEGV early (boost catch this signal from jvm, but it is ok for
> >> jvm).
> >> Flag is -catch_system_errors=no. I added it to CTest runner. You can
> >> invoke it manually and using make test ARGS="-V"
> >>
> >>
> >>
> >> пт, 29 мая 2020 г. в 11:54, Ilya Kasnacheev < ilya.kasnach...@gmail.com
> >:
> >>
> >> > Hello!
> >> >
> >> > Looks good to me! But we probably also ask Igor to take a look.
> >> >
> >> > Compiled debug and release, without and with odbc, checked running
> thick
> >> > node and ODBC connection on Linux.
> >> >
> >> > Regards,
> >> > --
> >> > Ilya Kasnacheev
> >> >
> >> >
> >> > чт, 28 мая 2020 г. в 17:31, Ivan Daschinsky < ivanda...@gmail.com >:
> >> >
> >> > > Ok, PR is ready
> >> > >  https://issues.apache.org/jira/browse/IGNITE-13078
> >> > >
> >> > > Build tested on Mac OS X 10.15 and Ubuntu 20.04 with CMake 3.17.2
> and
> >> > 3.6.1
> >> > > Unfortunately, I was not able to test on Windows, but principally it
> >> > should
> >> > > works, but minor issues are probable.
> >> > >
> >> > > Instruction is attached in PR.
> >> > > Any use reports are welcomed!
> >> > >
> >> > > вт, 26 мая 2020 г. в 18:51, Ivan Daschinsky < ivanda...@gmail.com
> >:
> >> > >
> >> > > > Stephen, looks great! I do mostly the same things in C++ code.
> Thank
> >> > you!
> >> > > >
> >> > > > вт, 26 мая 2020 г. в 18:33, Stephen Darlington <
> >> > > >  stephen.darling...@gridgain.com >:
> >> > > >
> >> > > >> Not sure if it’ll help, but I made some changes to get it working
> >> on a
> >> > > >> Mac with the current built system. There may be some code worth
> >> > taking.
> >> > > >>
> >> > > >>  https://github.com/apache/ignite/pull/4872 <
> >> > > >>  https://github.com/apache/ignite/pull/4872 >
> >> > > >>
> >> > > >> Regards,
> >> > > >> Stephen
> >> > > >>
> >> > > >> > On 26 May 2020, at 16:02, Ivan Daschinsky <
> ivanda...@gmail.com >
> >> > > wrote:
> >> > > >> >
> >> > > >> > I appreciate any help, thank you, Ilya.
> >> > > >> >
> >> > > >> > Currently I have a small PR without ticket (link in first
> >> post),but
> >> > I
> >> > > >> > decided not to file a jira issue before discussion.
> >> > > >> > Now I see, that this feature are of great interest to
> community.
> >> So
> >> > I
> >> > > >> file
> >> > > >> > a ticket, test myself on my home laptop (ubuntu 20.04)
> >> > > >> > and add detailed instructions to DEVNOTES.txt in a few days.
> >> > > >> >
> >> > > >> > I would be happy if my someone can follow the instruction and
> >> write
> >> > > >> > possible issues.
> >> > > >> >
> >> > > >> > I will notify about status update in this thread in next fe

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Also without with flag we can observe many failures of suites on TC

https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PlatformCLinux?branch=%3Cdefault%3E=builds#all-projects


пт, 29 мая 2020 г. в 12:20, Ivan Daschinsky :

> Ilya, thanks a lot! What about tests? I found one flag that must be
> supplied to boost.tests.
> This flag should fix JVM crash of C++ suites on Linux.
>
> Zhenya Stanilovsky and me have checked, that without this flag tests
> failed with SIGSEGV early (boost catch this signal from jvm, but it is ok
> for jvm).
> Flag is -catch_system_errors=no. I added it to CTest runner. You can
> invoke it manually and using make test ARGS="-V"
>
>
>
> пт, 29 мая 2020 г. в 11:54, Ilya Kasnacheev :
>
>> Hello!
>>
>> Looks good to me! But we probably also ask Igor to take a look.
>>
>> Compiled debug and release, without and with odbc, checked running thick
>> node and ODBC connection on Linux.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> чт, 28 мая 2020 г. в 17:31, Ivan Daschinsky :
>>
>> > Ok, PR is ready
>> > https://issues.apache.org/jira/browse/IGNITE-13078
>> >
>> > Build tested on Mac OS X 10.15 and Ubuntu 20.04 with CMake 3.17.2 and
>> 3.6.1
>> > Unfortunately, I was not able to test on Windows, but principally it
>> should
>> > works, but minor issues are probable.
>> >
>> > Instruction is attached in PR.
>> > Any use reports are welcomed!
>> >
>> > вт, 26 мая 2020 г. в 18:51, Ivan Daschinsky :
>> >
>> > > Stephen, looks great! I do mostly the same things in C++ code. Thank
>> you!
>> > >
>> > > вт, 26 мая 2020 г. в 18:33, Stephen Darlington <
>> > > stephen.darling...@gridgain.com>:
>> > >
>> > >> Not sure if it’ll help, but I made some changes to get it working on
>> a
>> > >> Mac with the current built system. There may be some code worth
>> taking.
>> > >>
>> > >> https://github.com/apache/ignite/pull/4872 <
>> > >> https://github.com/apache/ignite/pull/4872>
>> > >>
>> > >> Regards,
>> > >> Stephen
>> > >>
>> > >> > On 26 May 2020, at 16:02, Ivan Daschinsky 
>> > wrote:
>> > >> >
>> > >> > I appreciate any help, thank you, Ilya.
>> > >> >
>> > >> > Currently I have a small PR without ticket (link in first
>> post),but I
>> > >> > decided not to file a jira issue before discussion.
>> > >> > Now I see, that this feature are of great interest to community.
>> So I
>> > >> file
>> > >> > a ticket, test myself on my home laptop (ubuntu 20.04)
>> > >> > and add detailed instructions to DEVNOTES.txt in a few days.
>> > >> >
>> > >> > I would be happy if my someone can follow the instruction and write
>> > >> > possible issues.
>> > >> >
>> > >> > I will notify about status update in this thread in next few days.
>> > >> >
>> > >> > Thank you all very much for support!
>> > >> >
>> > >> >
>> > >> > вт, 26 мая 2020 г. в 17:50, Ilya Kasnacheev <
>> > ilya.kasnach...@gmail.com
>> > >> >:
>> > >> >
>> > >> >> Hello!
>> > >> >>
>> > >> >> I will assist with checking on Linux if you would contribute a
>> patch.
>> > >> >> Please start with a ticket (or even an IEP maybe?)
>> > >> >>
>> > >> >> Regards,
>> > >> >> --
>> > >> >> Ilya Kasnacheev
>> > >> >>
>> > >> >>
>> > >> >> вт, 26 мая 2020 г. в 16:47, Ivan Daschinsky > >:
>> > >> >>
>> > >> >>> Guys, I will certainly thoroughly test my fix not only unices,
>> but
>> > on
>> > >> >>> windows too.
>> > >> >>> And I will describe it very thoroughly.
>> > >> >>>
>> > >> >>> When I was C++ developer (more than 10 years ago), I have not any
>> > >> trouble
>> > >> >>> at all with CMake and Visual Studio 2005.
>> > >> >>> Everything works and works good. Moreover, you

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Ilya, thanks a lot! What about tests? I found one flag that must be
supplied to boost.tests.
This flag should fix JVM crash of C++ suites on Linux.

Zhenya Stanilovsky and me have checked, that without this flag tests failed
with SIGSEGV early (boost catch this signal from jvm, but it is ok for jvm).
Flag is -catch_system_errors=no. I added it to CTest runner. You can
invoke it manually and using make test ARGS="-V"



пт, 29 мая 2020 г. в 11:54, Ilya Kasnacheev :

> Hello!
>
> Looks good to me! But we probably also ask Igor to take a look.
>
> Compiled debug and release, without and with odbc, checked running thick
> node and ODBC connection on Linux.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 28 мая 2020 г. в 17:31, Ivan Daschinsky :
>
> > Ok, PR is ready
> > https://issues.apache.org/jira/browse/IGNITE-13078
> >
> > Build tested on Mac OS X 10.15 and Ubuntu 20.04 with CMake 3.17.2 and
> 3.6.1
> > Unfortunately, I was not able to test on Windows, but principally it
> should
> > works, but minor issues are probable.
> >
> > Instruction is attached in PR.
> > Any use reports are welcomed!
> >
> > вт, 26 мая 2020 г. в 18:51, Ivan Daschinsky :
> >
> > > Stephen, looks great! I do mostly the same things in C++ code. Thank
> you!
> > >
> > > вт, 26 мая 2020 г. в 18:33, Stephen Darlington <
> > > stephen.darling...@gridgain.com>:
> > >
> > >> Not sure if it’ll help, but I made some changes to get it working on a
> > >> Mac with the current built system. There may be some code worth
> taking.
> > >>
> > >> https://github.com/apache/ignite/pull/4872 <
> > >> https://github.com/apache/ignite/pull/4872>
> > >>
> > >> Regards,
> > >> Stephen
> > >>
> > >> > On 26 May 2020, at 16:02, Ivan Daschinsky 
> > wrote:
> > >> >
> > >> > I appreciate any help, thank you, Ilya.
> > >> >
> > >> > Currently I have a small PR without ticket (link in first post),but
> I
> > >> > decided not to file a jira issue before discussion.
> > >> > Now I see, that this feature are of great interest to community. So
> I
> > >> file
> > >> > a ticket, test myself on my home laptop (ubuntu 20.04)
> > >> > and add detailed instructions to DEVNOTES.txt in a few days.
> > >> >
> > >> > I would be happy if my someone can follow the instruction and write
> > >> > possible issues.
> > >> >
> > >> > I will notify about status update in this thread in next few days.
> > >> >
> > >> > Thank you all very much for support!
> > >> >
> > >> >
> > >> > вт, 26 мая 2020 г. в 17:50, Ilya Kasnacheev <
> > ilya.kasnach...@gmail.com
> > >> >:
> > >> >
> > >> >> Hello!
> > >> >>
> > >> >> I will assist with checking on Linux if you would contribute a
> patch.
> > >> >> Please start with a ticket (or even an IEP maybe?)
> > >> >>
> > >> >> Regards,
> > >> >> --
> > >> >> Ilya Kasnacheev
> > >> >>
> > >> >>
> > >> >> вт, 26 мая 2020 г. в 16:47, Ivan Daschinsky :
> > >> >>
> > >> >>> Guys, I will certainly thoroughly test my fix not only unices, but
> > on
> > >> >>> windows too.
> > >> >>> And I will describe it very thoroughly.
> > >> >>>
> > >> >>> When I was C++ developer (more than 10 years ago), I have not any
> > >> trouble
> > >> >>> at all with CMake and Visual Studio 2005.
> > >> >>> Everything works and works good. Moreover, you can build with
> NMake,
> > >> >>> msbuild and generate solutions for development.
> > >> >>>
> > >> >>> I suppose, for CI purposes, using NMake is a way better, than use
> vs
> > >> >>> solutions.
> > >> >>>
> > >> >>> вт, 26 мая 2020 г. в 16:42, Nikolay Izhikov  >:
> > >> >>>
> > >> >>>> Hello, Igor.
> > >> >>>>
> > >> >>>>> Nikolay, removing support for a certain build system is a
> breaking
> > >> >>>> change.
> > >

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Stephen, looks great! I do mostly the same things in C++ code. Thank you!

вт, 26 мая 2020 г. в 18:33, Stephen Darlington <
stephen.darling...@gridgain.com>:

> Not sure if it’ll help, but I made some changes to get it working on a Mac
> with the current built system. There may be some code worth taking.
>
> https://github.com/apache/ignite/pull/4872 <
> https://github.com/apache/ignite/pull/4872>
>
> Regards,
> Stephen
>
> > On 26 May 2020, at 16:02, Ivan Daschinsky  wrote:
> >
> > I appreciate any help, thank you, Ilya.
> >
> > Currently I have a small PR without ticket (link in first post),but I
> > decided not to file a jira issue before discussion.
> > Now I see, that this feature are of great interest to community. So I
> file
> > a ticket, test myself on my home laptop (ubuntu 20.04)
> > and add detailed instructions to DEVNOTES.txt in a few days.
> >
> > I would be happy if my someone can follow the instruction and write
> > possible issues.
> >
> > I will notify about status update in this thread in next few days.
> >
> > Thank you all very much for support!
> >
> >
> > вт, 26 мая 2020 г. в 17:50, Ilya Kasnacheev :
> >
> >> Hello!
> >>
> >> I will assist with checking on Linux if you would contribute a patch.
> >> Please start with a ticket (or even an IEP maybe?)
> >>
> >> Regards,
> >> --
> >> Ilya Kasnacheev
> >>
> >>
> >> вт, 26 мая 2020 г. в 16:47, Ivan Daschinsky :
> >>
> >>> Guys, I will certainly thoroughly test my fix not only unices, but on
> >>> windows too.
> >>> And I will describe it very thoroughly.
> >>>
> >>> When I was C++ developer (more than 10 years ago), I have not any
> trouble
> >>> at all with CMake and Visual Studio 2005.
> >>> Everything works and works good. Moreover, you can build with NMake,
> >>> msbuild and generate solutions for development.
> >>>
> >>> I suppose, for CI purposes, using NMake is a way better, than use vs
> >>> solutions.
> >>>
> >>> вт, 26 мая 2020 г. в 16:42, Nikolay Izhikov :
> >>>
> >>>> Hello, Igor.
> >>>>
> >>>>> Nikolay, removing support for a certain build system is a breaking
> >>>> change.
> >>>>
> >>>> No, it’s not.
> >>>> Why do you think so?
> >>>>
> >>>> Development environment and build system is a subject to change in any
> >>>> project.
> >>>> We can drop or add support of any build system any time we want.
> >>>>
> >>>>> 26 мая 2020 г., в 16:35, Ilya Kasnacheev 
> >>>> написал(а):
> >>>>>
> >>>>> Hello!
> >>>>>
> >>>>> I don't see why we can't get rid of autotools in a minor release,
> >>>> provided
> >>>>> that cmake actually works. Removing native VS support may be a
> >>> different
> >>>>> thing.
> >>>>>
> >>>>> Build system and precise set of dependencies is not a part of public
> >>> API
> >>>> in
> >>>>> my opinion.
> >>>>>
> >>>>> Regards,
> >>>>> --
> >>>>> Ilya Kasnacheev
> >>>>>
> >>>>>
> >>>>> вт, 26 мая 2020 г. в 16:02, Igor Sapego :
> >>>>>
> >>>>>> Great!
> >>>>>>
> >>>>>> Let's start with creating a TC suite for it.
> >>>>>>
> >>>>>> The only concern I have is that it is one more build system
> >>>>>> to support. Should we get rid of autotools in 3.0?
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Igor
> >>>>>>
> >>>>>>
> >>>>>> On Tue, May 26, 2020 at 2:44 PM Alexey Kukushkin <
> >>>>>> kukushkinale...@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> +1. I recently completed a cross-IDE (MS Visual Studio & GCC/GDB)
> >>>> Ignite
> >>>>>>> C++ project and CMake in Ignite C++ would save me a day of effort.
> >>>>>>>
> >>>>>>> вт, 26 мая 2020 г. в 12:09, Pavel Tupitsyn :
> >>>>>&g

Re: [DISCUSSION] Add autocompletion for commands in control.sh

[Ivan Daschinsky]

> By the way, according to README.md [1] the picocli is already using bythe
Apache Ignite, right? :-)
Yes, you are right, but a little bit outdated.

> Would it be better going the `control.sh` extensions-way?
Sounds good. Also, it would be great to incorporate in command line
utilities some features from visor-console also. And after developing new
tool, deprecate both (visor-console and control.sh)

вт, 2 июн. 2020 г. в 16:15, Maxim Muzafarov :

> Folks,
>
> +1
>
> However, AFAIK control.sh is the part of the ignite-core module with
> zero dependencies from external resources.
> Would it be better going the `control.sh` extensions-way?
>
>
> By the way, according to README.md [1] the picocli is already using by
> the Apache Ignite, right? :-)
>
> > Picocli is used in the Apache Hadoop Ozone/HDDS command line tools, the
> Apache Hive benchmark CLI, has ** Apache [Ignite TensorFlow] **, and Apache
> Sling.
>
>
> [1] https://github.com/remkop/picocli/blame/master/README.md#L199
>
> On Tue, 2 Jun 2020 at 16:09, Ivan Daschinsky  wrote:
> >
> > +1 But this is not only usability improvement, but also a huge code
> > improvement. With picocli developers can add custom command without
> writing
> > a lot of boilerplate and error prone code to do a trivial task
> > of parsing CLI arguments. Cleaner code, less bugs also matter.
> >
> > вт, 2 июн. 2020 г. в 16:02, Sergey Antonov :
> >
> > > It would be a great usability improvement!
> > >
> > > +1 From me.
> > >
> > > вт, 2 июн. 2020 г. в 15:54, Zhenya Stanilovsky
>  > > >:
> > >
> > > >
> > > >
> > > > good catch ! it`s a little bit pain for now to working with it.
> > > >
> > > >
> > > > >Hi, Igniters!
> > > > >
> > > > >At the moment to work with the control.sh we need to know exactly
> what
> > > > the name of the command and its options are and so the user can often
> > > make
> > > > mistakes when using it. So I think it would be useful to do
> control.sh
> > > more
> > > > user-friendly by adding autocomplete as in modern command-line
> utilities.
> > > > >
> > > > >For this purpose, I suggest using framework [1] and to do this,
> take out
> > > > control.sh together with its associated classes in a separate module
> such
> > > > as "modules/control-utility".
> > > > >
> > > > >Comments, suggestions?
> > > > >
> > > > >[1] -  https://picocli.info/
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > > --
> > > BR, Sergey Antonov
> > >
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSSION] Add autocompletion for commands in control.sh

[Ivan Daschinsky]

+1 But this is not only usability improvement, but also a huge code
improvement. With picocli developers can add custom command without writing
a lot of boilerplate and error prone code to do a trivial task
of parsing CLI arguments. Cleaner code, less bugs also matter.

вт, 2 июн. 2020 г. в 16:02, Sergey Antonov :

> It would be a great usability improvement!
>
> +1 From me.
>
> вт, 2 июн. 2020 г. в 15:54, Zhenya Stanilovsky  >:
>
> >
> >
> > good catch ! it`s a little bit pain for now to working with it.
> >
> >
> > >Hi, Igniters!
> > >
> > >At the moment to work with the control.sh we need to know exactly what
> > the name of the command and its options are and so the user can often
> make
> > mistakes when using it. So I think it would be useful to do control.sh
> more
> > user-friendly by adding autocomplete as in modern command-line utilities.
> > >
> > >For this purpose, I suggest using framework [1] and to do this, take out
> > control.sh together with its associated classes in a separate module such
> > as "modules/control-utility".
> > >
> > >Comments, suggestions?
> > >
> > >[1] -  https://picocli.info/
> >
> >
> >
> >
>
>
>
> --
> BR, Sergey Antonov
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: New Ignite Docs: Status and Incomplete Items

[Ivan Daschinsky]

Hi! As we introduce CMake build system for Ignite C++ for Linux and Mac OS
X and this improvement is in ignite 2.9, we should document it. (CMake for
windows is not ready yet and will be in 2.10)
I created ticket for it and assigned it to me. Patch will be available
soon.

[1] - https://issues.apache.org/jira/browse/IGNITE-13459

пт, 18 сент. 2020 г. в 09:59, Nikolay Izhikov :

> Hello, Denis.
>
> >   • @Nikolay Izhikov, is there any chance you can document the new
> management commands the next week? Use me as a reviewer.
>
> Yes.
>
> > 18 сент. 2020 г., в 02:24, Denis Magda  написал(а):
> >
> > Folks,
> >
> > While Artem is unavailable, Mauricio and I keep pushing the new docs
> project to the finish line: https://ignite.apache.org/docs/latest/preface
> >
> > Overall, we hope to copy the last bits from the readme.io to the new
> engine, document the release and contribution process, and settle down with
> the search plug-in throughout the next week or so. The docs will be ready
> by the time of the 2.9 release.
> >
> > But there are some items that need the involvement of some of you:
> >   • @Pavel Tupitsyn, I heard from Artem that you are working on some
> .NET-specific tasks. Are you going to document the features added in 2.9
> and is there is anything else you planned to move from readme.io?
> >   • @Nikolay Izhikov, is there any chance you can document the new
> management commands the next week? Use me as a reviewer.
> >
> > Once, all the features are documented and old pages are moved from
> readme.io, I'll go ahead and merge the new docs to the master and
> ignite-2.9 branch.
> >
> > -
> > Denis
>
>

-- 
Sincerely yours, Ivan Daschinskiy

Re: Apache Ignite 3.0

[Ivan Daschinsky]

Hi!
Alexey,
> If we want to support etcd as a metastorage - let's do this as a concrete
configuration option, a
> first-class citizen of the system rather than an SPI implementation with
a rigid interface.
On one side this is quite reasonable. But on the other side, if someone
wants to adopt, for example Apache Zookeeper or
some other proprietary external lock service, we could provide basic
interfaces to do the job.

> Thus, by default, they will be mixed which will significantly simplify
cluster setup and usability.
According to raft specs, the leader processes all requests from clients.
Leader's response latency is a crucial thing for the whole cluster
stability.
Cluster setup simplicity is a subject of documentation, scripts and so on,
i.e. starting kafka is quite easy.

Also, if we use mixed approach, service discovery protocol should be
implemented.This is necessary, because we should discover nodes firstly in
order to choose finite subset for RAFT ensemble.
For example, Consul by HashiCorp uses gossip protocol to do the job. (Nodes
participating in RAFT are called servers,  [1]

If we use separated approach, we could use service discovery pattern that
is common for zookeeper or etcd (data node create record with TTL and renew
it. (EPHEMERAL node approach for zk),
other data nodes watches for new records)

Some words about PacificA
Article  [2] -- is just brief descriptions and ideas. Alexey, is there any
formal specification of this protocol? Preferrably in TLA+?


[1] -- https://www.consul.io/docs/architecture/gossip
[2] --
https://www.microsoft.com/en-us/research/wp-content/uploads/2008/02/tr-2008-25.pdf




пт, 23 окт. 2020 г. в 13:05, Alexey Goncharuk :

> Hello Ivan,
>
> Thanks for the feedback, see my comments inline:
>
> чт, 22 окт. 2020 г. в 17:59, Ivan Daschinsky :
>
> > Hi!
> > Alexey, your proposal looks great. Can I ask you some questions?
> > 1. Is nodes, that take part of metastorage replication group (raft
> > candidates and leader) are expected to also bear cache data and
> participate
> > in cache transactions?
> >As for me, it seems quite dangerous to mix roles. For example, heavy
> > load from users can cause long GC pauses on leader of replication group
> and
> > therefore failure, new leader election, etc.
> >
> I think both ways should be possible. The set of nodes that hold
> metastorage should be defined declaratively in runtime, as well as the set
> of nodes holding table data. Thus, by default, they will be mixed which
> will significantly simplify cluster setup and usability, but when needed,
> this should be easily adjusted in runtime by the cluster administrator.
>
>
> > 2. If previous statement is true, other question arises. If one of
> > candidates or leader fails, how will a insufficient node will be chosen
> > from regular nodes to form full ensemble? Random one?
> >
> Similarly - by default, a 'best' node will be chosen from the available
> ones, but the administrator can override this.
>
>
> > 3. Do you think, that this metastorage implementation can be pluggable?
> it
> > can be implemented on top of etcd, for example.
>
> I think the metastorage abstraction must be clearly separated so it is
> possible to change the implementation. Moreover, I was thinking that we may
> use etcd to speed up the development of other system components while we
> are working on our own protocol implementation. However, I do not think we
> should expose it as a pluggable public API. If we want to support etcd as a
> metastorage - let's do this as a concrete configuration option, a
> first-class citizen of the system rather than an SPI implementation with a
> rigid interface.
>
> WDYT?
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: Re[2]: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]

[Ivan Daschinsky]

I recently found, that control.sh is broken since 2.8.0 a little bit.
Script returns always 0 code, despite the fact, that CommandHandler returns
code correctly.

Here is the issue https://issues.apache.org/jira/browse/IGNITE-13328, patch
is available.

пт, 31 июл. 2020 г. в 14:09, Alex Plehanov :

> Ivan,
>
> IGNITE-13306 cherry-picked to 2.9
>
> пт, 31 июл. 2020 г. в 13:43, Ivan Rakov :
>
> > Hi Alex,
> >
> > https://issues.apache.org/jira/browse/IGNITE-13306 is merged to master.
> > Can you please cherry-pick to 2.9?
> >
> > On Thu, Jul 30, 2020 at 7:42 PM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com>
> > wrote:
> >
> >> Hello!
> >>
> >> I don't think that IGNITE-13006
> >>  is a blocker in
> any
> >> way. It is a good candidate for 3.0.
> >>
> >> ignite-spring will work with 4.x Spring as well as 5.x and the user is
> >> free
> >> to bump Spring version. I think bumping this dependency explicitly is
> >> infeasible since it may break existing code.
> >>
> >> Regards,
> >> --
> >> Ilya Kasnacheev
> >>
> >>
> >> ср, 22 июл. 2020 г. в 10:22, Alex Plehanov :
> >>
> >> > Guys,
> >> >
> >> > We are in code-freeze phase now. I've moved almost all non-blocker
> >> > unresolved tickets from 2.9 to the next release. If you think that
> >> > some ticket is a blocker and should be included into 2.9 release,
> please
> >> > write a note in this thread.
> >> >
> >> > There are some tickets with "blocker" priority targeted to 2.9, some
> of
> >> > them in "open" state and still unassigned, and I'm not sure we need
> all
> >> of
> >> > these tickets in 2.9:
> >> >
> >> > IGNITE-13006 [1] (Apache Ignite spring libs upgrade from version 4x to
> >> > spring 5.2 version or later) - Is it really a blocker for 2.9 release?
> >> If
> >> > yes, can somebody help with resolving this ticket?
> >> >
> >> > IGNITE-11942 [2] (IGFS and Hadoop Accelerator Discontinuation) -
> ticket
> >> in
> >> > "Patch available" state. There is a thread on dev-list related to this
> >> > ticket ([6]), but as far as I understand we still don't have consensus
> >> > about version for this patch (2.9, 2.10, 3.0).
> >> >
> >> > IGNITE-12489 [3] (Error during purges by expiration: Unknown page
> type)
> >> -
> >> > perhaps issue is already resolved by some related tickets, there is
> >> still
> >> > no reproducer, no additional details and no work in progress. I
> propose
> >> to
> >> > move this ticket to the next release.
> >> >
> >> > IGNITE-12911 [4] (B+Tree Corrupted exception when using a key
> extracted
> >> > from a BinaryObject value object --- and SQL enabled) - ticket in
> "Patch
> >> > available" state, but there is no activity since May 2020. Anton
> >> > Kalashnikov, Ilya Kasnacheev, do we have any updates on this ticket?
> Is
> >> it
> >> > still in progress?
> >> >
> >> > IGNITE-12553 [5] ([IEP-35] public Java metric API) - since the new
> >> metrics
> >> > framework is already released in 2.8 and it's still marked with
> >> > @IgniteExperemental annotation, I think this ticket is not a blocker.
> I
> >> > propose to change the ticket priority and move it to the next release.
> >> >
> >> >
> >> > [1]: https://issues.apache.org/jira/browse/IGNITE-13006
> >> > [2]: https://issues.apache.org/jira/browse/IGNITE-11942
> >> > [3]: https://issues.apache.org/jira/browse/IGNITE-12489
> >> > [4]: https://issues.apache.org/jira/browse/IGNITE-12911
> >> > [5]: https://issues.apache.org/jira/browse/IGNITE-12553
> >> > [6]:
> >> >
> >>
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42282.html
> >> >
> >> > пт, 17 июл. 2020 г. в 11:50, Alex Plehanov :
> >> >
> >> >> Ivan,
> >> >>
> >> >> Merged to 2.9.
> >> >>
> >> >> Thanks
> >> >>
> >> >> пт, 17 июл. 2020 г. в 01:35, Ivan Rakov :
> >> >>
> >> >>> Alex,
> >> >>>
> >> >>> Tracing is merged to master:
> >> >>> https://issues.apache.org/jira/browse/IGNITE-13060
> >> >>>
> >> >>> Can you please port it to 2.9?
> >> >>> For you convenience, there's PR versus 2.9 with conflicts resolved:
> >> >>> https://github.com/apache/ignite/pull/8046/files
> >> >>>
> >> >>> --
> >> >>> Best Regards,
> >> >>> Ivan Rakov
> >> >>>
> >> >>> On Wed, Jul 15, 2020 at 5:33 PM Alex Plehanov <
> >> plehanov.a...@gmail.com>
> >> >>> wrote:
> >> >>>
> >>  Ivan,
> >> 
> >>  Looks like master is broken after IGNITE-13246 (but everything is
> ok
> >> in
> >>  2.9
> >>  branch)
> >> 
> >>  ср, 15 июл. 2020 г. в 18:54, Alex Plehanov <
> plehanov.a...@gmail.com
> >> >:
> >> 
> >>  > Zhenya, Ivan,
> >>  >
> >>  > I've cherry-picked IGNITE-13229 and IGNITE-13246 to ignite-2.9
> >> branch.
> >>  > Thank you.
> >>  >
> >>  > ср, 15 июл. 2020 г. в 18:31, Ivan Bessonov <
> bessonov...@gmail.com
> >> >:
> >>  >
> >>  >> Guys,
> >>  >>
> >>  >> can you please backport
> >>  >> https://issues.apache.org/jira/browse/IGNITE-13246
> >>  

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Ok, CMake now is the only option to build C++ on non-windows platforms.
Autotools is removed from master.

Many thanks to Igor Sapego, Zhenya Stanilovsky, Nickolay Izhikov and Ilya
Kasnacheev for review, testing and suggestions.

вт, 23 июн. 2020 г. в 18:42, Ivan Daschinsky :

> I suppose, that removal of autotools from source code is a question of a
> week. There is no need to support it.
>
> вт, 23 июн. 2020 г., 18:38 Ilya Kasnacheev :
>
>> Hello!
>>
>> Once you do that, I think it would make sense to remove autotools
>> invocation from artifact build process. In the future we may choose to
>> remove autotools support entirely.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> вт, 23 июн. 2020 г. в 18:00, Petr Ivanov :
>>
>> > That host was down for some time and got up only recently.
>> > Installed cmake there also.
>> >
>> >
>> >
>> > > On 23 Jun 2020, at 17:39, Ivan Daschinsky 
>> wrote:
>> > >
>> > > Petr, I see, that cmake is missing on aitc10_05 but it is in pool and
>> > > trying to run build.
>> > >
>> > > [1] --
>> > >
>> >
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PlatformCPPCMakeLinux/5412025
>> > >
>> > > вт, 23 июн. 2020 г. в 17:33, Ivan Daschinsky :
>> > >
>> > >>>> For the sake of discretion, I'd purpose to remove old suites from
>> Run
>> > >> All and archive or pause them for some time before removal.
>> > >> Agree, I was talking about removal from RunAll chain, complete
>> removal
>> > is
>> > >> not necessary.
>> > >> But after private discussion with Igor Sapego, me and he decided to
>> wait
>> > >> for a week before exclusion of these suites from RunAll.
>> > >>
>> > >> вт, 23 июн. 2020 г. в 17:21, Petr Ivanov :
>> > >>
>> > >>> For the sake of discretion, I'd purpose to remove old suites from
>> Run
>> > All
>> > >>> and archive or pause them for some time before removal.
>> > >>>
>> > >>>
>> > >>>> On 23 Jun 2020, at 13:26, Nikolay Izhikov 
>> > wrote:
>> > >>>>
>> > >>>>> So, folks, is it ok to remove old suites from build chain
>> > >>>>
>> > >>>> +1 from me.
>> > >>>>
>> > >>>>
>> > >>>>> 23 июня 2020 г., в 13:15, Ivan Daschinsky 
>> > >>> написал(а):
>> > >>>>>
>> > >>>>> Well, new suites added to RunAll and all seems to be ok.
>> > >>>>>
>> > >>>>> I think it is time to remove old suites.
>> > >>>>> WDYT?
>> > >>>>>
>> > >>>>> Also, I created patch with removal of autotools, but old suites
>> > should
>> > >>> be
>> > >>>>> removed first.
>> > >>>>>
>> > >>>>> So, folks, is it ok to remove old suites from build chain?
>> > >>>>>
>> > >>>>> вт, 23 июн. 2020 г. в 10:36, Ivan Daschinsky > >:
>> > >>>>>
>> > >>>>>> Ok, I changed agents requirements to builds and add them to
>> runAll
>> > >>>>>>
>> > >>>>>> пн, 22 июн. 2020 г. в 22:39, Petr Ivanov :
>> > >>>>>>
>> > >>>>>>> Cmake is installed on all agents (except 10 which is currently
>> down
>> > >>> and
>> > >>>>>>> will be updated later).
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>> On 22 Jun 2020, at 21:03, Ivan Daschinsky > >
>> > >>> wrote:
>> > >>>>>>>>
>> > >>>>>>>> Well, patch is merged. Thank a lot to Igor Sapego for review.
>> > >>>>>>>>
>> > >>>>>>>> Peter, well, cmake now can be installed on all agent.
>> > >>>>>>>>
>> > >>>>>>>> I think it's time to add cmake suites to runAll and exclude old
>> > >>> suites
>> > >>>>>>> from
>> > >>>>>>>> it.
>> 

Re: Re[2]: IGNITE-6499 Compact NULL fields

[Ivan Daschinsky]

I think that this feature can be handled as compactFooter. For example, C++
doesn't support compactFooter and it is not an issue.
Of course, this feature should be disabled by default, and should be
enabled explicitly in BinaryConfiguration.
Also, subsequent issues in jira about this feature support in platforms
should be created.

ср, 8 июл. 2020 г. в 14:31, Ilya Kasnacheev :

> Hello!
>
> I think this is a blocker for this change. We already have binary format
> published:
>
> https://apacheignite.readme.io/docs/binary-client-protocol-data-format#complex-object
>
> Arguably, we cannot change it in a minor version of Apache Ignite, so this
> change has to target AI 3.0.
>
> Extending this binary format with e.g. new operations could probably be OK.
> But we have clients released on a different schedule in their own repos
> (and there are some 3rd party clients too), we can't release a minor
> version which will change this format unilaterally without any change of
> operation (same data, same calls, different result after upgrade, broken
> clients).
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 8 июл. 2020 г. в 13:43, Ivan Daschinsky :
>
> > Hi!
> > Ilya, unfortunatelly yes, subsequent changes should be made in C++, .NET
> > and other platform code.
> >
> > ср, 8 июл. 2020 г. в 12:22, Ilya Kasnacheev :
> >
> > > Hello fellow devs,
> > >
> > > I just wanted to ask, how would this Binary Object format change affect
> > > thin clients? C++/.Net nodes? Etc?
> > >
> > > Is it fully backwards compatible or not?
> > >
> > > I think that realistically, we can only add binary-incompatible changes
> > to
> > > Binary Object format in 3.0.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > ср, 8 июл. 2020 г. в 09:05, Ivan Pavlukhin :
> > >
> > > > A side note. Now we have a neat URL for TC bot
> > > > https://mtcga.ignite.apache.org/ (along with one in a gridgain
> > > > domain).
> > > >
> > > > 2020-07-07 18:43 GMT+03:00, Zhenya Stanilovsky
> > >  > > > >:
> > > > >
> > > > > request it, check for example [1]
> > > > >
> > > > > also you need to run [2] tests.
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Phani-Introduction-td47788.html
> > > > > [2] https://mtcga.gridgain.com
> > > > >>Hello,
> > > > >>
> > > > >>Look at the ticket and the only comment I can see is creating a
> > branch
> > > on
> > > > >>git in the main repo and not in my fork. I do not have the right to
> > > > create
> > > > >> a
> > > > >>branch in the main repository. Am i missing something?
> > > > >>
> > > > >>Sorry I probably misread the document but I though that I should
> fork
> > > the
> > > > >>repo and then pull request as I do not have the rights to create a
> > > > branch.
> > > > >>
> > > > >>Thanks for your help
> > > > >>
> > > > >>
> > > > >>
> > > > >>--
> > > > >>Sent from:  http://apache-ignite-developers.2346864.n4.nabble.com/
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> > >
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: Re[2]: IGNITE-6499 Compact NULL fields

[Ivan Daschinsky]

Hi!
Ilya, unfortunatelly yes, subsequent changes should be made in C++, .NET
and other platform code.

ср, 8 июл. 2020 г. в 12:22, Ilya Kasnacheev :

> Hello fellow devs,
>
> I just wanted to ask, how would this Binary Object format change affect
> thin clients? C++/.Net nodes? Etc?
>
> Is it fully backwards compatible or not?
>
> I think that realistically, we can only add binary-incompatible changes to
> Binary Object format in 3.0.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 8 июл. 2020 г. в 09:05, Ivan Pavlukhin :
>
> > A side note. Now we have a neat URL for TC bot
> > https://mtcga.ignite.apache.org/ (along with one in a gridgain
> > domain).
> >
> > 2020-07-07 18:43 GMT+03:00, Zhenya Stanilovsky
>  > >:
> > >
> > > request it, check for example [1]
> > >
> > > also you need to run [2] tests.
> > >
> > > [1]
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Phani-Introduction-td47788.html
> > > [2] https://mtcga.gridgain.com
> > >>Hello,
> > >>
> > >>Look at the ticket and the only comment I can see is creating a branch
> on
> > >>git in the main repo and not in my fork. I do not have the right to
> > create
> > >> a
> > >>branch in the main repository. Am i missing something?
> > >>
> > >>Sorry I probably misread the document but I though that I should fork
> the
> > >>repo and then pull request as I do not have the rights to create a
> > branch.
> > >>
> > >>Thanks for your help
> > >>
> > >>
> > >>
> > >>--
> > >>Sent from:  http://apache-ignite-developers.2346864.n4.nabble.com/
> > >
> > >
> > >
> > >
> >
> >
> > --
> >
> > Best regards,
> > Ivan Pavlukhin
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: Checkstyle fails Build Apache Ignite - can we split it?

[Ivan Daschinsky]

I think, that we should enable checkstyle profile by default. Checkstyle
should run even before compile lifecycle, may be on process-source.
Codestyle problem should be handled immediately, but not on quite expensive
TC run.

пн, 6 июл. 2020 г. в 20:34, Maxim Muzafarov :

> Hello Ilya,
>
> Why do you think that splitting the checkstyle build is better option
> than fixing code style issues reporting by the checkstyle plugin?
>
> On Mon, 6 Jul 2020 at 19:43, Ilya Kasnacheev 
> wrote:
> >
> > Hello!
> >
> > I have just noticed today that Checkstyle will fail Apache Ignite build:
> >
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite/5443282?buildTab=log=3=683.4289
> >
> > This means that I have completely lost an option to run tests against
> pull
> > requests by new contributors - they usually compile but will not pass
> > Checkstyle. That's a blocker.
> >
> > Can we please split Checkstyle as a separate build which is triggered
> with
> > Run All?
> > I think we even have
> >
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_CheckCodeStyle?mode=builds#all-projects
> >
> > WDYT?
> >
> > Regards,
> > --
> > Ilya Kasnacheev
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Pavel, pay attention to two dots at the end. All build files should stay at
current dir, and cmake-build-release is supposed to be a build root. And
this directory is added to gitignore. Actually, build directory can be
everywhere, but you should pass path to dir, that contains root
CMakeLists.txt. Usually it is parent directory of cmake-build-release, so
double dot is used. It is so common pattern of usage and I simply cannot
understand why it is misleading.

ср, 1 июл. 2020 г., 0:48 Pavel Tupitsyn :

> This is awesome, thanks everyone!
>
> I've tried following the instructions, and the following two steps seem to
> be misleading:
> * mkdir cmake-build-[release|debug]
> * cd ./cmake-build-[release|debug]
>
> When I run cmake in the newly created cmake-build-debug dir, I get:
> CMake Error: The source directory
> "/home/pavel/w/ignite/modules/platforms/cpp/cmake-build-release" does not
> appear to contain CMakeLists.txt.
>
> If I skip those 2 steps, I can build Ignite C++ and run tests with ctest on
> Ubuntu 20.04.
>
>
> Another observation: building the project produces untracked git files.
> Should we update gitignore accordingly?
>
>
> Thanks,
> Pavel
>
>
> On Tue, Jun 30, 2020 at 9:33 PM Ivan Daschinsky 
> wrote:
>
> > Ok, CMake now is the only option to build C++ on non-windows platforms.
> > Autotools is removed from master.
> >
> > Many thanks to Igor Sapego, Zhenya Stanilovsky, Nickolay Izhikov and Ilya
> > Kasnacheev for review, testing and suggestions.
> >
> > вт, 23 июн. 2020 г. в 18:42, Ivan Daschinsky :
> >
> > > I suppose, that removal of autotools from source code is a question of
> a
> > > week. There is no need to support it.
> > >
> > > вт, 23 июн. 2020 г., 18:38 Ilya Kasnacheev  >:
> > >
> > >> Hello!
> > >>
> > >> Once you do that, I think it would make sense to remove autotools
> > >> invocation from artifact build process. In the future we may choose to
> > >> remove autotools support entirely.
> > >>
> > >> Regards,
> > >> --
> > >> Ilya Kasnacheev
> > >>
> > >>
> > >> вт, 23 июн. 2020 г. в 18:00, Petr Ivanov :
> > >>
> > >> > That host was down for some time and got up only recently.
> > >> > Installed cmake there also.
> > >> >
> > >> >
> > >> >
> > >> > > On 23 Jun 2020, at 17:39, Ivan Daschinsky 
> > >> wrote:
> > >> > >
> > >> > > Petr, I see, that cmake is missing on aitc10_05 but it is in pool
> > and
> > >> > > trying to run build.
> > >> > >
> > >> > > [1] --
> > >> > >
> > >> >
> > >>
> >
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PlatformCPPCMakeLinux/5412025
> > >> > >
> > >> > > вт, 23 июн. 2020 г. в 17:33, Ivan Daschinsky  >:
> > >> > >
> > >> > >>>> For the sake of discretion, I'd purpose to remove old suites
> from
> > >> Run
> > >> > >> All and archive or pause them for some time before removal.
> > >> > >> Agree, I was talking about removal from RunAll chain, complete
> > >> removal
> > >> > is
> > >> > >> not necessary.
> > >> > >> But after private discussion with Igor Sapego, me and he decided
> to
> > >> wait
> > >> > >> for a week before exclusion of these suites from RunAll.
> > >> > >>
> > >> > >> вт, 23 июн. 2020 г. в 17:21, Petr Ivanov :
> > >> > >>
> > >> > >>> For the sake of discretion, I'd purpose to remove old suites
> from
> > >> Run
> > >> > All
> > >> > >>> and archive or pause them for some time before removal.
> > >> > >>>
> > >> > >>>
> > >> > >>>> On 23 Jun 2020, at 13:26, Nikolay Izhikov  >
> > >> > wrote:
> > >> > >>>>
> > >> > >>>>> So, folks, is it ok to remove old suites from build chain
> > >> > >>>>
> > >> > >>>> +1 from me.
> > >> > >>>>
> > >> > >>>>
> > >> > >>>>> 23 июня 2020 г., в 13:15, Ivan Daschinsky <
> ivanda...@gmail

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Let me explain with details.

CMake is unusual from other build systems, that it can separate build root
and source root. Moreover, it is strongly discouraged to mix them.
Benefits:
1. You can have many build with different options using same source.
2. You can build project with different toolchain using same source.
3. You can build project for different platforms using same source
(cross-compilation, docker, etc.)

Common patter is:
1. Create build root.
2. Change current dir to build root.
3. Invoke cmake in build root, pointing to source root.

Usually build root is subdirectory, of source root. So, as it mentioned in
DEVNOTES.txt, usual steps are:
1. mdkir  (usually named cmake-build-[-]
2. cd 
3. cmake [-DOPTION1=VALUE1 -DOPTION2=VALUE2 ... ] .. (NB! This double dots
are important, they points to source root)

This is so common, that you can hardly find any tutorial about CMake in
google without thes double dots. This is basic of CMake.

So if you think that we should explain for C++ developers basics of the
most common build system (CMake is really very popular and nowadays usually
must for new projects),
we can add this to DEVNOTES.txt. But I think that is not necessary at all.
For example, we don't explain basics of maven, though it is not obvious at
all as it seems to us, Ignite developers.

ср, 1 июл. 2020 г. в 06:18, Ivan Daschinsky :

> Pavel, pay attention to two dots at the end. All build files should stay
> at current dir, and cmake-build-release is supposed to be a build root. And
> this directory is added to gitignore. Actually, build directory can be
> everywhere, but you should pass path to dir, that contains root
> CMakeLists.txt. Usually it is parent directory of cmake-build-release, so
> double dot is used. It is so common pattern of usage and I simply cannot
> understand why it is misleading.
>
> ср, 1 июл. 2020 г., 0:48 Pavel Tupitsyn :
>
>> This is awesome, thanks everyone!
>>
>> I've tried following the instructions, and the following two steps seem to
>> be misleading:
>> * mkdir cmake-build-[release|debug]
>> * cd ./cmake-build-[release|debug]
>>
>> When I run cmake in the newly created cmake-build-debug dir, I get:
>> CMake Error: The source directory
>> "/home/pavel/w/ignite/modules/platforms/cpp/cmake-build-release" does not
>> appear to contain CMakeLists.txt.
>>
>> If I skip those 2 steps, I can build Ignite C++ and run tests with ctest
>> on
>> Ubuntu 20.04.
>>
>>
>> Another observation: building the project produces untracked git files.
>> Should we update gitignore accordingly?
>>
>>
>> Thanks,
>> Pavel
>>
>>
>> On Tue, Jun 30, 2020 at 9:33 PM Ivan Daschinsky 
>> wrote:
>>
>> > Ok, CMake now is the only option to build C++ on non-windows platforms.
>> > Autotools is removed from master.
>> >
>> > Many thanks to Igor Sapego, Zhenya Stanilovsky, Nickolay Izhikov and
>> Ilya
>> > Kasnacheev for review, testing and suggestions.
>> >
>> > вт, 23 июн. 2020 г. в 18:42, Ivan Daschinsky :
>> >
>> > > I suppose, that removal of autotools from source code is a question
>> of a
>> > > week. There is no need to support it.
>> > >
>> > > вт, 23 июн. 2020 г., 18:38 Ilya Kasnacheev > >:
>> > >
>> > >> Hello!
>> > >>
>> > >> Once you do that, I think it would make sense to remove autotools
>> > >> invocation from artifact build process. In the future we may choose
>> to
>> > >> remove autotools support entirely.
>> > >>
>> > >> Regards,
>> > >> --
>> > >> Ilya Kasnacheev
>> > >>
>> > >>
>> > >> вт, 23 июн. 2020 г. в 18:00, Petr Ivanov :
>> > >>
>> > >> > That host was down for some time and got up only recently.
>> > >> > Installed cmake there also.
>> > >> >
>> > >> >
>> > >> >
>> > >> > > On 23 Jun 2020, at 17:39, Ivan Daschinsky 
>> > >> wrote:
>> > >> > >
>> > >> > > Petr, I see, that cmake is missing on aitc10_05 but it is in pool
>> > and
>> > >> > > trying to run build.
>> > >> > >
>> > >> > > [1] --
>> > >> > >
>> > >> >
>> > >>
>> >
>> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PlatformCPPCMakeLinux/5412025
>> > >> > >
>> > >> > > вт, 23 июн. 2

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

No, I am really curious if this instruction complete or not. Actually, Ilya
Kasnacheev asked me about the same thing as you. And may be instruction is
not so obvious and should be reworked. You know, that developers are not
usually great in writing documentation.

ср, 1 июл. 2020 г., 10:37 Pavel Tupitsyn :

> Ivan,
>
> Thank you for such a detailed explanation!
> I am sorry if I upset you.
>
> I just wanted to make sure that instructions are correct,
> and I simply missed those 2 dots. Now it all works perfectly.
>
> Pavel
>
> On Wed, Jul 1, 2020 at 8:09 AM Ivan Daschinsky 
> wrote:
>
> > Let me explain with details.
> >
> > CMake is unusual from other build systems, that it can separate build
> root
> > and source root. Moreover, it is strongly discouraged to mix them.
> > Benefits:
> > 1. You can have many build with different options using same source.
> > 2. You can build project with different toolchain using same source.
> > 3. You can build project for different platforms using same source
> > (cross-compilation, docker, etc.)
> >
> > Common patter is:
> > 1. Create build root.
> > 2. Change current dir to build root.
> > 3. Invoke cmake in build root, pointing to source root.
> >
> > Usually build root is subdirectory, of source root. So, as it mentioned
> in
> > DEVNOTES.txt, usual steps are:
> > 1. mdkir  (usually named
> cmake-build-[-]
> > 2. cd 
> > 3. cmake [-DOPTION1=VALUE1 -DOPTION2=VALUE2 ... ] .. (NB! This double
> dots
> > are important, they points to source root)
> >
> > This is so common, that you can hardly find any tutorial about CMake in
> > google without thes double dots. This is basic of CMake.
> >
> > So if you think that we should explain for C++ developers basics of the
> > most common build system (CMake is really very popular and nowadays
> usually
> > must for new projects),
> > we can add this to DEVNOTES.txt. But I think that is not necessary at
> all.
> > For example, we don't explain basics of maven, though it is not obvious
> at
> > all as it seems to us, Ignite developers.
> >
> > ср, 1 июл. 2020 г. в 06:18, Ivan Daschinsky :
> >
> > > Pavel, pay attention to two dots at the end. All build files should
> stay
> > > at current dir, and cmake-build-release is supposed to be a build root.
> > And
> > > this directory is added to gitignore. Actually, build directory can be
> > > everywhere, but you should pass path to dir, that contains root
> > > CMakeLists.txt. Usually it is parent directory of cmake-build-release,
> so
> > > double dot is used. It is so common pattern of usage and I simply
> cannot
> > > understand why it is misleading.
> > >
> > > ср, 1 июл. 2020 г., 0:48 Pavel Tupitsyn :
> > >
> > >> This is awesome, thanks everyone!
> > >>
> > >> I've tried following the instructions, and the following two steps
> seem
> > to
> > >> be misleading:
> > >> * mkdir cmake-build-[release|debug]
> > >> * cd ./cmake-build-[release|debug]
> > >>
> > >> When I run cmake in the newly created cmake-build-debug dir, I get:
> > >> CMake Error: The source directory
> > >> "/home/pavel/w/ignite/modules/platforms/cpp/cmake-build-release" does
> > not
> > >> appear to contain CMakeLists.txt.
> > >>
> > >> If I skip those 2 steps, I can build Ignite C++ and run tests with
> ctest
> > >> on
> > >> Ubuntu 20.04.
> > >>
> > >>
> > >> Another observation: building the project produces untracked git
> files.
> > >> Should we update gitignore accordingly?
> > >>
> > >>
> > >> Thanks,
> > >> Pavel
> > >>
> > >>
> > >> On Tue, Jun 30, 2020 at 9:33 PM Ivan Daschinsky 
> > >> wrote:
> > >>
> > >> > Ok, CMake now is the only option to build C++ on non-windows
> > platforms.
> > >> > Autotools is removed from master.
> > >> >
> > >> > Many thanks to Igor Sapego, Zhenya Stanilovsky, Nickolay Izhikov and
> > >> Ilya
> > >> > Kasnacheev for review, testing and suggestions.
> > >> >
> > >> > вт, 23 июн. 2020 г. в 18:42, Ivan Daschinsky :
> > >> >
> > >> > > I suppose, that removal of autotools from source code is a
> question
> > >> of a
> > >> > > week. There

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Well, patch is merged. Thank a lot to Igor Sapego for review.

Peter, well, cmake now can be installed on all agent.

I think it's time to add cmake suites to runAll and exclude old suites from
it.

Patch with removing autotools I Will submit in few days.

пн, 22 июн. 2020 г., 20:25 Ivan Daschinsky :

> Hi folks!
>
> Good news -- I successfully created 2 CMake suites [1] [2] and they works
> as a charm.
> Many thanks to Ilya Kasnacheev for giving me permissions and to Peter
> Ivanov for installing cmake and giving a whole agent for testing.
>
> [1] --
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PlatformCPPCMakeLinuxClang_IgniteTests24Java8=pull%2F7854%2Fhead=buildTypeStatusDiv
> [2] --
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PlatformCPPCMakeLinux_IgniteTests24Java8=pull%2F7854%2Fhead=buildTypeStatusDiv
>
> пн, 22 июн. 2020 г. в 12:03, Ilya Kasnacheev :
>
>> Hello!
>>
>> I think you should contact Peter Ivanov if you want anything to be
>> installed on agents.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 22 июн. 2020 г. в 12:00, Ivan Daschinsky :
>>
>> > Actually, I'm going to create 2 new suites based on existing ones:
>> > 1. Platform C++ CMake (Linux)
>> > 2. Platform C++ CMake (Linux Clang)
>> >
>> > Ilya, thank you very much. But, despite the fact, that I have
>> permissions,
>> > it seems that it's impossible to install something (namely, CMake) on
>> > agents.
>> > How can I do this obviously essential task?
>> >
>> > пн, 22 июн. 2020 г. в 11:30, Ilya Kasnacheev > >:
>> >
>> > > Hello!
>> > >
>> > > I have assigned roles on TC, you can now work on these builds.
>> > >
>> > > Please describe your changes on development list so that people
>> > understand
>> > > what is going on.
>> > >
>> > > Regards,
>> > > --
>> > > Ilya Kasnacheev
>> > >
>> > >
>> > > вс, 21 июн. 2020 г. в 17:35, Ivan Daschinsky :
>> > >
>> > > > Hi folks.
>> > > >
>> > > > Patch is rigorously tested and with Igor Sapego help it is possible
>> to
>> > > > build with cmake Ignite.C++  even on windows.
>> > > > But, it's required to make some TC suites and tune them, in order to
>> > > patch
>> > > > be merged.
>> > > > Unfortunately, I am not a commiter and I doesn't have rights to
>> create
>> > > > suites. Also, AFAIK, Igor currently is quite busy and doesn't have
>> much
>> > > > time to do this task by himself.
>> > > > So I need some help from community. Is it possible to grant me some
>> > > rights
>> > > > to TC agents?
>> > > >
>> > > > ср, 3 июн. 2020 г. в 18:03, Nikolay Izhikov :
>> > > >
>> > > > > Hello.
>> > > > >
>> > > > > I will do review of this changes.
>> > > > >
>> > > > > > 1 июня 2020 г., в 13:21, Ivan Daschinsky 
>> > > > > написал(а):
>> > > > > >
>> > > > > > Igor, could you please check my PR?
>> > > > > >
>> > > > > > пт, 29 мая 2020 г. в 15:28, Ivan Daschinsky <
>> ivanda...@gmail.com>:
>> > > > > >
>> > > > > >> Thanks you all. Run patch (I've changed some code also) on TC
>> --
>> > all
>> > > > CPP
>> > > > > >> suites are green (GCC, CLang, Win64)
>> > > > > >>
>> > > > > >> пт, 29 мая 2020 г. в 15:02, Zhenya Stanilovsky
>> > > > > > > > > > >>> :
>> > > > > >>
>> > > > > >>>
>> > > > > >>>
>> > > > > >>> Ivan besides documentation [1]
>> > > > > >>> -s no —  no works
>> > > > > >>> -- catch_system_errors =no — works properly well, tests are
>> > passed.
>> > > > > >>>
>> > > > > >>> boost 1.65
>> > > > > >>>
>> > > > > >>> [1]
>> > > > > >>>
>> > > > >
>> > > >
>> > >
>> >
>> https://www.boost.org/do

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Hi folks.

Patch is rigorously tested and with Igor Sapego help it is possible to
build with cmake Ignite.C++  even on windows.
But, it's required to make some TC suites and tune them, in order to patch
be merged.
Unfortunately, I am not a commiter and I doesn't have rights to create
suites. Also, AFAIK, Igor currently is quite busy and doesn't have much
time to do this task by himself.
So I need some help from community. Is it possible to grant me some rights
to TC agents?

ср, 3 июн. 2020 г. в 18:03, Nikolay Izhikov :

> Hello.
>
> I will do review of this changes.
>
> > 1 июня 2020 г., в 13:21, Ivan Daschinsky 
> написал(а):
> >
> > Igor, could you please check my PR?
> >
> > пт, 29 мая 2020 г. в 15:28, Ivan Daschinsky :
> >
> >> Thanks you all. Run patch (I've changed some code also) on TC -- all CPP
> >> suites are green (GCC, CLang, Win64)
> >>
> >> пт, 29 мая 2020 г. в 15:02, Zhenya Stanilovsky
>  >>> :
> >>
> >>>
> >>>
> >>> Ivan besides documentation [1]
> >>> -s no —  no works
> >>> -- catch_system_errors =no — works properly well, tests are passed.
> >>>
> >>> boost 1.65
> >>>
> >>> [1]
> >>>
> https://www.boost.org/doc/libs/1_65_0/libs/test/doc/html/boost_test/utf_reference/rt_param_reference/catch_system.html
> >>>
> >>>> Hello!
> >>>>
> >>>> I didn't check tests since I don't develop AI C++, merely use it as
> user.
> >>>> That's where we should wait for Igor Sapego to check.
> >>>>
> >>>> Regards,
> >>>> --
> >>>> Ilya Kasnacheev
> >>>>
> >>>>
> >>>> пт, 29 мая 2020 г. в 12:20, Ivan Daschinsky < ivanda...@gmail.com >:
> >>>>
> >>>>> Ilya, thanks a lot! What about tests? I found one flag that must be
> >>>>> supplied to boost.tests.
> >>>>> This flag should fix JVM crash of C++ suites on Linux.
> >>>>>
> >>>>> Zhenya Stanilovsky and me have checked, that without this flag tests
> >>> failed
> >>>>> with SIGSEGV early (boost catch this signal from jvm, but it is ok
> for
> >>>>> jvm).
> >>>>> Flag is -catch_system_errors=no. I added it to CTest runner. You can
> >>>>> invoke it manually and using make test ARGS="-V"
> >>>>>
> >>>>>
> >>>>>
> >>>>> пт, 29 мая 2020 г. в 11:54, Ilya Kasnacheev <
> >>> ilya.kasnach...@gmail.com >:
> >>>>>
> >>>>>> Hello!
> >>>>>>
> >>>>>> Looks good to me! But we probably also ask Igor to take a look.
> >>>>>>
> >>>>>> Compiled debug and release, without and with odbc, checked running
> >>> thick
> >>>>>> node and ODBC connection on Linux.
> >>>>>>
> >>>>>> Regards,
> >>>>>> --
> >>>>>> Ilya Kasnacheev
> >>>>>>
> >>>>>>
> >>>>>> чт, 28 мая 2020 г. в 17:31, Ivan Daschinsky < ivanda...@gmail.com
> >:
> >>>>>>
> >>>>>>> Ok, PR is ready
> >>>>>>> https://issues.apache.org/jira/browse/IGNITE-13078
> >>>>>>>
> >>>>>>> Build tested on Mac OS X 10.15 and Ubuntu 20.04 with CMake 3.17.2
> >>> and
> >>>>>> 3.6.1
> >>>>>>> Unfortunately, I was not able to test on Windows, but principally
> >>> it
> >>>>>> should
> >>>>>>> works, but minor issues are probable.
> >>>>>>>
> >>>>>>> Instruction is attached in PR.
> >>>>>>> Any use reports are welcomed!
> >>>>>>>
> >>>>>>> вт, 26 мая 2020 г. в 18:51, Ivan Daschinsky < ivanda...@gmail.com
> >>>> :
> >>>>>>>
> >>>>>>>> Stephen, looks great! I do mostly the same things in C++ code.
> >>> Thank
> >>>>>> you!
> >>>>>>>>
> >>>>>>>> вт, 26 мая 2020 г. в 18:33, Stephen Darlington <
> >>>>>>>> stephen.darling...@gridgain.com >:
> >>>>>>>>
> >>>>>>>>> N

MTCGA spam about BinaryConfigurationTest in .NET Core suite

[Ivan Daschinsky]

Hi folks!

It seems, that this test mysteriously fails only on one agent aitc-lin01_04
with this error. [1]
Locally this .NET Core tests doesn't fail for about 20 minutes of
continuously run (Mac OS X 10.15, .NET Core 3.1.200)

According to stacktrace, it seems that there is problem with build
configuration and invalid jars are in classpath.

Petr Ivanov, may be it is good idea to clean up aitc-lin01_04?

-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Ok, I changed agents requirements to builds and add them to runAll

пн, 22 июн. 2020 г. в 22:39, Petr Ivanov :

> Cmake is installed on all agents (except 10 which is currently down and
> will be updated later).
>
>
> > On 22 Jun 2020, at 21:03, Ivan Daschinsky  wrote:
> >
> > Well, patch is merged. Thank a lot to Igor Sapego for review.
> >
> > Peter, well, cmake now can be installed on all agent.
> >
> > I think it's time to add cmake suites to runAll and exclude old suites
> from
> > it.
> >
> > Patch with removing autotools I Will submit in few days.
> >
> > пн, 22 июн. 2020 г., 20:25 Ivan Daschinsky :
> >
> >> Hi folks!
> >>
> >> Good news -- I successfully created 2 CMake suites [1] [2] and they
> works
> >> as a charm.
> >> Many thanks to Ilya Kasnacheev for giving me permissions and to Peter
> >> Ivanov for installing cmake and giving a whole agent for testing.
> >>
> >> [1] --
> >>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PlatformCPPCMakeLinuxClang_IgniteTests24Java8=pull%2F7854%2Fhead=buildTypeStatusDiv
> >> [2] --
> >>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PlatformCPPCMakeLinux_IgniteTests24Java8=pull%2F7854%2Fhead=buildTypeStatusDiv
> >>
> >> пн, 22 июн. 2020 г. в 12:03, Ilya Kasnacheev  >:
> >>
> >>> Hello!
> >>>
> >>> I think you should contact Peter Ivanov if you want anything to be
> >>> installed on agents.
> >>>
> >>> Regards,
> >>> --
> >>> Ilya Kasnacheev
> >>>
> >>>
> >>> пн, 22 июн. 2020 г. в 12:00, Ivan Daschinsky :
> >>>
> >>>> Actually, I'm going to create 2 new suites based on existing ones:
> >>>> 1. Platform C++ CMake (Linux)
> >>>> 2. Platform C++ CMake (Linux Clang)
> >>>>
> >>>> Ilya, thank you very much. But, despite the fact, that I have
> >>> permissions,
> >>>> it seems that it's impossible to install something (namely, CMake) on
> >>>> agents.
> >>>> How can I do this obviously essential task?
> >>>>
> >>>> пн, 22 июн. 2020 г. в 11:30, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> >>>> :
> >>>>
> >>>>> Hello!
> >>>>>
> >>>>> I have assigned roles on TC, you can now work on these builds.
> >>>>>
> >>>>> Please describe your changes on development list so that people
> >>>> understand
> >>>>> what is going on.
> >>>>>
> >>>>> Regards,
> >>>>> --
> >>>>> Ilya Kasnacheev
> >>>>>
> >>>>>
> >>>>> вс, 21 июн. 2020 г. в 17:35, Ivan Daschinsky :
> >>>>>
> >>>>>> Hi folks.
> >>>>>>
> >>>>>> Patch is rigorously tested and with Igor Sapego help it is possible
> >>> to
> >>>>>> build with cmake Ignite.C++  even on windows.
> >>>>>> But, it's required to make some TC suites and tune them, in order to
> >>>>> patch
> >>>>>> be merged.
> >>>>>> Unfortunately, I am not a commiter and I doesn't have rights to
> >>> create
> >>>>>> suites. Also, AFAIK, Igor currently is quite busy and doesn't have
> >>> much
> >>>>>> time to do this task by himself.
> >>>>>> So I need some help from community. Is it possible to grant me some
> >>>>> rights
> >>>>>> to TC agents?
> >>>>>>
> >>>>>> ср, 3 июн. 2020 г. в 18:03, Nikolay Izhikov :
> >>>>>>
> >>>>>>> Hello.
> >>>>>>>
> >>>>>>> I will do review of this changes.
> >>>>>>>
> >>>>>>>> 1 июня 2020 г., в 13:21, Ivan Daschinsky 
> >>>>>>> написал(а):
> >>>>>>>>
> >>>>>>>> Igor, could you please check my PR?
> >>>>>>>>
> >>>>>>>> пт, 29 мая 2020 г. в 15:28, Ivan Daschinsky <
> >>> ivanda...@gmail.com>:
> >>>>>>>>
> >>>>>>>>> Thanks you all. Run patch (I've cha

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Well, new suites added to RunAll and all seems to be ok.

I think it is time to remove old suites.
WDYT?

Also, I created patch with removal of autotools, but old suites should be
removed first.

So, folks, is it ok to remove old suites from build chain?

вт, 23 июн. 2020 г. в 10:36, Ivan Daschinsky :

> Ok, I changed agents requirements to builds and add them to runAll
>
> пн, 22 июн. 2020 г. в 22:39, Petr Ivanov :
>
>> Cmake is installed on all agents (except 10 which is currently down and
>> will be updated later).
>>
>>
>> > On 22 Jun 2020, at 21:03, Ivan Daschinsky  wrote:
>> >
>> > Well, patch is merged. Thank a lot to Igor Sapego for review.
>> >
>> > Peter, well, cmake now can be installed on all agent.
>> >
>> > I think it's time to add cmake suites to runAll and exclude old suites
>> from
>> > it.
>> >
>> > Patch with removing autotools I Will submit in few days.
>> >
>> > пн, 22 июн. 2020 г., 20:25 Ivan Daschinsky :
>> >
>> >> Hi folks!
>> >>
>> >> Good news -- I successfully created 2 CMake suites [1] [2] and they
>> works
>> >> as a charm.
>> >> Many thanks to Ilya Kasnacheev for giving me permissions and to Peter
>> >> Ivanov for installing cmake and giving a whole agent for testing.
>> >>
>> >> [1] --
>> >>
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PlatformCPPCMakeLinuxClang_IgniteTests24Java8=pull%2F7854%2Fhead=buildTypeStatusDiv
>> >> [2] --
>> >>
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PlatformCPPCMakeLinux_IgniteTests24Java8=pull%2F7854%2Fhead=buildTypeStatusDiv
>> >>
>> >> пн, 22 июн. 2020 г. в 12:03, Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com>:
>> >>
>> >>> Hello!
>> >>>
>> >>> I think you should contact Peter Ivanov if you want anything to be
>> >>> installed on agents.
>> >>>
>> >>> Regards,
>> >>> --
>> >>> Ilya Kasnacheev
>> >>>
>> >>>
>> >>> пн, 22 июн. 2020 г. в 12:00, Ivan Daschinsky :
>> >>>
>> >>>> Actually, I'm going to create 2 new suites based on existing ones:
>> >>>> 1. Platform C++ CMake (Linux)
>> >>>> 2. Platform C++ CMake (Linux Clang)
>> >>>>
>> >>>> Ilya, thank you very much. But, despite the fact, that I have
>> >>> permissions,
>> >>>> it seems that it's impossible to install something (namely, CMake) on
>> >>>> agents.
>> >>>> How can I do this obviously essential task?
>> >>>>
>> >>>> пн, 22 июн. 2020 г. в 11:30, Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com
>> >>>> :
>> >>>>
>> >>>>> Hello!
>> >>>>>
>> >>>>> I have assigned roles on TC, you can now work on these builds.
>> >>>>>
>> >>>>> Please describe your changes on development list so that people
>> >>>> understand
>> >>>>> what is going on.
>> >>>>>
>> >>>>> Regards,
>> >>>>> --
>> >>>>> Ilya Kasnacheev
>> >>>>>
>> >>>>>
>> >>>>> вс, 21 июн. 2020 г. в 17:35, Ivan Daschinsky :
>> >>>>>
>> >>>>>> Hi folks.
>> >>>>>>
>> >>>>>> Patch is rigorously tested and with Igor Sapego help it is possible
>> >>> to
>> >>>>>> build with cmake Ignite.C++  even on windows.
>> >>>>>> But, it's required to make some TC suites and tune them, in order
>> to
>> >>>>> patch
>> >>>>>> be merged.
>> >>>>>> Unfortunately, I am not a commiter and I doesn't have rights to
>> >>> create
>> >>>>>> suites. Also, AFAIK, Igor currently is quite busy and doesn't have
>> >>> much
>> >>>>>> time to do this task by himself.
>> >>>>>> So I need some help from community. Is it possible to grant me some
>> >>>>> rights
>> >>>>>> to TC agents?
>> >>>>>>
>> >>>>>> ср, 3 июн. 2020 г. в 18:03,

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

>> For the sake of discretion, I'd purpose to remove old suites from Run
All and archive or pause them for some time before removal.
Agree, I was talking about removal from RunAll chain, complete removal is
not necessary.
But after private discussion with Igor Sapego, me and he decided to wait
for a week before exclusion of these suites from RunAll.

вт, 23 июн. 2020 г. в 17:21, Petr Ivanov :

> For the sake of discretion, I'd purpose to remove old suites from Run All
> and archive or pause them for some time before removal.
>
>
> > On 23 Jun 2020, at 13:26, Nikolay Izhikov  wrote:
> >
> >> So, folks, is it ok to remove old suites from build chain
> >
> > +1 from me.
> >
> >
> >> 23 июня 2020 г., в 13:15, Ivan Daschinsky 
> написал(а):
> >>
> >> Well, new suites added to RunAll and all seems to be ok.
> >>
> >> I think it is time to remove old suites.
> >> WDYT?
> >>
> >> Also, I created patch with removal of autotools, but old suites should
> be
> >> removed first.
> >>
> >> So, folks, is it ok to remove old suites from build chain?
> >>
> >> вт, 23 июн. 2020 г. в 10:36, Ivan Daschinsky :
> >>
> >>> Ok, I changed agents requirements to builds and add them to runAll
> >>>
> >>> пн, 22 июн. 2020 г. в 22:39, Petr Ivanov :
> >>>
> >>>> Cmake is installed on all agents (except 10 which is currently down
> and
> >>>> will be updated later).
> >>>>
> >>>>
> >>>>> On 22 Jun 2020, at 21:03, Ivan Daschinsky 
> wrote:
> >>>>>
> >>>>> Well, patch is merged. Thank a lot to Igor Sapego for review.
> >>>>>
> >>>>> Peter, well, cmake now can be installed on all agent.
> >>>>>
> >>>>> I think it's time to add cmake suites to runAll and exclude old
> suites
> >>>> from
> >>>>> it.
> >>>>>
> >>>>> Patch with removing autotools I Will submit in few days.
> >>>>>
> >>>>> пн, 22 июн. 2020 г., 20:25 Ivan Daschinsky :
> >>>>>
> >>>>>> Hi folks!
> >>>>>>
> >>>>>> Good news -- I successfully created 2 CMake suites [1] [2] and they
> >>>> works
> >>>>>> as a charm.
> >>>>>> Many thanks to Ilya Kasnacheev for giving me permissions and to
> Peter
> >>>>>> Ivanov for installing cmake and giving a whole agent for testing.
> >>>>>>
> >>>>>> [1] --
> >>>>>>
> >>>>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PlatformCPPCMakeLinuxClang_IgniteTests24Java8=pull%2F7854%2Fhead=buildTypeStatusDiv
> >>>>>> [2] --
> >>>>>>
> >>>>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PlatformCPPCMakeLinux_IgniteTests24Java8=pull%2F7854%2Fhead=buildTypeStatusDiv
> >>>>>>
> >>>>>> пн, 22 июн. 2020 г. в 12:03, Ilya Kasnacheev <
> >>>> ilya.kasnach...@gmail.com>:
> >>>>>>
> >>>>>>> Hello!
> >>>>>>>
> >>>>>>> I think you should contact Peter Ivanov if you want anything to be
> >>>>>>> installed on agents.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> --
> >>>>>>> Ilya Kasnacheev
> >>>>>>>
> >>>>>>>
> >>>>>>> пн, 22 июн. 2020 г. в 12:00, Ivan Daschinsky  >:
> >>>>>>>
> >>>>>>>> Actually, I'm going to create 2 new suites based on existing ones:
> >>>>>>>> 1. Platform C++ CMake (Linux)
> >>>>>>>> 2. Platform C++ CMake (Linux Clang)
> >>>>>>>>
> >>>>>>>> Ilya, thank you very much. But, despite the fact, that I have
> >>>>>>> permissions,
> >>>>>>>> it seems that it's impossible to install something (namely,
> CMake) on
> >>>>>>>> agents.
> >>>>>>>> How can I do this obviously essential task?
> >>>>>>>>
> >>>>>>>> пн, 22 июн. 2020 г. в 11:30, Ilya Kasnacheev <
> >>>> ilya.kasnach...@gmail.com
> >

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Petr, I see, that cmake is missing on aitc10_05 but it is in pool and
trying to run build.

[1] --
https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PlatformCPPCMakeLinux/5412025

вт, 23 июн. 2020 г. в 17:33, Ivan Daschinsky :

> >> For the sake of discretion, I'd purpose to remove old suites from Run
> All and archive or pause them for some time before removal.
> Agree, I was talking about removal from RunAll chain, complete removal is
> not necessary.
> But after private discussion with Igor Sapego, me and he decided to wait
> for a week before exclusion of these suites from RunAll.
>
> вт, 23 июн. 2020 г. в 17:21, Petr Ivanov :
>
>> For the sake of discretion, I'd purpose to remove old suites from Run All
>> and archive or pause them for some time before removal.
>>
>>
>> > On 23 Jun 2020, at 13:26, Nikolay Izhikov  wrote:
>> >
>> >> So, folks, is it ok to remove old suites from build chain
>> >
>> > +1 from me.
>> >
>> >
>> >> 23 июня 2020 г., в 13:15, Ivan Daschinsky 
>> написал(а):
>> >>
>> >> Well, new suites added to RunAll and all seems to be ok.
>> >>
>> >> I think it is time to remove old suites.
>> >> WDYT?
>> >>
>> >> Also, I created patch with removal of autotools, but old suites should
>> be
>> >> removed first.
>> >>
>> >> So, folks, is it ok to remove old suites from build chain?
>> >>
>> >> вт, 23 июн. 2020 г. в 10:36, Ivan Daschinsky :
>> >>
>> >>> Ok, I changed agents requirements to builds and add them to runAll
>> >>>
>> >>> пн, 22 июн. 2020 г. в 22:39, Petr Ivanov :
>> >>>
>> >>>> Cmake is installed on all agents (except 10 which is currently down
>> and
>> >>>> will be updated later).
>> >>>>
>> >>>>
>> >>>>> On 22 Jun 2020, at 21:03, Ivan Daschinsky 
>> wrote:
>> >>>>>
>> >>>>> Well, patch is merged. Thank a lot to Igor Sapego for review.
>> >>>>>
>> >>>>> Peter, well, cmake now can be installed on all agent.
>> >>>>>
>> >>>>> I think it's time to add cmake suites to runAll and exclude old
>> suites
>> >>>> from
>> >>>>> it.
>> >>>>>
>> >>>>> Patch with removing autotools I Will submit in few days.
>> >>>>>
>> >>>>> пн, 22 июн. 2020 г., 20:25 Ivan Daschinsky :
>> >>>>>
>> >>>>>> Hi folks!
>> >>>>>>
>> >>>>>> Good news -- I successfully created 2 CMake suites [1] [2] and they
>> >>>> works
>> >>>>>> as a charm.
>> >>>>>> Many thanks to Ilya Kasnacheev for giving me permissions and to
>> Peter
>> >>>>>> Ivanov for installing cmake and giving a whole agent for testing.
>> >>>>>>
>> >>>>>> [1] --
>> >>>>>>
>> >>>>
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PlatformCPPCMakeLinuxClang_IgniteTests24Java8=pull%2F7854%2Fhead=buildTypeStatusDiv
>> >>>>>> [2] --
>> >>>>>>
>> >>>>
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PlatformCPPCMakeLinux_IgniteTests24Java8=pull%2F7854%2Fhead=buildTypeStatusDiv
>> >>>>>>
>> >>>>>> пн, 22 июн. 2020 г. в 12:03, Ilya Kasnacheev <
>> >>>> ilya.kasnach...@gmail.com>:
>> >>>>>>
>> >>>>>>> Hello!
>> >>>>>>>
>> >>>>>>> I think you should contact Peter Ivanov if you want anything to be
>> >>>>>>> installed on agents.
>> >>>>>>>
>> >>>>>>> Regards,
>> >>>>>>> --
>> >>>>>>> Ilya Kasnacheev
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> пн, 22 июн. 2020 г. в 12:00, Ivan Daschinsky > >:
>> >>>>>>>
>> >>>>>>>> Actually, I'm going to create 2 new suites based on existing
>> ones:
>> >>>>>>>> 1. Platform C++ CMake (Linux)
>> >>

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

I suppose, that removal of autotools from source code is a question of a
week. There is no need to support it.

вт, 23 июн. 2020 г., 18:38 Ilya Kasnacheev :

> Hello!
>
> Once you do that, I think it would make sense to remove autotools
> invocation from artifact build process. In the future we may choose to
> remove autotools support entirely.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 23 июн. 2020 г. в 18:00, Petr Ivanov :
>
> > That host was down for some time and got up only recently.
> > Installed cmake there also.
> >
> >
> >
> > > On 23 Jun 2020, at 17:39, Ivan Daschinsky  wrote:
> > >
> > > Petr, I see, that cmake is missing on aitc10_05 but it is in pool and
> > > trying to run build.
> > >
> > > [1] --
> > >
> >
> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PlatformCPPCMakeLinux/5412025
> > >
> > > вт, 23 июн. 2020 г. в 17:33, Ivan Daschinsky :
> > >
> > >>>> For the sake of discretion, I'd purpose to remove old suites from
> Run
> > >> All and archive or pause them for some time before removal.
> > >> Agree, I was talking about removal from RunAll chain, complete removal
> > is
> > >> not necessary.
> > >> But after private discussion with Igor Sapego, me and he decided to
> wait
> > >> for a week before exclusion of these suites from RunAll.
> > >>
> > >> вт, 23 июн. 2020 г. в 17:21, Petr Ivanov :
> > >>
> > >>> For the sake of discretion, I'd purpose to remove old suites from Run
> > All
> > >>> and archive or pause them for some time before removal.
> > >>>
> > >>>
> > >>>> On 23 Jun 2020, at 13:26, Nikolay Izhikov 
> > wrote:
> > >>>>
> > >>>>> So, folks, is it ok to remove old suites from build chain
> > >>>>
> > >>>> +1 from me.
> > >>>>
> > >>>>
> > >>>>> 23 июня 2020 г., в 13:15, Ivan Daschinsky 
> > >>> написал(а):
> > >>>>>
> > >>>>> Well, new suites added to RunAll and all seems to be ok.
> > >>>>>
> > >>>>> I think it is time to remove old suites.
> > >>>>> WDYT?
> > >>>>>
> > >>>>> Also, I created patch with removal of autotools, but old suites
> > should
> > >>> be
> > >>>>> removed first.
> > >>>>>
> > >>>>> So, folks, is it ok to remove old suites from build chain?
> > >>>>>
> > >>>>> вт, 23 июн. 2020 г. в 10:36, Ivan Daschinsky  >:
> > >>>>>
> > >>>>>> Ok, I changed agents requirements to builds and add them to runAll
> > >>>>>>
> > >>>>>> пн, 22 июн. 2020 г. в 22:39, Petr Ivanov :
> > >>>>>>
> > >>>>>>> Cmake is installed on all agents (except 10 which is currently
> down
> > >>> and
> > >>>>>>> will be updated later).
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> On 22 Jun 2020, at 21:03, Ivan Daschinsky 
> > >>> wrote:
> > >>>>>>>>
> > >>>>>>>> Well, patch is merged. Thank a lot to Igor Sapego for review.
> > >>>>>>>>
> > >>>>>>>> Peter, well, cmake now can be installed on all agent.
> > >>>>>>>>
> > >>>>>>>> I think it's time to add cmake suites to runAll and exclude old
> > >>> suites
> > >>>>>>> from
> > >>>>>>>> it.
> > >>>>>>>>
> > >>>>>>>> Patch with removing autotools I Will submit in few days.
> > >>>>>>>>
> > >>>>>>>> пн, 22 июн. 2020 г., 20:25 Ivan Daschinsky  >:
> > >>>>>>>>
> > >>>>>>>>> Hi folks!
> > >>>>>>>>>
> > >>>>>>>>> Good news -- I successfully created 2 CMake suites [1] [2] and
> > they
> > >>>>>>> works
> > >>>>>>>>> as a charm.
> > >>>>>>>>> Many thanks to Ilya K

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Hi folks!

Good news -- I successfully created 2 CMake suites [1] [2] and they works
as a charm.
Many thanks to Ilya Kasnacheev for giving me permissions and to Peter
Ivanov for installing cmake and giving a whole agent for testing.

[1] --
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PlatformCPPCMakeLinuxClang_IgniteTests24Java8=pull%2F7854%2Fhead=buildTypeStatusDiv
[2] --
https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PlatformCPPCMakeLinux_IgniteTests24Java8=pull%2F7854%2Fhead=buildTypeStatusDiv

пн, 22 июн. 2020 г. в 12:03, Ilya Kasnacheev :

> Hello!
>
> I think you should contact Peter Ivanov if you want anything to be
> installed on agents.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 22 июн. 2020 г. в 12:00, Ivan Daschinsky :
>
> > Actually, I'm going to create 2 new suites based on existing ones:
> > 1. Platform C++ CMake (Linux)
> > 2. Platform C++ CMake (Linux Clang)
> >
> > Ilya, thank you very much. But, despite the fact, that I have
> permissions,
> > it seems that it's impossible to install something (namely, CMake) on
> > agents.
> > How can I do this obviously essential task?
> >
> > пн, 22 июн. 2020 г. в 11:30, Ilya Kasnacheev  >:
> >
> > > Hello!
> > >
> > > I have assigned roles on TC, you can now work on these builds.
> > >
> > > Please describe your changes on development list so that people
> > understand
> > > what is going on.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > вс, 21 июн. 2020 г. в 17:35, Ivan Daschinsky :
> > >
> > > > Hi folks.
> > > >
> > > > Patch is rigorously tested and with Igor Sapego help it is possible
> to
> > > > build with cmake Ignite.C++  even on windows.
> > > > But, it's required to make some TC suites and tune them, in order to
> > > patch
> > > > be merged.
> > > > Unfortunately, I am not a commiter and I doesn't have rights to
> create
> > > > suites. Also, AFAIK, Igor currently is quite busy and doesn't have
> much
> > > > time to do this task by himself.
> > > > So I need some help from community. Is it possible to grant me some
> > > rights
> > > > to TC agents?
> > > >
> > > > ср, 3 июн. 2020 г. в 18:03, Nikolay Izhikov :
> > > >
> > > > > Hello.
> > > > >
> > > > > I will do review of this changes.
> > > > >
> > > > > > 1 июня 2020 г., в 13:21, Ivan Daschinsky 
> > > > > написал(а):
> > > > > >
> > > > > > Igor, could you please check my PR?
> > > > > >
> > > > > > пт, 29 мая 2020 г. в 15:28, Ivan Daschinsky  >:
> > > > > >
> > > > > >> Thanks you all. Run patch (I've changed some code also) on TC --
> > all
> > > > CPP
> > > > > >> suites are green (GCC, CLang, Win64)
> > > > > >>
> > > > > >> пт, 29 мая 2020 г. в 15:02, Zhenya Stanilovsky
> > > > >  > > > > >>> :
> > > > > >>
> > > > > >>>
> > > > > >>>
> > > > > >>> Ivan besides documentation [1]
> > > > > >>> -s no —  no works
> > > > > >>> -- catch_system_errors =no — works properly well, tests are
> > passed.
> > > > > >>>
> > > > > >>> boost 1.65
> > > > > >>>
> > > > > >>> [1]
> > > > > >>>
> > > > >
> > > >
> > >
> >
> https://www.boost.org/doc/libs/1_65_0/libs/test/doc/html/boost_test/utf_reference/rt_param_reference/catch_system.html
> > > > > >>>
> > > > > >>>> Hello!
> > > > > >>>>
> > > > > >>>> I didn't check tests since I don't develop AI C++, merely use
> it
> > > as
> > > > > user.
> > > > > >>>> That's where we should wait for Igor Sapego to check.
> > > > > >>>>
> > > > > >>>> Regards,
> > > > > >>>> --
> > > > > >>>> Ilya Kasnacheev
> > > > > >>>>
> > > > > >>>>
> > > > > >>>&

Re: MTCGA spam about BinaryConfigurationTest in .NET Core suite

[Ivan Daschinsky]

Pavel, yes, I know and this was my argument for this approach.
I will create fix for both platform asap.

чт, 25 июн. 2020 г. в 17:13, Pavel Tupitsyn :

> I think such a fix is fine, this is only for tests.
> We actually have a similar thing in .NET to skip rest-http [1]
>
> [1]
>
> https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/Apache.Ignite.Core/Impl/Common/Classpath.cs#L136
>
>
> On Thu, Jun 25, 2020 at 5:10 PM Ivan Daschinsky 
> wrote:
>
> > I think that just add an array of ignored modules for test platform
> > classpath is absolutely ok fix.
> >
> > But I have a discussion with Igor Sapego and he thinks that this fix is a
> > little bit crutchy.
> >
> > чт, 25 июн. 2020 г. в 17:06, Ivan Daschinsky :
> >
> > > Actually, We have 3 (sic!) versions of spring data (1x, 2.0 and 2.2)
> and
> > > all of them are correct.
> > >
> > > But any versions of spring-data is not needed for platforms test.
> > >
> > > чт, 25 июн. 2020 г. в 16:50, Pavel Tupitsyn :
> > >
> > >> Ivan, thank you so much for getting to the bottom of this issue!
> > >>
> > >> Why do we have two versions of spring-data, and which one is correct?
> > >>
> > >> On Thu, Jun 25, 2020 at 4:34 PM Ivan Daschinsky 
> > >> wrote:
> > >>
> > >> > The main reason of failure is jar-hell. When .NET or C++ tests are
> > >> started,
> > >> > if IGNITE_NATIVE_TEST_CLASSPATH is set to true, source directory is
> > >> > iterated and files libs, target/classes etc.
> > >> > are added to classpath. But neither readdir(), FindNextFileA()
> > >> > or Directory.EnumerateDirectories() do guarantee any ordering.
> > >> > But in spring-data-2.0 and spring-data-2.2 there are different
> version
> > >> of
> > >> > spring. So jar hell occurs and tests fails.
> > >> >
> > >> > I created small fix (ignore module dir that contains spring-data),
> and
> > >> > tests start working on aitc-lin11_02.
> > >> >
> > >> > Ticket for this problem is [1]
> > >> >
> > >> > [1] -- https://issues.apache.org/jira/browse/IGNITE-13187
> > >> >
> > >> >
> > >> >
> > >> > чт, 25 июн. 2020 г. в 11:46, Ivan Daschinsky :
> > >> >
> > >> > > Hi igniters!
> > >> > >
> > >> > > I found that .NET Core suite and C++ suites fails on two agents --
> > >> > > aitc-lin01_04 and aitc-lin11_02.
> > >> > >
> > >> > > So I think the best solution is to exclude these agents as
> > requirement
> > >> > for
> > >> > > these suites and debug problems on these
> > >> > > agents separately
> > >> > >
> > >> > > вт, 23 июн. 2020 г. в 09:33, Ivan Daschinsky  >:
> > >> > >
> > >> > >> Hi folks!
> > >> > >>
> > >> > >> It seems, that this test mysteriously fails only on one
> > >> > >> agent aitc-lin01_04 with this error. [1]
> > >> > >> Locally this .NET Core tests doesn't fail for about 20 minutes of
> > >> > >> continuously run (Mac OS X 10.15, .NET Core 3.1.200)
> > >> > >>
> > >> > >> According to stacktrace, it seems that there is problem with
> build
> > >> > >> configuration and invalid jars are in classpath.
> > >> > >>
> > >> > >> Petr Ivanov, may be it is good idea to clean up aitc-lin01_04?
> > >> > >>
> > >> > >> --
> > >> > >> Sincerely yours, Ivan Daschinskiy
> > >> > >>
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Sincerely yours, Ivan Daschinskiy
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> > Sincerely yours, Ivan Daschinskiy
> > >> >
> > >>
> > >
> > >
> > > --
> > > Sincerely yours, Ivan Daschinskiy
> > >
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: [MTCGA]: new failures in builds [5395772] needs to be handled

[Ivan Daschinsky]

It seems, that this test mysteriously fails only on one agent aitc-lin01_04
with this error. [1]
Locally this .NET Core tests doesn't fail for about 20 minutes of
continuously run (Mac OS X 10.15, .NET Core 3.1.200)

According to stacktrace, it seems that there is problem with build
configuration and invalid jars are in classpath.

пн, 22 июн. 2020 г. в 10:38, Alexei Scherbakov :

> Not clear why a bot reports my change, first failure was 3 days after the
> commit [1]
>
> [1]
>
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-2134815735276887535=%3Cdefault%3E=testDetails
>
> пн, 22 июн. 2020 г. в 05:20, :
>
> > Hi Igniters,
> >
> >  I've detected some new issue on TeamCity to be handled. You are more
> than
> > welcomed to help.
> >
> >  If your changes can lead to this failure(s): We're grateful that you
> were
> > a volunteer to make the contribution to this project, but things change
> and
> > you may no longer be able to finalize your contribution.
> >  Could you respond to this email and indicate if you wish to continue and
> > fix test failures or step down and some committer may revert you commit.
> >
> >  *Test with high flaky rate in master
> > BinaryConfigurationTest.TestXmlConfiguration
> >
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=4488201393739220401=%3Cdefault%3E=testDetails
> >
> >  *Test with high flaky rate in master
> > BinaryConfigurationTest.TestCodeConfiguration
> >
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-2134815735276887535=%3Cdefault%3E=testDetails
> >  Changes may lead to failure were done by
> >  - alexey scherbakov 
> > https://ci.ignite.apache.org/viewModification.html?modId=903321
> >
> >  - Here's a reminder of what contributors were agreed to do
> > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
> >  - Should you have any questions please contact
> > dev@ignite.apache.org
> >
> > Best Regards,
> > Apache Ignite TeamCity Bot
> > https://github.com/apache/ignite-teamcity-bot
> > Notification generated at 05:20:15 22-06-2020
> >
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSSION] Ignite.C++ and CMake

[Ivan Daschinsky]

Actually, I'm going to create 2 new suites based on existing ones:
1. Platform C++ CMake (Linux)
2. Platform C++ CMake (Linux Clang)

Ilya, thank you very much. But, despite the fact, that I have permissions,
it seems that it's impossible to install something (namely, CMake) on
agents.
How can I do this obviously essential task?

пн, 22 июн. 2020 г. в 11:30, Ilya Kasnacheev :

> Hello!
>
> I have assigned roles on TC, you can now work on these builds.
>
> Please describe your changes on development list so that people understand
> what is going on.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вс, 21 июн. 2020 г. в 17:35, Ivan Daschinsky :
>
> > Hi folks.
> >
> > Patch is rigorously tested and with Igor Sapego help it is possible to
> > build with cmake Ignite.C++  even on windows.
> > But, it's required to make some TC suites and tune them, in order to
> patch
> > be merged.
> > Unfortunately, I am not a commiter and I doesn't have rights to create
> > suites. Also, AFAIK, Igor currently is quite busy and doesn't have much
> > time to do this task by himself.
> > So I need some help from community. Is it possible to grant me some
> rights
> > to TC agents?
> >
> > ср, 3 июн. 2020 г. в 18:03, Nikolay Izhikov :
> >
> > > Hello.
> > >
> > > I will do review of this changes.
> > >
> > > > 1 июня 2020 г., в 13:21, Ivan Daschinsky 
> > > написал(а):
> > > >
> > > > Igor, could you please check my PR?
> > > >
> > > > пт, 29 мая 2020 г. в 15:28, Ivan Daschinsky :
> > > >
> > > >> Thanks you all. Run patch (I've changed some code also) on TC -- all
> > CPP
> > > >> suites are green (GCC, CLang, Win64)
> > > >>
> > > >> пт, 29 мая 2020 г. в 15:02, Zhenya Stanilovsky
> > >  > > >>> :
> > > >>
> > > >>>
> > > >>>
> > > >>> Ivan besides documentation [1]
> > > >>> -s no —  no works
> > > >>> -- catch_system_errors =no — works properly well, tests are passed.
> > > >>>
> > > >>> boost 1.65
> > > >>>
> > > >>> [1]
> > > >>>
> > >
> >
> https://www.boost.org/doc/libs/1_65_0/libs/test/doc/html/boost_test/utf_reference/rt_param_reference/catch_system.html
> > > >>>
> > > >>>> Hello!
> > > >>>>
> > > >>>> I didn't check tests since I don't develop AI C++, merely use it
> as
> > > user.
> > > >>>> That's where we should wait for Igor Sapego to check.
> > > >>>>
> > > >>>> Regards,
> > > >>>> --
> > > >>>> Ilya Kasnacheev
> > > >>>>
> > > >>>>
> > > >>>> пт, 29 мая 2020 г. в 12:20, Ivan Daschinsky < ivanda...@gmail.com
> > >:
> > > >>>>
> > > >>>>> Ilya, thanks a lot! What about tests? I found one flag that must
> be
> > > >>>>> supplied to boost.tests.
> > > >>>>> This flag should fix JVM crash of C++ suites on Linux.
> > > >>>>>
> > > >>>>> Zhenya Stanilovsky and me have checked, that without this flag
> > tests
> > > >>> failed
> > > >>>>> with SIGSEGV early (boost catch this signal from jvm, but it is
> ok
> > > for
> > > >>>>> jvm).
> > > >>>>> Flag is -catch_system_errors=no. I added it to CTest runner. You
> > can
> > > >>>>> invoke it manually and using make test ARGS="-V"
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> пт, 29 мая 2020 г. в 11:54, Ilya Kasnacheev <
> > > >>> ilya.kasnach...@gmail.com >:
> > > >>>>>
> > > >>>>>> Hello!
> > > >>>>>>
> > > >>>>>> Looks good to me! But we probably also ask Igor to take a look.
> > > >>>>>>
> > > >>>>>> Compiled debug and release, without and with odbc, checked
> running
> > > >>> thick
> > > >>>>>> node and ODBC connection on Linux.
> > > >>>>>>
> > > >>>>>> Regards,
> > > >>>&g

Re: Tool for performance statistics reports

[Ivan Daschinsky]

According to [1], it's absolutely OK to include files under MIT licence.
But we should mention these components or modules in LICENCE file.


[1] -- https://www.apache.org/legal/resolved.html

вс, 21 июн. 2020 г. в 23:34, Saikat Maitra :

> Hi Nikita,
>
> I have reviewed the PR and shared comments. I also had a question on
> including files with a MIT license.
>
> Are we ok to use both Apache and MIT licenses in our source files?
>
> Regards,
> Saikat
>
> On Fri, Jun 12, 2020 at 8:45 PM Saikat Maitra 
> wrote:
>
> > Hello Nikita,
> >
> > Thank you for sharing the information, it is very helpful.
> >
> > Regards,
> > Saikat
> >
> > On Thu, Jun 11, 2020 at 3:20 AM Nikita Amelchev 
> > wrote:
> >
> >> Hello,
> >>
> >> > Can you please share more info on how we can use the profiling tool
> with
> >> > ignite-extensions modules?
> >>
> >> I have updated the module readme file. You can find instructions
> >> there. For now, I am working on review fixes and marked PR as a draft.
> >>
> >> чт, 11 июн. 2020 г. в 03:21, Saikat Maitra :
> >> >
> >> > Hello Nikita,
> >> >
> >> > I observed we have an open PR in ignite repo for this feature with
> >> > different set of changes compared to ignite extensions repo.
> >> >
> >> >
> >> > apache/ignite#7693 
> >> >
> >> > https://github.com/apache/ignite-extensions/pull/16
> >> >
> >> > Can you please share more info on how we can use the profiling tool
> with
> >> > ignite-extensions modules?
> >> >
> >> >
> >> > Regards,
> >> >
> >> > Saikat
> >> >
> >> > On Mon, Jun 8, 2020 at 5:51 AM Nikolay Izhikov 
> >> wrote:
> >> >
> >> > > Hello, Alexey.
> >> > >
> >> > > Thanks for the review.
> >> > >
> >> > > My understanding if the following:
> >> > >
> >> > > We will have 3 in-depth tool to find issues in cluster:
> >> > >
> >> > > 1. Metrics + System views - data that describe Ignite entities very
> >> > > high-level.
> >> > >
> >> > > 2. Profiling - tool to know what specific query of transactions are
> >> slow.
> >> > > In many cases, this knowledge is enough to fix the issue(rewrite
> >> query,
> >> > > redesign transactions flow, etc)
> >> > >
> >> > > 3. Tracing - tool to know why one of 1000 of the same queries was
> >> slow.
> >> > > The most detailed view of the Ignite internal processes.
> >> > >
> >> > > > For example, a user would not be able to match a long task with a
> >> long
> >> > > job in that task.
> >> > >
> >> > > This is not true.
> >> > > Profiling report will aggregate data from all nodes.
> >> > > So there will be both
> >> > >
> >> > >  * summary time of the task
> >> > >  * time of the each job in the task.
> >> > >
> >> > >
> >> > > > 8 июня 2020 г., в 12:52, Alexey Goncharuk <
> >> alexey.goncha...@gmail.com>
> >> > > написал(а):
> >> > > >
> >> > > > Nikita, Igniters,
> >> > > >
> >> > > > I left a few comments on the tool itself in the PR.
> >> > > >
> >> > > > However, I would like to reiterate and discuss why a user would
> >> prefer to
> >> > > > use the profiling tool over tracing? Profiling tool only captures
> >> very
> >> > > > high-level details of the operations (a single cache operation,
> for
> >> > > > example), and does not interconnect operations happened on
> different
> >> > > nodes.
> >> > > > For example, a user would not be able to match a long task with a
> >> long
> >> > > job
> >> > > > in that task. In other words, profiling data is always a subset of
> >> data
> >> > > > collected by tracing.
> >> > > >
> >> > > > Maybe it makes sense to adopt local log file approach to write
> >> spans so
> >> > > we
> >> > > > can process those spans later to build a report?
> >> > > >
> >> > > > чт, 4 июн. 2020 г. в 11:16, Nikita Amelchev  >:
> >> > > >
> >> > > >> Hi, Igniters.
> >> > > >>
> >> > > >> I have implemented cluster profiling and tool to build the
> >> performance
> >> > > >> report. It's ready to be reviewed. [1, 2]
> >> > > >>
> >> > > >> Profiling can be managed by JMX bean. I have plans to implement
> it
> >> to
> >> > > >> control.sh also.
> >> > > >>
> >> > > >> Nodes write statistics to the temporary off heap buffer and then
> >> one
> >> > > >> thread flushes to the profiling files. The write mechanics and
> >> format
> >> > > >> is like WAL logging.
> >> > > >> The report contains the following statistics:
> >> > > >> - nodes and caches info
> >> > > >> - cache operations and transaction statistics
> >> > > >> - SQL and scan queries statistics (include logical and physical
> >> reads
> >> > > per
> >> > > >> query)
> >> > > >> - tasks and jobs statistics.
> >> > > >>
> >> > > >> More details in the IEP [3].
> >> > > >>
> >> > > >> [1] https://github.com/apache/ignite/pull/7693
> >> > > >> [2] https://issues.apache.org/jira/browse/IGNITE-12666
> >> > > >> [3]
> >> > > >>
> >> > >
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Cluster+performance+profiling+tool
> >> > > >>
> >> > > >> вс, 26 апр. 2020 г. в 17:29, Вячеслав Коптилин <
> >> > > 

Re: MTCGA spam about BinaryConfigurationTest in .NET Core suite

[Ivan Daschinsky]

Hi igniters!

I found that .NET Core suite and C++ suites fails on two agents --
aitc-lin01_04 and aitc-lin11_02.

So I think the best solution is to exclude these agents as requirement for
these suites and debug problems on these
agents separately

вт, 23 июн. 2020 г. в 09:33, Ivan Daschinsky :

> Hi folks!
>
> It seems, that this test mysteriously fails only on one
> agent aitc-lin01_04 with this error. [1]
> Locally this .NET Core tests doesn't fail for about 20 minutes of
> continuously run (Mac OS X 10.15, .NET Core 3.1.200)
>
> According to stacktrace, it seems that there is problem with build
> configuration and invalid jars are in classpath.
>
> Petr Ivanov, may be it is good idea to clean up aitc-lin01_04?
>
> --
> Sincerely yours, Ivan Daschinskiy
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: MTCGA spam about BinaryConfigurationTest in .NET Core suite

[Ivan Daschinsky]

The main reason of failure is jar-hell. When .NET or C++ tests are started,
if IGNITE_NATIVE_TEST_CLASSPATH is set to true, source directory is
iterated and files libs, target/classes etc.
are added to classpath. But neither readdir(), FindNextFileA()
or Directory.EnumerateDirectories() do guarantee any ordering.
But in spring-data-2.0 and spring-data-2.2 there are different version of
spring. So jar hell occurs and tests fails.

I created small fix (ignore module dir that contains spring-data), and
tests start working on aitc-lin11_02.

Ticket for this problem is [1]

[1] -- https://issues.apache.org/jira/browse/IGNITE-13187



чт, 25 июн. 2020 г. в 11:46, Ivan Daschinsky :

> Hi igniters!
>
> I found that .NET Core suite and C++ suites fails on two agents --
> aitc-lin01_04 and aitc-lin11_02.
>
> So I think the best solution is to exclude these agents as requirement for
> these suites and debug problems on these
> agents separately
>
> вт, 23 июн. 2020 г. в 09:33, Ivan Daschinsky :
>
>> Hi folks!
>>
>> It seems, that this test mysteriously fails only on one
>> agent aitc-lin01_04 with this error. [1]
>> Locally this .NET Core tests doesn't fail for about 20 minutes of
>> continuously run (Mac OS X 10.15, .NET Core 3.1.200)
>>
>> According to stacktrace, it seems that there is problem with build
>> configuration and invalid jars are in classpath.
>>
>> Petr Ivanov, may be it is good idea to clean up aitc-lin01_04?
>>
>> --
>> Sincerely yours, Ivan Daschinskiy
>>
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: MTCGA spam about BinaryConfigurationTest in .NET Core suite

[Ivan Daschinsky]

Actually, We have 3 (sic!) versions of spring data (1x, 2.0 and 2.2) and
all of them are correct.

But any versions of spring-data is not needed for platforms test.

чт, 25 июн. 2020 г. в 16:50, Pavel Tupitsyn :

> Ivan, thank you so much for getting to the bottom of this issue!
>
> Why do we have two versions of spring-data, and which one is correct?
>
> On Thu, Jun 25, 2020 at 4:34 PM Ivan Daschinsky 
> wrote:
>
> > The main reason of failure is jar-hell. When .NET or C++ tests are
> started,
> > if IGNITE_NATIVE_TEST_CLASSPATH is set to true, source directory is
> > iterated and files libs, target/classes etc.
> > are added to classpath. But neither readdir(), FindNextFileA()
> > or Directory.EnumerateDirectories() do guarantee any ordering.
> > But in spring-data-2.0 and spring-data-2.2 there are different version of
> > spring. So jar hell occurs and tests fails.
> >
> > I created small fix (ignore module dir that contains spring-data), and
> > tests start working on aitc-lin11_02.
> >
> > Ticket for this problem is [1]
> >
> > [1] -- https://issues.apache.org/jira/browse/IGNITE-13187
> >
> >
> >
> > чт, 25 июн. 2020 г. в 11:46, Ivan Daschinsky :
> >
> > > Hi igniters!
> > >
> > > I found that .NET Core suite and C++ suites fails on two agents --
> > > aitc-lin01_04 and aitc-lin11_02.
> > >
> > > So I think the best solution is to exclude these agents as requirement
> > for
> > > these suites and debug problems on these
> > > agents separately
> > >
> > > вт, 23 июн. 2020 г. в 09:33, Ivan Daschinsky :
> > >
> > >> Hi folks!
> > >>
> > >> It seems, that this test mysteriously fails only on one
> > >> agent aitc-lin01_04 with this error. [1]
> > >> Locally this .NET Core tests doesn't fail for about 20 minutes of
> > >> continuously run (Mac OS X 10.15, .NET Core 3.1.200)
> > >>
> > >> According to stacktrace, it seems that there is problem with build
> > >> configuration and invalid jars are in classpath.
> > >>
> > >> Petr Ivanov, may be it is good idea to clean up aitc-lin01_04?
> > >>
> > >> --
> > >> Sincerely yours, Ivan Daschinskiy
> > >>
> > >
> > >
> > > --
> > > Sincerely yours, Ivan Daschinskiy
> > >
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: MTCGA spam about BinaryConfigurationTest in .NET Core suite

[Ivan Daschinsky]

I think that just add an array of ignored modules for test platform
classpath is absolutely ok fix.

But I have a discussion with Igor Sapego and he thinks that this fix is a
little bit crutchy.

чт, 25 июн. 2020 г. в 17:06, Ivan Daschinsky :

> Actually, We have 3 (sic!) versions of spring data (1x, 2.0 and 2.2) and
> all of them are correct.
>
> But any versions of spring-data is not needed for platforms test.
>
> чт, 25 июн. 2020 г. в 16:50, Pavel Tupitsyn :
>
>> Ivan, thank you so much for getting to the bottom of this issue!
>>
>> Why do we have two versions of spring-data, and which one is correct?
>>
>> On Thu, Jun 25, 2020 at 4:34 PM Ivan Daschinsky 
>> wrote:
>>
>> > The main reason of failure is jar-hell. When .NET or C++ tests are
>> started,
>> > if IGNITE_NATIVE_TEST_CLASSPATH is set to true, source directory is
>> > iterated and files libs, target/classes etc.
>> > are added to classpath. But neither readdir(), FindNextFileA()
>> > or Directory.EnumerateDirectories() do guarantee any ordering.
>> > But in spring-data-2.0 and spring-data-2.2 there are different version
>> of
>> > spring. So jar hell occurs and tests fails.
>> >
>> > I created small fix (ignore module dir that contains spring-data), and
>> > tests start working on aitc-lin11_02.
>> >
>> > Ticket for this problem is [1]
>> >
>> > [1] -- https://issues.apache.org/jira/browse/IGNITE-13187
>> >
>> >
>> >
>> > чт, 25 июн. 2020 г. в 11:46, Ivan Daschinsky :
>> >
>> > > Hi igniters!
>> > >
>> > > I found that .NET Core suite and C++ suites fails on two agents --
>> > > aitc-lin01_04 and aitc-lin11_02.
>> > >
>> > > So I think the best solution is to exclude these agents as requirement
>> > for
>> > > these suites and debug problems on these
>> > > agents separately
>> > >
>> > > вт, 23 июн. 2020 г. в 09:33, Ivan Daschinsky :
>> > >
>> > >> Hi folks!
>> > >>
>> > >> It seems, that this test mysteriously fails only on one
>> > >> agent aitc-lin01_04 with this error. [1]
>> > >> Locally this .NET Core tests doesn't fail for about 20 minutes of
>> > >> continuously run (Mac OS X 10.15, .NET Core 3.1.200)
>> > >>
>> > >> According to stacktrace, it seems that there is problem with build
>> > >> configuration and invalid jars are in classpath.
>> > >>
>> > >> Petr Ivanov, may be it is good idea to clean up aitc-lin01_04?
>> > >>
>> > >> --
>> > >> Sincerely yours, Ivan Daschinskiy
>> > >>
>> > >
>> > >
>> > > --
>> > > Sincerely yours, Ivan Daschinskiy
>> > >
>> >
>> >
>> > --
>> > Sincerely yours, Ivan Daschinskiy
>> >
>>
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>


-- 
Sincerely yours, Ivan Daschinskiy

Discussion: Existence of manual pinger for ZK client

[Ivan Daschinsky]

Igniters, I suppose, that existence of manual pinger, introduced in [1], in
our ZkDiscoveryImpl is under a big question. This weird thing was
introduced to solve specific issue, specifically strange freeze of all
threads of process during deallocate (Unsafe.freeMemory) of large amount of
data.
Actually, fix [2] introduced workaround for this problem.  Existence of
this "pinger" is not only weird, but actually a little bit ridiculous, even
more this is absolutely nonsense. See [3] for example.

We should remove this strange thing.

[1] - https://issues.apache.org/jira/browse/IGNITE-9683
[2] - https://issues.apache.org/jira/browse/IGNITE-9658
[3] - https://www.usenix.org/legacy/event/atc10/tech/full_papers/Hunt.pdf


-- 
Sincerely yours, Ivan Daschinskiy

IGNITE-12808 Allow create tables for existing caches

[Ivan Daschinsky]

Hi Igniters.

I've recently submitted patch[1] that implemented subj.
Any help with review would be appreciated.


[1] - https://issues.apache.org/jira/browse/IGNITE-12808
-- 
Sincerely yours, Ivan Daschinskiy

Re: PDS suites fail with exit code 137

[Ivan Daschinsky]

Ivan, I think that we should use mmap/munmap to allocate huge chunks of
memory.

I've experimented with JNA and invoke mmap/munmap with it and it works fine.
May be we can create module (similar to direct-io) that use mmap/munap on
platforms, that support them
and fallback to Unsafe if not?

чт, 23 июл. 2020 г. в 13:31, Ivan Bessonov :

> Hello Igniters,
>
> I'd like to discuss the current issue with "out of memory" fails on
> TeamCity. Particularly suites [1]
> and [2], they have quite a lot of "Exit code 137" failures.
>
> I investigated the "PDS (Indexing)" suite under [3]. There's another
> similar issue as well: [4].
> I came to the conclusion that the main problem is inside the default memory
> allocator (malloc).
> Let me explain the way I see it right now:
>
> "malloc" is allowed to allocate (for internal usages) up to 8 * (number of
> cores) blocks called
> ARENA, 64 mb each. This may happen when a program creates/stops threads
> frequently and
> allocates a lot of memory all the time, which is exactly what our tests do.
> Given that TC agents
> have 32 cores, 8 * 32 * 64 mb gives 16 gigabytes, that's like the whole
> amount of RAM on the
> single agent.
>
> The total amount of arenas can be manually lowered by setting
> the MALLOC_ARENA_MAX
> environment variable to 4 (or other small value). I tried it locally and in
> PDS (Indexing) suite
> settings on TC, results look very promising: [5]
>
> It is said that changing this variable may lead to some performance
> degradation, but it's hard to tell whether we have it or not, because the
> suite usually failed before it was completed.
>
> So, I have two questions right now:
>
> - can those of you, who are into hardcore Linux and C, confirm that the
> solution can help us? Experiments show that it completely solves the
> problem.
> - can you please point me to a person who usually does TC maintenance? I'm
> not entirely sure
> that I can propagate this environment variable to all suites by myself,
> which is necessary to
> avoid occasional error 137 (resulted from the same problem) in future. I
> just don't know all the
> details about suites structure.
>
> Thank you!
>
> [1]
>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PdsIndexing=buildTypeHistoryList=failed_IgniteTests24Java8=%3Cdefault%3E
> [2]
>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_Pds4=buildTypeHistoryList_IgniteTests24Java8=%3Cdefault%3E=failed
> [3] https://issues.apache.org/jira/browse/IGNITE-13266
> [4] https://issues.apache.org/jira/browse/IGNITE-13263
> [5]
>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PdsIndexing=buildTypeHistoryList_IgniteTests24Java8=pull%2F8051%2Fhead
>
> --
> Sincerely yours,
> Ivan Bessonov
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: PDS suites fail with exit code 137

[Ivan Daschinsky]

AFAIK, glibc allocator uses arenas for minimize contention between threads
when they trying to access
or free preallocated bit of memory. But seems that we
use -XX:+AlwaysPreTouch, so heap is allocated
and committed at start time. We allocate memory for durable memory in one
thread.
So I think there will be not so much contention between threads for native
memory pools.

Also, there is another approach -- try to use jemalloc.
This allocator shows better result than default glibc malloc in our
scenarios. (memory consumption) [1]

[1] --
http://ithare.com/testing-memory-allocators-ptmalloc2-tcmalloc-hoard-jemalloc-while-trying-to-simulate-real-world-loads/



чт, 23 июл. 2020 г. в 14:19, Ivan Bessonov :

> Hello Ivan,
>
> It feels like the problem is more about new starting threads rather than
> the
> allocation of offheap regions. Plus I'd like to see results soon, your
> proposal is
> a major change for Ignite that can't be implemented fast enough.
>
> Anyway, I think this makes sense, considering that one day Unsafe will be
> removed. But I wouldn't think about it right now, maybe as a separate
> proposal...
>
>
>
> чт, 23 июл. 2020 г. в 13:40, Ivan Daschinsky :
>
> > Ivan, I think that we should use mmap/munmap to allocate huge chunks of
> > memory.
> >
> > I've experimented with JNA and invoke mmap/munmap with it and it works
> > fine.
> > May be we can create module (similar to direct-io) that use mmap/munap on
> > platforms, that support them
> > and fallback to Unsafe if not?
> >
> > чт, 23 июл. 2020 г. в 13:31, Ivan Bessonov :
> >
> > > Hello Igniters,
> > >
> > > I'd like to discuss the current issue with "out of memory" fails on
> > > TeamCity. Particularly suites [1]
> > > and [2], they have quite a lot of "Exit code 137" failures.
> > >
> > > I investigated the "PDS (Indexing)" suite under [3]. There's another
> > > similar issue as well: [4].
> > > I came to the conclusion that the main problem is inside the default
> > memory
> > > allocator (malloc).
> > > Let me explain the way I see it right now:
> > >
> > > "malloc" is allowed to allocate (for internal usages) up to 8 * (number
> > of
> > > cores) blocks called
> > > ARENA, 64 mb each. This may happen when a program creates/stops threads
> > > frequently and
> > > allocates a lot of memory all the time, which is exactly what our tests
> > do.
> > > Given that TC agents
> > > have 32 cores, 8 * 32 * 64 mb gives 16 gigabytes, that's like the whole
> > > amount of RAM on the
> > > single agent.
> > >
> > > The total amount of arenas can be manually lowered by setting
> > > the MALLOC_ARENA_MAX
> > > environment variable to 4 (or other small value). I tried it locally
> and
> > in
> > > PDS (Indexing) suite
> > > settings on TC, results look very promising: [5]
> > >
> > > It is said that changing this variable may lead to some performance
> > > degradation, but it's hard to tell whether we have it or not, because
> the
> > > suite usually failed before it was completed.
> > >
> > > So, I have two questions right now:
> > >
> > > - can those of you, who are into hardcore Linux and C, confirm that the
> > > solution can help us? Experiments show that it completely solves the
> > > problem.
> > > - can you please point me to a person who usually does TC maintenance?
> > I'm
> > > not entirely sure
> > > that I can propagate this environment variable to all suites by myself,
> > > which is necessary to
> > > avoid occasional error 137 (resulted from the same problem) in future.
> I
> > > just don't know all the
> > > details about suites structure.
> > >
> > > Thank you!
> > >
> > > [1]
> > >
> > >
> >
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PdsIndexing=buildTypeHistoryList=failed_IgniteTests24Java8=%3Cdefault%3E
> > > [2]
> > >
> > >
> >
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_Pds4=buildTypeHistoryList_IgniteTests24Java8=%3Cdefault%3E=failed
> > > [3] https://issues.apache.org/jira/browse/IGNITE-13266
> > > [4] https://issues.apache.org/jira/browse/IGNITE-13263
> > > [5]
> > >
> > >
> >
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PdsIndexing=buildTypeHistoryList_IgniteTests24Java8=pull%2F8051%2Fhead
> > >
> > > --
> > > Sincerely yours,
> > > Ivan Bessonov
> > >
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
> >
>
>
> --
> Sincerely yours,
> Ivan Bessonov
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: PDS suites fail with exit code 137

[Ivan Daschinsky]

>
> About "jemalloc" - it's also an option, but it also requires reconfiguring
> suites on
> TC, maybe in a more complicated way. It requires additional installation,
> right?
> Can we stick to the solution that I already tested or should we update TC
> agents? :)


Yes, if you want to use jemalloc, you should install it and configure a
specific env variable.
This is just an option to consider, nothing more. I suppose that your
approach is may be the
best variant right now.


чт, 23 июл. 2020 г. в 15:28, Ivan Bessonov :

> >
> > glibc allocator uses arenas for minimize contention between threads
>
>
> I understand it the same way. I did testing with running of Indexing suite
> locally
> and periodically executing "pmap ", it showed that the number of 64mb
> arenas grows constantly and never shrinks. By the middle of the suite the
> amount
> of virtual memory was close to 50 Gb and used physical memory was at least
> 6-7 Gb, if I recall it correctly. I have only 8 cores BTW, so it should be
> worse on TC.
> It means that there is enough contention somewhere in tests.
>
> About "jemalloc" - it's also an option, but it also requires reconfiguring
> suites on
> TC, maybe in a more complicated way. It requires additional installation,
> right?
> Can we stick to the solution that I already tested or should we update TC
> agents? :)
>
> чт, 23 июл. 2020 г. в 15:02, Ivan Daschinsky :
>
> > AFAIK, glibc allocator uses arenas for minimize contention between
> threads
> > when they trying to access
> > or free preallocated bit of memory. But seems that we
> > use -XX:+AlwaysPreTouch, so heap is allocated
> > and committed at start time. We allocate memory for durable memory in one
> > thread.
> > So I think there will be not so much contention between threads for
> native
> > memory pools.
> >
> > Also, there is another approach -- try to use jemalloc.
> > This allocator shows better result than default glibc malloc in our
> > scenarios. (memory consumption) [1]
> >
> > [1] --
> >
> >
> http://ithare.com/testing-memory-allocators-ptmalloc2-tcmalloc-hoard-jemalloc-while-trying-to-simulate-real-world-loads/
> >
> >
> >
> > чт, 23 июл. 2020 г. в 14:19, Ivan Bessonov :
> >
> > > Hello Ivan,
> > >
> > > It feels like the problem is more about new starting threads rather
> than
> > > the
> > > allocation of offheap regions. Plus I'd like to see results soon, your
> > > proposal is
> > > a major change for Ignite that can't be implemented fast enough.
> > >
> > > Anyway, I think this makes sense, considering that one day Unsafe will
> be
> > > removed. But I wouldn't think about it right now, maybe as a separate
> > > proposal...
> > >
> > >
> > >
> > > чт, 23 июл. 2020 г. в 13:40, Ivan Daschinsky :
> > >
> > > > Ivan, I think that we should use mmap/munmap to allocate huge chunks
> of
> > > > memory.
> > > >
> > > > I've experimented with JNA and invoke mmap/munmap with it and it
> works
> > > > fine.
> > > > May be we can create module (similar to direct-io) that use
> mmap/munap
> > on
> > > > platforms, that support them
> > > > and fallback to Unsafe if not?
> > > >
> > > > чт, 23 июл. 2020 г. в 13:31, Ivan Bessonov :
> > > >
> > > > > Hello Igniters,
> > > > >
> > > > > I'd like to discuss the current issue with "out of memory" fails on
> > > > > TeamCity. Particularly suites [1]
> > > > > and [2], they have quite a lot of "Exit code 137" failures.
> > > > >
> > > > > I investigated the "PDS (Indexing)" suite under [3]. There's
> another
> > > > > similar issue as well: [4].
> > > > > I came to the conclusion that the main problem is inside the
> default
> > > > memory
> > > > > allocator (malloc).
> > > > > Let me explain the way I see it right now:
> > > > >
> > > > > "malloc" is allowed to allocate (for internal usages) up to 8 *
> > (number
> > > > of
> > > > > cores) blocks called
> > > > > ARENA, 64 mb each. This may happen when a program creates/stops
> > threads
> > > > > frequently and
> > > > > allocates a lot of memory all the time, which is exactly what our
> > tests
> > > > do.
> > > > > G

Re: New Committer: Sergey Chugunov

[Ivan Daschinsky]

Congrats! Well deserved!

пт, 17 июл. 2020 г. в 20:46, Sergey Antonov :

> Congratulations, Sergey!
>
> сб, 18 июл. 2020 г., 0:24 Andrey Mashenkov :
>
> > Congratulations, Sergey.
> >
> > пт, 17 июл. 2020 г., 15:06 Вячеслав Коптилин :
> >
> > > Hi,
> > >
> > > Well deserved! Keep it up, Sergey!
> > >
> > > Thanks,
> > > S.
> > >
> > > пт, 17 июл. 2020 г. в 14:36, Ivan Pavlukhin :
> > >
> > > > Sergey, congratulations!
> > > >
> > > > 2020-07-17 14:25 GMT+03:00, Petr Ivanov :
> > > > > Congratulations!
> > > > >
> > > > >
> > > > >> On 17 Jul 2020, at 14:14, Roman Kondakov
>  > >
> > > > >> wrote:
> > > > >>
> > > > >> Sergey, congratulations!
> > > > >>
> > > > >> --
> > > > >> Roman Kondakov
> > > > >>
> > > > >> On 17.07.2020 12:32, Dmitriy Pavlov wrote:
> > > > >>> Dear Ignite Community,
> > > > >>> The Project Management Committee (PMC) for Apache Ignite has
> > invited
> > > > >>> Sergey
> > > > >>> Chugunov to become a committer and we are pleased to announce
> that
> > he
> > > > >>> has
> > > > >>> accepted.
> > > > >>> Sergey has a long story of Ignite contributions. Besides the code
> > > > >>> contributions, Sergey has contributed the TCP Discovery Under The
> > > Hood
> > > > >>> wiki
> > > > >>> page, posted a blog post about running Apache Ignite on AWS, and
> > also
> > > > >>> mentors newcomer contributors.
> > > > >>> Being a committer enables easier contribution to the project
> since
> > > > there
> > > > >>> is
> > > > >>> no need to go via the patch submission process. This should
> enable
> > > > >>> better
> > > > >>> productivity.
> > > > >>> Sergey, congratulations on a new role in the Apache Ignite
> > Community.
> > > > >>> Best Regards,
> > > > >>> Dmitriy Pavlov
> > > > >>> on behalf of Apache Ignite PMC
> > > > >>> P.S. this announce is a little bit overdue, but I guess Sergey's
> > new
> > > > >>> role
> > > > >>> was not announced to the community.
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best regards,
> > > > Ivan Pavlukhin
> > > >
> > >
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Move TcpDiscoveryZookeeperIpFinder to ignite-extensions as part of IEP-36

[Ivan Daschinsky]

Hello, Igniters.

Currently, in module ignite-zookeeper, that contains full implementation of
ZookeeperDiscoverySpi, also presents one class that looks like a little
snippet and I have some concerns about presence of this class in the module.

1) This class is a simple snippet-like implementation of TcpDiscoverIpFinder
2) It creates EPHEMERAL_SEQUENTIAL ZNode in root directory with json,
containing IP addresses of joining node.
3) It reads registered children znodes from root directory  and use these
addresses to participate in common TcpDiscovery process.

1) This class has nothing in common with ZookeeperDiscovery, but
2) It brings to module additional dependencies (apache-curator framework,
jackson and so on)
3) All of these dependencies are not required to ZookeeperDiscoverySpi.
4) The usefulness of it is doubtful, because we have already fully
functional ZookeeperDiscovery and use Zookeeper quorum as just simple store
for IP addresses without utilizing full zookeeper power is questionable
decision.

So I suggest to move this single class with tests to separate module in
ignite-extensions.

What do you think?

-- 
Sincerely yours, Ivan Daschinskiy

Re: 2.9.1 release scope and dates

[Ivan Daschinsky]

hin Vladimir 
> > wrote:
> > >>
> > >> Yaroslav, Hi.
> > >>
> > >>
> > >> I suggest to merge minor fix of javadoc: [1]. It should have appeared
> in
> > >> 2.9. Commits in master:
> > >>
> > >> d3e5b7c11ed037670700eea75851e619d5d1b6b1
> > >>
> > >> and
> > >>
> > >> 1654e9fac61842424c08d26a08ef67569f74746a
> > >>
> > >>
> > >> [1] https://github.com/apache/ignite/pull/8448
> > >>
> > >>
> > >>
> > >> 19.11.2020 17:15, Ivan Daschinsky пишет:
> > >>> Hi!
> > >>> Yaroslav, Max -- I have another ticket that will be nice to have in
> > 2.9.1
> > >>> https://issues.apache.org/jira/browse/IGNITE-13699
> > >>>
> > >>> пт, 13 нояб. 2020 г. в 15:08, Yaroslav Molochkov <
> > molochko...@gmail.com>:
> > >>>
> > >>>> Igniters, hello!
> > >>>>
> > >>>> I think the scope of 2.9.1 is finalized.
> > >>>>
> > >>>>> On 9 Nov 2020, at 12:04, Yaroslav Molochkov  >
> > >>>> wrote:
> > >>>>> Ivan, thanks!
> > >>>>>
> > >>>>> Added it to the list.
> > >>>>>
> > >>>>>> On 8 Nov 2020, at 14:13, Ivan Daschinsky 
> > wrote:
> > >>>>>>
> > >>>>>> Yaroslav, there is another bug for 2.9.1 release
> > >>>>>> https://issues.apache.org/jira/browse/IGNITE-13572
> > >>>>>>
> > >>>>>> чт, 5 нояб. 2020 г., 19:23 Yaroslav Molochkov <
> > molochko...@gmail.com>:
> > >>>>>>
> > >>>>>>> Ivan, hi!
> > >>>>>>> Sure.
> > >>>>>>>
> > >>>>>>> UPD: i am the release manager and will be doing this with Maxim's
> > help
> > >>>>>>> (since i don't have some user permissions)
> > >>>>>>>
> > >>>>>>>> On Thu, Nov 5, 2020 at 6:24 PM Ivan Daschinsky <
> > ivanda...@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Hi. I'd suggest to add this issue. This is a usability
> improvement
> > >>>> for zk
> > >>>>>>>> discovery, and also this patch incorporates fixes for JMX
> metrics
> > >>>>>>>> concurrency issues
> > >>>>>>>>
> > >>>>>>>> [1] -- https://issues.apache.org/jira/browse/IGNITE-13577
> > >>>>>>>>
> > >>>>>>>> чт, 5 нояб. 2020 г., 16:20 Yaroslav Molochkov <
> > molochko...@gmail.com
> > >>>>> :
> > >>>>>>>>> Igniters!
> > >>>>>>>>>
> > >>>>>>>>> I'd like to help with the 2.9.1 release. The scope of this
> > release
> > >>>>>>>> includes
> > >>>>>>>>> following issues:
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>
> >
> https://issues.apache.org/jira/browse/IGNITE-13676?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.9.1
> > >>>>>>>>> Maxim Muzafarov agreed to help me with the process and he will
> > be the
> > >>>>>>>>> release manager.
> > >>>>>>>>>
> > >>>>>>>>> Scope freeze: Nov. 12th
> > >>>>>>>>> Code freeze: Nov. 19th
> > >>>>>>>>> Voting date: Nov. 26th
> > >>>>>>>>> Release date: Nov. 31st
> > >>>>>>>>>
> > >>>>>>>>> Tickets that were added (or to be added) to the scope don't
> > bring new
> > >>>>>>>>> features but various bug fixes.
> > >>>>>>>>>
> > >>>
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: IEP-61 Technical discussion

[Ivan Daschinsky]

Alexey, I kindly ask you to move the meeting a little bit earlier, ideal
variant -- in the morning.

ср, 25 нояб. 2020 г. в 20:10, Alexey Goncharuk :

> Folks, let's have the call on Friday, Nov 27th at 18:00 MSK? We can use the
> following waiting room link:
>  https://zoom.us/j/99450012496?pwd=RWZmOGhCNWlRK0ZpamdOOTZsYTJ0dz09
>
> Let me know if this time works for everybody.
>
> ср, 25 нояб. 2020 г. в 16:42, Alexey Goncharuk  >:
>
> > Folks,
> >
> > I've made some edits in IEP-61 [1] regarding the group membership service
> > and transaction protocol interaction with the replication infrastructure,
> > please take a look before our Friday call.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-61%3A+Common+Replication+Infrastructure
> >
> > пн, 23 нояб. 2020 г. в 13:28, Alexey Goncharuk <
> alexey.goncha...@gmail.com
> > >:
> >
> >> Thanks, Ivan,
> >>
> >> Another protocol for group membership worth checking out is RAPID [1] (a
> >> recent one). Not sure though if there are any available implementations
> for
> >> it already.
> >>
> >> [1]
> https://www.usenix.org/system/files/conference/atc18/atc18-suresh.pdf
> >>
> >> пн, 23 нояб. 2020 г. в 10:46, Ivan Daschinsky :
> >>
> >>> Also, here is some interesting reading about gossip, SWIM etc.
> >>>
> >>> 1 --
> >>> http://www.cs.cornell.edu/Info/Projects/Spinglass/public_pdfs/SWIM.pdf
> >>> 2 --
> >>>
> >>>
> http://www.antonkharenko.com/2015/09/swim-distributed-group-membership.html
> >>> 3 -- https://github.com/hashicorp/memberlist (Foundation library of
> >>> hashicorp serf)
> >>> 4 -- https://github.com/scalecube/scalecube-cluster -- (Java
> >>> implementation
> >>> of SWIM)
> >>>
> >>> чт, 19 нояб. 2020 г. в 16:35, Ivan Daschinsky :
> >>>
> >>> > >> Friday, Nov 27th work for you? If ok, let's have an open call
> then.
> >>> > Yes, great
> >>> > >> As for the protocol port - we will not be dealing with the
> >>> > concurrency...
> >>> > >>Judging by the Rust port, it seems fairly straightforward.
> >>> > Yes, they chose split transport and logic. But original Go package
> from
> >>> > etcd (see raft/node.go) contains some  heartbeats mechanism etc.
> >>> > I agree with you, this seems not to be a huge deal to port.
> >>> >
> >>> > чт, 19 нояб. 2020 г. в 16:13, Alexey Goncharuk <
> >>> alexey.goncha...@gmail.com
> >>> > >:
> >>> >
> >>> >> Ivan,
> >>> >>
> >>> >> Agree, let's have a call to discuss the IEP. I have some more
> thoughts
> >>> >> regarding how the replication infrastructure works with
> >>> >> atomic/transactional caches, will put this info to the IEP. Does
> next
> >>> >> Friday, Nov 27th work for you? If ok, let's have an open call then.
> >>> >>
> >>> >> As for the protocol port - we will not be dealing with the
> concurrency
> >>> >> model if we choose this way, this is what I like about their code
> >>> >> structure. Essentially, the raft module is a single-threaded
> automata
> >>> >> which
> >>> >> has a callback to process a message, process a tick (timeout) and
> >>> produces
> >>> >> messages that should be sent and log entries that should be
> persisted.
> >>> >> Judging by the Rust port, it seems fairly straightforward. Will be
> >>> happy
> >>> >> to
> >>> >> discuss this and other alternatives on the call as well.
> >>> >>
> >>> >> чт, 19 нояб. 2020 г. в 14:41, Ivan Daschinsky  >:
> >>> >>
> >>> >> > > Any existing library that can be used to avoid re-implementing
> the
> >>> >> > protocol ourselves? Perhaps, porting the existing implementation
> to
> >>> Java
> >>> >> > Personally, I like this idea. Go libraries (either raft module of
> >>> etcd
> >>> >> or
> >>> >> > serf by Hashicorp) are famous for clean code, good design,
> >>> stability,
> >>> >> not
> >>> >> > enormous size.
> >>> >> > But, on other side, Go has dif

Re: IEP-61 Technical discussion

[Ivan Daschinsky]

Alexey, is it possible to manage call at 16:00 MSK?

чт, 26 нояб. 2020 г. в 12:30, Alexey Goncharuk :

> Hi Ivan,
>
> Unfortunately, the earliest window available for us is 12:00 MSK (1 hour
> slot), or after 14:30 MSK. Let me know what time works best for you.
>
> ср, 25 нояб. 2020 г. в 21:38, Ivan Daschinsky :
>
> > Alexey, I kindly ask you to move the meeting a little bit earlier, ideal
> > variant -- in the morning.
> >
> > ср, 25 нояб. 2020 г. в 20:10, Alexey Goncharuk <
> alexey.goncha...@gmail.com
> > >:
> >
> > > Folks, let's have the call on Friday, Nov 27th at 18:00 MSK? We can use
> > the
> > > following waiting room link:
> > >  https://zoom.us/j/99450012496?pwd=RWZmOGhCNWlRK0ZpamdOOTZsYTJ0dz09
> > >
> > > Let me know if this time works for everybody.
> > >
> > > ср, 25 нояб. 2020 г. в 16:42, Alexey Goncharuk <
> > alexey.goncha...@gmail.com
> > > >:
> > >
> > > > Folks,
> > > >
> > > > I've made some edits in IEP-61 [1] regarding the group membership
> > service
> > > > and transaction protocol interaction with the replication
> > infrastructure,
> > > > please take a look before our Friday call.
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-61%3A+Common+Replication+Infrastructure
> > > >
> > > > пн, 23 нояб. 2020 г. в 13:28, Alexey Goncharuk <
> > > alexey.goncha...@gmail.com
> > > > >:
> > > >
> > > >> Thanks, Ivan,
> > > >>
> > > >> Another protocol for group membership worth checking out is RAPID
> [1]
> > (a
> > > >> recent one). Not sure though if there are any available
> > implementations
> > > for
> > > >> it already.
> > > >>
> > > >> [1]
> > > https://www.usenix.org/system/files/conference/atc18/atc18-suresh.pdf
> > > >>
> > > >> пн, 23 нояб. 2020 г. в 10:46, Ivan Daschinsky  >:
> > > >>
> > > >>> Also, here is some interesting reading about gossip, SWIM etc.
> > > >>>
> > > >>> 1 --
> > > >>>
> > http://www.cs.cornell.edu/Info/Projects/Spinglass/public_pdfs/SWIM.pdf
> > > >>> 2 --
> > > >>>
> > > >>>
> > >
> >
> http://www.antonkharenko.com/2015/09/swim-distributed-group-membership.html
> > > >>> 3 -- https://github.com/hashicorp/memberlist (Foundation library
> of
> > > >>> hashicorp serf)
> > > >>> 4 -- https://github.com/scalecube/scalecube-cluster -- (Java
> > > >>> implementation
> > > >>> of SWIM)
> > > >>>
> > > >>> чт, 19 нояб. 2020 г. в 16:35, Ivan Daschinsky  >:
> > > >>>
> > > >>> > >> Friday, Nov 27th work for you? If ok, let's have an open call
> > > then.
> > > >>> > Yes, great
> > > >>> > >> As for the protocol port - we will not be dealing with the
> > > >>> > concurrency...
> > > >>> > >>Judging by the Rust port, it seems fairly straightforward.
> > > >>> > Yes, they chose split transport and logic. But original Go
> package
> > > from
> > > >>> > etcd (see raft/node.go) contains some  heartbeats mechanism etc.
> > > >>> > I agree with you, this seems not to be a huge deal to port.
> > > >>> >
> > > >>> > чт, 19 нояб. 2020 г. в 16:13, Alexey Goncharuk <
> > > >>> alexey.goncha...@gmail.com
> > > >>> > >:
> > > >>> >
> > > >>> >> Ivan,
> > > >>> >>
> > > >>> >> Agree, let's have a call to discuss the IEP. I have some more
> > > thoughts
> > > >>> >> regarding how the replication infrastructure works with
> > > >>> >> atomic/transactional caches, will put this info to the IEP. Does
> > > next
> > > >>> >> Friday, Nov 27th work for you? If ok, let's have an open call
> > then.
> > > >>> >>
> > > >>> >> As for the protocol port - we will not be dealing with the
> > > concurrency
> > > >>> >> model i

Re: [DISCUSS] Use GridNioServer in Java thin client

[Ivan Daschinsky]

Pavel, good job and great benchmark results!

чт, 26 нояб. 2020 г. в 15:01, Pavel Tupitsyn :

> PR is ready for review [1]
>
> I've added a simple put/get benchmark, there is some performance
> improvement over existing implementation, see results in the PR
> description.
>
> [1] https://github.com/apache/ignite/pull/8483
>
> On Fri, Nov 20, 2020 at 10:39 AM Pavel Tupitsyn 
> wrote:
>
> > Since there are no objections, I've updated the IEP accordingly [1]
> > and started working on it [2]
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-60%3A+Java+Thin+Client+Non-Blocking+Async+IO
> > [2] https://github.com/apache/ignite/pull/8483
> >
> > On Mon, Nov 9, 2020 at 4:07 PM Ivan Daschinsky 
> > wrote:
> >
> >> I suppose that the best variant -- ability to switch to netty if this
> lib
> >> is in classpath
> >>
> >> пн, 9 нояб. 2020 г. в 15:58, Igor Sapego :
> >>
> >> > Sounds like a good idea to me.
> >> >
> >> > Best Regards,
> >> > Igor
> >> >
> >> >
> >> > On Mon, Nov 9, 2020 at 3:32 PM Alex Plehanov  >
> >> > wrote:
> >> >
> >> > > +1 for using GridNioServer as java thin client communication layer.
> >> > >
> >> > > вс, 8 нояб. 2020 г. в 19:12, Pavel Tupitsyn :
> >> > >
> >> > > > Igniters,
> >> > > >
> >> > > > This is a continuation of "Use Netty for Java thin client" [1],
> >> > > > I'm starting a new thread for better visibility.
> >> > > >
> >> > > > The problems with current Java thin client are:
> >> > > > * Socket writes block user threads
> >> > > > * Every connection uses a separate listener thread (with partition
> >> > > > awareness there is a thread for every server node within a single
> >> > > > IgniteClient)
> >> > > >
> >> > > > GridNioServer can work in client mode and solves both of these
> >> > problems.
> >> > > > It is the most practical choice as well at the moment - no extra
> >> > > > dependencies required.
> >> > > >
> >> > > > A potential drawback is increased coupling between thin client and
> >> core
> >> > > > code,
> >> > > > which I'm going to mitigate by abstracting GridNioServer behind a
> >> > simpler
> >> > > > facade,
> >> > > > so we can replace it with Netty or something else easier if we
> >> decide
> >> > to
> >> > > > split the code.
> >> > > >
> >> > > > Thoughts, objections?
> >> > > >
> >> > > > [1]
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Use-Netty-for-Java-thin-client-td49732.html
> >> > > >
> >> > >
> >> >
> >>
> >>
> >> --
> >> Sincerely yours, Ivan Daschinskiy
> >>
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: Replace Future.get with Future.get(int timeout) in tests

[Ivan Daschinsky]

Not only futures, there are a lot of latches, barriers etc. with same
problem.

вт, 8 дек. 2020 г. в 18:41, ткаленко кирилл :

> It seems to be a good topic, but it seems to be left to the reviewer's
> discretion.
>
> 08.12.2020, 18:36, "Nikolay Izhikov" :
> > Hello, Igniters.
> >
> > Currently, we have a lot of usages `Future.get` without a timeout in
> tests.
> > In case the test that uses `Future.get` is flaky it can lead to the
> whole suite hang.
> >
> > Is there any reason to use the get method without a timeout?
> > Can we
> >
> > a. Replace all invocation of get with the timeoutable companion.
> > b. Introduce code style check that will prohibit to use non-timeout
> method in tests.
> >
> > What do you think?
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: Migrating NodeJS client to TypeScript

[Ivan Daschinsky]

> Is there any value in keeping both versions - the plain JavaScript one
and the TypeScript specific
Hi! No, there is no value. TS sources should be transpiled to JS before
execution.

вт, 1 дек. 2020 г. в 01:31, Denis Magda :

> Hi Semyon,
>
> Is there any value in keeping both versions - the plain JavaScript one and
> the TypeScript specific?
>
> -
> Denis
>
>
> On Mon, Nov 30, 2020 at 12:16 PM Данилов Семён  wrote:
>
> > Hello Igniters!
> >
> > I'd like to propose a big change for the nodejs thin client: a full
> > rewrite from JavaScript to TypeScript.
> >
> > Strong typing will not only help us in future while developing new
> > features, but will also indicate existing type inconsistencies (I know
> > there is a couple).
> > Also, we will have out of the box types for API documentation, which is
> > very handy for users.
> >
> > So what do you think?
> >
> > Kind regards,
> > Semyon.
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: Re[2]: 2.9.1 release scope and dates

[Ivan Daschinsky]

Yaroslav, lets add another bug fix
https://issues.apache.org/jira/browse/IGNITE-13572
It's quite small fix, but bug is quite severe.

ср, 2 дек. 2020 г. в 10:48, Yaroslav Molochkov :

> Guys,
>
> can anyone grant me necessary permissions to create a wiki page with
> release info, please?
>
> On Tue, Dec 1, 2020 at 12:58 PM Yaroslav Molochkov 
> wrote:
>
> > Hi!
> > I see it's merged into 2.9.1 branch and master.
> >
> > I guess that's it for the issues, I will run nightly tests again just to
> > be sure that everything is okay and we will proceed.
> >
> > On Mon, Nov 30, 2020 at 7:31 PM Zhenya Stanilovsky
> >  wrote:
> >
> >>
> >> hello !
> >> seems it [1] need to be included too, if it`s not too late, of course.
> >> May be you can bump reviewer somehow?)
> >>
> >> [1]  https://issues.apache.org/jira/browse/IGNITE-13765
> >>
> >> >Ivan, it's added, thanks for pointing that out
> >> >
> >> >On Mon, Nov 30, 2020 at 5:44 PM Ivan Daschinsky < ivanda...@gmail.com
> >
> >> wrote:
> >> >
> >> >> Yaroslav, could you explain why you decided to remove from scope
> >> >>  https://issues.apache.org/jira/browse/IGNITE-13699 ?
> >> >> This ticket also incorporate few fixes for metrics. Currently metrics
> >> is a
> >> >> little bit broken (JoinedCount shows invalid results for example)
> >> >>
> >> >>
> >> >> пн, 30 нояб. 2020 г. в 17:26, Yaroslav Molochkov <
> >> molochko...@gmail.com >:
> >> >>
> >> >> > Igniters, hello!
> >> >> >
> >> >> > First of all, sorry that the release process is taking so long.
> >> >> >
> >> >> > Secondly, the release build seems stable and no blockers were
> >> introduced
> >> >> > within that list
> >> >> > <
> >> >> >
> >> >>
> >>
> https://issues.apache.org/jira/browse/IGNITE-11312?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.9.1%20and%20status%20%3D%20Resolved
> >> >> > >
> >> >> > (at
> >> >> > least on RunAll compared to 2.9)
> >> >> >
> >> >> > I've also prepared release notes:
> >> >> >
> >> >> > Ignite Core:
> >> >> > * Added support to graceful shutdown for ZookeeperDiscoverySpi
> >> >> > * Added System view for binary metadata
> >> >> > * Added System view for metastorage items
> >> >> > * Added RebalancingPartitionsTotal metrics
> >> >> > * Improved checkpoint concurrent behaviour
> >> >> > * Fixed critical system error when unregister a JMX bean
> >> >> > * Fixed IgniteCache#isClosed() returns false on server node even if
> >> >> > the cache had been closed before
> >> >> > * Fixed inability to eagerly remove expired cache entries
> >> >> > * Fixed partial index rebuild issue in case indexed cache contains
> >> >> > different datatypes
> >> >> > * Fixed local metastorage system view error if unmarshallable
> values
> >> >> > present
> >> >> > * Fixed deadlock between grid-timeout-worker and a thread opening a
> >> >> > communication connection
> >> >> > * Fixed deadlock in IgniteServiceProcessor
> >> >> > * Fixed issue when rebalance future might hang in no final state
> >> >> > though all partitions are owned
> >> >> > * Fixed issue when scan query fails with an assertion error:
> >> Unexpected
> >> >> > row key
> >> >> > * Fixed issue with archiving and enabled wal compaction setting on
> >> >> > server restart
> >> >> > * Fixed NPE during Cassandra Store initialization with PRIMITIVE
> >> strategy
> >> >> > * Fixed synchronization problems when different classloaders are
> used
> >> >> > for deployment of same class
> >> >> > * Fixed exception on SQL caches when client reconnect
> >> >> > * Fixed deadlock on multiple cache delete
> >> >> > * Fixed NPE in IgniteServiceProcessor when destroying a cache
> >> >> > * Fixed issue when DurableBackgroundTask can abandon incomplete
> task
> >> >> > * Fixed issue related to cache interceptor deserializatio

Re: 2.9.1 release scope and dates

[Ivan Daschinsky]

Yaroslav, there is another bug for 2.9.1 release
https://issues.apache.org/jira/browse/IGNITE-13572

чт, 5 нояб. 2020 г., 19:23 Yaroslav Molochkov :

> Ivan, hi!
> Sure.
>
> UPD: i am the release manager and will be doing this with Maxim's help
> (since i don't have some user permissions)
>
> On Thu, Nov 5, 2020 at 6:24 PM Ivan Daschinsky 
> wrote:
>
> > Hi. I'd suggest to add this issue. This is a usability improvement for zk
> > discovery, and also this patch incorporates fixes for JMX metrics
> > concurrency issues
> >
> > [1] -- https://issues.apache.org/jira/browse/IGNITE-13577
> >
> > чт, 5 нояб. 2020 г., 16:20 Yaroslav Molochkov :
> >
> > > Igniters!
> > >
> > > I'd like to help with the 2.9.1 release. The scope of this release
> > includes
> > > following issues:
> > >
> > >
> >
> https://issues.apache.org/jira/browse/IGNITE-13676?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.9.1
> > >
> > > Maxim Muzafarov agreed to help me with the process and he will be the
> > > release manager.
> > >
> > > Scope freeze: Nov. 12th
> > > Code freeze: Nov. 19th
> > > Voting date: Nov. 26th
> > > Release date: Nov. 31st
> > >
> > > Tickets that were added (or to be added) to the scope don't bring new
> > > features but various bug fixes.
> > >
> >
>

Re: Incorrect failure handler exceptions

[Ivan Daschinsky]

Ilya, thanks for your effort.

ср, 11 нояб. 2020 г. в 14:28, Ilya Kasnacheev :

> Hello!
>
> The fix is now merged.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 3 нояб. 2020 г. в 22:51, Ivan Daschinsky :
>
> > There is not a big problem to fix it.
> >
> > 1. Blocked thread is detected by another worker. Currently we take dump
> > before printing it to log with zero depth. This can be easily fixed.
> >
> > 2. We should propagate BlockedThreadExceptiob to failureCtx with original
> > stacktrace of blocked thread. StackFrames, obtained in previous step, can
> > be used to construct stacktrace of BlockedThreadException.
> >
> >
> > вт, 3 нояб. 2020 г., 19:15 Ilya Kasnacheev :
> >
> > > Hello!
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-13665
> > >
> > > The notorious problem with failure handler's thread dump, which nobody
> > has
> > > bothered to investigate, until now.
> > > "IgnitionEx$IgniteNamedInstance$2.apply"
> > >
> > > If anybody knows how to fix it, please suggest.
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: [jira] [Created] (IGNITE-13696) [ML] Tutorial examples fails

[Ivan Daschinsky]

I'm sorry, but this is so funny, can't stop laughing :)

ср, 11 нояб. 2020 г. в 23:19, Alexey Zinoviev :

> What errors did you found?
>
> ср, 11 нояб. 2020 г., 22:13 Stepan Pilschikov (Jira) :
>
> > Stepan Pilschikov created IGNITE-13696:
> > --
> >
> >  Summary: [ML] Tutorial examples fails
> >  Key: IGNITE-13696
> >  URL: https://issues.apache.org/jira/browse/IGNITE-13696
> >  Project: Ignite
> >   Issue Type: Bug
> >   Components: examples, ml
> > Affects Versions: 2.9
> > Reporter: Stepan Pilschikov
> >
> >
> > Trying to run Tutorial examples from repository or from binary release
> and
> > meet error results. Looks like a bug
> >
> > org.apache.ignite.examples.ml.tutorial.Step_1_Read_and_Learn (all other
> > tutorials have same issue)
> > {code}
> > >>> Trained model: if (x0 > 2.5000) then if (x1 > 2.5000) then if (x2 >
> > 0.5000) then if (x1 > 4.5000) then return 0. else if (x2 > 1.5000)
> then
> > return 0. else return 0. else return 0. else if (x2 > 0.5000)
> > then if (x2 > 3.5000) then if (x1 > 0.5000) then return 0. else
> return
> > 0. else if (x2 > 1.5000) then return 0. else return 1. else
> if
> > (x1 > 0.5000) then if (x1 > 1.5000) then return 0. else return 0.
> > else return 0. else if (x2 > 0.5000) then if (x1 > 0.5000) then if
> (x1
> > > 1.5000) then if (x1 > 2.5000) then return 1. else return 1.
> else
> > if (x2 > 3.5000) then return 0. else return 1. else if (x2 >
> > 1.5000) then if (x0 > 1.5000) then return 1. else return 1. else
> if
> > (x0 > 1.5000) then return 1. else return 1. else if (x0 > 1.5000)
> > then if (x1 > 2.5000) then return 1. else if (x1 > 1.5000) then
> return
> > 0. else return 0. else if (x1 > 0.5000) then if (x1 > 1.5000)
> then
> > return 1. else return 1. else return 1.
> >
> > >>> Accuracy 0.7117737003058104
> >
> > >>> Test Error 0.28822629969418956
> > >>> Tutorial step 1 (read and learn) example completed.
> > {code}
> >
> >
> >
> > --
> > This message was sent by Atlassian Jira
> > (v8.3.4#803005)
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSS] Ignite 3.0 development approach

[Ivan Daschinsky]

>> b. Implement IEP-61 - Common Replication Infrastructure
I suppose, that this is the main cause of the current discussion.
I hardly believe that this activity can be done without at least creating a
completely new branch.

пт, 13 нояб. 2020 г. в 11:12, Nikolay Izhikov :

> My suggestion:
>
> 1. Reduce Ignite3 scope to the following:
> a. Delete all deprecated API and support of obsolete internal
> protocols.
> b. Implement IEP-61 - Common Replication Infrastructure
> c. Implement new Ignite management tool ignitectl as suggested
> during Ignite3 discussion.
>
> 2. Implement and release following improvements like transactions, Calcite
> based SQL, etc in the ongoing releases - Ignite 4, 5, 6
>
> My concern against separate Ignite 3 repo is the following:
>
> 1. We spread community to the two very separated part - Ignite3 developers
> and Ignite2 maintainers.  believe it’s bad for our community.
> That can lead to the situation when we don’t fix critical or
> blocker issueds «because they will not exists in Ignite3»
> That will lead to the solutions never reviewed or reviewed poorly.
>
> 2. It seems for me that current scope of Ignite3 is too big to be
> implemented in any reasonable time.
>
>
> > 13 нояб. 2020 г., в 10:57, Nikolay Izhikov 
> написал(а):
> >
> > Hello, Valentin.
> >
> >> Nikolay, Maxim, are you OK with this route?
> >
> > -1 to have another repo for Ignite3 development.
> >
> >> 13 нояб. 2020 г., в 03:04, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> написал(а):
> >>
> >> Folks,
> >>
> >> We already have multiple IEPs for Ignite 3.0, and as far as I know,
> there are contributors that would like to work on them (or probably already
> started). That said, we should make a decision as soon as possible.
> >>
> >> At this point, it doesn't seem that there are any strong objections to
> the technical side of things. So I would suggest the following:
> >>
> >> 1. Proceed with Alexey's approach to the development process, as it
> seems to be the best (in my opinion - the only) way to address all the
> technical concerns and issues expressed in the thread. We'll start by
> creating a new repo and a new TC project.
> >> 2. Start a separate discussion around transparency. If there are any
> changes we need to make to our contributor guidelines, I am happy to talk
> them through, but I don't think it's reasonable to delay feature
> development because of this. In the short term, I will make sure that
> everything that happens within the new repo is as open to the community as
> possible.
> >>
> >> Nikolay, Maxim, are you OK with this route?
> >>
> >> -Val
> >>
> >> On Tue, Nov 10, 2020 at 4:55 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
> >> Maxim,
> >>
> >> 2.x and 3.x will have to coexist for some time - I don't see how we can
> avoid this considering the set of proposed changes. That said, we
> effectively will need to have two "masters" - one for each major version.
> Master for 3.x can technically be a branch in the existing repo, but having
> a separate repo seems cleaner, simply because it will not be a "branch" in
> the traditional sense.
> >>
> >> Note that the new repo will still be under the Apache org, with the
> same set of committers, managed by the community, etc. All the development
> happening for 3.0 must follow the rules that we currently have (if
> anything, it's an opportunity to improve those rules).
> >>
> >> As I said during the call on Friday, I strongly believe that if there
> is a transparency issue, it will exist regardless of the approach we choose
> for 3.0. If community members develop without IEPs or public discussions,
> this will happen for both 2.x and 3.x unless we address this separately. I
> don't see how this is related to Alexey's suggestion, which targets
> *technical* issues with the product more than anything else. This a way to
> achieve better modularity, introduce better coverage with unit tests,
> reduce conflicts during development, etc.
> >>
> >> Coming back to transparency, let's identify the issues and fix them. It
> probably makes sense to have a separate discussion on this topic.
> >>
> >> -Val
> >>
> >> On Tue, Nov 10, 2020 at 1:05 PM Maxim Muzafarov 
> wrote:
> >> Sergey,
> >>
> >>
> >> Your summary makes sense to me.
> >>
> >> However, how we come up from *Development transparency* to *create a
> >> separate public repository dedicated for 3.0*? For me *development
> >> transparency* is about making changes in the master branch. These
> >> changes will definitely be seen by all the Ignite developers.
> >>
> >> A dedicated public repository is technically public and visible for
> >> everyone, but it allows development without IEPs, without public
> >> discussion (since all the code changes are not related to the master
> >> branch) it also allows a large number of assumptions and deviations
> >> (like code-style violations). It also not about *development
> >> transparency* 

Re: IEP-61 Technical discussion

[Ivan Daschinsky]

> Any existing library that can be used to avoid re-implementing the
protocol ourselves? Perhaps, porting the existing implementation to Java
Personally, I like this idea. Go libraries (either raft module of etcd or
serf by Hashicorp) are famous for clean code, good design, stability, not
enormous size.
But, on other side, Go has different model for concurrency and porting
probably will not be so straightforward.



чт, 19 нояб. 2020 г. в 13:48, Ivan Daschinsky :

> I'd suggest to discuss this IEP and technical details in open ZOOM
> meeting.
>
> чт, 19 нояб. 2020 г. в 13:47, Ivan Daschinsky :
>
>>
>>
>> -- Forwarded message -
>> От: Ivan Daschinsky 
>> Date: чт, 19 нояб. 2020 г. в 13:02
>> Subject: Re: IEP-61 Technical discussion
>> To: Alexey Goncharuk 
>>
>>
>> Alexey, let's arise another question. Specifically, how nodes initially
>> find each other (discovery) and how they detect failures.
>>
>> I suppose, that gossip protocol is an ideal candidate. For example,
>> consul [1] uses this approach, using serf [2] library to discover members
>> of cluster.
>> Then consul forms raft ensemble (server nodes) and client use raft
>> ensemble only as lock service.
>>
>> PacificA suggests internal heartbeats mechanism for failure detection of
>> replicated group, but it says nothing about initial discovery of nodes.
>>
>> WDYT?
>>
>> [1] -- https://www.consul.io/docs/architecture/gossip
>> [2] -- https://www.serf.io/
>>
>> чт, 19 нояб. 2020 г. в 12:46, Alexey Goncharuk <
>> alexey.goncha...@gmail.com>:
>>
>>> Following up the Ignite 3.0 scope/development approach threads, this is
>>> a separate thread to discuss technical aspects of the IEP.
>>>
>>> Let's reiterate one more time on the questions raised by Ivan and also
>>> see if there are any other thoughts on the IEP:
>>>
>>>- *Whether to deploy metastorage on a separate subset of the nodes
>>>or allow Ignite to choose these nodes automatically.* I think it is
>>>feasible to maintain both modes: by default, Ignite will choose
>>>metastorage nodes automatically which essentially will provide the same
>>>seamless user experience as TCP discovery SPI - no separate roles,
>>>simplistic deployment. For deployments where people want to have more
>>>fine-grained control over the nodes' assignments, we will provide a 
>>> runtime
>>>configuration which will allow pinning metastorage group to certain 
>>> nodes,
>>>thus eliminating the latency concerns.
>>>- *Whether there are any TLA+ specs for the PacificA protocol.* Not
>>>to my knowledge, but it is known to be used in production by Microsoft 
>>> and
>>>other projects, e.g. [1]
>>>
>>> I would like to collect general feedback on the IEP, as well as feedback
>>> on specific parts of it, such as:
>>>
>>>- Metastorage API
>>>- Any existing library that can be used to avoid re-implementing the
>>>protocol ourselves? Perhaps, porting the existing implementation to Java
>>>(the way TiKV did with etcd-raft [2] [3]? This is a very neat way btw in 
>>> my
>>>opinion because I like the finite automata-like approach of the 
>>> replication
>>>module, and, additionally, we could sync bug fixes and improvements from
>>>the upstream project)
>>>
>>>
>>> Thanks,
>>> --AG
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/INCUBATOR/PegasusProposal
>>> [2] https://github.com/etcd-io/etcd/tree/master/raft
>>> [3] https://github.com/tikv/raft-rs
>>>
>>
>>
>> --
>> Sincerely yours, Ivan Daschinskiy
>>
>>
>> --
>> Sincerely yours, Ivan Daschinskiy
>>
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>


-- 
Sincerely yours, Ivan Daschinskiy

Fwd: IEP-61 Technical discussion

[Ivan Daschinsky]

-- Forwarded message -
От: Ivan Daschinsky 
Date: чт, 19 нояб. 2020 г. в 13:02
Subject: Re: IEP-61 Technical discussion
To: Alexey Goncharuk 


Alexey, let's arise another question. Specifically, how nodes initially
find each other (discovery) and how they detect failures.

I suppose, that gossip protocol is an ideal candidate. For example, consul
[1] uses this approach, using serf [2] library to discover members of
cluster.
Then consul forms raft ensemble (server nodes) and client use raft ensemble
only as lock service.

PacificA suggests internal heartbeats mechanism for failure detection of
replicated group, but it says nothing about initial discovery of nodes.

WDYT?

[1] -- https://www.consul.io/docs/architecture/gossip
[2] -- https://www.serf.io/

чт, 19 нояб. 2020 г. в 12:46, Alexey Goncharuk :

> Following up the Ignite 3.0 scope/development approach threads, this is a
> separate thread to discuss technical aspects of the IEP.
>
> Let's reiterate one more time on the questions raised by Ivan and also see
> if there are any other thoughts on the IEP:
>
>- *Whether to deploy metastorage on a separate subset of the nodes or
>allow Ignite to choose these nodes automatically.* I think it is
>feasible to maintain both modes: by default, Ignite will choose
>metastorage nodes automatically which essentially will provide the same
>seamless user experience as TCP discovery SPI - no separate roles,
>simplistic deployment. For deployments where people want to have more
>fine-grained control over the nodes' assignments, we will provide a runtime
>configuration which will allow pinning metastorage group to certain nodes,
>thus eliminating the latency concerns.
>- *Whether there are any TLA+ specs for the PacificA protocol.* Not to
>my knowledge, but it is known to be used in production by Microsoft and
>other projects, e.g. [1]
>
> I would like to collect general feedback on the IEP, as well as feedback
> on specific parts of it, such as:
>
>- Metastorage API
>- Any existing library that can be used to avoid re-implementing the
>protocol ourselves? Perhaps, porting the existing implementation to Java
>(the way TiKV did with etcd-raft [2] [3]? This is a very neat way btw in my
>opinion because I like the finite automata-like approach of the replication
>module, and, additionally, we could sync bug fixes and improvements from
>the upstream project)
>
>
> Thanks,
> --AG
>
> [1] https://cwiki.apache.org/confluence/display/INCUBATOR/PegasusProposal
> [2] https://github.com/etcd-io/etcd/tree/master/raft
> [3] https://github.com/tikv/raft-rs
>


-- 
Sincerely yours, Ivan Daschinskiy


-- 
Sincerely yours, Ivan Daschinskiy

Re: IEP-61 Technical discussion

[Ivan Daschinsky]

I'd suggest to discuss this IEP and technical details in open ZOOM meeting.

чт, 19 нояб. 2020 г. в 13:47, Ivan Daschinsky :

>
>
> -- Forwarded message -----
> От: Ivan Daschinsky 
> Date: чт, 19 нояб. 2020 г. в 13:02
> Subject: Re: IEP-61 Technical discussion
> To: Alexey Goncharuk 
>
>
> Alexey, let's arise another question. Specifically, how nodes initially
> find each other (discovery) and how they detect failures.
>
> I suppose, that gossip protocol is an ideal candidate. For example, consul
> [1] uses this approach, using serf [2] library to discover members of
> cluster.
> Then consul forms raft ensemble (server nodes) and client use raft
> ensemble only as lock service.
>
> PacificA suggests internal heartbeats mechanism for failure detection of
> replicated group, but it says nothing about initial discovery of nodes.
>
> WDYT?
>
> [1] -- https://www.consul.io/docs/architecture/gossip
> [2] -- https://www.serf.io/
>
> чт, 19 нояб. 2020 г. в 12:46, Alexey Goncharuk  >:
>
>> Following up the Ignite 3.0 scope/development approach threads, this is a
>> separate thread to discuss technical aspects of the IEP.
>>
>> Let's reiterate one more time on the questions raised by Ivan and also
>> see if there are any other thoughts on the IEP:
>>
>>- *Whether to deploy metastorage on a separate subset of the nodes or
>>allow Ignite to choose these nodes automatically.* I think it is
>>feasible to maintain both modes: by default, Ignite will choose
>>metastorage nodes automatically which essentially will provide the same
>>seamless user experience as TCP discovery SPI - no separate roles,
>>simplistic deployment. For deployments where people want to have more
>>fine-grained control over the nodes' assignments, we will provide a 
>> runtime
>>configuration which will allow pinning metastorage group to certain nodes,
>>thus eliminating the latency concerns.
>>- *Whether there are any TLA+ specs for the PacificA protocol.* Not
>>to my knowledge, but it is known to be used in production by Microsoft and
>>other projects, e.g. [1]
>>
>> I would like to collect general feedback on the IEP, as well as feedback
>> on specific parts of it, such as:
>>
>>- Metastorage API
>>- Any existing library that can be used to avoid re-implementing the
>>protocol ourselves? Perhaps, porting the existing implementation to Java
>>(the way TiKV did with etcd-raft [2] [3]? This is a very neat way btw in 
>> my
>>opinion because I like the finite automata-like approach of the 
>> replication
>>module, and, additionally, we could sync bug fixes and improvements from
>>the upstream project)
>>
>>
>> Thanks,
>> --AG
>>
>> [1] https://cwiki.apache.org/confluence/display/INCUBATOR/PegasusProposal
>> [2] https://github.com/etcd-io/etcd/tree/master/raft
>> [3] https://github.com/tikv/raft-rs
>>
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>
>
> --
> Sincerely yours, Ivan Daschinskiy
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: IEP-61 Technical discussion

[Ivan Daschinsky]

>> Friday, Nov 27th work for you? If ok, let's have an open call then.
Yes, great
>> As for the protocol port - we will not be dealing with the concurrency...
>>Judging by the Rust port, it seems fairly straightforward.
Yes, they chose split transport and logic. But original Go package from
etcd (see raft/node.go) contains some  heartbeats mechanism etc.
I agree with you, this seems not to be a huge deal to port.

чт, 19 нояб. 2020 г. в 16:13, Alexey Goncharuk :

> Ivan,
>
> Agree, let's have a call to discuss the IEP. I have some more thoughts
> regarding how the replication infrastructure works with
> atomic/transactional caches, will put this info to the IEP. Does next
> Friday, Nov 27th work for you? If ok, let's have an open call then.
>
> As for the protocol port - we will not be dealing with the concurrency
> model if we choose this way, this is what I like about their code
> structure. Essentially, the raft module is a single-threaded automata which
> has a callback to process a message, process a tick (timeout) and produces
> messages that should be sent and log entries that should be persisted.
> Judging by the Rust port, it seems fairly straightforward. Will be happy to
> discuss this and other alternatives on the call as well.
>
> чт, 19 нояб. 2020 г. в 14:41, Ivan Daschinsky :
>
> > > Any existing library that can be used to avoid re-implementing the
> > protocol ourselves? Perhaps, porting the existing implementation to Java
> > Personally, I like this idea. Go libraries (either raft module of etcd or
> > serf by Hashicorp) are famous for clean code, good design, stability, not
> > enormous size.
> > But, on other side, Go has different model for concurrency and porting
> > probably will not be so straightforward.
> >
> >
> >
> > чт, 19 нояб. 2020 г. в 13:48, Ivan Daschinsky :
> >
> > > I'd suggest to discuss this IEP and technical details in open ZOOM
> > > meeting.
> > >
> > > чт, 19 нояб. 2020 г. в 13:47, Ivan Daschinsky :
> > >
> > >>
> > >>
> > >> -- Forwarded message -
> > >> От: Ivan Daschinsky 
> > >> Date: чт, 19 нояб. 2020 г. в 13:02
> > >> Subject: Re: IEP-61 Technical discussion
> > >> To: Alexey Goncharuk 
> > >>
> > >>
> > >> Alexey, let's arise another question. Specifically, how nodes
> initially
> > >> find each other (discovery) and how they detect failures.
> > >>
> > >> I suppose, that gossip protocol is an ideal candidate. For example,
> > >> consul [1] uses this approach, using serf [2] library to discover
> > members
> > >> of cluster.
> > >> Then consul forms raft ensemble (server nodes) and client use raft
> > >> ensemble only as lock service.
> > >>
> > >> PacificA suggests internal heartbeats mechanism for failure detection
> of
> > >> replicated group, but it says nothing about initial discovery of
> nodes.
> > >>
> > >> WDYT?
> > >>
> > >> [1] -- https://www.consul.io/docs/architecture/gossip
> > >> [2] -- https://www.serf.io/
> > >>
> > >> чт, 19 нояб. 2020 г. в 12:46, Alexey Goncharuk <
> > >> alexey.goncha...@gmail.com>:
> > >>
> > >>> Following up the Ignite 3.0 scope/development approach threads, this
> is
> > >>> a separate thread to discuss technical aspects of the IEP.
> > >>>
> > >>> Let's reiterate one more time on the questions raised by Ivan and
> also
> > >>> see if there are any other thoughts on the IEP:
> > >>>
> > >>>- *Whether to deploy metastorage on a separate subset of the nodes
> > >>>or allow Ignite to choose these nodes automatically.* I think it
> is
> > >>>feasible to maintain both modes: by default, Ignite will choose
> > >>>metastorage nodes automatically which essentially will provide the
> > same
> > >>>seamless user experience as TCP discovery SPI - no separate roles,
> > >>>simplistic deployment. For deployments where people want to have
> > more
> > >>>fine-grained control over the nodes' assignments, we will provide
> a
> > runtime
> > >>>configuration which will allow pinning metastorage group to
> certain
> > nodes,
> > >>>thus eliminating the latency concerns.
> > >>>- *Whether there are any TLA+ specs for the PacificA protocol.*
> No

Re: 2.9.1 release scope and dates

[Ivan Daschinsky]

Hi!
Yaroslav, Max -- I have another ticket that will be nice to have in 2.9.1
https://issues.apache.org/jira/browse/IGNITE-13699

пт, 13 нояб. 2020 г. в 15:08, Yaroslav Molochkov :

>
> Igniters, hello!
>
> I think the scope of 2.9.1 is finalized.
>
> > On 9 Nov 2020, at 12:04, Yaroslav Molochkov 
> wrote:
> >
> > Ivan, thanks!
> >
> > Added it to the list.
> >
> >> On 8 Nov 2020, at 14:13, Ivan Daschinsky  wrote:
> >>
> >> Yaroslav, there is another bug for 2.9.1 release
> >> https://issues.apache.org/jira/browse/IGNITE-13572
> >>
> >> чт, 5 нояб. 2020 г., 19:23 Yaroslav Molochkov :
> >>
> >>> Ivan, hi!
> >>> Sure.
> >>>
> >>> UPD: i am the release manager and will be doing this with Maxim's help
> >>> (since i don't have some user permissions)
> >>>
> >>>> On Thu, Nov 5, 2020 at 6:24 PM Ivan Daschinsky 
> >>>> wrote:
> >>>>
> >>>> Hi. I'd suggest to add this issue. This is a usability improvement
> for zk
> >>>> discovery, and also this patch incorporates fixes for JMX metrics
> >>>> concurrency issues
> >>>>
> >>>> [1] -- https://issues.apache.org/jira/browse/IGNITE-13577
> >>>>
> >>>> чт, 5 нояб. 2020 г., 16:20 Yaroslav Molochkov  >:
> >>>>
> >>>>> Igniters!
> >>>>>
> >>>>> I'd like to help with the 2.9.1 release. The scope of this release
> >>>> includes
> >>>>> following issues:
> >>>>>
> >>>>>
> >>>>
> >>>
> https://issues.apache.org/jira/browse/IGNITE-13676?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.9.1
> >>>>>
> >>>>> Maxim Muzafarov agreed to help me with the process and he will be the
> >>>>> release manager.
> >>>>>
> >>>>> Scope freeze: Nov. 12th
> >>>>> Code freeze: Nov. 19th
> >>>>> Voting date: Nov. 26th
> >>>>> Release date: Nov. 31st
> >>>>>
> >>>>> Tickets that were added (or to be added) to the scope don't bring new
> >>>>> features but various bug fixes.
> >>>>>
> >>>>
> >>>
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSS] Use GridNioServer in Java thin client

[Ivan Daschinsky]

I suppose that the best variant -- ability to switch to netty if this lib
is in classpath

пн, 9 нояб. 2020 г. в 15:58, Igor Sapego :

> Sounds like a good idea to me.
>
> Best Regards,
> Igor
>
>
> On Mon, Nov 9, 2020 at 3:32 PM Alex Plehanov 
> wrote:
>
> > +1 for using GridNioServer as java thin client communication layer.
> >
> > вс, 8 нояб. 2020 г. в 19:12, Pavel Tupitsyn :
> >
> > > Igniters,
> > >
> > > This is a continuation of "Use Netty for Java thin client" [1],
> > > I'm starting a new thread for better visibility.
> > >
> > > The problems with current Java thin client are:
> > > * Socket writes block user threads
> > > * Every connection uses a separate listener thread (with partition
> > > awareness there is a thread for every server node within a single
> > > IgniteClient)
> > >
> > > GridNioServer can work in client mode and solves both of these
> problems.
> > > It is the most practical choice as well at the moment - no extra
> > > dependencies required.
> > >
> > > A potential drawback is increased coupling between thin client and core
> > > code,
> > > which I'm going to mitigate by abstracting GridNioServer behind a
> simpler
> > > facade,
> > > so we can replace it with Netty or something else easier if we decide
> to
> > > split the code.
> > >
> > > Thoughts, objections?
> > >
> > > [1]
> > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Use-Netty-for-Java-thin-client-td49732.html
> > >
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: IEP-61 Technical discussion

[Ivan Daschinsky]

Also, here is some interesting reading about gossip, SWIM etc.

1 -- http://www.cs.cornell.edu/Info/Projects/Spinglass/public_pdfs/SWIM.pdf
2 --
http://www.antonkharenko.com/2015/09/swim-distributed-group-membership.html
3 -- https://github.com/hashicorp/memberlist (Foundation library of
hashicorp serf)
4 -- https://github.com/scalecube/scalecube-cluster -- (Java implementation
of SWIM)

чт, 19 нояб. 2020 г. в 16:35, Ivan Daschinsky :

> >> Friday, Nov 27th work for you? If ok, let's have an open call then.
> Yes, great
> >> As for the protocol port - we will not be dealing with the
> concurrency...
> >>Judging by the Rust port, it seems fairly straightforward.
> Yes, they chose split transport and logic. But original Go package from
> etcd (see raft/node.go) contains some  heartbeats mechanism etc.
> I agree with you, this seems not to be a huge deal to port.
>
> чт, 19 нояб. 2020 г. в 16:13, Alexey Goncharuk  >:
>
>> Ivan,
>>
>> Agree, let's have a call to discuss the IEP. I have some more thoughts
>> regarding how the replication infrastructure works with
>> atomic/transactional caches, will put this info to the IEP. Does next
>> Friday, Nov 27th work for you? If ok, let's have an open call then.
>>
>> As for the protocol port - we will not be dealing with the concurrency
>> model if we choose this way, this is what I like about their code
>> structure. Essentially, the raft module is a single-threaded automata
>> which
>> has a callback to process a message, process a tick (timeout) and produces
>> messages that should be sent and log entries that should be persisted.
>> Judging by the Rust port, it seems fairly straightforward. Will be happy
>> to
>> discuss this and other alternatives on the call as well.
>>
>> чт, 19 нояб. 2020 г. в 14:41, Ivan Daschinsky :
>>
>> > > Any existing library that can be used to avoid re-implementing the
>> > protocol ourselves? Perhaps, porting the existing implementation to Java
>> > Personally, I like this idea. Go libraries (either raft module of etcd
>> or
>> > serf by Hashicorp) are famous for clean code, good design, stability,
>> not
>> > enormous size.
>> > But, on other side, Go has different model for concurrency and porting
>> > probably will not be so straightforward.
>> >
>> >
>> >
>> > чт, 19 нояб. 2020 г. в 13:48, Ivan Daschinsky :
>> >
>> > > I'd suggest to discuss this IEP and technical details in open ZOOM
>> > > meeting.
>> > >
>> > > чт, 19 нояб. 2020 г. в 13:47, Ivan Daschinsky :
>> > >
>> > >>
>> > >>
>> > >> -- Forwarded message -
>> > >> От: Ivan Daschinsky 
>> > >> Date: чт, 19 нояб. 2020 г. в 13:02
>> > >> Subject: Re: IEP-61 Technical discussion
>> > >> To: Alexey Goncharuk 
>> > >>
>> > >>
>> > >> Alexey, let's arise another question. Specifically, how nodes
>> initially
>> > >> find each other (discovery) and how they detect failures.
>> > >>
>> > >> I suppose, that gossip protocol is an ideal candidate. For example,
>> > >> consul [1] uses this approach, using serf [2] library to discover
>> > members
>> > >> of cluster.
>> > >> Then consul forms raft ensemble (server nodes) and client use raft
>> > >> ensemble only as lock service.
>> > >>
>> > >> PacificA suggests internal heartbeats mechanism for failure
>> detection of
>> > >> replicated group, but it says nothing about initial discovery of
>> nodes.
>> > >>
>> > >> WDYT?
>> > >>
>> > >> [1] -- https://www.consul.io/docs/architecture/gossip
>> > >> [2] -- https://www.serf.io/
>> > >>
>> > >> чт, 19 нояб. 2020 г. в 12:46, Alexey Goncharuk <
>> > >> alexey.goncha...@gmail.com>:
>> > >>
>> > >>> Following up the Ignite 3.0 scope/development approach threads,
>> this is
>> > >>> a separate thread to discuss technical aspects of the IEP.
>> > >>>
>> > >>> Let's reiterate one more time on the questions raised by Ivan and
>> also
>> > >>> see if there are any other thoughts on the IEP:
>> > >>>
>> > >>>- *Whether to deploy metastorage on a separate subset of the
>> nodes
>> > >>>or 

Re: Why WAL archives enabled by default?

[Ivan Daschinsky]

Dmitriy, as far as I understand, Denis have adviced user to "disable"
archiving by setting wal and wal archive path to the same value. I tried to
explain that this measure doesn't prevent from storing wal segments needed
for recovery but bring additional performance penalty.

> In older versions of Apache Ignite, WAL archive could contain valid
records
needed for recovery. If something was changed since then, my comment may be
not valid.

Nothing changed since that time.


ср, 11 нояб. 2020 г., 03:31 Dmitriy Pavlov :

> In older versions of Apache Ignite, WAL archive could contain valid records
> needed for recovery. If something was changed since then, my comment may be
> not valid.
>
> We've discussed that before, that naming this directory as 'archive' was
> not the best possible option. The archive is often considered by users as
> something not needed and sometimes it was deleted.
>
> See also page related to internals and directory structure:
>
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Persistent+Store+-+under+the+hood#IgnitePersistentStoreunderthehood-WALstructure
>
> So infinite storage of archive is definitely not necessary for vanilla
> open-source version, but archive itself is needed.
>
> Sincerely,
> Dmitriy Pavlov
>
> ср, 11 нояб. 2020 г. в 01:21, Raymond Wilson :
>
> > Isn't the discussion here related to the WAL archive? If you disable that
> > don't you still have the WAL containing un-checkpointed changes?
> >
> > On Wed, Nov 11, 2020 at 11:01 AM Dmitriy Pavlov 
> > wrote:
> >
> > > Hi Denis,
> > >
> > > the short answer here, Apache Ignite guarantees ACID, and for
> > D-Durability
> > > it is required to save all changes in some WAL/Redo Log to have a safe
> > way
> > > to recover from any hardware failures/disk outage.
> > >
> > > Should the user disable WAL, he/she could potentially lose durability.
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > > вт, 10 нояб. 2020 г. в 09:57, ткаленко кирилл :
> > >
> > > > Hello guys again!
> > > >
> > > > Does anyone know why we are doing any calculation here
> > > > IgniteUtils#adjustedWalHistorySize at all?
> > > > Would it be easier to always take the
> > > > DataStorageConfiguration#maxWalArchiveSize? It seems that the user
> can
> > > > easily do this himself by changing the value by 1 byte.
> > > >
> > > > 06.11.2020, 13:56, "Ivan Daschinsky" :
> > > > > Alex, thanks for pointing that out. Shame that I missed it.
> > > > >
> > > > > пт, 6 нояб. 2020 г. в 13:45, Alex Plehanov <
> plehanov.a...@gmail.com
> > >:
> > > > >
> > > > >>  Guys,
> > > > >>
> > > > >>  We already have
> > > FileWriteAheadLogManager#maxSegCountWithoutCheckpoint.
> > > > >>  Checkpoint triggered if there are too many WAL segments without
> > > > checkpoint.
> > > > >>  Looks like you are talking about this feature.
> > > > >>
> > > > >>  пт, 6 нояб. 2020 г. в 13:21, Ivan Daschinsky <
> ivanda...@gmail.com
> > >:
> > > > >>
> > > > >>  > Kirill and I discussed privately proposed approach. As far as I
> > > > >>  understand,
> > > > >>  > Kirill suggests to implement some
> > > > >>  > heuristic to do a force checkpoint in some cases if user by
> > mistake
> > > > >>  > misconfigured cluster in order to preserve
> > > > >>  > requested size of WAL archive.
> > > > >>  > Currently, as for me, this approach is questionable, because it
> > can
> > > > cause
> > > > >>  > some performance problems. But as an option,
> > > > >>  > it can be used and should be switchable.
> > > > >>  >
> > > > >>  > пт, 6 нояб. 2020 г. в 12:36, Ivan Daschinsky <
> > ivanda...@gmail.com
> > > >:
> > > > >>  >
> > > > >>  > > Kirill, how your approach will help if user tuned a cluster
> to
> > do
> > > > >>  > > checkpoints rarely under load?
> > > > >>  > > No way.
> > > > >>  > >
> > > > >>  > > пт, 6 нояб. 2020 г. в 12:19, ткаленко кирилл <
> > > tkalkir...@yandex.ru
> > > > >:
> > > > >&g

Re: [DISCUSS] Disable socket linger by default in TCP discovery SPI.

[Ivan Daschinsky]

Ilya, yes, there is an option TcpDiscoverySpi#soLinger. The main question
is why the default value is true, 5 instead of false,0

пн, 2 нояб. 2020 г., 20:14 Ilya Kasnacheev :

> Hello!
>
> Is there any option to re-enable linger on SSL sockets?
>
> Telling people to re-configure does not help if they can't.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 30 окт. 2020 г. в 15:21, Anton Vinogradov :
>
> > > When TLS 1.3 is introduced, whole sun.security.ssl.SSLSocketImpl was
> > > rewritten.
> > Correct, I meant rewritten TLSv1.3, the good news that 1.2- also were
> > fixed.
> > so,
> > -- brand new TLS with any linger
> > -- plain old TLS with linger>0
> >
> > On Fri, Oct 30, 2020 at 3:10 PM Ivan Daschinsky 
> > wrote:
> >
> > > Ilya, Anton.
> > > It means that not if TLS 1.3 is worked ok and with TLS < 1.2 is not ok.
> > >
> > > When TLS 1.3 is introduced, whole sun.security.ssl.SSLSocketImpl was
> > > rewritten.
> > > There is not any code anymore that could cause a deadlock.
> > > Therefore, in JDK, that supports TLS 1.3, this option is unnecessary,
> > even
> > > if you use TLS 1.2
> > >
> > >
> > > пт, 30 окт. 2020 г. в 14:46, Anton Vinogradov :
> > >
> > > > Ilya
> > > > > I think we should still keep setting linger if SSL is enabled
> > > > Modern (updated) JVMs do not require this.
> > > > AFAIK, Problem caused this workaround already fixed everywhere,
> > including
> > > > JDK 8.
> > > >
> > > > > If SSL only works with TLSv1.3 and no linger
> > > > SSL works if
> > > > -- TLSv1.3 with any linger
> > > > -- TLSv1.2- with linger>0
> > > >
> > > > > we should make TLSv1.3 a
> > > > > default. If JVM does not support it, users will have to reconfigure
> > > > > explicitly.
> > > > I don't think it's a good idea to reconfigure production environments
> > > this
> > > > way.
> > > >
> > > > P.s.
> > > > My +1 to zero linger as default + warning on SSL enabled on JVM
> before
> > > the
> > > > fix + warning at documentation + migration notes
> > > >
> > > > On Fri, Oct 30, 2020 at 2:19 PM Ilya Kasnacheev <
> > > ilya.kasnach...@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hello!
> > > > >
> > > > > I think we should still keep setting linger if SSL is enabled, and
> > not
> > > > > expect user to enable it (or face consequences).
> > > > >
> > > > > If SSL only works with TLSv1.3 and no linger, we should make
> TLSv1.3
> > a
> > > > > default. If JVM does not support it, user will have to reconfigure
> > > > > explicitly.
> > > > >
> > > > > Regards,
> > > > > --
> > > > > Ilya Kasnacheev
> > > > >
> > > > >
> > > > > пт, 30 окт. 2020 г. в 14:05, Steshin Vladimir  >:
> > > > >
> > > > > > *
> > > > > >
> > > > > > Hi, Igniters.
> > > > > >
> > > > > > We’ve found that enabled by default socket linger causes
> unexpected
> > > > > > delay in detection of node failure.
> > > > > >
> > > > > >
> > > > > > Moreover, long closing of socket works as Thread.sleep() within
> > > > > > algorithms of failure detection and connection recovery in TCP
> > > > > > discovery. These time gaps lead to hardly predictable behavior of
> > the
> > > > > > discovery. When the socket linger is enabled, it’s hard or even
> > > > > > impossible to figure out what time is taken to detect node
> failure
> > > and
> > > > > > restore connections with the provided settings.
> > > > > >
> > > > > > Socket linger was enabled only as a workaround for SSL bugs (i.e.
> > > [2],
> > > > > > [3]). It was enabled without including in failure processing
> > routines
> > > > in
> > > > > > TCP discovery SPI as described above. SSL bugs, mentioned above,
> > were
> > > > > > fixed and backported to various JDK, supporting TLS 1.3 ([4] and
> > > [5]).
> > > > > >
> > > > > >
> > > > > > I’d suggest to disable socket linger by default, because enabled
> > > socket
> > > > > > linger prolongs detection of node failure. The ticket is [1]. In
> > case
> > > > of
> > > > > > SSL issues the linger could be enabled. Or one may just update
> JDK.
> > > > > > We'll provide the documentation.
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-13643
> > > > > >
> > > > > > [2] https://bugs.openjdk.java.net/browse/JDK-8219658
> > > > > >
> > > > > > [3]https://issues.apache.org/jira/browse/IGNITE-12818
> > > > > >
> > > > > > [4]https://bugs.openjdk.java.net/browse/JDK-8245468
> > > > > >
> > > > > > [5]
> > > > https://www.oracle.com/java/technologies/javase/8u261-relnotes.html
> > > > > >
> > > > > > *
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Sincerely yours, Ivan Daschinskiy
> > >
> >
>

Re: Incorrect failure handler exceptions

[Ivan Daschinsky]

There is not a big problem to fix it.

1. Blocked thread is detected by another worker. Currently we take dump
before printing it to log with zero depth. This can be easily fixed.

2. We should propagate BlockedThreadExceptiob to failureCtx with original
stacktrace of blocked thread. StackFrames, obtained in previous step, can
be used to construct stacktrace of BlockedThreadException.


вт, 3 нояб. 2020 г., 19:15 Ilya Kasnacheev :

> Hello!
>
> https://issues.apache.org/jira/browse/IGNITE-13665
>
> The notorious problem with failure handler's thread dump, which nobody has
> bothered to investigate, until now.
> "IgnitionEx$IgniteNamedInstance$2.apply"
>
> If anybody knows how to fix it, please suggest.
>
> Regards,
> --
> Ilya Kasnacheev
>

Re: [DISCUSS] Missed (non-suited) tests

[Ivan Daschinsky]

Hi

WalCompactionAfterRestartTest -- yes we need it. This test failed because
of race (test shold be rewritten a little bit)

пт, 30 окт. 2020 г. в 16:15, Max Timonin :

> Hi!
>
> Yes, you're correct. I've developed the check and started to clean tests
> (move them to suites, mark some tests with Ignore, etc.). I finish work on
> the core module. I hope it was the biggest one, and others are less. If so,
> I think I will finish the work on other modules in 1 or 2 weeks, as I do
> this activity in the background (~10% of my work time). Actually I've found
> 3 failed tests in the core module that aren't in any suite, so I need time
> to discover reason of failures and if we actually need those tests:
>
> GridCacheMultithreadedFailoverAbstractTest
> WalCompactionAfterRestartTest
> GridTcpCommunicationSpiLogTest
>
> Also we should decide how to be with wrongly located es. As I understand we
> can't just move suites between modules, as TeamCity may depend on the path
> to them. So, for such cases I've just created suites in the right module,
> and replaced the test list with the new class suite. It does not look
> pretty enough, but I think It's a path of least resistance. WDYT?
>
> BEFORE:
> Module A -> SuiteA -> testA1, testA2, testB1, testB2
> Module B -> testB1, testB2
>
> AFTER:
> Module A -> SuiteA, SuiteB
> Module B -> SuiteB -> testB1, testB2
>
> On Fri, Oct 30, 2020 at 3:38 PM Anton Vinogradov  wrote:
>
> > Folks,
> > What's the current state of this thread?
> > AFAIU, we found unused and wrongly located tests and developed some
> > checker, could we split this to some PRs?
> > Let's merge tests usage fix and location fixes first, this will provide
> us
> > an ability to automate check using Travis.
> >
> > On Tue, Oct 20, 2020 at 12:06 PM Ivan Pavlukhin 
> > wrote:
> >
> > > Max, Ivan,
> > >
> > > Using explicit @Ignore and the automated check sounds good to me. If
> > > nobody has arguments against it I think we should do it.
> > >
> > > 2020-10-19 19:30 GMT+03:00, Max Timonin :
> > > > Hi Ivan,
> > > >
> > > > I've checked the ticket you provide. It contains subtasks to
> uncomment
> > or
> > > > to remove some unused tests. It definitely describes some cases I've
> > > found.
> > > > So what do you think if I uncomment them in suites, add @Ignore
> > > annotation
> > > > for those tests while the tickets are open? This will help to find
> out
> > > > tests that were forgiven in a recent time.
> > > >
> > > > Also I believe that this check must be automated. I didn't find a way
> > how
> > > > uncomment / unused tests are found in the ticket. If there is no any
> -
> > I
> > > > propose mine PR for this purpose.
> > > >
> > > >
> > > >
> > > > On Mon, Oct 19, 2020 at 5:24 PM Ivan Daschinsky  >
> > > > wrote:
> > > >
> > > >> Ivan, as far as I understand, Max also created verification check
> for
> > > not
> > > >> included test and found a few tests, that have never been included
> in
> > > any
> > > >> testsuites.
> > > >>
> > > >> Also, I suppose, that even if we cannot run some tests, these tests
> > > >> should
> > > >> be ignored using annotation, but not commented.
> > > >>
> > > >> пн, 19 окт. 2020 г. в 16:33, Ivan Pavlukhin :
> > > >>
> > > >> > Hi Max,
> > > >> >
> > > >> > There is an existing effort about "abandoned" tests
> > > >> > https://issues.apache.org/jira/browse/IGNITE-9210
> > > >> >
> > > >> > 2020-10-19 16:25 GMT+03:00, Max Timonin  >:
> > > >> > > Hi Igniters!
> > > >> > >
> > > >> > > I made a research into tests that aren't included in any test
> > suite.
> > > >> > > As
> > > >> > > TeamCity runs tests by suites so there could be tests that never
> > run
> > > >> > > on
> > > >> > TC.
> > > >> > > So I tried implementing a simple check for such tests and
> include
> > it
> > > >> > > in
> > > >> > > Ignite's travis config.
> > > >> > >
> > > >> > > The check runs while "mvn test" command and piggy-backs on the
> > maven
> > &g

Re: [DISCUSS] Missed (non-suited) tests

[Ivan Daschinsky]

I suggests to mark these tests with @Ignore and file tickets to fix them.

пт, 30 окт. 2020 г. в 16:26, Ivan Daschinsky :

> Hi
>
> WalCompactionAfterRestartTest -- yes we need it. This test failed because
> of race (test shold be rewritten a little bit)
>
> пт, 30 окт. 2020 г. в 16:15, Max Timonin :
>
>> Hi!
>>
>> Yes, you're correct. I've developed the check and started to clean tests
>> (move them to suites, mark some tests with Ignore, etc.). I finish work on
>> the core module. I hope it was the biggest one, and others are less. If
>> so,
>> I think I will finish the work on other modules in 1 or 2 weeks, as I do
>> this activity in the background (~10% of my work time). Actually I've
>> found
>> 3 failed tests in the core module that aren't in any suite, so I need time
>> to discover reason of failures and if we actually need those tests:
>>
>> GridCacheMultithreadedFailoverAbstractTest
>> WalCompactionAfterRestartTest
>> GridTcpCommunicationSpiLogTest
>>
>> Also we should decide how to be with wrongly located es. As I understand
>> we
>> can't just move suites between modules, as TeamCity may depend on the path
>> to them. So, for such cases I've just created suites in the right module,
>> and replaced the test list with the new class suite. It does not look
>> pretty enough, but I think It's a path of least resistance. WDYT?
>>
>> BEFORE:
>> Module A -> SuiteA -> testA1, testA2, testB1, testB2
>> Module B -> testB1, testB2
>>
>> AFTER:
>> Module A -> SuiteA, SuiteB
>> Module B -> SuiteB -> testB1, testB2
>>
>> On Fri, Oct 30, 2020 at 3:38 PM Anton Vinogradov  wrote:
>>
>> > Folks,
>> > What's the current state of this thread?
>> > AFAIU, we found unused and wrongly located tests and developed some
>> > checker, could we split this to some PRs?
>> > Let's merge tests usage fix and location fixes first, this will provide
>> us
>> > an ability to automate check using Travis.
>> >
>> > On Tue, Oct 20, 2020 at 12:06 PM Ivan Pavlukhin 
>> > wrote:
>> >
>> > > Max, Ivan,
>> > >
>> > > Using explicit @Ignore and the automated check sounds good to me. If
>> > > nobody has arguments against it I think we should do it.
>> > >
>> > > 2020-10-19 19:30 GMT+03:00, Max Timonin :
>> > > > Hi Ivan,
>> > > >
>> > > > I've checked the ticket you provide. It contains subtasks to
>> uncomment
>> > or
>> > > > to remove some unused tests. It definitely describes some cases I've
>> > > found.
>> > > > So what do you think if I uncomment them in suites, add @Ignore
>> > > annotation
>> > > > for those tests while the tickets are open? This will help to find
>> out
>> > > > tests that were forgiven in a recent time.
>> > > >
>> > > > Also I believe that this check must be automated. I didn't find a
>> way
>> > how
>> > > > uncomment / unused tests are found in the ticket. If there is no
>> any -
>> > I
>> > > > propose mine PR for this purpose.
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Oct 19, 2020 at 5:24 PM Ivan Daschinsky <
>> ivanda...@gmail.com>
>> > > > wrote:
>> > > >
>> > > >> Ivan, as far as I understand, Max also created verification check
>> for
>> > > not
>> > > >> included test and found a few tests, that have never been included
>> in
>> > > any
>> > > >> testsuites.
>> > > >>
>> > > >> Also, I suppose, that even if we cannot run some tests, these tests
>> > > >> should
>> > > >> be ignored using annotation, but not commented.
>> > > >>
>> > > >> пн, 19 окт. 2020 г. в 16:33, Ivan Pavlukhin :
>> > > >>
>> > > >> > Hi Max,
>> > > >> >
>> > > >> > There is an existing effort about "abandoned" tests
>> > > >> > https://issues.apache.org/jira/browse/IGNITE-9210
>> > > >> >
>> > > >> > 2020-10-19 16:25 GMT+03:00, Max Timonin > >:
>> > > >> > > Hi Igniters!
>> > > >> > >
>> > > >> > > I made a research into tests that aren't included in any 

Re: [DISCUSS] Disable socket linger by default in TCP discovery SPI.

[Ivan Daschinsky]

Hi!
Vova, I agree with you, this default behaviour is quite confusing. Even if
we want to workaround bug for old jdk's and SSL,
it's strange idea to affect all other users by default. I think that we
should add section in documentation how to workaround this issue,
and disable socket linger on close by default.

пт, 30 окт. 2020 г. в 14:05, Steshin Vladimir :

> *
>
> Hi, Igniters.
>
> We’ve found that enabled by default socket linger causes unexpected
> delay in detection of node failure.
>
>
> Moreover, long closing of socket works as Thread.sleep() within
> algorithms of failure detection and connection recovery in TCP
> discovery. These time gaps lead to hardly predictable behavior of the
> discovery. When the socket linger is enabled, it’s hard or even
> impossible to figure out what time is taken to detect node failure and
> restore connections with the provided settings.
>
> Socket linger was enabled only as a workaround for SSL bugs (i.e. [2],
> [3]). It was enabled without including in failure processing routines in
> TCP discovery SPI as described above. SSL bugs, mentioned above, were
> fixed and backported to various JDK, supporting TLS 1.3 ([4] and [5]).
>
>
> I’d suggest to disable socket linger by default, because enabled socket
> linger prolongs detection of node failure. The ticket is [1]. In case of
> SSL issues the linger could be enabled. Or one may just update JDK.
> We'll provide the documentation.
>
> WDYT?
>
>
> [1] https://issues.apache.org/jira/browse/IGNITE-13643
>
> [2] https://bugs.openjdk.java.net/browse/JDK-8219658
>
> [3]https://issues.apache.org/jira/browse/IGNITE-12818
>
> [4]https://bugs.openjdk.java.net/browse/JDK-8245468
>
> [5] https://www.oracle.com/java/technologies/javase/8u261-relnotes.html
>
> *
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSS] Disable socket linger by default in TCP discovery SPI.

[Ivan Daschinsky]

Ilya, Anton.
It means that not if TLS 1.3 is worked ok and with TLS < 1.2 is not ok.

When TLS 1.3 is introduced, whole sun.security.ssl.SSLSocketImpl was
rewritten.
There is not any code anymore that could cause a deadlock.
Therefore, in JDK, that supports TLS 1.3, this option is unnecessary, even
if you use TLS 1.2


пт, 30 окт. 2020 г. в 14:46, Anton Vinogradov :

> Ilya
> > I think we should still keep setting linger if SSL is enabled
> Modern (updated) JVMs do not require this.
> AFAIK, Problem caused this workaround already fixed everywhere, including
> JDK 8.
>
> > If SSL only works with TLSv1.3 and no linger
> SSL works if
> -- TLSv1.3 with any linger
> -- TLSv1.2- with linger>0
>
> > we should make TLSv1.3 a
> > default. If JVM does not support it, users will have to reconfigure
> > explicitly.
> I don't think it's a good idea to reconfigure production environments this
> way.
>
> P.s.
> My +1 to zero linger as default + warning on SSL enabled on JVM before the
> fix + warning at documentation + migration notes
>
> On Fri, Oct 30, 2020 at 2:19 PM Ilya Kasnacheev  >
> wrote:
>
> > Hello!
> >
> > I think we should still keep setting linger if SSL is enabled, and not
> > expect user to enable it (or face consequences).
> >
> > If SSL only works with TLSv1.3 and no linger, we should make TLSv1.3 a
> > default. If JVM does not support it, user will have to reconfigure
> > explicitly.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пт, 30 окт. 2020 г. в 14:05, Steshin Vladimir :
> >
> > > *
> > >
> > > Hi, Igniters.
> > >
> > > We’ve found that enabled by default socket linger causes unexpected
> > > delay in detection of node failure.
> > >
> > >
> > > Moreover, long closing of socket works as Thread.sleep() within
> > > algorithms of failure detection and connection recovery in TCP
> > > discovery. These time gaps lead to hardly predictable behavior of the
> > > discovery. When the socket linger is enabled, it’s hard or even
> > > impossible to figure out what time is taken to detect node failure and
> > > restore connections with the provided settings.
> > >
> > > Socket linger was enabled only as a workaround for SSL bugs (i.e. [2],
> > > [3]). It was enabled without including in failure processing routines
> in
> > > TCP discovery SPI as described above. SSL bugs, mentioned above, were
> > > fixed and backported to various JDK, supporting TLS 1.3 ([4] and [5]).
> > >
> > >
> > > I’d suggest to disable socket linger by default, because enabled socket
> > > linger prolongs detection of node failure. The ticket is [1]. In case
> of
> > > SSL issues the linger could be enabled. Or one may just update JDK.
> > > We'll provide the documentation.
> > >
> > > WDYT?
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-13643
> > >
> > > [2] https://bugs.openjdk.java.net/browse/JDK-8219658
> > >
> > > [3]https://issues.apache.org/jira/browse/IGNITE-12818
> > >
> > > [4]https://bugs.openjdk.java.net/browse/JDK-8245468
> > >
> > > [5]
> https://www.oracle.com/java/technologies/javase/8u261-relnotes.html
> > >
> > > *
> > >
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: 2.9.1 release scope and dates

[Ivan Daschinsky]

Hi. I'd suggest to add this issue. This is a usability improvement for zk
discovery, and also this patch incorporates fixes for JMX metrics
concurrency issues

[1] -- https://issues.apache.org/jira/browse/IGNITE-13577

чт, 5 нояб. 2020 г., 16:20 Yaroslav Molochkov :

> Igniters!
>
> I'd like to help with the 2.9.1 release. The scope of this release includes
> following issues:
>
> https://issues.apache.org/jira/browse/IGNITE-13676?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.9.1
>
> Maxim Muzafarov agreed to help me with the process and he will be the
> release manager.
>
> Scope freeze: Nov. 12th
> Code freeze: Nov. 19th
> Voting date: Nov. 26th
> Release date: Nov. 31st
>
> Tickets that were added (or to be added) to the scope don't bring new
> features but various bug fixes.
>

Re: Why WAL archives enabled by default?

[Ivan Daschinsky]

Kirill, how your approach will help if user tuned a cluster to do
checkpoints rarely under load?
No way.

пт, 6 нояб. 2020 г. в 12:19, ткаленко кирилл :

> Ivan, I agree with you that the archive is primarily about optimization.
>
> If the size of the archive is critical for the user, we have no protection
> against this, we can always go beyond this limit.
> Thus, the user needs to remember this and configure it in some way.
>
> I suggest not to exceed this limit and give the expected behavior for the
> user. At the same time, the segments needed for recovery will remain and
> there will be no data loss.
>
> 06.11.2020, 11:29, "Ivan Daschinsky" :
> > Guys, fisrt of all, archiving is not for PITR at all, this is
> optimization.
> > If we disable archiving, every rollover we need to create new file. If we
> > enable archiving, we reserve 10 (by default) segments filled with zeroes.
> > We use mmap by default, so if we use no-archiver approach:
> > 1. We firstly create new empty file
> > 2. Call on it sun.nio.ch.FileChannelImpl#map, thats under the hood
> > a. If file is shorter, than wal segment size, it
> > calls sun.nio.ch.FileDispatcherImpl#truncate0, this is under the hood
> just
> > a system call truncate [1]
> > b. Than it calls system call mmap on this
> > file sun.nio.ch.FileChannelImpl#map0, under the hood see [2]
> > These manipulation are not free and cheap. So rollover will be much much
> > slower.
> > If archiving is enabled, 10 segments are already preallocated at the
> moment
> > of node's start.
> >
> > When archiving is enabled, archiver just copy previous preallocated
> segment
> > and move it to archive directory.
> > This archived segment is crucial for recovery. When new checkpoints
> > finished, all eligible for trunocating segments are just removed.
> >
> > If archiving is disabled, we also write WAL segments in wal directory and
> > disabling archiving don't prevent you from storing segments, if they are
> > required for recovery.
> >
> >>> Before increasing the size of WAL archive (transferring to archive
> >
> > /rollOver, compression, decompression), we can make sure that there will
> be
> > enough space in the archive and if there is no such, then we will try to
> >>> clean it. We cannot delete those segments that are required for
> recovery
> >
> > (between the last two checkpoints) and reserved for example for
> historical
> > rebalancing.
> > First of all, compression/decompression is offtopic here.
> > Secondly, wal segments are required only with idx higher than LAST
> > checkpoint marker.
> > Thirdly, archiving and rolling over can be during checkpoint and we can
> > broke everything accidentially.
> > Fourthly, I see no benefits to overcomplicated already complicated logic.
> > This is basically problem of misunderstanding and tuning.
> > There are a lot of similar topics for almost every DB. [3]
> >
> > [1] -- https://man7.org/linux/man-pages/man2/ftruncate.2.html
> > [2] -- https://man7.org/linux/man-pages/man2/mmap.2.html
> > [3] --
> >
> https://www.google.com/search?q=pg_wal%2Fxlogtemp+no+space+left+on+device=pg+wal+no
> >
> > пт, 6 нояб. 2020 г. в 10:42, ткаленко кирилл :
> >
> >>  Hi, Ivan!
> >>
> >>  I have only described ideas. But here are a few more details.
> >>
> >>  We can take care not to go beyond
> >>  DataStorageConfiguration#maxWalArchiveSize.
> >>
> >>  Before increasing the size of WAL archive (transferring to archive
> >>  /rollOver, compression, decompression), we can make sure that there
> will be
> >>  enough space in the archive and if there is no such, then we will try
> to
> >>  clean it. We cannot delete those segments that are required for
> recovery
> >>  (between the last two checkpoints) and reserved for example for
> historical
> >>  rebalancing.
> >>
> >>  We can receive a notification about the change of checkpoints and the
> >>  reservation / release of segments, thus we can know how many segments
> we
> >>  can delete right now.
> >>
> >>  06.11.2020, 09:53, "Ivan Daschinsky" :
> >>  >>> For example, when trying to move a segment to the archive.
> >>  >
> >>  > We cannot do this, we will lost data. We can truncate archived
> segment if
> >>  > and only if it is not required for recovery. If last checkpoint
> marker
> >>  > points to segment
> >>  > with lower index, we cannot delete any se

Re: Why WAL archives enabled by default?

[Ivan Daschinsky]

Kirill and I discussed privately proposed approach. As far as I understand,
Kirill suggests to implement some
heuristic to do a force checkpoint in some cases if user by mistake
misconfigured cluster in order to preserve
requested size of WAL archive.
Currently, as for me, this approach is questionable, because it can cause
some performance problems. But as an option,
it can be used and should be switchable.

пт, 6 нояб. 2020 г. в 12:36, Ivan Daschinsky :

> Kirill, how your approach will help if user tuned a cluster to do
> checkpoints rarely under load?
> No way.
>
> пт, 6 нояб. 2020 г. в 12:19, ткаленко кирилл :
>
>> Ivan, I agree with you that the archive is primarily about optimization.
>>
>> If the size of the archive is critical for the user, we have no
>> protection against this, we can always go beyond this limit.
>> Thus, the user needs to remember this and configure it in some way.
>>
>> I suggest not to exceed this limit and give the expected behavior for the
>> user. At the same time, the segments needed for recovery will remain and
>> there will be no data loss.
>>
>> 06.11.2020, 11:29, "Ivan Daschinsky" :
>> > Guys, fisrt of all, archiving is not for PITR at all, this is
>> optimization.
>> > If we disable archiving, every rollover we need to create new file. If
>> we
>> > enable archiving, we reserve 10 (by default) segments filled with
>> zeroes.
>> > We use mmap by default, so if we use no-archiver approach:
>> > 1. We firstly create new empty file
>> > 2. Call on it sun.nio.ch.FileChannelImpl#map, thats under the hood
>> > a. If file is shorter, than wal segment size, it
>> > calls sun.nio.ch.FileDispatcherImpl#truncate0, this is under the hood
>> just
>> > a system call truncate [1]
>> > b. Than it calls system call mmap on this
>> > file sun.nio.ch.FileChannelImpl#map0, under the hood see [2]
>> > These manipulation are not free and cheap. So rollover will be much much
>> > slower.
>> > If archiving is enabled, 10 segments are already preallocated at the
>> moment
>> > of node's start.
>> >
>> > When archiving is enabled, archiver just copy previous preallocated
>> segment
>> > and move it to archive directory.
>> > This archived segment is crucial for recovery. When new checkpoints
>> > finished, all eligible for trunocating segments are just removed.
>> >
>> > If archiving is disabled, we also write WAL segments in wal directory
>> and
>> > disabling archiving don't prevent you from storing segments, if they are
>> > required for recovery.
>> >
>> >>> Before increasing the size of WAL archive (transferring to archive
>> >
>> > /rollOver, compression, decompression), we can make sure that there
>> will be
>> > enough space in the archive and if there is no such, then we will try to
>> >>> clean it. We cannot delete those segments that are required for
>> recovery
>> >
>> > (between the last two checkpoints) and reserved for example for
>> historical
>> > rebalancing.
>> > First of all, compression/decompression is offtopic here.
>> > Secondly, wal segments are required only with idx higher than LAST
>> > checkpoint marker.
>> > Thirdly, archiving and rolling over can be during checkpoint and we can
>> > broke everything accidentially.
>> > Fourthly, I see no benefits to overcomplicated already complicated
>> logic.
>> > This is basically problem of misunderstanding and tuning.
>> > There are a lot of similar topics for almost every DB. [3]
>> >
>> > [1] -- https://man7.org/linux/man-pages/man2/ftruncate.2.html
>> > [2] -- https://man7.org/linux/man-pages/man2/mmap.2.html
>> > [3] --
>> >
>> https://www.google.com/search?q=pg_wal%2Fxlogtemp+no+space+left+on+device=pg+wal+no
>> >
>> > пт, 6 нояб. 2020 г. в 10:42, ткаленко кирилл :
>> >
>> >>  Hi, Ivan!
>> >>
>> >>  I have only described ideas. But here are a few more details.
>> >>
>> >>  We can take care not to go beyond
>> >>  DataStorageConfiguration#maxWalArchiveSize.
>> >>
>> >>  Before increasing the size of WAL archive (transferring to archive
>> >>  /rollOver, compression, decompression), we can make sure that there
>> will be
>> >>  enough space in the archive and if there is no such, then we will try
>> to
>> >>  clean it. We cannot delete those segments that are required for
&

Re: Why WAL archives enabled by default?

[Ivan Daschinsky]

>> For example, when trying to move a segment to the archive.
We cannot do this, we will lost data. We can truncate archived segment if
and only if it is not required for recovery. If last checkpoint marker
points to segment
with lower index, we cannot delete any segment with higher index. So the
only moment where we can remove truncate segments is a finish of checkpoint.

пт, 6 нояб. 2020 г. в 09:46, ткаленко кирилл :

> Hello, everybody!
>
> As far as I know, WAL archive is used for PITP(GridGain feature) and
> historical rebalancing.
>
> Facundo seems to have a problem with running out of directory
> (/opt/work/walarchive) space.
> Currently, WAL archive is cleared at the end of checkpoint. Potentially
> long transaction may prevent checkpoint starting, thereby not cleaning WAL
> archive, which will lead to such an error.
> At the moment, I see such a WA to increase size of directory
> (/opt/work/walarchive) in k8s and avoid long transactions or something like
> that that modifies data and runs for a long time.
>
> And it is best to fix the logic of working with WAL archive. I think we
> should remove WAL archive cleanup from the end of the checkpoint and do it
> on demand. For example, when trying to move a segment to the archive.
>
>
> 06.11.2020, 01:58, "Denis Magda" :
> > Folks,
> >
> > In my understanding, you need the archives only for features such as
> PITR.
> > Considering, that the PITR functionality is not provided in Ignite why do
> > we have the archives enabled by default?
> >
> > How about having this feature disabled by default to prevent the
> following
> > issues experienced by our users:
> >
> http://apache-ignite-users.70518.x6.nabble.com/WAL-and-WAL-Archive-volume-size-recommendation-td34458.html
> >
> > -
> > Denis
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: Why WAL archives enabled by default?

[Ivan Daschinsky]

Guys, fisrt of all, archiving is not for PITR at all, this is optimization.
If we disable archiving, every rollover we need to create new file. If we
enable archiving, we reserve 10 (by default) segments filled with zeroes.
We use mmap by default, so if we use no-archiver approach:
1. We firstly create new empty file
2. Call on it sun.nio.ch.FileChannelImpl#map, thats under the hood
a. If file is shorter, than wal segment size, it
calls sun.nio.ch.FileDispatcherImpl#truncate0, this is under the hood just
a system call truncate [1]
b. Than it calls system call mmap on this
file sun.nio.ch.FileChannelImpl#map0, under the hood see [2]
These manipulation are not free and cheap. So rollover will be much much
slower.
If archiving is enabled, 10 segments are already preallocated at the moment
of node's start.

When archiving is enabled, archiver just copy previous preallocated segment
and move it to archive directory.
This archived segment is crucial for recovery. When new checkpoints
finished, all eligible for trunocating segments are just removed.

If archiving is disabled, we also write WAL segments in wal directory and
disabling archiving don't prevent you from storing segments, if they are
required for recovery.

>>Before increasing the size of WAL archive (transferring to archive
/rollOver, compression, decompression), we can make sure that there will be
enough space in the archive and if there is no such, then we will try to
>>clean it. We cannot delete those segments that are required for recovery
(between the last two checkpoints) and reserved for example for historical
rebalancing.
First of all, compression/decompression is offtopic here.
Secondly, wal segments are required only with idx higher than LAST
checkpoint marker.
Thirdly, archiving and rolling over can be during checkpoint and we can
broke everything accidentially.
Fourthly, I see no benefits to overcomplicated already complicated logic.
This is basically problem of misunderstanding and tuning.
There are a lot of similar topics for almost every DB. [3]



[1] -- https://man7.org/linux/man-pages/man2/ftruncate.2.html
[2] -- https://man7.org/linux/man-pages/man2/mmap.2.html
[3] --
https://www.google.com/search?q=pg_wal%2Fxlogtemp+no+space+left+on+device=pg+wal+no

пт, 6 нояб. 2020 г. в 10:42, ткаленко кирилл :

> Hi, Ivan!
>
> I have only described ideas. But here are a few more details.
>
> We can take care not to go beyond
> DataStorageConfiguration#maxWalArchiveSize.
>
> Before increasing the size of WAL archive (transferring to archive
> /rollOver, compression, decompression), we can make sure that there will be
> enough space in the archive and if there is no such, then we will try to
> clean it. We cannot delete those segments that are required for recovery
> (between the last two checkpoints) and reserved for example for historical
> rebalancing.
>
> We can receive a notification about the change of checkpoints and the
> reservation / release of segments, thus we can know how many segments we
> can delete right now.
>
> 06.11.2020, 09:53, "Ivan Daschinsky" :
> >>>  For example, when trying to move a segment to the archive.
> >
> > We cannot do this, we will lost data. We can truncate archived segment if
> > and only if it is not required for recovery. If last checkpoint marker
> > points to segment
> > with lower index, we cannot delete any segment with higher index. So the
> > only moment where we can remove truncate segments is a finish of
> checkpoint.
> >
> > пт, 6 нояб. 2020 г. в 09:46, ткаленко кирилл :
> >
> >>  Hello, everybody!
> >>
> >>  As far as I know, WAL archive is used for PITP(GridGain feature) and
> >>  historical rebalancing.
> >>
> >>  Facundo seems to have a problem with running out of directory
> >>  (/opt/work/walarchive) space.
> >>  Currently, WAL archive is cleared at the end of checkpoint. Potentially
> >>  long transaction may prevent checkpoint starting, thereby not cleaning
> WAL
> >>  archive, which will lead to such an error.
> >>  At the moment, I see such a WA to increase size of directory
> >>  (/opt/work/walarchive) in k8s and avoid long transactions or something
> like
> >>  that that modifies data and runs for a long time.
> >>
> >>  And it is best to fix the logic of working with WAL archive. I think we
> >>  should remove WAL archive cleanup from the end of the checkpoint and
> do it
> >>  on demand. For example, when trying to move a segment to the archive.
> >>
> >>  06.11.2020, 01:58, "Denis Magda" :
> >>  > Folks,
> >>  >
> >>  > In my understanding, you need the archives only for features such as
> >>  PITR.
> >

Re: Why WAL archives enabled by default?

[Ivan Daschinsky]

Alex, thanks for pointing that out. Shame that I missed it.

пт, 6 нояб. 2020 г. в 13:45, Alex Plehanov :

> Guys,
>
> We already have FileWriteAheadLogManager#maxSegCountWithoutCheckpoint.
> Checkpoint triggered if there are too many WAL segments without checkpoint.
> Looks like you are talking about this feature.
>
> пт, 6 нояб. 2020 г. в 13:21, Ivan Daschinsky :
>
> > Kirill and I discussed privately proposed approach. As far as I
> understand,
> > Kirill suggests to implement some
> > heuristic to do a force checkpoint in some cases if user by mistake
> > misconfigured cluster in order to preserve
> > requested size of WAL archive.
> > Currently, as for me, this approach is questionable, because it can cause
> > some performance problems. But as an option,
> > it can be used and should be switchable.
> >
> > пт, 6 нояб. 2020 г. в 12:36, Ivan Daschinsky :
> >
> > > Kirill, how your approach will help if user tuned a cluster to do
> > > checkpoints rarely under load?
> > > No way.
> > >
> > > пт, 6 нояб. 2020 г. в 12:19, ткаленко кирилл :
> > >
> > >> Ivan, I agree with you that the archive is primarily about
> optimization.
> > >>
> > >> If the size of the archive is critical for the user, we have no
> > >> protection against this, we can always go beyond this limit.
> > >> Thus, the user needs to remember this and configure it in some way.
> > >>
> > >> I suggest not to exceed this limit and give the expected behavior for
> > the
> > >> user. At the same time, the segments needed for recovery will remain
> and
> > >> there will be no data loss.
> > >>
> > >> 06.11.2020, 11:29, "Ivan Daschinsky" :
> > >> > Guys, fisrt of all, archiving is not for PITR at all, this is
> > >> optimization.
> > >> > If we disable archiving, every rollover we need to create new file.
> If
> > >> we
> > >> > enable archiving, we reserve 10 (by default) segments filled with
> > >> zeroes.
> > >> > We use mmap by default, so if we use no-archiver approach:
> > >> > 1. We firstly create new empty file
> > >> > 2. Call on it sun.nio.ch.FileChannelImpl#map, thats under the hood
> > >> > a. If file is shorter, than wal segment size, it
> > >> > calls sun.nio.ch.FileDispatcherImpl#truncate0, this is under the
> hood
> > >> just
> > >> > a system call truncate [1]
> > >> > b. Than it calls system call mmap on this
> > >> > file sun.nio.ch.FileChannelImpl#map0, under the hood see [2]
> > >> > These manipulation are not free and cheap. So rollover will be much
> > much
> > >> > slower.
> > >> > If archiving is enabled, 10 segments are already preallocated at the
> > >> moment
> > >> > of node's start.
> > >> >
> > >> > When archiving is enabled, archiver just copy previous preallocated
> > >> segment
> > >> > and move it to archive directory.
> > >> > This archived segment is crucial for recovery. When new checkpoints
> > >> > finished, all eligible for trunocating segments are just removed.
> > >> >
> > >> > If archiving is disabled, we also write WAL segments in wal
> directory
> > >> and
> > >> > disabling archiving don't prevent you from storing segments, if they
> > are
> > >> > required for recovery.
> > >> >
> > >> >>> Before increasing the size of WAL archive (transferring to archive
> > >> >
> > >> > /rollOver, compression, decompression), we can make sure that there
> > >> will be
> > >> > enough space in the archive and if there is no such, then we will
> try
> > to
> > >> >>> clean it. We cannot delete those segments that are required for
> > >> recovery
> > >> >
> > >> > (between the last two checkpoints) and reserved for example for
> > >> historical
> > >> > rebalancing.
> > >> > First of all, compression/decompression is offtopic here.
> > >> > Secondly, wal segments are required only with idx higher than LAST
> > >> > checkpoint marker.
> > >> > Thirdly, archiving and rolling over can be during checkpoint and we
> > can
> > >> > broke everything accidentially.
> > &g

[DISCUSS] Python thin client development approach.

[Ivan Daschinsky]

Hi folks!

Since we already have a separate repo for thin-clients [1], [2]
I'd like to propose some improvements in development process/

1. We should simplify and automate unit tests run for different versions of
python
2. We should add travis integration per commit and pr. Tests could be run
against latest dockered image of ignite
3. There should be ability to run tests against multiple pythons on TC
4. For thin client development process, travis should be the first option.
TC suite should be used only to check that compatibility is not broken
and when new functionality is developed (rare case).

I've prepared fix [3], you can see successful builds for travis. It uses
tox [5], the most common tool to run tests in multiple environments.
There are few environments set up in tox.ini -- with and without docker,
with or without ssl, etc. This helped a lot
to setup travis CI build (you can see in commits list of PR) and simplify
run tests for developers. Also docker-compose was introduced
to help python thin client developers.

But  I need some assistance for TC:
1. There is outdated python setuptools on TC agents, so tests cannot be run
with updated pytest etc.
2. Multiple pythons should be installed on TC agents (via
https://github.com/pyenv/pyenv), latest minor versions
for 3.6, 3.7 and 3.8
3. After that, TC job should be changed to utilize tox

WDYT about this initiative?


[1] -- https://issues.apache.org/jira/browse/IGNITE-13767
[2] -- https://github.com/apache/ignite-python-thin-client
[3] -- https://issues.apache.org/jira/browse/IGNITE-13903
[4] -- https://github.com/apache/ignite-python-thin-client/pull/1/commits
[5] -- https://tox.readthedocs.io/en/latest/

-- 
Sincerely yours, Ivan Daschinskiy

Re: Limiting number of active connection for thin client

[Ivan Daschinsky]

>> However, sometimes it can create excessive load for cluster in use cases
when there are a lot of thin clients.
Could you please provide some figures? I suppose, that our NIO server can
handle thousands connections easily, and problem
is not in multiple connections.

пт, 25 дек. 2020 г. в 11:33, Pavel Tupitsyn :

> Igor,
>
> The idea is to keep partition awareness enabled
> even when the limit is reached, right?
> So it works when the corresponding server is available,
> and ignored otherwise.
>
> I think this could be useful, but we should document it thoroughly,
> so the negative effect on partition awareness performance is clear.
>
>
> On Fri, Dec 25, 2020 at 12:02 AM Igor Sapego  wrote:
>
> > Hi Igniters,
> >
> > As you may know thin clients now can establish connections with
> > multiple servers simultaneously. It is implemented this way to make
> > partition awareness [1] work or for fast failover if partition awareness
> > is not used. However, sometimes it can create excessive load for
> > cluster in use cases when there are a lot of thin clients. I think, we
> > should give user a possibility to limit maximum number of active
> > connections established by a client. What do you think?
> >
> > [1] -
> >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-23%3A+Best+Effort+Affinity+for+thin+clients
> >
> > Best Regards,
> > Igor
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: [DISCUSS] Missed (non-suited) tests

[Ivan Daschinsky]

u leave in the PR. Also I checked
> > some
> > > > >> test
> > > > >> > > > suites
> > > > >> > > > > > and found that some tests are written in the core module
> > but
> > > > >> depend
> > > > >> > > on
> > > > >> > > > > the
> > > > >> > > > > > indexing module (or other modules). Some of such test
> > > classes
> > > > >> > contain
> > > > >> > > > > tests
> > > > >> > > > > > that are related to the core functionality, but some to
> > > > >> indexing.
> > > > >> > I'm
> > > > >> > > > not
> > > > >> > > > > > sure if it is correct to move a whole suite with all
> tests
> > > > from
> > > > >> the
> > > > >> > > > > > indexing module to the core, as it will hide some core
> > tests
> > > > >> from
> > > > >> > the
> > > > >> > > > > core
> > > > >> > > > > > module.
> > > > >> > > > > >
> > > > >> > > > > > I believe that the correct solution is to investigate
> > every
> > > > such
> > > > >> > test
> > > > >> > > > and
> > > > >> > > > > > move it to the right module. But I think this work will
> > > take a
> > > > >> lot
> > > > >> > of
> > > > >> > > > > time
> > > > >> > > > > > and should be performed in a separate ticket, I will do
> it
> > > in
> > > > >> the
> > > > >> > > > > > background.
> > > > >> > > > > >
> > > > >> > > > > > I think currently we should proceed with a way I
> > introduced
> > > in
> > > > >> PR:
> > > > >> > > > > > 1. Create fake suites for all such tests (written in
> core,
> > > > >> suited
> > > > >> > in
> > > > >> > > > > other
> > > > >> > > > > > modules: indexing/spring/zookeeper/etc) in the core
> > module.
> > > I
> > > > >> named
> > > > >> > > > such
> > > > >> > > > > > suites with prefix Core*.
> > > > >> > > > > > 2. Replace tests in modules with links to fake suites.
> > > > >> > > > > > 3. Create an umbrella Jira ticket to discover every fake
> > > suite
> > > > >> and
> > > > >> > > > > replace
> > > > >> > > > > > it with a new one in the right module.
> > > > >> > > > > > 4. Merge this PR for introducing a new travis check to
> > avoid
> > > > >> losing
> > > > >> > > > > > new tests.
> > > > >> > > > > >
> > > > >> > > > > > WDYT?
> > > > >> > > > > >
> > > > >> > > > > > List of such mixed suites:
> > > > >> > > > > >
> > > > >> > > > > > 1. suite IgniteBinaryCacheQueryTestSuite
> > > > >> > > > > >
> > > > >> > > > > > test GridCacheQueryIndexingDisabledSelfTest
> > > > >> > > > > > test IgniteCacheBinaryObjectsScanSelfTest
> > > > >> > > > > > test IgniteCacheBinaryObjectsScanWithEventsSelfTest)
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > 2. suite IgniteCacheQuerySelfTestSuite3
> > > > >> > > > > >
> > > > >> > > > > > test GridCacheContinuousQueryNodesFilteringTest
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > 3. suite IgniteCacheQuerySelfTestSuite5
> > > > >> > > > > >
> > > > >> > > > > > test
> ContinuousQuery

Re: [DISCUSS] Python thin client development approach.

[Ivan Daschinsky]

Igor, I've never talked about complete removal of TC builds.
I just suggested to add TC jobs for different python versions and use
travis heavily.

Currently, we have done most of the tasks, except of run TC tests on
different python's versions.



пт, 22 янв. 2021 г. в 15:16, Igor Sapego :

> Ivan,
>
> Though I generally agree with the approach you've suggested, I can see
> a problem here. Since we now have a separate repos for thin clients, for
> some features we may need to introduce changes to Ignite and python-thin
> repos in a single ticket and we should have an ability to run tests on with
> changes on both python client and server nodes. Current TC suites provide
> such ability, Travis does not. So I believe, it's too soon to abandon TC
> for thin
> clients, at least until we could solve the issue I've described.
>
> Best Regards,
> Igor
>
>
> On Fri, Dec 25, 2020 at 1:49 PM Nikolay Izhikov 
> wrote:
>
> > Hello, Ivan.
> >
> > I’m +1 for your proposal.
> >
> > > 25 дек. 2020 г., в 13:14, Ivan Daschinsky 
> > написал(а):
> > >
> > > Hi folks!
> > >
> > > Since we already have a separate repo for thin-clients [1], [2]
> > > I'd like to propose some improvements in development process/
> > >
> > > 1. We should simplify and automate unit tests run for different
> versions
> > of
> > > python
> > > 2. We should add travis integration per commit and pr. Tests could be
> run
> > > against latest dockered image of ignite
> > > 3. There should be ability to run tests against multiple pythons on TC
> > > 4. For thin client development process, travis should be the first
> > option.
> > > TC suite should be used only to check that compatibility is not broken
> > > and when new functionality is developed (rare case).
> > >
> > > I've prepared fix [3], you can see successful builds for travis. It
> uses
> > > tox [5], the most common tool to run tests in multiple environments.
> > > There are few environments set up in tox.ini -- with and without
> docker,
> > > with or without ssl, etc. This helped a lot
> > > to setup travis CI build (you can see in commits list of PR) and
> simplify
> > > run tests for developers. Also docker-compose was introduced
> > > to help python thin client developers.
> > >
> > > But  I need some assistance for TC:
> > > 1. There is outdated python setuptools on TC agents, so tests cannot be
> > run
> > > with updated pytest etc.
> > > 2. Multiple pythons should be installed on TC agents (via
> > > https://github.com/pyenv/pyenv), latest minor versions
> > > for 3.6, 3.7 and 3.8
> > > 3. After that, TC job should be changed to utilize tox
> > >
> > > WDYT about this initiative?
> > >
> > >
> > > [1] -- https://issues.apache.org/jira/browse/IGNITE-13767
> > > [2] -- https://github.com/apache/ignite-python-thin-client
> > > [3] -- https://issues.apache.org/jira/browse/IGNITE-13903
> > > [4] --
> > https://github.com/apache/ignite-python-thin-client/pull/1/commits
> > > [5] -- https://tox.readthedocs.io/en/latest/
> > >
> > > --
> > > Sincerely yours, Ivan Daschinskiy
> >
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: Ignite website css not loading in local dev

[Ivan Daschinsky]

Saikat, have you tried build and run locally docs using docker? I just
tried latest master -- everything is working as expected.
Checked this in chrome incognito mode. I usually check docs locally using
these steps:
1 cd docs
2. docker run -v "$PWD:/srv/jekyll" -p 4000:4000 jekyll/jekyll:latest
jekyll s
3. Go to http://0.0.0.0:4000/docs/ in chrome incognito tab.

пн, 1 февр. 2021 г. в 07:09, Denis Magda :

> Saikat,
>
> Attach the screenshot with what you see. Please also check what errors the
> browser is printing out while rendering the page.
>
> You don't need to install Jekyll unless you build the docs locally. So,
> skip this step. Also, you don't need to uncomment the "include virtual".
>
> -
> Denis
>
>
> On Sun, Jan 31, 2021 at 1:35 PM Saikat Maitra 
> wrote:
>
> > Hi Denis, Ivan
> >
> > Thank you for your email. I have verified the below lines in httpd.conf
> >
> > AddType text/html .html
> > AddOutputFilter INCLUDES .html
> >
> > I have also pull latest changes from master.
> >
> > I still see the issue in local, let me check if I need to make any
> > other changes.
> >
> > Also another change I did is I omitted the /trunk path from the Document
> > root and used path till ignite-website since we are no longer using svn.
> >
> > I have a question related to Jekyll, do I need to install Jekyll to run
> the
> > website in local?
> >
> > Also I have observed few additional lines commented in the index.html
> like
> > below
> >
> > 
> >
> > I will check if uncommenting them addresses the issue.
> >
> > Regards,
> > Saikat
> >
> >
> >
> > On Sun, Jan 31, 2021 at 12:08 PM Ivan Daschinsky 
> > wrote:
> >
> > > Hi, Saikat. Do you have
> > https://issues.apache.org/jira/browse/IGNITE-13659
> > > commit in yout local branch?
> > > In this patch, this issue was solved. Namely, docs/assets/css/docs.scss
> > > and docs/assets/css/styles.scss are corrected.
> > > Jekyll required that in custom stiles, first two lines contains this
> char
> > > sequence `---`
> > >
> > > вс, 31 янв. 2021 г. в 03:09, Denis Magda :
> > >
> > > > Hi Saikat,
> > > >
> > > > Please double check that you configured the Apache server strictly
> > > > following steps 3-5. As far as I remember, I had the exact CSS issue
> > > until
> > > > did this (step 5):
> > > >
> > > > AddType text/html .html
> > > > AddOutputFilter INCLUDES .html
> > > >
> > > >
> > > > -
> > > > Denis
> > > >
> > > >
> > > > On Sat, Jan 30, 2021 at 1:38 PM Saikat Maitra <
> saikat.mai...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I was working on updating the ignite-website page[1] for Ignite
> > > > Extensions
> > > > > and I observed that in local when we are running the ignite-website
> > > then
> > > > it
> > > > > is not loading the css.
> > > > >
> > > > > I have followed the instructions as mentioned  in confluence
> page[2]
> > > > >
> > > > > [1] https://ignite.apache.org/features/streaming.html
> > > > > [2]
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/Website+Development
> > > > >
> > > > > Are there any additional steps that need to be done?
> > > > >
> > > > > I also noted that the source in live website pages vs the local is
> > > > > different in terms of js and css files.
> > > > >
> > > > > Regards,
> > > > > Saikat
> > > > >
> > > >
> > >
> > >
> > > --
> > > Sincerely yours, Ivan Daschinskiy
> > >
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: Ignite website css not loading in local dev

[Ivan Daschinsky]

Hi, Saikat. Do you have https://issues.apache.org/jira/browse/IGNITE-13659
commit in yout local branch?
In this patch, this issue was solved. Namely, docs/assets/css/docs.scss
and docs/assets/css/styles.scss are corrected.
Jekyll required that in custom stiles, first two lines contains this char
sequence `---`

вс, 31 янв. 2021 г. в 03:09, Denis Magda :

> Hi Saikat,
>
> Please double check that you configured the Apache server strictly
> following steps 3-5. As far as I remember, I had the exact CSS issue until
> did this (step 5):
>
> AddType text/html .html
> AddOutputFilter INCLUDES .html
>
>
> -
> Denis
>
>
> On Sat, Jan 30, 2021 at 1:38 PM Saikat Maitra 
> wrote:
>
> > Hi,
> >
> > I was working on updating the ignite-website page[1] for Ignite
> Extensions
> > and I observed that in local when we are running the ignite-website then
> it
> > is not loading the css.
> >
> > I have followed the instructions as mentioned  in confluence page[2]
> >
> > [1] https://ignite.apache.org/features/streaming.html
> > [2]
> https://cwiki.apache.org/confluence/display/IGNITE/Website+Development
> >
> > Are there any additional steps that need to be done?
> >
> > I also noted that the source in live website pages vs the local is
> > different in terms of js and css files.
> >
> > Regards,
> > Saikat
> >
>


-- 
Sincerely yours, Ivan Daschinskiy

Re: Hard limit WAL archive size

[Ivan Daschinsky]

As for me, correct approach is to trigger checkpoint when we are too close
to WAL archive size limit.
The main purpose of these mechanism is to provide durability, so we
should think about not to fail node, nor to delete data voluntary,
but prevent possible data loss.

вт, 26 янв. 2021 г. в 19:13, Zhenya Stanilovsky :

>
>
> Hello !
> this is unclear for me, all you described near brings no info why node
> work improperly and why FH can possibly fail this node. Can you explain ?
>
> >Hello, everyone!
> >
> >Currently, property DataStorageConfiguration#maxWalArchiveSize is not
> working as expected by users. We can easily go beyond this limit and
> overflow the disk, which will lead to errors and a crash of the node. I
> propose to fix this behavior and not let WAL archive overflow.
> >
> >It is suggested not to add segments to the archive if we can exceed the
> DataStorageConfiguration#maxWalArchiveSize and wait until space becomes
> available for this.
> >
> >Thus, we may have a deadlock:
> >Get checkpontReadLock -> write to WAL -> need to rollover WAL segment ->
> need to clean WAL archive -> need to complete checkpoint (impossible
> because of checkpontReadLock taken).
> >
> >To avoid such situations, I suggest adding a custom heuristic - do not
> give a IgniteCacheDatabaseSharedManager#checkpointReadLock if there are few
> (default 1) segments left.
> >But this will not allow us to completely avoid archive overflow
> situations. Therefore, I suggest fail node by FH when a deadlock is
> detected, since it could be the same if there was no disk space left.
>
>
>
>



-- 
Sincerely yours, Ivan Daschinskiy

Re: Ignite website css not loading in local dev

[Ivan Daschinsky]

Ops, i'm very sorry. Really, nothing common with docs.

пн, 1 февр. 2021 г. в 16:10, Denis Magda :

> Ivan,
>
> Saikat is updating the page below which doesn’t not belong to the docs and
> doesn’t not require a Jekyll setup:
>
> https://ignite.apache.org/features/streaming.html
>
> On Sunday, January 31, 2021, Ivan Daschinsky  wrote:
>
> > Saikat, have you tried build and run locally docs using docker? I just
> > tried latest master -- everything is working as expected.
> > Checked this in chrome incognito mode. I usually check docs locally using
> > these steps:
> > 1 cd docs
> > 2. docker run -v "$PWD:/srv/jekyll" -p 4000:4000 jekyll/jekyll:latest
> > jekyll s
> > 3. Go to http://0.0.0.0:4000/docs/ in chrome incognito tab.
> >
> > пн, 1 февр. 2021 г. в 07:09, Denis Magda :
> >
> > > Saikat,
> > >
> > > Attach the screenshot with what you see. Please also check what errors
> > the
> > > browser is printing out while rendering the page.
> > >
> > > You don't need to install Jekyll unless you build the docs locally. So,
> > > skip this step. Also, you don't need to uncomment the "include
> virtual".
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Sun, Jan 31, 2021 at 1:35 PM Saikat Maitra  >
> > > wrote:
> > >
> > > > Hi Denis, Ivan
> > > >
> > > > Thank you for your email. I have verified the below lines in
> httpd.conf
> > > >
> > > > AddType text/html .html
> > > > AddOutputFilter INCLUDES .html
> > > >
> > > > I have also pull latest changes from master.
> > > >
> > > > I still see the issue in local, let me check if I need to make any
> > > > other changes.
> > > >
> > > > Also another change I did is I omitted the /trunk path from the
> > Document
> > > > root and used path till ignite-website since we are no longer using
> > svn.
> > > >
> > > > I have a question related to Jekyll, do I need to install Jekyll to
> run
> > > the
> > > > website in local?
> > > >
> > > > Also I have observed few additional lines commented in the index.html
> > > like
> > > > below
> > > >
> > > > 
> > > >
> > > > I will check if uncommenting them addresses the issue.
> > > >
> > > > Regards,
> > > > Saikat
> > > >
> > > >
> > > >
> > > > On Sun, Jan 31, 2021 at 12:08 PM Ivan Daschinsky <
> ivanda...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi, Saikat. Do you have
> > > > https://issues.apache.org/jira/browse/IGNITE-13659
> > > > > commit in yout local branch?
> > > > > In this patch, this issue was solved. Namely,
> > docs/assets/css/docs.scss
> > > > > and docs/assets/css/styles.scss are corrected.
> > > > > Jekyll required that in custom stiles, first two lines contains
> this
> > > char
> > > > > sequence `---`
> > > > >
> > > > > вс, 31 янв. 2021 г. в 03:09, Denis Magda :
> > > > >
> > > > > > Hi Saikat,
> > > > > >
> > > > > > Please double check that you configured the Apache server
> strictly
> > > > > > following steps 3-5. As far as I remember, I had the exact CSS
> > issue
> > > > > until
> > > > > > did this (step 5):
> > > > > >
> > > > > > AddType text/html .html
> > > > > > AddOutputFilter INCLUDES .html
> > > > > >
> > > > > >
> > > > > > -
> > > > > > Denis
> > > > > >
> > > > > >
> > > > > > On Sat, Jan 30, 2021 at 1:38 PM Saikat Maitra <
> > > saikat.mai...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I was working on updating the ignite-website page[1] for Ignite
> > > > > > Extensions
> > > > > > > and I observed that in local when we are running the
> > ignite-website
> > > > > then
> > > > > > it
> > > > > > > is not loading the css.
> > > > > > >
> > > > > > > I have followed the instructions as mentioned  in confluence
> > > page[2]
> > > > > > >
> > > > > > > [1] https://ignite.apache.org/features/streaming.html
> > > > > > > [2]
> > > > > >
> > > https://cwiki.apache.org/confluence/display/IGNITE/Website+Development
> > > > > > >
> > > > > > > Are there any additional steps that need to be done?
> > > > > > >
> > > > > > > I also noted that the source in live website pages vs the local
> > is
> > > > > > > different in terms of js and css files.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Saikat
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sincerely yours, Ivan Daschinskiy
> > > > >
> > > >
> > >
> >
> >
> > --
> > Sincerely yours, Ivan Daschinskiy
> >
>
>
> --
> -
> Denis
>


-- 
Sincerely yours, Ivan Daschinskiy