[MTCGA]: new failures in builds [4483209] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 If your changes can lead to this failure(s): We're grateful that you were a 
volunteer to make the contribution to this project, but things change and you 
may no longer be able to finalize your contribution.
 Could you respond to this email and indicate if you wish to continue and fix 
test failures or step down and some committer may revert you commit. 

 *New test failure in master-nightly 
DiskPageCompressionConfigValidationTest.testIncorrectStaticCacheConfiguration 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-2692660105095122533=%3Cdefault%3E=testDetails

 *New test failure in master-nightly 
DiskPageCompressionConfigValidationTest.testIncorrectDynamicCacheStartRequest 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=1915110918646717850=%3Cdefault%3E=testDetails
 Changes may lead to failure were done by 
 - eduard shangareev  
https://ci.ignite.apache.org/viewModification.html?modId=889258

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 07:57:11 10-08-2019 

Re: SQL query timeout: in progress or abandoned

[Denis Magda]

Ivan, thanks for sharing this discussion. Let's use it for our conversation.

-
Denis


On Thu, Aug 8, 2019 at 11:15 PM Павлухин Иван  wrote:

> Just for the protocol. There was an original dev-list discussion [1].
> Added a link to the ticket as well.
>
> [1]
> http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-7285-Add-default-query-timeout-td41828.html
>
> пт, 9 авг. 2019 г. в 01:22, Denis Magda :
> >
> > Hey Saikat,
> >
> > Are you still working on this ticket?
> > https://issues.apache.org/jira/browse/IGNITE-7285
> >
> > Seems that's the last API that doesn't support timeouts - JDBC and ODBC
> > drivers already go with it.
> >
> > If you don't have time to complete the changes then someone else from the
> > community can take over. We see a lot of demand for this API and here is
> > one example:
> >
> https://stackoverflow.com/questions/57275301/how-to-set-a-query-timeout-for-apache-ignite-cache
> >
> > -
> > Denis
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

[Pavel Tupitsyn]

>  the invoker thread is going to come back eventually
Invoker thread can be anything, it can be busy, waiting or aborted.
Async is just syntactic sugar to solve "callback hell", compiler transforms
it into a state machine with callbacks.

In general, the behavior is the same in the end - callbacks/continuations
are invoked on the future/task completion thread.
Yes, there are mechanisms that are used in .NET UI frameworks to actually
dispatch the callback to the "main" thread, but
that main thread is controlled by the framework and runs a
producer/consumer queue. But that's not possible for console apps and so on.

Back to our problem - yes, using public pool for callbacks seems to be the
most obvious step.

On Fri, Aug 9, 2019 at 7:31 PM Ilya Kasnacheev 
wrote:

> I'm not sure, but I think that in case of async, the invoker thread is
> going to come back eventually?
>
> In case of Java futures with listeners, I think listeners should be invoked
> from public pool.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 9 авг. 2019 г. в 19:16, Pavel Tupitsyn :
>
> > Well, you can't just take any running thread and make it run your code
> > instead, right?
> >
> > On Fri, Aug 9, 2019 at 1:32 PM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com>
> > wrote:
> >
> > > Hello!
> > >
> > > Why can't we dispatch those callbacks in the caller thread?
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > пт, 9 авг. 2019 г. в 09:50, Pavel Tupitsyn :
> > >
> > > > Иван,
> > > >
> > > > The fix is to dispatch those callbacks (future listeners) to a
> > different
> > > > thread pool, not sure which one though.
> > > > If I would do a .NET-only fix, I would use the default thread pool
> (non
> > > > Ignite-specific), for Java-side there is no such thing as I
> understand.
> > > >
> > > > Yes, let's have a ticket to track the issue.
> > > >
> > > > On Fri, Aug 9, 2019 at 9:17 AM Павлухин Иван 
> > > wrote:
> > > >
> > > > > Ilya, Pavel,
> > > > >
> > > > > Do we a have a proposal how to fix the root cause of the problem?
> > > > > Should we a have a ticket for it?
> > > > >
> > > > > ср, 7 авг. 2019 г. в 17:48, Ilya Kasnacheev <
> > ilya.kasnach...@gmail.com
> > > >:
> > > > > >
> > > > > > Hello!
> > > > > >
> > > > > > I think we should definitely stop running futures out of striped
> > > pool,
> > > > > > while holding any cache logs (stripe thread counts as one).
> > > > > >
> > > > > > Regards,
> > > > > > --
> > > > > > Ilya Kasnacheev
> > > > > >
> > > > > >
> > > > > > ср, 7 авг. 2019 г. в 17:20, Pavel Tupitsyn  >:
> > > > > >
> > > > > > > Yes, this can be done purely on .NET side, which is an option
> > that
> > > I
> > > > > > > consider.
> > > > > > > However, the root problem is on Java side, and I believe that
> we
> > > > > should fix
> > > > > > > the root problem.
> > > > > > >
> > > > > > > > violate some of Ignite assumptions: that we never run user
> code
> > > > from
> > > > > > > certain thread pools
> > > > > > > We actually do run user code from Ignite thread pools:
> > > > > > >
> > > > > > > cache.getAsync(1).listen(fut ->
> > > > > > > System.out.println("Get operation completed [value=" +
> > > fut.get()
> > > > +
> > > > > > > ']'));
> > > > > > >
> > > > > > > `println` here is executed on the striped pool. This is stated
> in
> > > the
> > > > > > > docs that I linked above.
> > > > > > >
> > > > > > > Users have to be aware of this and they have to be very careful
> > > with
> > > > > > > every future listener. IMO, this is a tricky gotcha and a bad
> > > > > > > usability.
> > > > > > >
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev <
> > > > > ilya.kasnach...@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello!
> > > > > > > >
> > > > > > > > + dev@
> > > > > > > >
> > > > > > > > I think the current behavior, where .Net callbacks may be run
> > > from
> > > > > > > striped
> > > > > > > > pool, violate some of Ignite assumptions: that we never run
> > user
> > > > code
> > > > > > > from
> > > > > > > > certain thread pools (like sys-stripe) and that we try to
> limit
> > > > > options
> > > > > > > of
> > > > > > > > running user-supplied code from our internals.
> > > > > > > >
> > > > > > > > I think that future versions of .Net integration should
> remove
> > > the
> > > > > > > ability
> > > > > > > > of async callbacks to be called from non-user threads, even
> if
> > it
> > > > can
> > > > > > > lead
> > > > > > > > to performance degradation in some cases. I suggest removing
> > this
> > > > > mode,
> > > > > > > if
> > > > > > > > possible, while keeping only the safe one, where internal
> > threads
> > > > > are not
> > > > > > > > waiting upon completion of user code.
> > > > > > > >
> > > > > > > > In this case my issue IGNITE-12033 could be used to track
> this
> > > > work.
> > > > > > > >
> > > > > > > > WDYT?
> > > > > > > >
> > > > > > > > Regards,
> > > 

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

[Ilya Kasnacheev]

I'm not sure, but I think that in case of async, the invoker thread is
going to come back eventually?

In case of Java futures with listeners, I think listeners should be invoked
from public pool.

Regards,
-- 
Ilya Kasnacheev


пт, 9 авг. 2019 г. в 19:16, Pavel Tupitsyn :

> Well, you can't just take any running thread and make it run your code
> instead, right?
>
> On Fri, Aug 9, 2019 at 1:32 PM Ilya Kasnacheev 
> wrote:
>
> > Hello!
> >
> > Why can't we dispatch those callbacks in the caller thread?
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пт, 9 авг. 2019 г. в 09:50, Pavel Tupitsyn :
> >
> > > Иван,
> > >
> > > The fix is to dispatch those callbacks (future listeners) to a
> different
> > > thread pool, not sure which one though.
> > > If I would do a .NET-only fix, I would use the default thread pool (non
> > > Ignite-specific), for Java-side there is no such thing as I understand.
> > >
> > > Yes, let's have a ticket to track the issue.
> > >
> > > On Fri, Aug 9, 2019 at 9:17 AM Павлухин Иван 
> > wrote:
> > >
> > > > Ilya, Pavel,
> > > >
> > > > Do we a have a proposal how to fix the root cause of the problem?
> > > > Should we a have a ticket for it?
> > > >
> > > > ср, 7 авг. 2019 г. в 17:48, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> > >:
> > > > >
> > > > > Hello!
> > > > >
> > > > > I think we should definitely stop running futures out of striped
> > pool,
> > > > > while holding any cache logs (stripe thread counts as one).
> > > > >
> > > > > Regards,
> > > > > --
> > > > > Ilya Kasnacheev
> > > > >
> > > > >
> > > > > ср, 7 авг. 2019 г. в 17:20, Pavel Tupitsyn :
> > > > >
> > > > > > Yes, this can be done purely on .NET side, which is an option
> that
> > I
> > > > > > consider.
> > > > > > However, the root problem is on Java side, and I believe that we
> > > > should fix
> > > > > > the root problem.
> > > > > >
> > > > > > > violate some of Ignite assumptions: that we never run user code
> > > from
> > > > > > certain thread pools
> > > > > > We actually do run user code from Ignite thread pools:
> > > > > >
> > > > > > cache.getAsync(1).listen(fut ->
> > > > > > System.out.println("Get operation completed [value=" +
> > fut.get()
> > > +
> > > > > > ']'));
> > > > > >
> > > > > > `println` here is executed on the striped pool. This is stated in
> > the
> > > > > > docs that I linked above.
> > > > > >
> > > > > > Users have to be aware of this and they have to be very careful
> > with
> > > > > > every future listener. IMO, this is a tricky gotcha and a bad
> > > > > > usability.
> > > > > >
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev <
> > > > ilya.kasnach...@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello!
> > > > > > >
> > > > > > > + dev@
> > > > > > >
> > > > > > > I think the current behavior, where .Net callbacks may be run
> > from
> > > > > > striped
> > > > > > > pool, violate some of Ignite assumptions: that we never run
> user
> > > code
> > > > > > from
> > > > > > > certain thread pools (like sys-stripe) and that we try to limit
> > > > options
> > > > > > of
> > > > > > > running user-supplied code from our internals.
> > > > > > >
> > > > > > > I think that future versions of .Net integration should remove
> > the
> > > > > > ability
> > > > > > > of async callbacks to be called from non-user threads, even if
> it
> > > can
> > > > > > lead
> > > > > > > to performance degradation in some cases. I suggest removing
> this
> > > > mode,
> > > > > > if
> > > > > > > possible, while keeping only the safe one, where internal
> threads
> > > > are not
> > > > > > > waiting upon completion of user code.
> > > > > > >
> > > > > > > In this case my issue IGNITE-12033 could be used to track this
> > > work.
> > > > > > >
> > > > > > > WDYT?
> > > > > > >
> > > > > > > Regards,
> > > > > > > --
> > > > > > > Ilya Kasnacheev
> > > > > > >
> > > > > > >
> > > > > > > ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn <
> ptupit...@apache.org
> > >:
> > > > > > >
> > > > > > >> Sorry guys, I've completely missed this thread, and the topic
> is
> > > > very
> > > > > > >> important.
> > > > > > >>
> > > > > > >> First, a simple fix for the given example. Add the following
> on
> > > the
> > > > > > first
> > > > > > >> line of Main:
> > > > > > >> SynchronizationContext.SetSynchronizationContext(new
> > > > > > >> ThreadPoolSynchronizationContext());
> > > > > > >>
> > > > > > >> And put the ThreadPoolSynchronizationContext class somewhere:
> > > > > > >> class ThreadPoolSynchronizationContext :
> SynchronizationContext
> > > > > > >> {
> > > > > > >> // No-op.
> > > > > > >> }
> > > > > > >>
> > > > > > >>
> > > > > > >> Now, detailed explanation. The problem exists forever in
> Ignite
> > > and
> > > > is
> > > > > > >> mentioned in the docs briefly [1].
> > > > > > >> Also mentioned in .NET docs (I've updated them a bit) [2].
> > > > > > >>
> > > > > > >> 

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

[Pavel Tupitsyn]

Well, you can't just take any running thread and make it run your code
instead, right?

On Fri, Aug 9, 2019 at 1:32 PM Ilya Kasnacheev 
wrote:

> Hello!
>
> Why can't we dispatch those callbacks in the caller thread?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 9 авг. 2019 г. в 09:50, Pavel Tupitsyn :
>
> > Иван,
> >
> > The fix is to dispatch those callbacks (future listeners) to a different
> > thread pool, not sure which one though.
> > If I would do a .NET-only fix, I would use the default thread pool (non
> > Ignite-specific), for Java-side there is no such thing as I understand.
> >
> > Yes, let's have a ticket to track the issue.
> >
> > On Fri, Aug 9, 2019 at 9:17 AM Павлухин Иван 
> wrote:
> >
> > > Ilya, Pavel,
> > >
> > > Do we a have a proposal how to fix the root cause of the problem?
> > > Should we a have a ticket for it?
> > >
> > > ср, 7 авг. 2019 г. в 17:48, Ilya Kasnacheev  >:
> > > >
> > > > Hello!
> > > >
> > > > I think we should definitely stop running futures out of striped
> pool,
> > > > while holding any cache logs (stripe thread counts as one).
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > ср, 7 авг. 2019 г. в 17:20, Pavel Tupitsyn :
> > > >
> > > > > Yes, this can be done purely on .NET side, which is an option that
> I
> > > > > consider.
> > > > > However, the root problem is on Java side, and I believe that we
> > > should fix
> > > > > the root problem.
> > > > >
> > > > > > violate some of Ignite assumptions: that we never run user code
> > from
> > > > > certain thread pools
> > > > > We actually do run user code from Ignite thread pools:
> > > > >
> > > > > cache.getAsync(1).listen(fut ->
> > > > > System.out.println("Get operation completed [value=" +
> fut.get()
> > +
> > > > > ']'));
> > > > >
> > > > > `println` here is executed on the striped pool. This is stated in
> the
> > > > > docs that I linked above.
> > > > >
> > > > > Users have to be aware of this and they have to be very careful
> with
> > > > > every future listener. IMO, this is a tricky gotcha and a bad
> > > > > usability.
> > > > >
> > > > >
> > > > > Thoughts?
> > > > >
> > > > >
> > > > > On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev <
> > > ilya.kasnach...@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hello!
> > > > > >
> > > > > > + dev@
> > > > > >
> > > > > > I think the current behavior, where .Net callbacks may be run
> from
> > > > > striped
> > > > > > pool, violate some of Ignite assumptions: that we never run user
> > code
> > > > > from
> > > > > > certain thread pools (like sys-stripe) and that we try to limit
> > > options
> > > > > of
> > > > > > running user-supplied code from our internals.
> > > > > >
> > > > > > I think that future versions of .Net integration should remove
> the
> > > > > ability
> > > > > > of async callbacks to be called from non-user threads, even if it
> > can
> > > > > lead
> > > > > > to performance degradation in some cases. I suggest removing this
> > > mode,
> > > > > if
> > > > > > possible, while keeping only the safe one, where internal threads
> > > are not
> > > > > > waiting upon completion of user code.
> > > > > >
> > > > > > In this case my issue IGNITE-12033 could be used to track this
> > work.
> > > > > >
> > > > > > WDYT?
> > > > > >
> > > > > > Regards,
> > > > > > --
> > > > > > Ilya Kasnacheev
> > > > > >
> > > > > >
> > > > > > ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn  >:
> > > > > >
> > > > > >> Sorry guys, I've completely missed this thread, and the topic is
> > > very
> > > > > >> important.
> > > > > >>
> > > > > >> First, a simple fix for the given example. Add the following on
> > the
> > > > > first
> > > > > >> line of Main:
> > > > > >> SynchronizationContext.SetSynchronizationContext(new
> > > > > >> ThreadPoolSynchronizationContext());
> > > > > >>
> > > > > >> And put the ThreadPoolSynchronizationContext class somewhere:
> > > > > >> class ThreadPoolSynchronizationContext : SynchronizationContext
> > > > > >> {
> > > > > >> // No-op.
> > > > > >> }
> > > > > >>
> > > > > >>
> > > > > >> Now, detailed explanation. The problem exists forever in Ignite
> > and
> > > is
> > > > > >> mentioned in the docs briefly [1].
> > > > > >> Also mentioned in .NET docs (I've updated them a bit) [2].
> > > > > >>
> > > > > >> Breakdown:
> > > > > >> * Ignite (Java side) runs async callbacks (continuations) on
> > system
> > > > > >> threads, and those threads have limitations (you should not call
> > > Ignite
> > > > > >> APIs from them in general)
> > > > > >> * Ignite.NET wraps async operations into native .NET Tasks
> > > > > >> * Usually `await ...` call in .NET will continue execution on
> the
> > > > > >> original Thread (simply put, actually it is more complex), so
> > Ignite
> > > > > system
> > > > > >> thread issue is avoided
> > > > > >> * However, Console applications have no
> `SynchronizationContext`,
> > > so the
> > > > > >> continuation can't be dispatched to 

New Branch created: ignite-2.7.6

[Dmitriy Pavlov]

Hi Igniters,

This email is for informing about new branch available: 2.7.6 (based on
2.7.5).

https://gitbox.apache.org/repos/asf?p=ignite.git;a=shortlog;h=refs/heads/ignite-2.7.6


The branch was set up in the GG's AI Teamcity Bot instance as triggered and
tracked:
https://mtcga.gridgain.com/current.html?branch=ignite-2.7.6-nightly
(no builds yet).


Sincerely,
Dmitriy Pavlov

Re: [MTCGA]: new failures in builds [4480005] needs to be handled

[Dmitriy Pavlov]

The test is now Ignored. Thanks Ivan for fixing this

пт, 9 авг. 2019 г. в 07:03, :

> Hi Igniters,
>
>  I've detected some new issue on TeamCity to be handled. You are more than
> welcomed to help.
>
>  If your changes can lead to this failure(s): We're grateful that you were
> a volunteer to make the contribution to this project, but things change and
> you may no longer be able to finalize your contribution.
>  Could you respond to this email and indicate if you wish to continue and
> fix test failures or step down and some committer may revert you commit.
>
>  *Recently contributed test failed in master
> IgniteNodeStoppedDuringDisableWALTest.test[AFTER_DISABLE_WAL]
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-4415134685125470652=%3Cdefault%3E=testDetails
>  Changes may lead to failure were done by
>  - dmitriy govorukhin 
> https://ci.ignite.apache.org/viewModification.html?modId=889238
>
>  - Here's a reminder of what contributors were agreed to do
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
>  - Should you have any questions please contact
> dev@ignite.apache.org
>
> Best Regards,
> Apache Ignite TeamCity Bot
> https://github.com/apache/ignite-teamcity-bot
> Notification generated at 07:03:39 09-08-2019
>

Re: SecurityTestSuite as a separate test suite at TC

[Denis Garus]

Excuse me! I was wrong.
I try to find that parameter on Step 4: Run test suite.
One more time, thank you!

пт, 9 авг. 2019 г. в 14:05, Petr Ivanov :

> Why do you think I did not use it?
>
>
> On 9 Aug 2019, at 13:25, Denis Garus  wrote:
>
> Great!
> Could you please add surefire-fork-count-1 to additional Maven command
> line parameters?
> It's crucial.
>
> Thank you!
>
> пт, 9 авг. 2019 г. в 12:42, Petr Ivanov :
>
>> Done [1]
>>
>>
>> [1] https://ci.ignite.apache.org/viewLog.html?buildId=4482200
>>
>> On 9 Aug 2019, at 12:02, Denis Garus  wrote:
>>
>> Sure! I created the task [1].
>>
>> Thank you!
>>
>> 1. https://issues.apache.org/jira/browse/IGNITE-12055
>>
>> пт, 9 авг. 2019 г. в 11:38, Petr Ivanov :
>>
>>> Hi, Denis!
>>>
>>>
>>> Could file a ticket with description, please?
>>>
>>> On 9 Aug 2019, at 11:35, Denis Garus  wrote:
>>>
>>> Thanks all for the feedback!
>>>
>>> I think no one is against of proposal.
>>>
>>> Petr, could you please assist with wit separation of SecurityTestSuite?
>>>
>>> чт, 8 авг. 2019 г. в 14:43, Denis Garus :
>>>
 Hello, Ivan!

 >> Could you please provide more details why do we need to run these
 tests in forked JVM?

 Surefite documentation [1] says:
 If forkCount=0, it's impossible to use the system class loader or a
 plain old Java classpath; we have to use an isolated class loader.

 When using isolated class loader will cause compiler error:
 package org.apache.ignite.lang does not exist

 We cannot compile the TestIgniteCallable class.

 1.
 https://maven.apache.org/surefire/maven-surefire-plugin/examples/class-loading.html

 чт, 8 авг. 2019 г. в 09:44, Павлухин Иван :

> Denis,
>
> Could you please provide more details why do we need to run these
> tests in forked JVM?
>
> Still, having separate security suite on TC sounds not bad.
>
> ср, 7 авг. 2019 г. в 09:35, Vyacheslav Daradur :
> >
> > Hi Denis.
> >
> > I think it is fine to extract security tests in a separate build
> plan on TC.
> >
> > BTW, if you are going to write a lot of Sandbox's tests pay attention
> > to 'extdata' module and an approach of P2P tests
> > (IgniteP2PSelfTestSuite) - this may help you to avoid Maven's
> > classloading issues.
> >
> > On Tue, Aug 6, 2019 at 3:25 PM Denis Garus 
> wrote:
> > >
> > > Hello Igniters!
> > >
> > > I made the test DoPrivelegedOnRemoteNodeTest[1]
> (SecurityTestSuite) for the
> > > task "Sandbox for user-defined code" [2]
> > > that uses p2p deploy like the test
> > > ServiceHotRedeploymentViaDeploymentSpiTest [3] from
> > > IgniteServiceGridTestSuite.
> > > That test requires additional Maven command line parameter -P
> > > surefire-fork-count-1.
> > > The suite Basic 1 contains the SecurityTestSuite and many other
> test suites
> > > at TeamCity that do not need that additional Maven parameter.
> > > I suggest extracting SecurityTestSuite as a separate test suite to
> define
> > > additional Maven command line parameter for it.
> > >
> > > WDYT?
> > >
> > >
> > > 1. https://github.com/apache/ignite/pull/6707
> > > 2. https://issues.apache.org/jira/browse/IGNITE-11410
> > > 3.
> > >
> https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/processors/service/ServiceHotRedeploymentViaDeploymentSpiTest.java
> >
> >
> >
> > --
> > Best Regards, Vyacheslav D.
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

>>>
>>
>

Re: SecurityTestSuite as a separate test suite at TC

[Petr Ivanov]

Why do you think I did not use it?


> On 9 Aug 2019, at 13:25, Denis Garus  wrote:
> 
> Great!
> Could you please add surefire-fork-count-1 to additional Maven command line 
> parameters?
> It's crucial.
> 
> Thank you!
> 
> пт, 9 авг. 2019 г. в 12:42, Petr Ivanov  >:
> Done [1]
> 
> 
> [1] https://ci.ignite.apache.org/viewLog.html?buildId=4482200 
> 
> 
>> On 9 Aug 2019, at 12:02, Denis Garus > > wrote:
>> 
>> Sure! I created the task [1].
>> 
>> Thank you!
>> 
>> 1. https://issues.apache.org/jira/browse/IGNITE-12055 
>> 
>> 
>> пт, 9 авг. 2019 г. в 11:38, Petr Ivanov > >:
>> Hi, Denis!
>> 
>> 
>> Could file a ticket with description, please?
>> 
>>> On 9 Aug 2019, at 11:35, Denis Garus >> > wrote:
>>> 
>>> Thanks all for the feedback!
>>> 
>>> I think no one is against of proposal.
>>> 
>>> Petr, could you please assist with wit separation of SecurityTestSuite?
>>> 
>>> чт, 8 авг. 2019 г. в 14:43, Denis Garus >> >:
>>> Hello, Ivan!
>>> 
>>> >> Could you please provide more details why do we need to run these tests 
>>> >> in forked JVM?
>>> 
>>> Surefite documentation [1] says:
>>> If forkCount=0, it's impossible to use the system class loader or a plain 
>>> old Java classpath; we have to use an isolated class loader.
>>> 
>>> When using isolated class loader will cause compiler error: 
>>> package org.apache.ignite.lang does not exist
>>> 
>>> We cannot compile the TestIgniteCallable class.
>>> 
>>> 1. 
>>> https://maven.apache.org/surefire/maven-surefire-plugin/examples/class-loading.html
>>>  
>>> 
>>> 
>>> чт, 8 авг. 2019 г. в 09:44, Павлухин Иван >> >:
>>> Denis,
>>> 
>>> Could you please provide more details why do we need to run these
>>> tests in forked JVM?
>>> 
>>> Still, having separate security suite on TC sounds not bad.
>>> 
>>> ср, 7 авг. 2019 г. в 09:35, Vyacheslav Daradur >> >:
>>> >
>>> > Hi Denis.
>>> >
>>> > I think it is fine to extract security tests in a separate build plan on 
>>> > TC.
>>> >
>>> > BTW, if you are going to write a lot of Sandbox's tests pay attention
>>> > to 'extdata' module and an approach of P2P tests
>>> > (IgniteP2PSelfTestSuite) - this may help you to avoid Maven's
>>> > classloading issues.
>>> >
>>> > On Tue, Aug 6, 2019 at 3:25 PM Denis Garus >> > > wrote:
>>> > >
>>> > > Hello Igniters!
>>> > >
>>> > > I made the test DoPrivelegedOnRemoteNodeTest[1] (SecurityTestSuite) for 
>>> > > the
>>> > > task "Sandbox for user-defined code" [2]
>>> > > that uses p2p deploy like the test
>>> > > ServiceHotRedeploymentViaDeploymentSpiTest [3] from
>>> > > IgniteServiceGridTestSuite.
>>> > > That test requires additional Maven command line parameter -P
>>> > > surefire-fork-count-1.
>>> > > The suite Basic 1 contains the SecurityTestSuite and many other test 
>>> > > suites
>>> > > at TeamCity that do not need that additional Maven parameter.
>>> > > I suggest extracting SecurityTestSuite as a separate test suite to 
>>> > > define
>>> > > additional Maven command line parameter for it.
>>> > >
>>> > > WDYT?
>>> > >
>>> > >
>>> > > 1. https://github.com/apache/ignite/pull/6707 
>>> > > 
>>> > > 2. https://issues.apache.org/jira/browse/IGNITE-11410 
>>> > > 
>>> > > 3.
>>> > > https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/processors/service/ServiceHotRedeploymentViaDeploymentSpiTest.java
>>> > >  
>>> > > 
>>> >
>>> >
>>> >
>>> > --
>>> > Best Regards, Vyacheslav D.
>>> 
>>> 
>>> 
>>> -- 
>>> Best regards,
>>> Ivan Pavlukhin
>> 
> 

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

[Ilya Kasnacheev]

Hello!

Why can't we dispatch those callbacks in the caller thread?

Regards,
-- 
Ilya Kasnacheev


пт, 9 авг. 2019 г. в 09:50, Pavel Tupitsyn :

> Иван,
>
> The fix is to dispatch those callbacks (future listeners) to a different
> thread pool, not sure which one though.
> If I would do a .NET-only fix, I would use the default thread pool (non
> Ignite-specific), for Java-side there is no such thing as I understand.
>
> Yes, let's have a ticket to track the issue.
>
> On Fri, Aug 9, 2019 at 9:17 AM Павлухин Иван  wrote:
>
> > Ilya, Pavel,
> >
> > Do we a have a proposal how to fix the root cause of the problem?
> > Should we a have a ticket for it?
> >
> > ср, 7 авг. 2019 г. в 17:48, Ilya Kasnacheev :
> > >
> > > Hello!
> > >
> > > I think we should definitely stop running futures out of striped pool,
> > > while holding any cache logs (stripe thread counts as one).
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > ср, 7 авг. 2019 г. в 17:20, Pavel Tupitsyn :
> > >
> > > > Yes, this can be done purely on .NET side, which is an option that I
> > > > consider.
> > > > However, the root problem is on Java side, and I believe that we
> > should fix
> > > > the root problem.
> > > >
> > > > > violate some of Ignite assumptions: that we never run user code
> from
> > > > certain thread pools
> > > > We actually do run user code from Ignite thread pools:
> > > >
> > > > cache.getAsync(1).listen(fut ->
> > > > System.out.println("Get operation completed [value=" + fut.get()
> +
> > > > ']'));
> > > >
> > > > `println` here is executed on the striped pool. This is stated in the
> > > > docs that I linked above.
> > > >
> > > > Users have to be aware of this and they have to be very careful with
> > > > every future listener. IMO, this is a tricky gotcha and a bad
> > > > usability.
> > > >
> > > >
> > > > Thoughts?
> > > >
> > > >
> > > > On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev <
> > ilya.kasnach...@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hello!
> > > > >
> > > > > + dev@
> > > > >
> > > > > I think the current behavior, where .Net callbacks may be run from
> > > > striped
> > > > > pool, violate some of Ignite assumptions: that we never run user
> code
> > > > from
> > > > > certain thread pools (like sys-stripe) and that we try to limit
> > options
> > > > of
> > > > > running user-supplied code from our internals.
> > > > >
> > > > > I think that future versions of .Net integration should remove the
> > > > ability
> > > > > of async callbacks to be called from non-user threads, even if it
> can
> > > > lead
> > > > > to performance degradation in some cases. I suggest removing this
> > mode,
> > > > if
> > > > > possible, while keeping only the safe one, where internal threads
> > are not
> > > > > waiting upon completion of user code.
> > > > >
> > > > > In this case my issue IGNITE-12033 could be used to track this
> work.
> > > > >
> > > > > WDYT?
> > > > >
> > > > > Regards,
> > > > > --
> > > > > Ilya Kasnacheev
> > > > >
> > > > >
> > > > > ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn :
> > > > >
> > > > >> Sorry guys, I've completely missed this thread, and the topic is
> > very
> > > > >> important.
> > > > >>
> > > > >> First, a simple fix for the given example. Add the following on
> the
> > > > first
> > > > >> line of Main:
> > > > >> SynchronizationContext.SetSynchronizationContext(new
> > > > >> ThreadPoolSynchronizationContext());
> > > > >>
> > > > >> And put the ThreadPoolSynchronizationContext class somewhere:
> > > > >> class ThreadPoolSynchronizationContext : SynchronizationContext
> > > > >> {
> > > > >> // No-op.
> > > > >> }
> > > > >>
> > > > >>
> > > > >> Now, detailed explanation. The problem exists forever in Ignite
> and
> > is
> > > > >> mentioned in the docs briefly [1].
> > > > >> Also mentioned in .NET docs (I've updated them a bit) [2].
> > > > >>
> > > > >> Breakdown:
> > > > >> * Ignite (Java side) runs async callbacks (continuations) on
> system
> > > > >> threads, and those threads have limitations (you should not call
> > Ignite
> > > > >> APIs from them in general)
> > > > >> * Ignite.NET wraps async operations into native .NET Tasks
> > > > >> * Usually `await ...` call in .NET will continue execution on the
> > > > >> original Thread (simply put, actually it is more complex), so
> Ignite
> > > > system
> > > > >> thread issue is avoided
> > > > >> * However, Console applications have no `SynchronizationContext`,
> > so the
> > > > >> continuation can't be dispatched to original thread, and is
> > executed on
> > > > >> current (Ignite) thread
> > > > >> * Setting custom SynchronizationContext fixes the issue: all async
> > > > >> continuations will be dispatched to .NET thread pool and never
> > executed
> > > > on
> > > > >> Ignite threads
> > > > >>
> > > > >> However, dispatching callbacks to a different thread causes
> > performance
> > > > >> hit, and Ignite favors performance over usability right now.
> > > > >> So 

Re: SecurityTestSuite as a separate test suite at TC

[Denis Garus]

Great!
Could you please add surefire-fork-count-1 to additional Maven command line
parameters?
It's crucial.

Thank you!

пт, 9 авг. 2019 г. в 12:42, Petr Ivanov :

> Done [1]
>
>
> [1] https://ci.ignite.apache.org/viewLog.html?buildId=4482200
>
> On 9 Aug 2019, at 12:02, Denis Garus  wrote:
>
> Sure! I created the task [1].
>
> Thank you!
>
> 1. https://issues.apache.org/jira/browse/IGNITE-12055
>
> пт, 9 авг. 2019 г. в 11:38, Petr Ivanov :
>
>> Hi, Denis!
>>
>>
>> Could file a ticket with description, please?
>>
>> On 9 Aug 2019, at 11:35, Denis Garus  wrote:
>>
>> Thanks all for the feedback!
>>
>> I think no one is against of proposal.
>>
>> Petr, could you please assist with wit separation of SecurityTestSuite?
>>
>> чт, 8 авг. 2019 г. в 14:43, Denis Garus :
>>
>>> Hello, Ivan!
>>>
>>> >> Could you please provide more details why do we need to run these
>>> tests in forked JVM?
>>>
>>> Surefite documentation [1] says:
>>> If forkCount=0, it's impossible to use the system class loader or a
>>> plain old Java classpath; we have to use an isolated class loader.
>>>
>>> When using isolated class loader will cause compiler error:
>>> package org.apache.ignite.lang does not exist
>>>
>>> We cannot compile the TestIgniteCallable class.
>>>
>>> 1.
>>> https://maven.apache.org/surefire/maven-surefire-plugin/examples/class-loading.html
>>>
>>> чт, 8 авг. 2019 г. в 09:44, Павлухин Иван :
>>>
 Denis,

 Could you please provide more details why do we need to run these
 tests in forked JVM?

 Still, having separate security suite on TC sounds not bad.

 ср, 7 авг. 2019 г. в 09:35, Vyacheslav Daradur :
 >
 > Hi Denis.
 >
 > I think it is fine to extract security tests in a separate build plan
 on TC.
 >
 > BTW, if you are going to write a lot of Sandbox's tests pay attention
 > to 'extdata' module and an approach of P2P tests
 > (IgniteP2PSelfTestSuite) - this may help you to avoid Maven's
 > classloading issues.
 >
 > On Tue, Aug 6, 2019 at 3:25 PM Denis Garus 
 wrote:
 > >
 > > Hello Igniters!
 > >
 > > I made the test DoPrivelegedOnRemoteNodeTest[1] (SecurityTestSuite)
 for the
 > > task "Sandbox for user-defined code" [2]
 > > that uses p2p deploy like the test
 > > ServiceHotRedeploymentViaDeploymentSpiTest [3] from
 > > IgniteServiceGridTestSuite.
 > > That test requires additional Maven command line parameter -P
 > > surefire-fork-count-1.
 > > The suite Basic 1 contains the SecurityTestSuite and many other
 test suites
 > > at TeamCity that do not need that additional Maven parameter.
 > > I suggest extracting SecurityTestSuite as a separate test suite to
 define
 > > additional Maven command line parameter for it.
 > >
 > > WDYT?
 > >
 > >
 > > 1. https://github.com/apache/ignite/pull/6707
 > > 2. https://issues.apache.org/jira/browse/IGNITE-11410
 > > 3.
 > >
 https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/processors/service/ServiceHotRedeploymentViaDeploymentSpiTest.java
 >
 >
 >
 > --
 > Best Regards, Vyacheslav D.



 --
 Best regards,
 Ivan Pavlukhin

>>>
>>
>

[jira] [Created] (IGNITE-12056) Index creation sequence may influence on execution plan.

[Stanilovsky Evgeny (JIRA)]

Stanilovsky Evgeny created IGNITE-12056:
---

 Summary: Index creation sequence may influence on execution plan.
 Key: IGNITE-12056
 URL: https://issues.apache.org/jira/browse/IGNITE-12056
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.7, 2.5
Reporter: Stanilovsky Evgeny
 Attachments: index.diff

Different index creation, for example on new attached nodes, may breaks 
expected index usage. Reproducer in attach.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: SecurityTestSuite as a separate test suite at TC

[Petr Ivanov]

Done [1]


[1] https://ci.ignite.apache.org/viewLog.html?buildId=4482200

> On 9 Aug 2019, at 12:02, Denis Garus  wrote:
> 
> Sure! I created the task [1].
> 
> Thank you!
> 
> 1. https://issues.apache.org/jira/browse/IGNITE-12055 
> 
> 
> пт, 9 авг. 2019 г. в 11:38, Petr Ivanov  >:
> Hi, Denis!
> 
> 
> Could file a ticket with description, please?
> 
>> On 9 Aug 2019, at 11:35, Denis Garus > > wrote:
>> 
>> Thanks all for the feedback!
>> 
>> I think no one is against of proposal.
>> 
>> Petr, could you please assist with wit separation of SecurityTestSuite?
>> 
>> чт, 8 авг. 2019 г. в 14:43, Denis Garus > >:
>> Hello, Ivan!
>> 
>> >> Could you please provide more details why do we need to run these tests 
>> >> in forked JVM?
>> 
>> Surefite documentation [1] says:
>> If forkCount=0, it's impossible to use the system class loader or a plain 
>> old Java classpath; we have to use an isolated class loader.
>> 
>> When using isolated class loader will cause compiler error: 
>> package org.apache.ignite.lang does not exist
>> 
>> We cannot compile the TestIgniteCallable class.
>> 
>> 1. 
>> https://maven.apache.org/surefire/maven-surefire-plugin/examples/class-loading.html
>>  
>> 
>> 
>> чт, 8 авг. 2019 г. в 09:44, Павлухин Иван > >:
>> Denis,
>> 
>> Could you please provide more details why do we need to run these
>> tests in forked JVM?
>> 
>> Still, having separate security suite on TC sounds not bad.
>> 
>> ср, 7 авг. 2019 г. в 09:35, Vyacheslav Daradur > >:
>> >
>> > Hi Denis.
>> >
>> > I think it is fine to extract security tests in a separate build plan on 
>> > TC.
>> >
>> > BTW, if you are going to write a lot of Sandbox's tests pay attention
>> > to 'extdata' module and an approach of P2P tests
>> > (IgniteP2PSelfTestSuite) - this may help you to avoid Maven's
>> > classloading issues.
>> >
>> > On Tue, Aug 6, 2019 at 3:25 PM Denis Garus > > > wrote:
>> > >
>> > > Hello Igniters!
>> > >
>> > > I made the test DoPrivelegedOnRemoteNodeTest[1] (SecurityTestSuite) for 
>> > > the
>> > > task "Sandbox for user-defined code" [2]
>> > > that uses p2p deploy like the test
>> > > ServiceHotRedeploymentViaDeploymentSpiTest [3] from
>> > > IgniteServiceGridTestSuite.
>> > > That test requires additional Maven command line parameter -P
>> > > surefire-fork-count-1.
>> > > The suite Basic 1 contains the SecurityTestSuite and many other test 
>> > > suites
>> > > at TeamCity that do not need that additional Maven parameter.
>> > > I suggest extracting SecurityTestSuite as a separate test suite to define
>> > > additional Maven command line parameter for it.
>> > >
>> > > WDYT?
>> > >
>> > >
>> > > 1. https://github.com/apache/ignite/pull/6707 
>> > > 
>> > > 2. https://issues.apache.org/jira/browse/IGNITE-11410 
>> > > 
>> > > 3.
>> > > https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/processors/service/ServiceHotRedeploymentViaDeploymentSpiTest.java
>> > >  
>> > > 
>> >
>> >
>> >
>> > --
>> > Best Regards, Vyacheslav D.
>> 
>> 
>> 
>> -- 
>> Best regards,
>> Ivan Pavlukhin
> 

Re: SecurityTestSuite as a separate test suite at TC

[Denis Garus]

Sure! I created the task [1].


Thank you!


1. https://issues.apache.org/jira/browse/IGNITE-12055

пт, 9 авг. 2019 г. в 11:38, Petr Ivanov :

> Hi, Denis!
>
>
> Could file a ticket with description, please?
>
> On 9 Aug 2019, at 11:35, Denis Garus  wrote:
>
> Thanks all for the feedback!
>
> I think no one is against of proposal.
>
> Petr, could you please assist with wit separation of SecurityTestSuite?
>
> чт, 8 авг. 2019 г. в 14:43, Denis Garus :
>
>> Hello, Ivan!
>>
>> >> Could you please provide more details why do we need to run these
>> tests in forked JVM?
>>
>> Surefite documentation [1] says:
>> If forkCount=0, it's impossible to use the system class loader or a plain
>> old Java classpath; we have to use an isolated class loader.
>>
>> When using isolated class loader will cause compiler error:
>> package org.apache.ignite.lang does not exist
>>
>> We cannot compile the TestIgniteCallable class.
>>
>> 1.
>> https://maven.apache.org/surefire/maven-surefire-plugin/examples/class-loading.html
>>
>> чт, 8 авг. 2019 г. в 09:44, Павлухин Иван :
>>
>>> Denis,
>>>
>>> Could you please provide more details why do we need to run these
>>> tests in forked JVM?
>>>
>>> Still, having separate security suite on TC sounds not bad.
>>>
>>> ср, 7 авг. 2019 г. в 09:35, Vyacheslav Daradur :
>>> >
>>> > Hi Denis.
>>> >
>>> > I think it is fine to extract security tests in a separate build plan
>>> on TC.
>>> >
>>> > BTW, if you are going to write a lot of Sandbox's tests pay attention
>>> > to 'extdata' module and an approach of P2P tests
>>> > (IgniteP2PSelfTestSuite) - this may help you to avoid Maven's
>>> > classloading issues.
>>> >
>>> > On Tue, Aug 6, 2019 at 3:25 PM Denis Garus 
>>> wrote:
>>> > >
>>> > > Hello Igniters!
>>> > >
>>> > > I made the test DoPrivelegedOnRemoteNodeTest[1] (SecurityTestSuite)
>>> for the
>>> > > task "Sandbox for user-defined code" [2]
>>> > > that uses p2p deploy like the test
>>> > > ServiceHotRedeploymentViaDeploymentSpiTest [3] from
>>> > > IgniteServiceGridTestSuite.
>>> > > That test requires additional Maven command line parameter -P
>>> > > surefire-fork-count-1.
>>> > > The suite Basic 1 contains the SecurityTestSuite and many other test
>>> suites
>>> > > at TeamCity that do not need that additional Maven parameter.
>>> > > I suggest extracting SecurityTestSuite as a separate test suite to
>>> define
>>> > > additional Maven command line parameter for it.
>>> > >
>>> > > WDYT?
>>> > >
>>> > >
>>> > > 1. https://github.com/apache/ignite/pull/6707
>>> > > 2. https://issues.apache.org/jira/browse/IGNITE-11410
>>> > > 3.
>>> > >
>>> https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/processors/service/ServiceHotRedeploymentViaDeploymentSpiTest.java
>>> >
>>> >
>>> >
>>> > --
>>> > Best Regards, Vyacheslav D.
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Ivan Pavlukhin
>>>
>>
>

[jira] [Created] (IGNITE-12055) SecurityTestSuite as a separate test suite at TC

[Denis Garus (JIRA)]

Denis Garus created IGNITE-12055:


 Summary: SecurityTestSuite as a separate test suite at TC
 Key: IGNITE-12055
 URL: https://issues.apache.org/jira/browse/IGNITE-12055
 Project: Ignite
  Issue Type: Task
  Components: security
Reporter: Denis Garus


Need exclude SecurityTestSuite from IgniteBasicTestSuite (Basic 1).

Create suite Security with SecurityTestSuite at TC. Config is the same as Basic 
1 except 
additional Maven command line parameters that should be: 
-P 
all-java,all-other,scala-2.10,tensorflow,scala,scala-test,surefire-fork-count-1 
-pl %MAVEN_MODULES% -am -Dmaven.test.failure.ignore=true -DfailIfNoTests=false 
-Dtest=%TEST_SUITE% -Dmaven.javadoc.skip=true %MAVEN_OPTS%


Include SecurityTestSuite to Run::All, Run::All (Nightly), Run::Basic Tests at 
TC.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: SecurityTestSuite as a separate test suite at TC

[Petr Ivanov]

Hi, Denis!


Could file a ticket with description, please?

> On 9 Aug 2019, at 11:35, Denis Garus  wrote:
> 
> Thanks all for the feedback!
> 
> I think no one is against of proposal.
> 
> Petr, could you please assist with wit separation of SecurityTestSuite?
> 
> чт, 8 авг. 2019 г. в 14:43, Denis Garus  >:
> Hello, Ivan!
> 
> >> Could you please provide more details why do we need to run these tests in 
> >> forked JVM?
> 
> Surefite documentation [1] says:
> If forkCount=0, it's impossible to use the system class loader or a plain old 
> Java classpath; we have to use an isolated class loader.
> 
> When using isolated class loader will cause compiler error: 
> package org.apache.ignite.lang does not exist
> 
> We cannot compile the TestIgniteCallable class.
> 
> 1. 
> https://maven.apache.org/surefire/maven-surefire-plugin/examples/class-loading.html
>  
> 
> 
> чт, 8 авг. 2019 г. в 09:44, Павлухин Иван  >:
> Denis,
> 
> Could you please provide more details why do we need to run these
> tests in forked JVM?
> 
> Still, having separate security suite on TC sounds not bad.
> 
> ср, 7 авг. 2019 г. в 09:35, Vyacheslav Daradur  >:
> >
> > Hi Denis.
> >
> > I think it is fine to extract security tests in a separate build plan on TC.
> >
> > BTW, if you are going to write a lot of Sandbox's tests pay attention
> > to 'extdata' module and an approach of P2P tests
> > (IgniteP2PSelfTestSuite) - this may help you to avoid Maven's
> > classloading issues.
> >
> > On Tue, Aug 6, 2019 at 3:25 PM Denis Garus  > > wrote:
> > >
> > > Hello Igniters!
> > >
> > > I made the test DoPrivelegedOnRemoteNodeTest[1] (SecurityTestSuite) for 
> > > the
> > > task "Sandbox for user-defined code" [2]
> > > that uses p2p deploy like the test
> > > ServiceHotRedeploymentViaDeploymentSpiTest [3] from
> > > IgniteServiceGridTestSuite.
> > > That test requires additional Maven command line parameter -P
> > > surefire-fork-count-1.
> > > The suite Basic 1 contains the SecurityTestSuite and many other test 
> > > suites
> > > at TeamCity that do not need that additional Maven parameter.
> > > I suggest extracting SecurityTestSuite as a separate test suite to define
> > > additional Maven command line parameter for it.
> > >
> > > WDYT?
> > >
> > >
> > > 1. https://github.com/apache/ignite/pull/6707 
> > > 
> > > 2. https://issues.apache.org/jira/browse/IGNITE-11410 
> > > 
> > > 3.
> > > https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/processors/service/ServiceHotRedeploymentViaDeploymentSpiTest.java
> > >  
> > > 
> >
> >
> >
> > --
> > Best Regards, Vyacheslav D.
> 
> 
> 
> -- 
> Best regards,
> Ivan Pavlukhin

Re: SecurityTestSuite as a separate test suite at TC

[Denis Garus]

Thanks all for the feedback!

I think no one is against of proposal.

Petr, could you please assist with wit separation of SecurityTestSuite?

чт, 8 авг. 2019 г. в 14:43, Denis Garus :

> Hello, Ivan!
>
> >> Could you please provide more details why do we need to run these tests
> in forked JVM?
>
> Surefite documentation [1] says:
> If forkCount=0, it's impossible to use the system class loader or a plain
> old Java classpath; we have to use an isolated class loader.
>
> When using isolated class loader will cause compiler error:
> package org.apache.ignite.lang does not exist
>
> We cannot compile the TestIgniteCallable class.
>
> 1.
> https://maven.apache.org/surefire/maven-surefire-plugin/examples/class-loading.html
>
> чт, 8 авг. 2019 г. в 09:44, Павлухин Иван :
>
>> Denis,
>>
>> Could you please provide more details why do we need to run these
>> tests in forked JVM?
>>
>> Still, having separate security suite on TC sounds not bad.
>>
>> ср, 7 авг. 2019 г. в 09:35, Vyacheslav Daradur :
>> >
>> > Hi Denis.
>> >
>> > I think it is fine to extract security tests in a separate build plan
>> on TC.
>> >
>> > BTW, if you are going to write a lot of Sandbox's tests pay attention
>> > to 'extdata' module and an approach of P2P tests
>> > (IgniteP2PSelfTestSuite) - this may help you to avoid Maven's
>> > classloading issues.
>> >
>> > On Tue, Aug 6, 2019 at 3:25 PM Denis Garus  wrote:
>> > >
>> > > Hello Igniters!
>> > >
>> > > I made the test DoPrivelegedOnRemoteNodeTest[1] (SecurityTestSuite)
>> for the
>> > > task "Sandbox for user-defined code" [2]
>> > > that uses p2p deploy like the test
>> > > ServiceHotRedeploymentViaDeploymentSpiTest [3] from
>> > > IgniteServiceGridTestSuite.
>> > > That test requires additional Maven command line parameter -P
>> > > surefire-fork-count-1.
>> > > The suite Basic 1 contains the SecurityTestSuite and many other test
>> suites
>> > > at TeamCity that do not need that additional Maven parameter.
>> > > I suggest extracting SecurityTestSuite as a separate test suite to
>> define
>> > > additional Maven command line parameter for it.
>> > >
>> > > WDYT?
>> > >
>> > >
>> > > 1. https://github.com/apache/ignite/pull/6707
>> > > 2. https://issues.apache.org/jira/browse/IGNITE-11410
>> > > 3.
>> > >
>> https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/processors/service/ServiceHotRedeploymentViaDeploymentSpiTest.java
>> >
>> >
>> >
>> > --
>> > Best Regards, Vyacheslav D.
>>
>>
>>
>> --
>> Best regards,
>> Ivan Pavlukhin
>>
>

Re: Apache Ignite 2.7.6 (Time, Scope, and Release manager)

[Nikolay Izhikov]

Hello, Deni.

> Nickolay, could you check if that's a quick upgrade?

Yes, I will take a look.


В Чт, 08/08/2019 в 11:08 -0700, Denis Magda пишет:
> Dmitry,
> 
> Please add this BTree corruption fix to the scope:
> https://issues.apache.org/jira/browse/IGNITE-11953
> 
> Plus, I would upgrade our Spark integration to version 2.4 as long as 2.3
> goes with limitations reported by our users:
> https://issues.apache.org/jira/browse/IGNITE-12054
> 
> Nickolay, could you check if that's a quick upgrade?
> 
> -
> Denis
> 
> 
> On Thu, Aug 8, 2019 at 10:40 AM Dmitriy Pavlov  wrote:
> 
> > Hi Ivan, Ilya, Igniters,
> > 
> > 
> > 
> > I would like this release would be as minimal as possible.
> > 
> > 
> > 
> > According to dates proposed we could freeze scope at 12.08, 4 days is more
> > than enough to stand up and say, ‘Hey, I have an urgent fix’. But it is
> > also ok for me if we decide to have more relaxed dates.
> > 
> > 
> > 
> > For now, I suppose the following fixed should be cherry-picked:
> > 
> > https://issues.apache.org/jira/browse/IGNITE-11767 (Blocker)
> > GridDhtPartitionsFullMessage retains huge maps on heap in exchange history
> > 
> > 
> > 
> > https://issues.apache.org/jira/browse/IGNITE-10451 (Major Bug) .NET:
> > Persistence does not work with custom affinity function
> > 
> > 
> > 
> > https://issues.apache.org/jira/browse/IGNITE-9562 (Critical Bug) Destroyed
> > cache that resurrected on an old offline node breaks PME
> > 
> > 
> > 
> > But I will continue to research JIRA.
> > 
> > 
> > 
> > Sincerely,
> > 
> > Dmitriy Pavlov
> > 
> > 
> > чт, 8 авг. 2019 г. в 17:30, Павлухин Иван :
> > 
> > > > What's the scope for this release?
> > > 
> > > Same question.
> > > 
> > > On the other hand an idea of 2.7.6 release attracts me because having
> > > a practice of doing frequent minor releases can help us to build
> > > reliable and predictable release rails.
> > > 
> > > чт, 8 авг. 2019 г. в 15:09, Ilya Kasnacheev :
> > > > 
> > > > Hello!
> > > > 
> > > > What's the scope for this release?
> > > > 
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > > 
> > > > 
> > > > чт, 8 авг. 2019 г. в 15:07, Dmitriy Pavlov :
> > > > 
> > > > > Hi Apache Ignite Developers,
> > > > > 
> > > > > 
> > > > > 
> > > > > We seem to be on the same page about 2.8 release, but we’ve started
> > 
> > new
> > > > > practice - minor releases, the first release was 2.7.5.
> > 
> > Unfortunately,
> > > > > there is a couple of issues still not fixed in that release, so I
> > 
> > would
> > > > > like to propose to prepare one more minor release for Apache Ignite.
> > > > > 
> > > > > 
> > > > > 
> > > > > I propose my candidacy to be release manager once again.
> > > > > 
> > > > > 
> > > > > 
> > > > > I could (of course with help from Peter Ivanov) do some additional
> > > 
> > > efforts
> > > > > to complete and improve process doc
> > > > > https://cwiki.apache.org/confluence/display/IGNITE/Release+Process
> > > > > 
> > > > > 
> > > > > 
> > > > > I propose (most optimistic) dates for release stages:
> > > > > 
> > > > > Scope Freeze: August 12, 2019
> > > > > 
> > > > > Code Freeze: August 15, 2019
> > > > > 
> > > > > Voting Date: August 22, 2019
> > > > > 
> > > > > Release Date: August 27, 2019
> > > > > 
> > > > > 
> > > > > WDYT?
> > > > > 
> > > > > 
> > > > > If nobody minds, I will create branch 2.7.6 based on 2.7.5 and set up
> > > 
> > > in
> > > > > the TC Bot during the weekend.
> > > > > 
> > > > > 
> > > > > 
> > > > > Sincerely,
> > > > > 
> > > > > Dmitriy Pavlov
> > > > > 
> > > 
> > > 
> > > 
> > > --
> > > Best regards,
> > > Ivan Pavlukhin
> > > 


signature.asc
Description: This is a digitally signed message part

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

[Pavel Tupitsyn]

Иван,

The fix is to dispatch those callbacks (future listeners) to a different
thread pool, not sure which one though.
If I would do a .NET-only fix, I would use the default thread pool (non
Ignite-specific), for Java-side there is no such thing as I understand.

Yes, let's have a ticket to track the issue.

On Fri, Aug 9, 2019 at 9:17 AM Павлухин Иван  wrote:

> Ilya, Pavel,
>
> Do we a have a proposal how to fix the root cause of the problem?
> Should we a have a ticket for it?
>
> ср, 7 авг. 2019 г. в 17:48, Ilya Kasnacheev :
> >
> > Hello!
> >
> > I think we should definitely stop running futures out of striped pool,
> > while holding any cache logs (stripe thread counts as one).
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 7 авг. 2019 г. в 17:20, Pavel Tupitsyn :
> >
> > > Yes, this can be done purely on .NET side, which is an option that I
> > > consider.
> > > However, the root problem is on Java side, and I believe that we
> should fix
> > > the root problem.
> > >
> > > > violate some of Ignite assumptions: that we never run user code from
> > > certain thread pools
> > > We actually do run user code from Ignite thread pools:
> > >
> > > cache.getAsync(1).listen(fut ->
> > > System.out.println("Get operation completed [value=" + fut.get() +
> > > ']'));
> > >
> > > `println` here is executed on the striped pool. This is stated in the
> > > docs that I linked above.
> > >
> > > Users have to be aware of this and they have to be very careful with
> > > every future listener. IMO, this is a tricky gotcha and a bad
> > > usability.
> > >
> > >
> > > Thoughts?
> > >
> > >
> > > On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hello!
> > > >
> > > > + dev@
> > > >
> > > > I think the current behavior, where .Net callbacks may be run from
> > > striped
> > > > pool, violate some of Ignite assumptions: that we never run user code
> > > from
> > > > certain thread pools (like sys-stripe) and that we try to limit
> options
> > > of
> > > > running user-supplied code from our internals.
> > > >
> > > > I think that future versions of .Net integration should remove the
> > > ability
> > > > of async callbacks to be called from non-user threads, even if it can
> > > lead
> > > > to performance degradation in some cases. I suggest removing this
> mode,
> > > if
> > > > possible, while keeping only the safe one, where internal threads
> are not
> > > > waiting upon completion of user code.
> > > >
> > > > In this case my issue IGNITE-12033 could be used to track this work.
> > > >
> > > > WDYT?
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn :
> > > >
> > > >> Sorry guys, I've completely missed this thread, and the topic is
> very
> > > >> important.
> > > >>
> > > >> First, a simple fix for the given example. Add the following on the
> > > first
> > > >> line of Main:
> > > >> SynchronizationContext.SetSynchronizationContext(new
> > > >> ThreadPoolSynchronizationContext());
> > > >>
> > > >> And put the ThreadPoolSynchronizationContext class somewhere:
> > > >> class ThreadPoolSynchronizationContext : SynchronizationContext
> > > >> {
> > > >> // No-op.
> > > >> }
> > > >>
> > > >>
> > > >> Now, detailed explanation. The problem exists forever in Ignite and
> is
> > > >> mentioned in the docs briefly [1].
> > > >> Also mentioned in .NET docs (I've updated them a bit) [2].
> > > >>
> > > >> Breakdown:
> > > >> * Ignite (Java side) runs async callbacks (continuations) on system
> > > >> threads, and those threads have limitations (you should not call
> Ignite
> > > >> APIs from them in general)
> > > >> * Ignite.NET wraps async operations into native .NET Tasks
> > > >> * Usually `await ...` call in .NET will continue execution on the
> > > >> original Thread (simply put, actually it is more complex), so Ignite
> > > system
> > > >> thread issue is avoided
> > > >> * However, Console applications have no `SynchronizationContext`,
> so the
> > > >> continuation can't be dispatched to original thread, and is
> executed on
> > > >> current (Ignite) thread
> > > >> * Setting custom SynchronizationContext fixes the issue: all async
> > > >> continuations will be dispatched to .NET thread pool and never
> executed
> > > on
> > > >> Ignite threads
> > > >>
> > > >> However, dispatching callbacks to a different thread causes
> performance
> > > >> hit, and Ignite favors performance over usability right now.
> > > >> So it is up to the user to configure desired behavior.
> > > >>
> > > >> Let me know if you need more details.
> > > >>
> > > >> Thanks
> > > >>
> > > >> [1] https://apacheignite.readme.io/docs/async-support
> > > >> [2] https://apacheignite-net.readme.io/docs/asynchronous-support
> > > >>
> > > >> On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev <
> > > ilya.kasnach...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Hello!
> > > >>>
> > > >>> I 

Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang

[Павлухин Иван]

Ilya, Pavel,

Do we a have a proposal how to fix the root cause of the problem?
Should we a have a ticket for it?

ср, 7 авг. 2019 г. в 17:48, Ilya Kasnacheev :
>
> Hello!
>
> I think we should definitely stop running futures out of striped pool,
> while holding any cache logs (stripe thread counts as one).
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 7 авг. 2019 г. в 17:20, Pavel Tupitsyn :
>
> > Yes, this can be done purely on .NET side, which is an option that I
> > consider.
> > However, the root problem is on Java side, and I believe that we should fix
> > the root problem.
> >
> > > violate some of Ignite assumptions: that we never run user code from
> > certain thread pools
> > We actually do run user code from Ignite thread pools:
> >
> > cache.getAsync(1).listen(fut ->
> > System.out.println("Get operation completed [value=" + fut.get() +
> > ']'));
> >
> > `println` here is executed on the striped pool. This is stated in the
> > docs that I linked above.
> >
> > Users have to be aware of this and they have to be very careful with
> > every future listener. IMO, this is a tricky gotcha and a bad
> > usability.
> >
> >
> > Thoughts?
> >
> >
> > On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev  > >
> > wrote:
> >
> > > Hello!
> > >
> > > + dev@
> > >
> > > I think the current behavior, where .Net callbacks may be run from
> > striped
> > > pool, violate some of Ignite assumptions: that we never run user code
> > from
> > > certain thread pools (like sys-stripe) and that we try to limit options
> > of
> > > running user-supplied code from our internals.
> > >
> > > I think that future versions of .Net integration should remove the
> > ability
> > > of async callbacks to be called from non-user threads, even if it can
> > lead
> > > to performance degradation in some cases. I suggest removing this mode,
> > if
> > > possible, while keeping only the safe one, where internal threads are not
> > > waiting upon completion of user code.
> > >
> > > In this case my issue IGNITE-12033 could be used to track this work.
> > >
> > > WDYT?
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn :
> > >
> > >> Sorry guys, I've completely missed this thread, and the topic is very
> > >> important.
> > >>
> > >> First, a simple fix for the given example. Add the following on the
> > first
> > >> line of Main:
> > >> SynchronizationContext.SetSynchronizationContext(new
> > >> ThreadPoolSynchronizationContext());
> > >>
> > >> And put the ThreadPoolSynchronizationContext class somewhere:
> > >> class ThreadPoolSynchronizationContext : SynchronizationContext
> > >> {
> > >> // No-op.
> > >> }
> > >>
> > >>
> > >> Now, detailed explanation. The problem exists forever in Ignite and is
> > >> mentioned in the docs briefly [1].
> > >> Also mentioned in .NET docs (I've updated them a bit) [2].
> > >>
> > >> Breakdown:
> > >> * Ignite (Java side) runs async callbacks (continuations) on system
> > >> threads, and those threads have limitations (you should not call Ignite
> > >> APIs from them in general)
> > >> * Ignite.NET wraps async operations into native .NET Tasks
> > >> * Usually `await ...` call in .NET will continue execution on the
> > >> original Thread (simply put, actually it is more complex), so Ignite
> > system
> > >> thread issue is avoided
> > >> * However, Console applications have no `SynchronizationContext`, so the
> > >> continuation can't be dispatched to original thread, and is executed on
> > >> current (Ignite) thread
> > >> * Setting custom SynchronizationContext fixes the issue: all async
> > >> continuations will be dispatched to .NET thread pool and never executed
> > on
> > >> Ignite threads
> > >>
> > >> However, dispatching callbacks to a different thread causes performance
> > >> hit, and Ignite favors performance over usability right now.
> > >> So it is up to the user to configure desired behavior.
> > >>
> > >> Let me know if you need more details.
> > >>
> > >> Thanks
> > >>
> > >> [1] https://apacheignite.readme.io/docs/async-support
> > >> [2] https://apacheignite-net.readme.io/docs/asynchronous-support
> > >>
> > >> On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev <
> > ilya.kasnach...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hello!
> > >>>
> > >>> I have filed a ticket about this issue so it won't get lost.
> > >>> https://issues.apache.org/jira/browse/IGNITE-12033
> > >>>
> > >>> Regards,
> > >>> --
> > >>> Ilya Kasnacheev
> > >>>
> > >>>
> > >>> чт, 2 мая 2019 г. в 10:53, Barney Pippin <
> > james.pri...@uk.bnpparibas.com
> > >>> >:
> > >>>
> >  Thanks for the response Ilya. Did you get a chance to look at this
> >  Pavel?
> >  Thanks.
> > 
> > 
> > 
> >  --
> >  Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> > 
> > >>>
> >



-- 
Best regards,
Ivan Pavlukhin

Re: SQL query timeout: in progress or abandoned

[Павлухин Иван]

Just for the protocol. There was an original dev-list discussion [1].
Added a link to the ticket as well.

[1] 
http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-7285-Add-default-query-timeout-td41828.html

пт, 9 авг. 2019 г. в 01:22, Denis Magda :
>
> Hey Saikat,
>
> Are you still working on this ticket?
> https://issues.apache.org/jira/browse/IGNITE-7285
>
> Seems that's the last API that doesn't support timeouts - JDBC and ODBC
> drivers already go with it.
>
> If you don't have time to complete the changes then someone else from the
> community can take over. We see a lot of demand for this API and here is
> one example:
> https://stackoverflow.com/questions/57275301/how-to-set-a-query-timeout-for-apache-ignite-cache
>
> -
> Denis



-- 
Best regards,
Ivan Pavlukhin