Re: [ML] IGNITE-9978 Compound Naive Bayes Pull Request

[Dmitriy Pavlov]

Hi Ravil,

Thank you for your efforts. I've triggered tests for this PR.

I'm sure that ML experts will take care of it. Please always feel free to
remind the community about submitted PR.

Some Apache projects have a weekly digest about JIRAs in PA state, maybe we
can adapt this technique. And for now, I hope to add a similar feature to
Apache Ignite TC Bot (including information about tests failures). Since
JIRA digest nor TC Bot feature are not available yet, forgotten PR may
occur from time to time.

Sincerely,
Dmitriy Pavlov

сб, 1 июн. 2019 г. в 00:12, Ravil Galeyev :

> Hi Team,
>
> A week ago I submitted a pull-request
>  for IGNITE-9978
>  but it still
> unreviewed.
> Can somebody take a look at it?
>
> Dear
> @zaleslaw @ybabak @avplatonov @dmitrievanthony
> I mentioned you in the PR, but it looks like I did something wrong,
> if you didn't receive notifications.
>
> If you are busy now just let me know and I'll proceed with another task.
>
> Best regards,
> Ravil
>

[ML] IGNITE-9978 Compound Naive Bayes Pull Request

[Ravil Galeyev]

Hi Team,

A week ago I submitted a pull-request
 for IGNITE-9978
 but it still unreviewed.
Can somebody take a look at it?

Dear
@zaleslaw @ybabak @avplatonov @dmitrievanthony
I mentioned you in the PR, but it looks like I did something wrong,
if you didn't receive notifications.

If you are busy now just let me know and I'll proceed with another task.

Best regards,
Ravil

Why TDE doc says that memory encryption is supported?

[Denis Magda]

Nikolay,

Do we really support encryption for pages in RAM? That's what I found in
the docs:

Ignite uses JDK-provided encryption algorithms: "AES/CBC/PKCS5Padding" for
WAL records encryption and *"AES/CBC/NoPadding" for memory page encryption.*

Shouldn't we remove the highlighted from the docs?
-
Denis

Re: [ML] Deployment of user-defined preprocessors

[Alexey Zinoviev]

I'd like this approach, can't wait to review your PR!

пт, 31 мая 2019 г. в 15:33, Алексей Платонов :

> No, I won't change them.
> In context of this architecture we should pass learning environment to
> ComputeUtils and this concerns just DatasetBuilders and ComputeUtils APIs.
> APIs of Preprocessors and Trainers won't be changed. We just set
> DeploymentContext in fit method but it doens't require to change APIs of
> these classes.
>
> Sincerely
> Alexey Platonov
>
> пт, 31 мая 2019 г. в 14:51, Alexey Zinoviev :
>
> > It sounds great, could you please explain what are you going to change in
> > the main Vectorizer/Preprocessor API to support it?
> >
> > пт, 31 мая 2019 г. в 14:20, Алексей Платонов :
> >
> > > Hi, Igniters!
> > > Currently we don't have an ability to deploy automatically user-defined
> > > preprocessors and vectorizers. Client's code should be deployed
> manually
> > to
> > > Ignite server nodes.
> > >
> > > I have an idea how to fix it. If we pass user's classloader and one of
> > > user-defined classes from fit-level to
> > > ComputeUtils.affinityCallWithRetries() then we wiil be able to use
> > > GridPeerDeployAware interface to send informtation about this
> classloader
> > > to server nodes.
> > >
> > > To support this ability we can define interfaces like these:
> > >
> > > public interface DeployableObject {
> > > public List getDependencies();
> > > }
> > >
> > > and
> > >
> > > public interface DeployingContext {
> > > public Class userClass();
> > > public ClassLoader clientClassLoader();
> > > }
> > >
> > > DeployableObject will be mark for our ignite-ml final classes like
> > trainers
> > > or concrete preprocessors and it can be able to return all dependencies
> > > that should be deployed to server nodes if it's needed. If these
> > > dependencies are DeployableObjects too then depenndencies will be
> > unfolded
> > > recursively. Classes that isn't defined as DeployableObject will be
> > > recognized as user-defined (NOTE: all leaf classes in our hierarchy
> will
> > be
> > > DeployableObject).
> > >
> > > This list of DeployableObjects will be user for define user class
> loader
> > > and one of these objects will be used for passing to
> GridPeerDeployAware.
> > >
> > > So, this logic allows to pass user-defined Preprocessors and
> Vectorizers
> > to
> > > training algorithms and pipelines.
> > >
> > > What do you think?
> > >
> > > Sincerely
> > > Alexey Platonov
> > >
> >
>

[jira] [Created] (IGNITE-11888) CacheContinuousQueryAsyncFailoverAtomicSelfTest.testFailoverStartStopBackup fails on GG TC

[Ivan Pavlukhin (JIRA)]

Ivan Pavlukhin created IGNITE-11888:
---

 Summary: 
CacheContinuousQueryAsyncFailoverAtomicSelfTest.testFailoverStartStopBackup 
fails on GG TC
 Key: IGNITE-11888
 URL: https://issues.apache.org/jira/browse/IGNITE-11888
 Project: Ignite
  Issue Type: Bug
  Components: cache
Reporter: Ivan Pavlukhin
Assignee: Ivan Pavlukhin


A test 
CacheContinuousQueryAsyncFailoverAtomicSelfTest.testFailoverStartStopBackup 
fails.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11887) Add more test scenarious for OWNING -> RENTING -> MOVING scenario

[Alexei Scherbakov (JIRA)]

Alexei Scherbakov created IGNITE-11887:
--

 Summary: Add more test scenarious for OWNING -> RENTING -> MOVING 
scenario
 Key: IGNITE-11887
 URL: https://issues.apache.org/jira/browse/IGNITE-11887
 Project: Ignite
  Issue Type: Test
Reporter: Alexei Scherbakov
Assignee: Alexei Scherbakov


Relevant test 
GridCacheRebalancingWithAsyncClearingTest#testCorrectRebalancingCurrentlyRentingPartitions



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: IgniteConfigVariationsAbstractTest subclasses do not run

[Ivan Fedotov]

Igniters,

I created corresponding tickets [1][2][3][4] in Jira and outlined
description of problems in a nutshell. In the context of the current ticket
(IGNITE-11708), I ignored them.

If some of us have a comprehension of the problem why tests are failed,
please let know here or in the tickets.

[1] https://issues.apache.org/jira/browse/IGNITE-11883
[2] https://issues.apache.org/jira/browse/IGNITE-11884
[3] https://issues.apache.org/jira/browse/IGNITE-11885
[4] https://issues.apache.org/jira/browse/IGNITE-11886



чт, 30 мая 2019 г. в 14:55, Ivan Fedotov :

> Hi Igniters!
>
> I found the solution for how to run IgniteConfigVariationsAbstractTest. I
> removed garbage injection [1] and static variable [2].
> Now assigning of VariationsTestsConfig proceeds in constructor class which
> is created dynamically [3].
>
> But I faced with the problem that a certain amount of tests are failed. It
> seems that they failed because of the real reasons, not because
> of the test workflow. Despite those fact that a big amount of tests on TC
> became red, really failed tests are not so much. Derivatives tests appear
> from
> different configs.
>
> Could some of us take a look on failed tests? I am going to ignore them in
> the context of the current ticket [5] and create separate
> issues.
>
> [1]
> https://github.com/apache/ignite/pull/6434/files#diff-cd3a07ce10f21d396c1eac0c328850e0L102
> [2]
> https://github.com/apache/ignite/pull/6434/files#diff-cd3a07ce10f21d396c1eac0c328850e0L84
> [3]
> https://github.com/apache/ignite/pull/6434/files#diff-3da5dbb6f22e5bf3bf18734933f1bfc5R434
> [4]
> https://ci.ignite.apache.org/viewLog.html?buildId=3978807=IgniteTests24Java8_RunAllNightly
> [5]https://issues.apache.org/jira/browse/IGNITE-11708
>
> вт, 14 мая 2019 г. в 16:31, Ivan Pavlukhina :
>
>> Ivan F.,
>>
>> Agree with referring tickets in @Ignore annotation.
>>
>> Currently I have no access to a computer. Will be able to look at updated
>> PR next week.
>>
>> Sent from my iPhone
>>
>> > On 14 May 2019, at 10:55, Ivan Fedotov  wrote:
>> >
>> > Ivan P.,
>> > I managed with IgniteConfigVariationsAbstractTest - now subclasses
>> enable
>> > [1].
>> > I ignored all the failed tests that were mentioned in our conversation.
>> If
>> > you don't mind, I can create appropriate tickets and mention them in
>> Ignore
>> > annotation.
>> >
>> > [1] https://github.com/apache/ignite/pull/6434/files
>> >
>> > ср, 1 мая 2019 г. в 15:29, Павлухин Иван :
>> >
>> >> Ivan F.,
>> >>
>> >> I think that it is better to enable IgniteConfigVariationsAbstractTest
>> >> subclasses first, so no new broken tests will appear. After that we
>> >> can fix IgniteConfigVariationsAbstractTest subclasses one by one in
>> >> separate ticket(s).
>> >>
>> >> вт, 30 апр. 2019 г. в 13:23, Ivan Fedotov :
>> >>>
>> >>> Ivan R., Ivan P., thank you for the suggestions, I took a look at
>> them.
>> >>>
>> >>> According to checking CacheAtomicityMode on null in
>> >>> IgniteCacheConfigVariationsAbstractTest#atomicityMode - are you sure
>> >>> that checking should proceed on test level? May be it will be better
>> to
>> >>> make default cache value in case if cc.getAtomicityMode() == null
>> >>> in CacheConfiguration constructor [1]?
>> >>>
>> >>> Also let me suggest one minor change according cache specification in
>> >>> IgniteCacheReadThroughEvictionSelfTest. The same logic is used in
>> >>> GridCacheAbstractSelfTest#cacheConfiguration [2].
>> >>> I think we can excract this code block in a separate static methods
>> >>> (initCacheConfig, for example) in
>> IgniteCacheReadThroughEvictionSelfTest
>> >> and
>> >>> invoke it in IgniteCacheReadThroughEvictionSelfTest#variationConfig
>> and
>> >>> GridCacheAbstractSelfTest#cacheConfiguration methods.
>> >>>
>> >>> In addition, I want to draw your attention to
>> >>> IgniteContinuousQueryConfigVariationsSuite test runs [3].
>> >>> CacheContinuousQueryVariationsTest are generated dynamically.
>> >>> The first batch (12 tests) works fine, but on the next runs we got
>> NPE in
>> >>> IgniteCacheConfigVariationsAbstractTest#afterTest - default cache does
>> >> not
>> >>> exist and we can not
>> >>> invoke unwrap() on this [4].
>> >>> Seems that the problem is in
>> >>>
>> >>
>> IgniteCacheConfigVariationsAbstractTest#beforeTestsStarted/afterTestsStopped
>> >>> methods, cache is not properly created (or already existed cache is
>> >>> destroyed).
>> >>>
>> >>> By the way, should I resolve these issues in the context of
>> IGNITE-11708
>> >> or
>> >>> create a separate ticket(s)?
>> >>>
>> >>> [1]
>> >>>
>> >>
>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/configuration/CacheConfiguration.java#L434
>> >>> [2]
>> >>>
>> >>
>> https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/GridCacheAbstractSelfTest.java#L235
>> >>> [3]
>> >>>
>> >>
>> 

Re: [ML] Deployment of user-defined preprocessors

[Алексей Платонов]

No, I won't change them.
In context of this architecture we should pass learning environment to
ComputeUtils and this concerns just DatasetBuilders and ComputeUtils APIs.
APIs of Preprocessors and Trainers won't be changed. We just set
DeploymentContext in fit method but it doens't require to change APIs of
these classes.

Sincerely
Alexey Platonov

пт, 31 мая 2019 г. в 14:51, Alexey Zinoviev :

> It sounds great, could you please explain what are you going to change in
> the main Vectorizer/Preprocessor API to support it?
>
> пт, 31 мая 2019 г. в 14:20, Алексей Платонов :
>
> > Hi, Igniters!
> > Currently we don't have an ability to deploy automatically user-defined
> > preprocessors and vectorizers. Client's code should be deployed manually
> to
> > Ignite server nodes.
> >
> > I have an idea how to fix it. If we pass user's classloader and one of
> > user-defined classes from fit-level to
> > ComputeUtils.affinityCallWithRetries() then we wiil be able to use
> > GridPeerDeployAware interface to send informtation about this classloader
> > to server nodes.
> >
> > To support this ability we can define interfaces like these:
> >
> > public interface DeployableObject {
> > public List getDependencies();
> > }
> >
> > and
> >
> > public interface DeployingContext {
> > public Class userClass();
> > public ClassLoader clientClassLoader();
> > }
> >
> > DeployableObject will be mark for our ignite-ml final classes like
> trainers
> > or concrete preprocessors and it can be able to return all dependencies
> > that should be deployed to server nodes if it's needed. If these
> > dependencies are DeployableObjects too then depenndencies will be
> unfolded
> > recursively. Classes that isn't defined as DeployableObject will be
> > recognized as user-defined (NOTE: all leaf classes in our hierarchy will
> be
> > DeployableObject).
> >
> > This list of DeployableObjects will be user for define user class loader
> > and one of these objects will be used for passing to GridPeerDeployAware.
> >
> > So, this logic allows to pass user-defined Preprocessors and Vectorizers
> to
> > training algorithms and pipelines.
> >
> > What do you think?
> >
> > Sincerely
> > Alexey Platonov
> >
>

[jira] [Created] (IGNITE-11886) Unable to check query result in IgniteCacheConfigVariationsQueryTest

[Ivan Fedotov (JIRA)]

Ivan Fedotov created IGNITE-11886:
-

 Summary: Unable to check query result in 
IgniteCacheConfigVariationsQueryTest
 Key: IGNITE-11886
 URL: https://issues.apache.org/jira/browse/IGNITE-11886
 Project: Ignite
  Issue Type: Bug
Reporter: Ivan Fedotov


After turn on IgniteConfigVariationsAbstractTest 
IgniteCacheConfigVariationsQueryTest #testLocalScanQuery and 
testScanQueryLocalFilter became failed [1]. Not all predicates were executed 
during query - failed to check execEvtLatch.

[1] 
https://ci.ignite.apache.org/viewLog.html?buildId=3978709=IgniteTests24Java8_QueriesConfigVariations




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11885) Tests from IgniteCacheConfigVariationsFullApiTest failed on TC

[Ivan Fedotov (JIRA)]

Ivan Fedotov created IGNITE-11885:
-

 Summary: Tests from IgniteCacheConfigVariationsFullApiTest failed 
on TC
 Key: IGNITE-11885
 URL: https://issues.apache.org/jira/browse/IGNITE-11885
 Project: Ignite
  Issue Type: Bug
Reporter: Ivan Fedotov


After turn on IgniteConfigVariationsAbstractTest multiple tests in 
IgniteCacheConfigVariationsFullApiTest became failed [1]. The reason is that 
the expected value was not found in the cache, but deeper reason for each test 
can be different - this issue must be investigated.

[1] 
https://ci.ignite.apache.org/viewLog.html?buildId=3978681=IgniteTests24Java8_CacheFullApiBasicConfigVariations



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ML] Deployment of user-defined preprocessors

[Alexey Zinoviev]

It sounds great, could you please explain what are you going to change in
the main Vectorizer/Preprocessor API to support it?

пт, 31 мая 2019 г. в 14:20, Алексей Платонов :

> Hi, Igniters!
> Currently we don't have an ability to deploy automatically user-defined
> preprocessors and vectorizers. Client's code should be deployed manually to
> Ignite server nodes.
>
> I have an idea how to fix it. If we pass user's classloader and one of
> user-defined classes from fit-level to
> ComputeUtils.affinityCallWithRetries() then we wiil be able to use
> GridPeerDeployAware interface to send informtation about this classloader
> to server nodes.
>
> To support this ability we can define interfaces like these:
>
> public interface DeployableObject {
> public List getDependencies();
> }
>
> and
>
> public interface DeployingContext {
> public Class userClass();
> public ClassLoader clientClassLoader();
> }
>
> DeployableObject will be mark for our ignite-ml final classes like trainers
> or concrete preprocessors and it can be able to return all dependencies
> that should be deployed to server nodes if it's needed. If these
> dependencies are DeployableObjects too then depenndencies will be unfolded
> recursively. Classes that isn't defined as DeployableObject will be
> recognized as user-defined (NOTE: all leaf classes in our hierarchy will be
> DeployableObject).
>
> This list of DeployableObjects will be user for define user class loader
> and one of these objects will be used for passing to GridPeerDeployAware.
>
> So, this logic allows to pass user-defined Preprocessors and Vectorizers to
> training algorithms and pipelines.
>
> What do you think?
>
> Sincerely
> Alexey Platonov
>

[jira] [Created] (IGNITE-11884) Time outed invokeAll tests in WithKeepBinaryCacheFullApiTest

[Ivan Fedotov (JIRA)]

Ivan Fedotov created IGNITE-11884:
-

 Summary: Time outed invokeAll tests in 
WithKeepBinaryCacheFullApiTest
 Key: IGNITE-11884
 URL: https://issues.apache.org/jira/browse/IGNITE-11884
 Project: Ignite
  Issue Type: Bug
Reporter: Ivan Fedotov


After turn on IgniteConfigVariationsAbstractTest 2 tests in 
WithKeepBinaryCacheFullApiTest became failed: testInvokeAll and 
testInvokeAllAsync. Tests failed because of time out [1].

[1] 
https://ci.ignite.apache.org/viewLog.html?buildTypeId=IgniteTests24Java8_CacheFullApiConfigVariations=3978682




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11883) Unable to find keys in testKeyAffinityDeploy

[Ivan Fedotov (JIRA)]

Ivan Fedotov created IGNITE-11883:
-

 Summary: Unable to find keys in testKeyAffinityDeploy
 Key: IGNITE-11883
 URL: https://issues.apache.org/jira/browse/IGNITE-11883
 Project: Ignite
  Issue Type: Bug
Reporter: Ivan Fedotov


After turn on IgniteConfigVariationsAbstractTest 
IgniteServiceConfigVariationsFullApiTest#testKeyAffinityDeploy became failed - 
"Unable to find 1 required keys" [1].
It is necessary to investigate the reason and fix the test.

[1] 
https://ci.ignite.apache.org/viewLog.html?buildId=3978784=buildResultsDiv=IgniteTests24Java8_ServiceGridLegacyMode#testNameId5798659135386117314




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[ML] Deployment of user-defined preprocessors

[Алексей Платонов]

Hi, Igniters!
Currently we don't have an ability to deploy automatically user-defined
preprocessors and vectorizers. Client's code should be deployed manually to
Ignite server nodes.

I have an idea how to fix it. If we pass user's classloader and one of
user-defined classes from fit-level to
ComputeUtils.affinityCallWithRetries() then we wiil be able to use
GridPeerDeployAware interface to send informtation about this classloader
to server nodes.

To support this ability we can define interfaces like these:

public interface DeployableObject {
public List getDependencies();
}

and

public interface DeployingContext {
public Class userClass();
public ClassLoader clientClassLoader();
}

DeployableObject will be mark for our ignite-ml final classes like trainers
or concrete preprocessors and it can be able to return all dependencies
that should be deployed to server nodes if it's needed. If these
dependencies are DeployableObjects too then depenndencies will be unfolded
recursively. Classes that isn't defined as DeployableObject will be
recognized as user-defined (NOTE: all leaf classes in our hierarchy will be
DeployableObject).

This list of DeployableObjects will be user for define user class loader
and one of these objects will be used for passing to GridPeerDeployAware.

So, this logic allows to pass user-defined Preprocessors and Vectorizers to
training algorithms and pipelines.

What do you think?

Sincerely
Alexey Platonov

[jira] [Created] (IGNITE-11882) Bugs related to SPI & tests fixes

[Ivan Bessonov (JIRA)]

Ivan Bessonov created IGNITE-11882:
--

 Summary: Bugs related to SPI & tests fixes
 Key: IGNITE-11882
 URL: https://issues.apache.org/jira/browse/IGNITE-11882
 Project: Ignite
  Issue Type: Bug
Reporter: Ivan Bessonov
Assignee: Ivan Bessonov


This issue contains fixes for several issues:
 * Checkpointer thread waits for too long on pending futures without heartbeat.
 * Ignite build date is always shown in current timezone. UTC should be used.
 * Examples have troublesome "jackson" dependency version and can't be run as a 
result.
 * Baseline in discovery cache might be inconsistent with the actual baseline 
on in-memory clusters.
 * Distributed metastorage triggers failure handler on thread interruption 
while node stopping.
 * Sometimes node restore fails with no segments to read, but there are no 
useful logs for diagnostic.
 * Spring test suite has issues:
 ** Exchange manager may invoke the failure processor on node stop with 
NodeStoppingException.
 ** KillerLifecycleBean should wait for the start of all nodes before stop node 
otherwise start the second node may lead to G.start() return null.
 * IgnitePdsCorruptedStoreTest.testReadOnlyMetaStore fails when run under root 
permissions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: {DISCUSSION] Cluster read-only mode.

[Sergey Antonov]

Hello, Zhenya, Maxim!

Thank you for your replies!

>> Should we also allow writes to the DistributedMetaStorage and if not why?
Yes. DistributedMetastorage available for updates with enabled read-only
mode. I added test about it to ClusterReadOnlyModeSelfTest

>> What's the purpose for ignite-sys-cache updates still be available ?
ignite-sys-cache is using in the different subcomponents, for example,
security.

чт, 30 мая 2019 г. в 20:30, Zhenya Stanilovsky :

> hi, Sergey.
> What's the purpose for ignite-sys-cache updates still be available ?
>
> thanks !
>
> > Hello Igniters!
> >
> > I'm working on cluster read-only mode [1] feature. In this mode cluster
> > will be available only for read operations, all data modification
> > operations in user caches will be rejected
> > with ClusterReadOnlyModeCheckedException. This feature could be helpfull
> > for maintenance works (control.sh idle_verify/validate_indexes).
> >
> > A few points about cluster read-only mode:
> > 1) Read-only mode could be enabled on active cluster only.
> > 2) Read-only mode doens't store on PDS (i.e. after cluster restart
> > enabled
> > read-only mode will be forgotten)
> > 3) Updates to ignite-sys-cache will be available with enabled read-only
> > mode.
> >
> > More informartion about implementation you could find in PR [2].
> >
> > What do you think about this feature?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-11256
> > [2] https://github.com/apache/ignite/pull/6423
>


-- 
BR, Sergey Antonov