Virtual Ignite meetup tomorrow!

[Valentin Kulichenko]

Fellow Igniters,

I know many of you have to stay at home these days. I have good news for
you: this doesn't mean that you can't attend Ignite meetups, as we're
switching to a virtual format. The virus is not going to stop us :)

Tomorrow I will be talking about some of the behind-the-scenes design
decisions for distributed systems. You're welcome to join!

RSVP here:
https://www.meetup.com/Bay-Area-In-Memory-Computing/events/269533457/

-Val

[jira] [Created] (IGNITE-12850) Ignite cannot start (metastorage history loading fails)

[Sarunas Valaskevicius (Jira)]

Sarunas Valaskevicius created IGNITE-12850:
--

 Summary: Ignite cannot start (metastorage history loading fails)
 Key: IGNITE-12850
 URL: https://issues.apache.org/jira/browse/IGNITE-12850
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 2.7.6
Reporter: Sarunas Valaskevicius


# metastorage is using persistence
 # when a node is ready to write, writeBaselineTopology is called with null 
history, and generates base line topology with history in gaps
 # from that point, it is impossible to start the node as 
`{color:#569e16}restoreHistory{color}` throws an exception when it is 
processing the gap



{code:java}
2020-03-21_00:00:03.867 [fapi-main-0] INFO  
o.a.i.i.p.c.GridClusterStateProcessor:117 <> - Restoring history for 
BaselineTopology[id=9]
2020-03-21_00:00:03.904 [fapi-main-0] ERROR 
o.a.ignite.internal.IgniteKernal:137 <> - Exception during start processors, 
node will be stopped and close connections
org.apache.ignite.IgniteCheckedException: Restoring of BaselineTopology history 
has failed, expected history item not found for id=8
at 
org.apache.ignite.internal.processors.cluster.BaselineTopologyHistory.restoreHistory(BaselineTopologyHistory.java:54)
at 
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onReadyForRead(GridClusterStateProcessor.java:223)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetastorageReadyForRead(GridCacheDatabaseSharedManager.java:409)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:675)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.notifyMetaStorageSubscribersOnReadyForRead(GridCacheDatabaseSharedManager.java:4730)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1048)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2038)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1730)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:678)
 {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [jira] [Created] (IGNITE-12849) Add New BinaryObject Vectorizer for SparseVectors and Integer Coordinates

[Alexey Zinoviev]

Especially I'd like to add to String method for vectors, what information
could be added here, what do you think?

вт, 31 мар. 2020 г., 20:41 Alexey Zinoviev :

> Great suggestion! Are you ready to make a PR for any of suggested ideas?
>
>
> вт, 31 мар. 2020 г., 20:26 Glenn Wiebe (Jira) :
>
>> Glenn Wiebe created IGNITE-12849:
>> 
>>
>>  Summary: Add New BinaryObject Vectorizer for SparseVectors
>> and Integer Coordinates
>>  Key: IGNITE-12849
>>  URL: https://issues.apache.org/jira/browse/IGNITE-12849
>>  Project: Ignite
>>   Issue Type: New Feature
>>   Components: ml
>> Affects Versions: 2.8
>> Reporter: Glenn Wiebe
>>  Fix For: 2.8
>>
>>
>> A. DenseVector-based BinaryObjectVectorizer
>> When using existing caches as a source of Datasets, the
>> BinaryObjectVectorizer is used.
>> The existing BinaryObjectVectorizer only supports the creation of a
>> SparseVector.
>> The LUDecomposition utility that supports gaussian factorization for
>> models like GMM have a "Singularity indicator" for which a SparseVector and
>> its null handling will set a matrix column calculation to be zero/0.0 which
>> is below the minimum check value (1e-11) and thus indicate a matrix is not
>> square.
>>
>> This null handling of the SparseMatrix will restrict the use of some
>> algorithms like Gaussian Mixture Models where any Vector dimension that is
>> null will incorrectly signal that a matrix is not square.
>>
>> It would be great if we could:
>> - Have a BinaryObjectVectorizer that uses a DenseMatrix to eliminate this
>> singularity trigger and enable use of GMM Trainer.
>>
>> B. CacheBasedDatasets not treated as Temporary Cache
>> When using a cache-based dataset, the close() method destroys the Ignite
>> cache. This means that there is no ability to re-use the data loaded into
>> this dataset.
>>
>> It would be great if we could:
>> - Not destroy the Ignite Cache holding the dataset on close (of one step
>> in an ML processing flow)
>> - Allow for "attaching" to this prior, pre-calculated dataset in
>> subsequent use.
>>
>> C. Vector Visibility
>> Vectors (unlike other value types, e.g. BinaryObjects) are not visible in
>> standard mechanisms, like the Ignite Web Console, where the toString()
>> method does not present any information about the embedded vector values.
>>
>> It would be great if we could:
>> - have a Vector.toString() method implementation that presented some
>> information about what is actually in the Vector.
>>
>> I have implemented the above items and have used them at a customer where
>> I needed these capabilities (or at least it dramatically reduced the cost
>> and increased the value of the solution).
>>
>> It would be great if the community was supportive of this
>> expansion/improvement of the Ignite ML library.
>>
>> Thanks,
>>   Glenn
>>
>>
>>
>>
>> --
>> This message was sent by Atlassian Jira
>> (v8.3.4#803005)
>>
>

Re: [jira] [Created] (IGNITE-12849) Add New BinaryObject Vectorizer for SparseVectors and Integer Coordinates

[Alexey Zinoviev]

Great suggestion! Are you ready to make a PR for any of suggested ideas?


вт, 31 мар. 2020 г., 20:26 Glenn Wiebe (Jira) :

> Glenn Wiebe created IGNITE-12849:
> 
>
>  Summary: Add New BinaryObject Vectorizer for SparseVectors
> and Integer Coordinates
>  Key: IGNITE-12849
>  URL: https://issues.apache.org/jira/browse/IGNITE-12849
>  Project: Ignite
>   Issue Type: New Feature
>   Components: ml
> Affects Versions: 2.8
> Reporter: Glenn Wiebe
>  Fix For: 2.8
>
>
> A. DenseVector-based BinaryObjectVectorizer
> When using existing caches as a source of Datasets, the
> BinaryObjectVectorizer is used.
> The existing BinaryObjectVectorizer only supports the creation of a
> SparseVector.
> The LUDecomposition utility that supports gaussian factorization for
> models like GMM have a "Singularity indicator" for which a SparseVector and
> its null handling will set a matrix column calculation to be zero/0.0 which
> is below the minimum check value (1e-11) and thus indicate a matrix is not
> square.
>
> This null handling of the SparseMatrix will restrict the use of some
> algorithms like Gaussian Mixture Models where any Vector dimension that is
> null will incorrectly signal that a matrix is not square.
>
> It would be great if we could:
> - Have a BinaryObjectVectorizer that uses a DenseMatrix to eliminate this
> singularity trigger and enable use of GMM Trainer.
>
> B. CacheBasedDatasets not treated as Temporary Cache
> When using a cache-based dataset, the close() method destroys the Ignite
> cache. This means that there is no ability to re-use the data loaded into
> this dataset.
>
> It would be great if we could:
> - Not destroy the Ignite Cache holding the dataset on close (of one step
> in an ML processing flow)
> - Allow for "attaching" to this prior, pre-calculated dataset in
> subsequent use.
>
> C. Vector Visibility
> Vectors (unlike other value types, e.g. BinaryObjects) are not visible in
> standard mechanisms, like the Ignite Web Console, where the toString()
> method does not present any information about the embedded vector values.
>
> It would be great if we could:
> - have a Vector.toString() method implementation that presented some
> information about what is actually in the Vector.
>
> I have implemented the above items and have used them at a customer where
> I needed these capabilities (or at least it dramatically reduced the cost
> and increased the value of the solution).
>
> It would be great if the community was supportive of this
> expansion/improvement of the Ignite ML library.
>
> Thanks,
>   Glenn
>
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)
>

[jira] [Created] (IGNITE-12849) Add New BinaryObject Vectorizer for SparseVectors and Integer Coordinates

[Glenn Wiebe (Jira)]

Glenn Wiebe created IGNITE-12849:


 Summary: Add New BinaryObject Vectorizer for SparseVectors and 
Integer Coordinates
 Key: IGNITE-12849
 URL: https://issues.apache.org/jira/browse/IGNITE-12849
 Project: Ignite
  Issue Type: New Feature
  Components: ml
Affects Versions: 2.8
Reporter: Glenn Wiebe
 Fix For: 2.8


A. DenseVector-based BinaryObjectVectorizer
When using existing caches as a source of Datasets, the BinaryObjectVectorizer 
is used.
The existing BinaryObjectVectorizer only supports the creation of a 
SparseVector.
The LUDecomposition utility that supports gaussian factorization for models 
like GMM have a "Singularity indicator" for which a SparseVector and its null 
handling will set a matrix column calculation to be zero/0.0 which is below the 
minimum check value (1e-11) and thus indicate a matrix is not square. 

This null handling of the SparseMatrix will restrict the use of some algorithms 
like Gaussian Mixture Models where any Vector dimension that is null will 
incorrectly signal that a matrix is not square.

It would be great if we could:
- Have a BinaryObjectVectorizer that uses a DenseMatrix to eliminate this 
singularity trigger and enable use of GMM Trainer.

B. CacheBasedDatasets not treated as Temporary Cache
When using a cache-based dataset, the close() method destroys the Ignite cache. 
This means that there is no ability to re-use the data loaded into this dataset.

It would be great if we could:
- Not destroy the Ignite Cache holding the dataset on close (of one step in an 
ML processing flow)
- Allow for "attaching" to this prior, pre-calculated dataset in subsequent use.

C. Vector Visibility
Vectors (unlike other value types, e.g. BinaryObjects) are not visible in 
standard mechanisms, like the Ignite Web Console, where the toString() method 
does not present any information about the embedded vector values.

It would be great if we could:
- have a Vector.toString() method implementation that presented some 
information about what is actually in the Vector.

I have implemented the above items and have used them at a customer where I 
needed these capabilities (or at least it dramatically reduced the cost and 
increased the value of the solution).

It would be great if the community was supportive of this expansion/improvement 
of the Ignite ML library.

Thanks,
  Glenn




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: GridKernalContextImpl.isDaemon is calling System.Properties frequently

[Ilya Kasnacheev]

Hello!

Thank you for this contribution, I have merged it to master.

Are you interested in becoming a contributor to Apache Ignite so that you
can assign tickets to yourself?

Regards,
-- 
Ilya Kasnacheev


пт, 27 мар. 2020 г. в 07:44, Sunny Chan, CLSA :

> Hello,
>
>
>
> I was performing some tuning using JFR with our application, and I have
> found that in many of the cache get it makes a lot of calls to get System
> Properties, which is a synchronized call. Upon further examination of the
> code I have found that we can potentially optimized the code to cache the
> results.
>
>
>
> I have raised a bug report (
> https://issues.apache.org/jira/browse/IGNITE-12842) and proposed a patch (
> https://github.com/apache/ignite/pull/7579). Please review this and let
> me know whether this is acceptable
>
>
>
> *Sunny Chan*
>
> *Senior Lead Engineer, Executive Services*
>
> D  +852 2600 8907  |  M  +852 6386 1835  |  T  +852 2600 
>
> 5/F, One Island East, 18 Westlands Road, Island East, Hong Kong
>
>
>
> [image: :1. Social Media Icons:CLSA_Social Media Icons_linkedin.png]
> [image: :1. Social Media
> Icons:CLSA_Social Media Icons_twitter.png]
> [image: :1. Social Media
> Icons:CLSA_Social Media Icons_youtube.png]
> [image: :1.
> Social Media Icons:CLSA_Social Media Icons_facebook.png]
> 
>
>
>
> *clsa.com* 
>
> *Insights. Liquidity. Capital. *
>
>
>
> [image: CLSA_RGB] 
>
>
>
> *A CITIC Securities Company*
>
>
>
> The content of this communication is intended for the recipient and is
> subject to CLSA Legal and Regulatory Notices.
> These can be viewed at https://www.clsa.com/disclaimer.html or sent to
> you upon request.
> Please consider before printing. CLSA is ISO14001 certified and committed
> to reducing its impact on the environment.
>

Re: Inconsistent API IgniteClient and REST

[Andrey Kuznetsov]

I'd prefer marking ADMIN_CACHE as deprecated, but postpone its removal from
GridRestProcessor till next Ignte release (2.10 or 3.0?). For now we could
just add checks for CACHE_CREATE / CACHE_DESTROY there along
with ADMIN_CACHE.

вт, 31 мар. 2020 г. в 12:30, Nikolay Izhikov :

> Hello, Sergey.
>
>
> I’m +1 to make this change.
>
> I think we should make security consistent across all APIs.
>
> > 31 марта 2020 г., в 12:14, Sergei Ryzhov 
> написал(а):
> >
> > Hello!
> > Now the work of permissions for API IgniteClient and REST is different.
> > To create/delete a cache:
> > IgniteClient authorises
> CACHE_CREATE/CACHE_DESTROY.(GridCacheProcessor#authorizeCacheCreate <
> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java#L3983>,
> authorizeCacheDestroy <
> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java#L3973
> >)
> > REST authorises ADMIN_CACHE.(GridRestProcessor#authorize <
> https://github.com/apache/ignite/blob/aefad946ebd7720f81b460aa39e205c10dc24b26/modules/core/src/main/java/org/apache/ignite/internal/processors/rest/GridRestProcessor.java#L841
> >)
> > I think this is inconsistent.
> >
> > I suggest ADMIN_CACHE mark @Deprecated
> > and replace it in the GridRestProcessor with CACHE_CREATE /
> CACHE_DESTROY
> > while maintaining backward compatibility for ADMIN_CACHE.
> >
> > This will allow us to remove ADMIN_CACHE in the future.
> >
> >
> >
> > Sergei Ryzhov
> > s.vi.ryz...@gmail.com
> >
> >
> >
> >
> >
> >
> >
> >
>
>

-- 
Best regards,
  Andrey Kuznetsov.

Re: Thin client: compute support

[Alex Plehanov]

Pavel, thanks for the review. I've fixed your comments.

вт, 31 мар. 2020 г. в 13:47, Pavel Tupitsyn :

> Alex, I did a partial review - mostly server-side code. Please see my
> comments on GitHub.
> We will need more eyes on this, especially on Java client side.
>
> Feature masks are a great addition!
>
> On Tue, Mar 31, 2020 at 1:05 PM Alex Plehanov 
> wrote:
>
> > Also, I've updated IEP regarding features masks (new backward
> compatibility
> > mechanism for thin clients) which was discussed earlier in this thread.
> >
> > вт, 31 мар. 2020 г. в 02:42, Alex Plehanov :
> >
> > > Guys,
> > >
> > > I've finished IEP implementation for server-side and java thin client
> > side
> > > (ticket: [1], PR: [2]). Can you please review the patch?
> > >
> > > [1]: https://issues.apache.org/jira/browse/IGNITE-12835
> > > [2]: https://github.com/apache/ignite/pull/7572
> > >
> > > пт, 27 мар. 2020 г. в 19:11, Pavel Tupitsyn :
> > >
> > >> Agree with Igor, let's set deployment aside for now, it is out of
> scope
> > of
> > >> this IEP.
> > >>
> > >>
> > >>
> > >> On Fri, Mar 27, 2020 at 6:52 PM Igor Sapego 
> wrote:
> > >>
> > >> > Hi guys,
> > >> >
> > >> > I like the proposal in general.
> > >> >
> > >> > Now for the task deployment - I think we should have separate API
> > >> > for it (and separate IEP I believe). Also, I'm not sure that this
> API
> > >> > should be a part of the API of any thin client as it seems weird to
> me
> > >> > to use Python client to deploy Java tasks. Control.sh or visor
> > proposal
> > >> > sounds much better.
> > >> >
> > >> > Best Regards,
> > >> > Igor
> > >> >
> > >> >
> > >> > On Thu, Mar 26, 2020 at 10:56 PM Denis Magda 
> > wrote:
> > >> >
> > >> > > >
> > >> > > > Deployment API definitely needed as one of the next steps.
> > >> Currently,
> > >> > we
> > >> > > > are talking only about the first step (execution of already
> > deployed
> > >> > > > tasks).
> > >> > > > Also, I'm not sure about automatic redeploy and
> peer-class-loading
> > >> for
> > >> > > thin
> > >> > > > clients, I think it's better to have more control here and
> provide
> > >> API
> > >> > to
> > >> > > > explicitly deploy classes or jar files. WDYT?
> > >> > >
> > >> > >
> > >> > > Alex, agree that automatic redeployment is better suited for the
> > >> > management
> > >> > > APIs. How about adding this capability to our command-line tool
> > >> > > (control.sh, or visor cmd, or one new holistic tool).
> > >> > >
> > >> > > -
> > >> > > Denis
> > >> > >
> > >> > >
> > >> > > On Wed, Mar 25, 2020 at 1:04 PM Alex Plehanov <
> > >> plehanov.a...@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Pavel,
> > >> > > >
> > >> > > > 1. Actually it can be solved on the client-side (and already
> > solved
> > >> in
> > >> > > PoC
> > >> > > > implementation). But I agreed it brings extra complexity for
> > >> > client-side
> > >> > > > implementation, will try to provide such guarantees on the
> > >> server-side.
> > >> > > > 2. ComputeTask has also "reduce" step which is executed on the
> > >> > initiator
> > >> > > > node. Binary-rest client implementation, for example, has such
> > >> affinity
> > >> > > > methods (to execute the task by name). I'm ok with removing it.
> At
> > >> > least
> > >> > > if
> > >> > > > someone will need it we can implement it again at any time in
> the
> > >> > future
> > >> > > > without protocol change.
> > >> > > > I've fixed IEP.
> > >> > > >
> > >> > > > Denis,
> > >> > > >
> > >> > > > Deployment API definitely needed as one of the next steps.
> > >> Currently,
> > >> > we
> > >> > > > are talking only about the first step (execution of already
> > deployed
> > >> > > > tasks).
> > >> > > > Also, I'm not sure about automatic redeploy and
> peer-class-loading
> > >> for
> > >> > > thin
> > >> > > > clients, I think it's better to have more control here and
> provide
> > >> API
> > >> > to
> > >> > > > explicitly deploy classes or jar files. WDYT?
> > >> > > >
> > >> > > > ср, 25 мар. 2020 г. в 21:17, Denis Magda :
> > >> > > >
> > >> > > > > Alex, thanks for preparing the outline.
> > >> > > > >
> > >> > > > > I'd like us to discuss an approach for compute tasks update
> with
> > >> no
> > >> > > > > downtimes on the servers' end. For instance, let's assume
> that a
> > >> > > > > Python/C++/Node.JS developer requested to update a compute
> task
> > he
> > >> > > called
> > >> > > > > from the app. Should we introduce some system level API to the
> > >> binary
> > >> > > > > protocol that can take a jar file (or class) and redeploy it
> > >> > > > automatically
> > >> > > > > with the usage of peer-class-loading?
> > >> > > > >
> > >> > > > > -
> > >> > > > > Denis
> > >> > > > >
> > >> > > > >
> > >> > > > > On Wed, Mar 25, 2020 at 5:47 AM Alex Plehanov <
> > >> > plehanov.a...@gmail.com
> > >> > > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Hello guys.
> > >> > > > > >
> > >> > > > > > I've implemented PoC and created IEP [1] for thin client
> > compute
> > >> 

Re: JSR 381 (Visual Recognition in Java) and Apache Ignite

[Alexey Zinoviev]

Nice to see all of you guys (thanks Vladimir) here on dev-list of Apache
Ignite that includes distributed ML.

First of all, want to say thank you for your great work to build specs to
the Java world, hope it will be helpful for Java devs in the nearest
future.
In the second, a few days ago I found a link
 in Frank Greco
twitter to this JSR and read a few examples to understand the proposed API.
Also, we (in Apache Ignite ML) have experimental integration with
TensorFlow for inference and distributed training purposes and I'm
interested in TF Java API development  and DL4j/PyTorch possible
integration.
Will be great to have common API for different implementations.

P.S. From my side, I've ready to join your team if the specification is not
finished yet, because I have a few comments and thoughts based on my
experience of different API usage. A few weeks ago I also joined to the
developer group of JVM API for TensorFlow (SIG JVM), hope it will be
helpful in both directions to communicate each other.



пн, 30 мар. 2020 г. в 21:00, Vladimir Ozerov :

> Hi Alexey, Igniters,
>
> Let me introduce you Heather, Zoran, and Frank. Heather is the Chair of
> the JCP. Zoran and Frank are JSR 381 spec leads. They are interested in
> discussing the upcoming visual recognition specification with the Apache
> Ignite community, to understand whether the community has any interest in
> implementing it.
>
> Zoran and Frank,
>
> Please meet Alexey Zinoviev, who is the principal maintainer of the Apache
> ML module. I hope he will be able to help you with your questions.
>
> Regards,
> Vladimir.
>

[jira] [Created] (IGNITE-12848) SQL: H2Connection leaks on INSERT

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-12848:
-

 Summary: SQL: H2Connection leaks on INSERT
 Key: IGNITE-12848
 URL: https://issues.apache.org/jira/browse/IGNITE-12848
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.8
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.8.1


H2 connection leaks on INSERT query when one row is inserted and not constant 
values are used:
e.g.:
{{INSERT INTO T VALUES(1, CURRENT_TIMESTAM());}}

*Root cause:*
Query cursor isn't closed at the IgniteH2Indexing#executeUpdateNonTransactional 
after {{DmlUtils#processSelectResult}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Data vanished from cluster after INACTIVE/ACTIVE switch

[Vladimir Steshin]

Ivan, hi.

I absolutely agree this particular description is not enough to see the 
deactivation issue. I also vote for brief code.


There are about 15 places in inner logic with this description. I 
propose balance between code base size and comment completeness.


Should we enlarge code even if we got several full descriptions?


30.03.2020 20:02, Ivan Rakov пишет:

Vladimir,

@param forceDeactivation If {@code true}, cluster deactivation will be

forced.

It's true that it's possible to infer semantics of forced deactivation from
other parts of API. I just wanted to highlight that exactly this
description explains something that can be guessed by the parameter name.
I suppose to shorten the lookup path and shed a light on deactivation
semantics a bit:


@param forceDeactivation If {@code true}, cluster will be deactivated even
if running in-memory caches are present. All data in the corresponding
caches will be vanished as a result.

Does this make sense?

On Fri, Mar 27, 2020 at 12:00 PM Vladimir Steshin 
wrote:


Ivan, hi.


1) >>> Is it correct? If we are on the same page, let's proceed this way

It is correct.


2) - In many places in the code I can see the following javadoc


   @param forceDeactivation If {@code true}, cluster deactivation will be

forced.

In the internal params/flags. You can also find /@see
ClusterState#INACTIVE/ and full description with several public APIs (
like /Ignite.active(boolean)/ ):

//

/* /

//

/* NOTE:/

//

/* Deactivation clears in-memory caches (without persistence) including
the system caches./

Should be enough. Is not?


27.03.2020 10:51, Ivan Rakov пишет:

Vladimir, Igniters,

Let's emphasize our final plan.

We are going to add --force flags that will be necessary to pass for a
deactivation if there are in-memory caches to:
1) Rest API (already implemented in [1])
2) Command line utility (already implemented in [1])
3) JMX bean (going to be implemented in [2])
We are *not* going to change IgniteCluster or any other thick Java API,
thus we are *not* going to merge [3].
We plan to *fully rollback* [1] and [2] once cache data survival after
activation-deactivation cycle will be implemented.

Is it correct? If we are on the same page, let's proceed this way.
I propose to:
- Create a JIRA issue for in-memory-data-safe deactivation (possibly,
without IEP and detailed design so far)
- Describe in the issue description what exact parts of API should be
removed under the issue scope.

Also, a few questions on already merged [1]:
- We have removed GridClientClusterState#state(ClusterState) from Java
client API. Is it a legitimate thing to do? Don't we have to support API
compatibility for thin clients as well?
- In many places in the code I can see the following javadoc


   @param forceDeactivation If {@code true}, cluster deactivation will

be forced.

As for me, this javadoc doesn't clarify anything. I'd suggest to

describe

in which cases deactivation won't happen unless it's forced and which
impact forced deactivation will bring on the system.

[1]: https://issues.apache.org/jira/browse/IGNITE-12701
[2]: https://issues.apache.org/jira/browse/IGNITE-12779
[3]: https://issues.apache.org/jira/browse/IGNITE-12614

--
Ivan

On Tue, Mar 24, 2020 at 7:18 PM Vladimir Steshin 

wrote:

Hi, Igniters.

I'd like to remind you that cluster can be deactivated by user with 3
utilities: control.sh, *JMX and the REST*. Proposed in [1] solution is
not about control.sh. It suggests same approach regardless of the
utility user executes. The task touches *only* *API of the user calls*,
not the internal APIs.

The reasons why flag “--yes” and confirmation prompt hasn’t taken into
account for control.sh are:

-Various commands widely use “--yes” just to start. Even not dangerous
ones require “--yes” to begin. “--force” is dedicated for *harmless
actions*.

-Checking of probable data erasure works after command start and
“--force” may not be required at all.

-There are also JMX and REST. They have no “—yes” but should work alike.

   To get the deactivation safe I propose to merge last ticket with
the JMX fixes [2]. In future releases, I believe, we should estimate
jobs and fix memory erasure in general. For now, let’s prevent it. WDYT?


[1] https://issues.apache.org/jira/browse/IGNITE-12614

[2] https://issues.apache.org/jira/browse/IGNITE-12779


24.03.2020 15:55, Вячеслав Коптилин пишет:

Hello Nikolay,

I am talking about the interactive mode of the control utility, which
requires explicit confirmation from the user.
Please take a look at DeactivateCommand#prepareConfirmation and its

usages.

It seems to me, this mode has the same aim as the forceDeactivation

flag.

We can change the message returned by

DeactivateCommand#confirmationPrompt

as follows:
   "Warning: the command will deactivate the cluster nnn and clear
in-memory caches (without persistence) including system caches."

What do you think?

Thanks,
S.

вт, 24 мар. 2020 г. в 13:07, Nikolay Izhikov :


He

Re: Thin client: compute support

[Pavel Tupitsyn]

Alex, I did a partial review - mostly server-side code. Please see my
comments on GitHub.
We will need more eyes on this, especially on Java client side.

Feature masks are a great addition!

On Tue, Mar 31, 2020 at 1:05 PM Alex Plehanov 
wrote:

> Also, I've updated IEP regarding features masks (new backward compatibility
> mechanism for thin clients) which was discussed earlier in this thread.
>
> вт, 31 мар. 2020 г. в 02:42, Alex Plehanov :
>
> > Guys,
> >
> > I've finished IEP implementation for server-side and java thin client
> side
> > (ticket: [1], PR: [2]). Can you please review the patch?
> >
> > [1]: https://issues.apache.org/jira/browse/IGNITE-12835
> > [2]: https://github.com/apache/ignite/pull/7572
> >
> > пт, 27 мар. 2020 г. в 19:11, Pavel Tupitsyn :
> >
> >> Agree with Igor, let's set deployment aside for now, it is out of scope
> of
> >> this IEP.
> >>
> >>
> >>
> >> On Fri, Mar 27, 2020 at 6:52 PM Igor Sapego  wrote:
> >>
> >> > Hi guys,
> >> >
> >> > I like the proposal in general.
> >> >
> >> > Now for the task deployment - I think we should have separate API
> >> > for it (and separate IEP I believe). Also, I'm not sure that this API
> >> > should be a part of the API of any thin client as it seems weird to me
> >> > to use Python client to deploy Java tasks. Control.sh or visor
> proposal
> >> > sounds much better.
> >> >
> >> > Best Regards,
> >> > Igor
> >> >
> >> >
> >> > On Thu, Mar 26, 2020 at 10:56 PM Denis Magda 
> wrote:
> >> >
> >> > > >
> >> > > > Deployment API definitely needed as one of the next steps.
> >> Currently,
> >> > we
> >> > > > are talking only about the first step (execution of already
> deployed
> >> > > > tasks).
> >> > > > Also, I'm not sure about automatic redeploy and peer-class-loading
> >> for
> >> > > thin
> >> > > > clients, I think it's better to have more control here and provide
> >> API
> >> > to
> >> > > > explicitly deploy classes or jar files. WDYT?
> >> > >
> >> > >
> >> > > Alex, agree that automatic redeployment is better suited for the
> >> > management
> >> > > APIs. How about adding this capability to our command-line tool
> >> > > (control.sh, or visor cmd, or one new holistic tool).
> >> > >
> >> > > -
> >> > > Denis
> >> > >
> >> > >
> >> > > On Wed, Mar 25, 2020 at 1:04 PM Alex Plehanov <
> >> plehanov.a...@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Pavel,
> >> > > >
> >> > > > 1. Actually it can be solved on the client-side (and already
> solved
> >> in
> >> > > PoC
> >> > > > implementation). But I agreed it brings extra complexity for
> >> > client-side
> >> > > > implementation, will try to provide such guarantees on the
> >> server-side.
> >> > > > 2. ComputeTask has also "reduce" step which is executed on the
> >> > initiator
> >> > > > node. Binary-rest client implementation, for example, has such
> >> affinity
> >> > > > methods (to execute the task by name). I'm ok with removing it. At
> >> > least
> >> > > if
> >> > > > someone will need it we can implement it again at any time in the
> >> > future
> >> > > > without protocol change.
> >> > > > I've fixed IEP.
> >> > > >
> >> > > > Denis,
> >> > > >
> >> > > > Deployment API definitely needed as one of the next steps.
> >> Currently,
> >> > we
> >> > > > are talking only about the first step (execution of already
> deployed
> >> > > > tasks).
> >> > > > Also, I'm not sure about automatic redeploy and peer-class-loading
> >> for
> >> > > thin
> >> > > > clients, I think it's better to have more control here and provide
> >> API
> >> > to
> >> > > > explicitly deploy classes or jar files. WDYT?
> >> > > >
> >> > > > ср, 25 мар. 2020 г. в 21:17, Denis Magda :
> >> > > >
> >> > > > > Alex, thanks for preparing the outline.
> >> > > > >
> >> > > > > I'd like us to discuss an approach for compute tasks update with
> >> no
> >> > > > > downtimes on the servers' end. For instance, let's assume that a
> >> > > > > Python/C++/Node.JS developer requested to update a compute task
> he
> >> > > called
> >> > > > > from the app. Should we introduce some system level API to the
> >> binary
> >> > > > > protocol that can take a jar file (or class) and redeploy it
> >> > > > automatically
> >> > > > > with the usage of peer-class-loading?
> >> > > > >
> >> > > > > -
> >> > > > > Denis
> >> > > > >
> >> > > > >
> >> > > > > On Wed, Mar 25, 2020 at 5:47 AM Alex Plehanov <
> >> > plehanov.a...@gmail.com
> >> > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hello guys.
> >> > > > > >
> >> > > > > > I've implemented PoC and created IEP [1] for thin client
> compute
> >> > grid
> >> > > > > > functionality. Please have a look.
> >> > > > > >
> >> > > > > > [1]:
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-42+Thin+client%3A+compute+support
> >> > > > > >
> >> > > > > > пт, 24 янв. 2020 г. в 16:56, Alex Plehanov <
> >> > plehanov.a...@gmail.com
> >> > > >:
> >> > > > > >
> >> > > > > > > We've discussed t

[jira] [Created] (IGNITE-12847) Incorrect query result

[Alexey Melchakov (Jira)]

Alexey Melchakov created IGNITE-12847:
-

 Summary: Incorrect query result 
 Key: IGNITE-12847
 URL: https://issues.apache.org/jira/browse/IGNITE-12847
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.8
 Environment: * apache ignite 2.8.0
 * docker image: apacheignite/ignite:2.8.0
 * Reproduces both in sqlline.sh and IntelliJ IDEA SQL command console

Command for docker:

 
{code:java}
docker run -it --name \
-p 47500:47500 \
-p 47501:47501 \
-p 10800:10800 \
-e 
"CONFIG_URI=https://raw.githubusercontent.com/apache/ignite/master/examples/config/example-cache.xml";
 apacheignite/ignite:2.8.0
{code}
 
 
Reporter: Alexey Melchakov


This two queries should return same result. 
{code:java}
SELECT d.deptno, d.dname, e.empno,
e.ename FROM emp e
INNER JOIN dept d
ON ( e.deptno = d.deptno )
WHERE EXISTS (SELECT 1 FROM emp t
WHERE t.mgr = e.empno) AND d.deptno IN ( 10, 20, 30 );
{code}
 and
{code:java}
SELECT d.deptno, d.dname, e.empno,
e.ename FROM emp e
INNER JOIN dept d
ON ( e.deptno = d.deptno )
WHERE EXISTS (SELECT 1 FROM emp t
WHERE t.mgr = e.empno) AND e.deptno IN ( 10, 20, 30 );
{code}
*But in first case there are no results.*

In second case a have this one:

{{+++---+---+}}
{{| DEPTNO | DNAME      | EMPNO | ENAME |}}
{{+++---+---+}}
{{| 10     | ACCOUNTING | 7839  | KING  |}}
{{+++---+---+}}

*This issue is not reproduces on apache ignite 2.7.6*

 

Example data:
{code:java}
CREATE TABLE dept (deptno LONG,dname VARCHAR,loc VARCHAR,CONSTRAINT pk_dept 
PRIMARY KEY (deptno));

CREATE TABLE emp (empno LONG,ename VARCHAR, job VARCHAR, mgr INTEGER, hiredate 
DATE,sal LONG,comm LONG,deptno LONG,CONSTRAINT pk_emp PRIMARY KEY (empno));

INSERT INTO dept (deptno, dname, loc) VALUES (10,'ACCOUNTING', 'NEW 
YORK');INSERT INTO dept (deptno, dname, loc) VALUES(20,'RESEARCH', 'DALLAS');
INSERT INTO dept (deptno, dname, loc) VALUES(30,'SALES', 'CHICAGO');INSERT INTO 
emp (empno, ename, job, mgr, hiredate, sal, comm, deptno) 
VALUES(7839,'KING','PRESIDENT',NULL,to_date('17-11-1981', 'dd-mm-'), 
5000,NULL, 10);
INSERT INTO emp (empno, ename, job, mgr, hiredate, sal, comm, deptno) 
VALUES(7698,'BLAKE','MANAGER',7839,to_date('1-5-1981', 'dd-mm-'), 2850, 
NULL, 30);
INSERT INTO emp (empno, ename, job, mgr, hiredate, sal, comm, deptno) 
VALUES(7782,'CLARK','MANAGER',7839,to_date('9-6-1981', 'dd-mm-'), 
2450,NULL, 10);
{code}
 

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Thin client: compute support

[Alex Plehanov]

Also, I've updated IEP regarding features masks (new backward compatibility
mechanism for thin clients) which was discussed earlier in this thread.

вт, 31 мар. 2020 г. в 02:42, Alex Plehanov :

> Guys,
>
> I've finished IEP implementation for server-side and java thin client side
> (ticket: [1], PR: [2]). Can you please review the patch?
>
> [1]: https://issues.apache.org/jira/browse/IGNITE-12835
> [2]: https://github.com/apache/ignite/pull/7572
>
> пт, 27 мар. 2020 г. в 19:11, Pavel Tupitsyn :
>
>> Agree with Igor, let's set deployment aside for now, it is out of scope of
>> this IEP.
>>
>>
>>
>> On Fri, Mar 27, 2020 at 6:52 PM Igor Sapego  wrote:
>>
>> > Hi guys,
>> >
>> > I like the proposal in general.
>> >
>> > Now for the task deployment - I think we should have separate API
>> > for it (and separate IEP I believe). Also, I'm not sure that this API
>> > should be a part of the API of any thin client as it seems weird to me
>> > to use Python client to deploy Java tasks. Control.sh or visor proposal
>> > sounds much better.
>> >
>> > Best Regards,
>> > Igor
>> >
>> >
>> > On Thu, Mar 26, 2020 at 10:56 PM Denis Magda  wrote:
>> >
>> > > >
>> > > > Deployment API definitely needed as one of the next steps.
>> Currently,
>> > we
>> > > > are talking only about the first step (execution of already deployed
>> > > > tasks).
>> > > > Also, I'm not sure about automatic redeploy and peer-class-loading
>> for
>> > > thin
>> > > > clients, I think it's better to have more control here and provide
>> API
>> > to
>> > > > explicitly deploy classes or jar files. WDYT?
>> > >
>> > >
>> > > Alex, agree that automatic redeployment is better suited for the
>> > management
>> > > APIs. How about adding this capability to our command-line tool
>> > > (control.sh, or visor cmd, or one new holistic tool).
>> > >
>> > > -
>> > > Denis
>> > >
>> > >
>> > > On Wed, Mar 25, 2020 at 1:04 PM Alex Plehanov <
>> plehanov.a...@gmail.com>
>> > > wrote:
>> > >
>> > > > Pavel,
>> > > >
>> > > > 1. Actually it can be solved on the client-side (and already solved
>> in
>> > > PoC
>> > > > implementation). But I agreed it brings extra complexity for
>> > client-side
>> > > > implementation, will try to provide such guarantees on the
>> server-side.
>> > > > 2. ComputeTask has also "reduce" step which is executed on the
>> > initiator
>> > > > node. Binary-rest client implementation, for example, has such
>> affinity
>> > > > methods (to execute the task by name). I'm ok with removing it. At
>> > least
>> > > if
>> > > > someone will need it we can implement it again at any time in the
>> > future
>> > > > without protocol change.
>> > > > I've fixed IEP.
>> > > >
>> > > > Denis,
>> > > >
>> > > > Deployment API definitely needed as one of the next steps.
>> Currently,
>> > we
>> > > > are talking only about the first step (execution of already deployed
>> > > > tasks).
>> > > > Also, I'm not sure about automatic redeploy and peer-class-loading
>> for
>> > > thin
>> > > > clients, I think it's better to have more control here and provide
>> API
>> > to
>> > > > explicitly deploy classes or jar files. WDYT?
>> > > >
>> > > > ср, 25 мар. 2020 г. в 21:17, Denis Magda :
>> > > >
>> > > > > Alex, thanks for preparing the outline.
>> > > > >
>> > > > > I'd like us to discuss an approach for compute tasks update with
>> no
>> > > > > downtimes on the servers' end. For instance, let's assume that a
>> > > > > Python/C++/Node.JS developer requested to update a compute task he
>> > > called
>> > > > > from the app. Should we introduce some system level API to the
>> binary
>> > > > > protocol that can take a jar file (or class) and redeploy it
>> > > > automatically
>> > > > > with the usage of peer-class-loading?
>> > > > >
>> > > > > -
>> > > > > Denis
>> > > > >
>> > > > >
>> > > > > On Wed, Mar 25, 2020 at 5:47 AM Alex Plehanov <
>> > plehanov.a...@gmail.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hello guys.
>> > > > > >
>> > > > > > I've implemented PoC and created IEP [1] for thin client compute
>> > grid
>> > > > > > functionality. Please have a look.
>> > > > > >
>> > > > > > [1]:
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-42+Thin+client%3A+compute+support
>> > > > > >
>> > > > > > пт, 24 янв. 2020 г. в 16:56, Alex Plehanov <
>> > plehanov.a...@gmail.com
>> > > >:
>> > > > > >
>> > > > > > > We've discussed thin client compute protocol with Pavel
>> Tupitsyn
>> > > and
>> > > > > Igor
>> > > > > > > Sapego and come to the conclusion that approach with two-way
>> > > requests
>> > > > > > > should be used: client generates taskId and send a request to
>> the
>> > > > > server
>> > > > > > to
>> > > > > > > execute a task. The server responds that the request has been
>> > > > accepted.
>> > > > > > > After task has finished the server notifies the client (send a
>> > > > request
>> > > > > > > without waiting for a response). The client can cancel the
>> t

Re: [jira] [Created] (IGNITE-12846) Docs: ml package org.apache.ignite.ml.knn.utils.indices have no description in binary release

[Alexey Zinoviev]

Do you mean missed package-level docs, isn't it?

вт, 31 мар. 2020 г. в 11:52, Stepan Pilschikov (Jira) :

> Stepan Pilschikov created IGNITE-12846:
> --
>
>  Summary: Docs: ml package
> org.apache.ignite.ml.knn.utils.indices have no description in binary release
>  Key: IGNITE-12846
>  URL: https://issues.apache.org/jira/browse/IGNITE-12846
>  Project: Ignite
>   Issue Type: Bug
>   Components: documentation
> Affects Versions: 2.8
> Reporter: Stepan Pilschikov
>
>
> apache-ignite-2.8.0-bin/docs/javadoc/overview-summary.html does not
> contain description block for org.apache.ignite.ml.knn.utils.indices
>
> Actual:
> {code}
> 
> 
>  href="org/apache/ignite/ml/knn/utils/indices/package-summary.html">org.apache.ignite.ml.knn.utils.indices
>  
> 
> {code}
>
> Expected:
> {code}
> 
> 
>  href="org/apache/ignite/ml/knn/utils/indices/package-summary.html">org.apache.ignite.ml.knn.utils.indices
> 
> [Some description]
> 
> {code}
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)
>

Re: ignitevisorcmd.sh does not connect to grid

[Nikolay Izhikov]

Hello, Ali.

This fix would be available in 2.8.1.

> 30 марта 2020 г., в 21:09, Nikolai Kulagin  
> написал(а):
> 
> Ali, yes, this problem has been fixed.
> 
> пн, 30 мар. 2020 г. в 11:44, Nikolay Izhikov :
> 
>> Hello, Ali.
>> 
>> Looks like this is known issue and it was fixed by [1]
>> 
>> Nikolay, can you confirm it?
>> 
>> [1]
>> https://github.com/apache/ignite/commit/d0bae90bce33dbed8b9f3cd52663ce5efa545d97
>> 
>>> 28 марта 2020 г., в 12:30, ali hassan  написал(а):
>>> 
>>> Hi,
>>> I failed to connect ignitevisorcmd.sh to my grid. The result of open is:
>>> 
>>> +-+
>>> 
>>> | Status   | Disconnected |
>>> 
>>> | Ignite instance name | |
>>> 
>>> | Config path  | |
>>> 
>>> | Uptime   | |
>>> 
>>> +-+
>>> 
>>> visor>
>>> 
>>> 
>>> 
>>> even if I choose the right conf, and I repeat open many times.
>>> 
>>> 
>>> 
>>> The grid is running, and I can perform queries normally.
>>> 
>>> There is no exception, nor indication in the logs.
>>> 
>>> I used –v in both ignite.sh and ignitevisorcmd.sh.
>>> 
>>> 
>>> 
>>> I appreciate your help.
>>> 
>>> 
>>> 
>>> Best Regards,
>>> 
>>> Ali Hassan
>>> 
>>> Senior Scala SE
>>> 
>>> Senior BigData SE
>> 
>> 

Re: Do we have any Python experts in the community?

[Igor Sapego]

I'm not very experienced in Python, but if no one else wants
to take a look, I'll do it.

Best Regards,
Igor


On Sat, Mar 28, 2020 at 2:04 AM Denis Magda  wrote:

> Igniters,
>
> Is any of you is skillful enough to contribute improvements to our Python
> thin client? For instance, that's one of the tickets that has been recently
> reported on the user list:
> https://issues.apache.org/jira/browse/IGNITE-12809
>
> -
> Denis
>

Re: Inconsistent API IgniteClient and REST

[Nikolay Izhikov]

Hello, Sergey.


I’m +1 to make this change.

I think we should make security consistent across all APIs.

> 31 марта 2020 г., в 12:14, Sergei Ryzhov  написал(а):
> 
> Hello!
> Now the work of permissions for API IgniteClient and REST is different.
> To create/delete a cache:
> IgniteClient authorises 
> CACHE_CREATE/CACHE_DESTROY.(GridCacheProcessor#authorizeCacheCreate 
> ,
>  authorizeCacheDestroy 
> )
> REST authorises ADMIN_CACHE.(GridRestProcessor#authorize 
> )
> I think this is inconsistent.
> 
> I suggest ADMIN_CACHE mark @Deprecated
> and replace it in the GridRestProcessor with CACHE_CREATE / CACHE_DESTROY 
> while maintaining backward compatibility for ADMIN_CACHE.
> 
> This will allow us to remove ADMIN_CACHE in the future.
> 
> 
> 
> Sergei Ryzhov
> s.vi.ryz...@gmail.com
> 
> 
> 
> 
> 
> 
> 
> 

Inconsistent API IgniteClient and REST

[Sergei Ryzhov]

Hello!
Now the work of permissions for API IgniteClient and REST is different.
To create/delete a cache:
IgniteClient authorises 
CACHE_CREATE/CACHE_DESTROY.(GridCacheProcessor#authorizeCacheCreate 
,
 authorizeCacheDestroy 
)
REST authorises ADMIN_CACHE.(GridRestProcessor#authorize 
)
I think this is inconsistent.

I suggest ADMIN_CACHE mark @Deprecated
and replace it in the GridRestProcessor with CACHE_CREATE / CACHE_DESTROY 
while maintaining backward compatibility for ADMIN_CACHE.

This will allow us to remove ADMIN_CACHE in the future.



Sergei Ryzhov
s.vi.ryz...@gmail.com

[jira] [Created] (IGNITE-12846) Docs: ml package org.apache.ignite.ml.knn.utils.indices have no description in binary release

[Stepan Pilschikov (Jira)]

Stepan Pilschikov created IGNITE-12846:
--

 Summary: Docs: ml package org.apache.ignite.ml.knn.utils.indices 
have no description in binary release
 Key: IGNITE-12846
 URL: https://issues.apache.org/jira/browse/IGNITE-12846
 Project: Ignite
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.8
Reporter: Stepan Pilschikov


apache-ignite-2.8.0-bin/docs/javadoc/overview-summary.html does not contain 
description block for org.apache.ignite.ml.knn.utils.indices

Actual:
{code}


org.apache.ignite.ml.knn.utils.indices
 

{code}

Expected:
{code}


org.apache.ignite.ml.knn.utils.indices

[Some description]

{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)