Re: Professional Ignite Course

[sri hari kali charan Tummala]

I did went trough the course and I asked pluralsight to add more, in the
next course I would like to see ignite installation on aws and spark +
ignite integration.
Real time analytics with spark + ignite.


On Wednesday, September 25, 2019, Denis Magda  wrote:

> Igniters,
>
> Recently, I've come across a professionally crafted course about Apache
> Ignite. It will be interesting for some of you:
> https://www.pluralsight.com/courses/apache-ignite-getting-started
>
> Edward, thanks for the course! I've added it to Ignite Doc's menu right
> below the book.
>
> -
> Denis
>


-- 
Thanks & Regards
Sri Tummala

Professional Ignite Course

[Denis Magda]

Igniters,

Recently, I've come across a professionally crafted course about Apache
Ignite. It will be interesting for some of you:
https://www.pluralsight.com/courses/apache-ignite-getting-started

Edward, thanks for the course! I've added it to Ignite Doc's menu right
below the book.

-
Denis

Re: Application for Ignite Contributor

[Denis Magda]

Hi Deepak and welcome!

Sorry for the late response. You sent this message to the user list,
forwarded it to the dev list.

I added you to the Ignite contributors list in JIRA. Feel free to take over
any ticket you like. Do you have any specific interest (SQL, ML, caching,
etc.?

-
Denis


On Sat, Sep 21, 2019 at 2:35 AM Deepak Nigam 
wrote:

> Hello all,
>
> Please consider my application to become an Ignite Contributor. Here are
> the details:
>
> Full Name: Deepak Nigam
> ASF User Name: deepaknigam
> Email Address: deepaknigam.1...@gmail.com
>
> I am already a committer in the Apache OFBiz project, hence already signed
> ICLA.
>
> Regards,
> --
> Deepak Nigam
>

[jira] [Created] (IGNITE-12234) .NET: Fix flaky tests

[Pavel Tupitsyn (Jira)]

Pavel Tupitsyn created IGNITE-12234:
---

 Summary: .NET: Fix flaky tests
 Key: IGNITE-12234
 URL: https://issues.apache.org/jira/browse/IGNITE-12234
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn


Known flaky tests:

* CacheTestSsl.TestAsyncCompletionOrder
* DataStorageMetricsTest.TestDataStorageMetrics
* PersistenceTest.TestBaselineTopology
* CancellationTest.TestTask





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)

[Yuriy Shuliga]

Ivan,

Thank you for interesting question!

Text searches (or full text searches) are mostly human-oriented. And the
point of user's interest is topmost part of response.
Then user can read it, evaluate and use the given records for further
purposes.

Particularly in our case, we use Ignite for operations with financial data,
and there lots of text stuff like assets names, fin. instruments, companies
etc.
In order to operate with this quickly and reliably, users used to work with
text search, type-ahead completions, suggestions.

For this purposes we are indexing particular string data in separate caches.

Sorting capabilities and response size limitations are very important
there. As our API have to provide most relevant information in view of
limited size.

Now let me comment some Ignite/Lucene perspective.
Actually Ignite queries and Lucene returns *TopDocs.scoresDocs *already
sorted by *score *(relevance). So most relevant documents are on the top.
And currently distributed queries responses from different nodes are merged
into final query cursor queue in arbitrary way.
So in fact we already have the score order ruined here. Also Ignite
requests all possible documents from Lucene that is redundant and not good
for performance.

I'm implementing *limit* parameter to be part of *TextQuery *and have to
notice that we still have to add sorting for text queries processing in
order to have applicable results.

*Limit* parameter itself should improve the part of issues from above, but
definitely, sorting by document score at least  should be implemented along
with limit.

This is a pretty short commentary if you still have any questions, please
ask, do not hesitate)

BR,
Yuriy Shuliha

чт, 19 вер. 2019 о 11:38 Павлухин Иван  пише:

> Yuriy,
>
> Greatly appreciate your interest.
>
> Could you please elaborate a little bit about sorting? What tasks does
> it help to solve and how? It would be great to provide an example.
>
> ср, 18 сент. 2019 г. в 09:39, Alexei Scherbakov <
> alexey.scherbak...@gmail.com>:
> >
> > Denis,
> >
> > I like the idea of throwing an exception for enabled text queries on
> > persistent caches.
> >
> > Also I'm fine with proposed limit for unsorted searches.
> >
> > Yury, please proceed with ticket creation.
> >
> > вт, 17 сент. 2019 г., 22:06 Denis Magda :
> >
> > > Igniters,
> > >
> > > I see nothing wrong with Yury's proposal in regards full-text search
> API
> > > evolution as long as Yury is ready to push it forward.
> > >
> > > As for the in-memory mode only, it makes total sense for in-memory data
> > > grid deployments when Ignite caches data of an underlying DB like
> Postgres.
> > > As part of the changes, I would simply throw an exception (by default)
> if
> > > the one attempts to use text indices with the native persistence
> enabled.
> > > If the person is ready to live with that limitation that an explicit
> > > configuration change is needed to come around the exception.
> > >
> > > Thoughts?
> > >
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Tue, Sep 17, 2019 at 7:44 AM Yuriy Shuliga 
> wrote:
> > >
> > > > Hello to all again,
> > > >
> > > > Thank you for important comments and notes given below!
> > > >
> > > > Let me answer and continue the discussion.
> > > >
> > > > (I) Overall needs in Lucene indexing
> > > >
> > > > Alexei has referenced to
> > > > https://issues.apache.org/jira/browse/IGNITE-5371 where
> > > > absence of index persistence was declared as an obstacle to further
> > > > development.
> > > >
> > > > a) This ticket is already closed as not valid.b) There are definite
> needs
> > > > (and in our project as well) in just in-memory indexing of selected
> data.
> > > > We intend to use search capabilities for fetching limited amount of
> > > records
> > > > that should be used in type-ahead search / suggestions.
> > > > Not all of the data will be indexed and the are no need in Lucene
> index
> > > to
> > > > be persistence. Hope this is a wide pattern of text-search usage.
> > > >
> > > > (II) Necessary fixes in current implementation.
> > > >
> > > > a) Implementation of correct *limit *(*offset* seems to be not
> required
> > > in
> > > > text-search tasks for now)
> > > > I have investigated the data flow for distributed text queries. it
> was
> > > > simple test prefix query, like 'name'*='ene*'*
> > > > For now each server-node returns all response records to the
> client-node
> > > > and it may contain ~thousands, ~hundred thousands records.
> > > > Event if we need only first 10-100. Again, all the results are added
> to
> > > > queue in GridCacheQueryFutureAdapter in arbitrary order by pages.
> > > > I did not find here any means to deliver deterministic result.
> > > > So implementing limit as part of query and (GridCacheQueryRequest)
> will
> > > not
> > > > change the nature of response but will limit load on nodes and
> > > networking.
> > > >
> > > > Can we consider to open a ticket for this?
> > > >
> > > > (III) Further extension of 

Re: [MTCGA]: new failures in builds [4627134] needs to be handled

[Вячеслав Коптилин]

Hi Dmitrii,

> I prepared fix, waiting for TC results.
Thank you a lot!

Could you please share the JIRA ticket?

Thanks,
S.


ср, 25 сент. 2019 г. в 17:46, Dmitrii Ryabov :

> Bug looks strange. I prepared fix, waiting for TC results.
>

Re: [MTCGA]: new failures in builds [4627134] needs to be handled

[Dmitrii Ryabov]

Bug looks strange. I prepared fix, waiting for TC results.

[jira] [Created] (IGNITE-12233) Merge benchmarks sql index benchmarks from ignite-2.7 to master

[Ilya Suntsov (Jira)]

Ilya Suntsov created IGNITE-12233:
-

 Summary: Merge benchmarks sql index benchmarks from ignite-2.7 to 
master
 Key: IGNITE-12233
 URL: https://issues.apache.org/jira/browse/IGNITE-12233
 Project: Ignite
  Issue Type: Task
  Components: yardstick
Reporter: Ilya Suntsov
Assignee: Ilya Suntsov


The following benchmarks are missed in master but exist in ignite-2.7:
* IgniteSqlMergeIndexedValue1Benchmark
* IgniteSqlMergeIndexedValue2Benchmark
* IgniteSqlMergeIndexedValue8Benchmark
* IgniteSqlDeleteFilteredBenchmark
* IgniteSqlUpdateBenchmark
* IgniteSqlUpdateFilteredBenchmark
* IgniteSqlInsertIndexedValue1Benchmark
* IgniteSqlInsertIndexedValue2Benchmark
* IgniteSqlInsertIndexedValue8Benchmark 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12232) NPE while node join processing

[PetrovMikhail (Jira)]

PetrovMikhail created IGNITE-12232:
--

 Summary: NPE while node join processing
 Key: IGNITE-12232
 URL: https://issues.apache.org/jira/browse/IGNITE-12232
 Project: Ignite
  Issue Type: Bug
Reporter: PetrovMikhail


ServerImpl.RingMessageWorker#processNodeAddedMessage method throws npe 
exception in case DiscoverySpiNodeAuthenticator#authenticateNode returns null.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [MTCGA]: new failures in builds [4627134] needs to be handled

[Вячеслав Коптилин]

Hello Dmitrii Ryabov,

It seems that the following tests fail due to your change
https://issues.apache.org/jira/browse/IGNITE-11094
 *IgniteSinkConnectorTest.testSinkPutsWithoutTransformation
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=7952366899750322781=%3Cdefault%3E=testDetails
 *IgniteSinkConnectorTest.testSinkPutsWithTransformation
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-6495637752134628320=%3Cdefault%3E=testDetails


Could you please take a look?

Thanks,
S.

ср, 25 сент. 2019 г. в 04:09, :

> Hi Igniters,
>
>  I've detected some new issue on TeamCity to be handled. You are more than
> welcomed to help.
>
>  If your changes can lead to this failure(s): We're grateful that you were
> a volunteer to make the contribution to this project, but things change and
> you may no longer be able to finalize your contribution.
>  Could you respond to this email and indicate if you wish to continue and
> fix test failures or step down and some committer may revert you commit.
>
>  *New stable failure of a flaky test in master
> IgniteSinkConnectorTest.testSinkPutsWithoutTransformation
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=7952366899750322781=%3Cdefault%3E=testDetails
>
>  *New stable failure of a flaky test in master
> IgniteSinkConnectorTest.testSinkPutsWithTransformation
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-6495637752134628320=%3Cdefault%3E=testDetails
>  Changes may lead to failure were done by
>  - dmitrii ryabov 
> https://ci.ignite.apache.org/viewModification.html?modId=890732
>  - mstepachev 
> https://ci.ignite.apache.org/viewModification.html?modId=890735
>  - andrey gura 
> https://ci.ignite.apache.org/viewModification.html?modId=890728
>
>  - Here's a reminder of what contributors were agreed to do
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
>  - Should you have any questions please contact
> dev@ignite.apache.org
>
> Best Regards,
> Apache Ignite TeamCity Bot
> https://github.com/apache/ignite-teamcity-bot
> Notification generated at 04:08:55 25-09-2019
>

Re: Improvements for P2P class loading

[Denis Garus]

Denis, yes you are right, there is a reason for this.
It's the case when one of a participated node leaves a cluster and loader
send a request to the next node.
Also, the list of participated nodes is limited enough.
Thus, the proposed improvement a) is wrong.

But I think the point b) is still actual.

Thank you for the feedback!

ср, 25 сент. 2019 г. в 01:32, Denis Magda :

> Denis,
>
> Why do we broadcast the request to all the nodes in case of scenario a)?
> There should be a reason for that.
>
> Alex Goncharuk, do you remember we planned to revisit the P2P as part of
> Ignite 3.0 to remove certain limitations? How about linking all the P2P
> tasks together and create either an IEP or an umbrella ticket.
>
> -
> Denis
>
>
> On Tue, Sep 24, 2019 at 12:55 AM Denis Garus  wrote:
>
> > Igniters!
> >
> > I would like to propose a few improvements for the P2P class loading
> > feature in Ignite.
> > These improvements have the aim to reduce the number of requests that may
> > be needing to get a class on a remote node.
> >
> > a. We should send a request for a class to the node initiator firstly.
> > Currently, GridDeploymentClassLoader sends a request for a class to all
> > participated nodes one by one until it gets a successful response.
> However,
> > in most cases, the required byte code would be loaded from the node that
> > initiates execution. To find out what is the node initiator, we can use
> the
> > way that we use to determine a security context to execute a user-defined
> > code (see GridIoManager#invokeListener). If the node initiator doesn't
> have
> > a required class, we should ask other participated nodes as it is
> > currently.
> >
> > b. Some classes in a single response.
> > When a required class contains anonymous and/or inner classes, Ignite
> tries
> > to get these classes using separate requests. However, we can determine
> > that case and send data, that we send anyway, in a single response as an
> > answer for the first request. In this way, we may significantly reduce
> the
> > number of communications required for a class loading.
> >
> > What do you think?
> >
> > I would like to create a separate task for a. and b.
> >
>

[jira] [Created] (IGNITE-12231) RollbackRecord's must be flushed after logging to WAL

[Andrey Gura (Jira)]

Andrey Gura created IGNITE-12231:


 Summary: RollbackRecord's must be flushed after logging to WAL
 Key: IGNITE-12231
 URL: https://issues.apache.org/jira/browse/IGNITE-12231
 Project: Ignite
  Issue Type: Improvement
Reporter: Andrey Gura
Assignee: Andrey Gura
 Fix For: 2.8


Every record or records batch logged to WAL must call {{wal().flush()}} in 
order to save data on storage device.

{{TxPartitionCounterStateOnePrimaryTwoBackupsHistoryRebalanceTest.testPartialPrepare_2TX_1_1}}
 test fails periodically with disabled MMAP.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Dmitriy Pavlov]

Hi Maxim,

Thank you for preparing the release page!

Could you please add Require release notes filter? You can find an example
in  https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.7.6

Sincerely
Dmitriy Pavlov

ср, 25 сент. 2019 г. в 11:58, Maxim Muzafarov :

> Igniters,
>
>
> It's true that we are still discussing the release dates. But
> nevertheless, all the release blockers are important since some of
> them may require more than one month to be fixed. Let's discuss today
> how we will handle these issues and track Monitoring and ML major
> features to get them into the next release.
>
> The meeting already scheduled. We will use the ASF Slack on September
> 25-th, 17-00 (MSK).
> I've created the channel [2] #ignite-release-2_8 please, join.
> (Discussion will be on Russin language).
>
>
>
> Please, also note that I've created the 2.8 release confluence page
> [1] with additional information. I will review all the issues we have
> and will move some of them to 2.9.
> But currently, we've had pinned to 2.8:
>
> - 604 open issues
> - 57 in progress issues
> - 34 patch available issues
>
> - 17 issues marked as the release blockers
> - 4 of release blocker issues are unassigned
>
>
> [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.8
> [2] https://app.slack.com/client/T4S1WH2J3/CNQ51M4FQ
>
> On Wed, 25 Sep 2019 at 10:21, Dmitriy Pavlov  wrote:
> >
> > Hi Igniters,
> >
> > I suppose discussion is still at phase 0-Initializing
> > https://cwiki.apache.org/confluence/display/IGNITE/Release+Process
> >
> > So it is probably no reason to discuss particular blockers. It would make
> > sense when the process of removal irrelevant tickets starts (phase 1.2)
> and
> > till phase 4-Release candidate building.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > вт, 24 сент. 2019 г. в 18:53, Anton Kalashnikov :
> >
> > > Hello, Igniters.
> > >
> > > I want to notice one more blocker for release [1]. This bug can lead to
> > > some incorrect baseline default enabled flag calculation(more details
> in
> > > the ticket).
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-12227
> > >
> > > --
> > > Best regards,
> > > Anton Kalashnikov
> > >
> > >
> > > 24.09.2019, 17:01, "Andrey Gura" :
> > > > Sergey,
> > > >
> > > > As I know, scope freeze is not announced yet.
> > > >
> > > > On Tue, Sep 24, 2019 at 4:41 PM Sergey Antonov
> > > >  wrote:
> > > >>  Hi, I would add to release scope my ticket [1].
> > > >>
> > > >>  Any objections?
> > > >>
> > > >>  [1] https://issues.apache.org/jira/browse/IGNITE-12225
> > > >>
> > > >>  вт, 24 сент. 2019 г. в 09:21, Nikolay Izhikov  >:
> > > >>
> > > >>  > > merge to master only fully finished features
> > > >>  >
> > > >>  > It's already true for Ignite master branch.
> > > >>  >
> > > >>  >
> > > >>  > В Вт, 24/09/2019 в 09:03 +0300, Alexey Zinoviev пишет:
> > > >>  > > The planned before 2_3 months release dates are good defender
> from
> > > >>  > > partially merged features, In my opinion
> > > >>  > >
> > > >>  > > Or we should have Master and dev branch separetely, and merge
> to
> > > master
> > > >>  > > only fully finished features
> > > >>  > >
> > > >>  > > пн, 23 сент. 2019 г., 20:27 Maxim Muzafarov  >:
> > > >>  > >
> > > >>  > > > Andrey,
> > > >>  > > >
> > > >>  > > > Agree with you. It can affect the user impression.
> > > >>  > > >
> > > >>  > > > Can you advise, how can we guarantee in our case when we
> > > complete with
> > > >>  > > > current partially merged features that someone will not
> partially
> > > >>  > > > merge the new one? Should we monitor the master branch
> commits
> > > for
> > > >>  > > > such purpose?
> > > >>  > > >
> > > >>  > > > On Mon, 23 Sep 2019 at 20:18, Andrey Gura 
> > > wrote:
> > > >>  > > > >
> > > >>  > > > > Maxim,
> > > >>  > > > >
> > > >>  > > > > > > From my point, if some components will not be ready by
> > > >>  > > > > > > previously discussed `scope freeze` date it is
> absolutely
> > > OK to
> > > >>  > > > > > > perform the next (e.g. 2.8.1, 2.8.2) releases.
> > > >>  > > > >
> > > >>  > > > > It is good approach if partial implemented features aren't
> > > merged to
> > > >>  > > > > master branch. Unfortunately this is not our case.
> > > >>  > > > >
> > > >>  > > > > I don't see any reasons to force new Apache Ignite release.
> > > Time is
> > > >>  > > > > not driver for release. If we want release Ignite
> periodically
> > > we
> > > >>  > must
> > > >>  > > > > significantly review the process. And most valuable change
> in
> > > this
> > > >>  > > > > process is feature branches that will not block new
> release by
> > > >>  > design.
> > > >>  > > > >
> > > >>  > > > > On Mon, Sep 23, 2019 at 8:12 PM Andrey Gura <
> ag...@apache.org>
> > > >>  > wrote:
> > > >>  > > > > >
> > > >>  > > > > > > > From my point of view monitoring isn't ready for
> release.
> > > >>  > > > > > > Can you clarify, what exactly is not ready?
> > > >>  > > > > > > Can we track planned 

[jira] [Created] (IGNITE-12230) Partition eviction during cache stop / deactivation may cause errors leading to node failure and storage corruption

[Stepachev Maksim (Jira)]

Stepachev Maksim created IGNITE-12230:
-

 Summary: Partition eviction during cache stop / deactivation may 
cause errors leading to node failure and storage corruption
 Key: IGNITE-12230
 URL: https://issues.apache.org/jira/browse/IGNITE-12230
 Project: Ignite
  Issue Type: Bug
  Components: cache
Reporter: Stepachev Maksim
Assignee: Stepachev Maksim
 Fix For: 2.8


PartitionEvictionTask may produce NullPointerException if cache / cache group / 
cluser is stopping / deactivating.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [IEP-35] Metrics management in Ignite

[Nikolay Izhikov]

Hello, Andrey.

> My proposal target is simplification of metrics adding to the code.
> There is no any complexity.

That's great.

Looking forward for your PR.
Please, don't hesitate to request my review for it.

В Ср, 25/09/2019 в 13:23 +0300, Andrey Gura пишет:
> Nikolay,
> 
> > > 1. There is no unified approach to adding metrics during development.
> > Yes, we have it.
> > Just call `MetricRegistry#longMetric` or other method and you got your 
> > counter.
> 
> I talk about code structure, not API.
> 
> > > Now the logic is spread throughout the code base and there is now any
> > > way to find proper place where particular metric could be added to the
> > > registry
> > You should obtain your counter in the place where it be updated or exported.
> 
> And again, I talk about code structure, not API.
> 
> > > Moreover, we can create registry in one place and add some
> > > set of metrics to the registry in another place and we have many
> > > examples of it.
> > What's wrong with this approach?
> 
> Bad code organization. Redundant synchronization because there is no
> real life cases for changing metric registry content  at runtime.
> 
> > > Of course, metrics API allows it and it is meaningful drawback from my 
> > > point view
> > I know many cases where specific metrics was not implemented because of 
> > development complexity.
> >  From my point of view, we shouldn't complicate process of creation of 
> > specific metric.
> > It will slow down creation of new metric therefore slow down product 
> > evolving.
> 
> My proposal target is simplification of metrics adding to the code.
> There is no any complexity.
> 
> > > 2. Introduce MetricRegistryBuilder that returns immutable instance of 
> > > MetricRegistry
> > We can extend `AttibuteWalkerGenerator` to generate all required 
> > boilerplace code (for enabling/disabling metrics, etc.)
> 
> I think it is redundant complexity. Code generation is is a
> controversial approach. Metrics and system vies is not a case for code
> generation. IMHO.
> 
> > I want to bold my statement: We should keep metric creation as simple as we 
> > can.
> 
> Please, read my proposal again. Main point is code structure changing
> in order to simplify adding new metrics for developer.
> 
> > Developer of specific metric shouldn't change dozens of static interfaces, 
> > implementations, enable/disable methods.
> 
> Absolutely agree. But metric sources should have unified life-cycle methods.
> 
> > It should only update metric in the specific places in core code.
> 
> It is not related with my proposal and will true after implementation.
> 
> On Fri, Sep 20, 2019 at 1:03 PM Nikolay Izhikov  wrote:
> > 
> > Hello, Andrey.
> > 
> > Thanks for starting this discussion.
> > 
> > > 1. There is no unified approach to adding metrics during development.
> > 
> > Yes, we have it.
> > Just call `MetricRegistry#longMetric` or other method and you got your 
> > counter.
> > 
> > > Now the logic is spread throughout the code base and there is now any
> > > way to find proper place where particular metric could be added to the
> > > registry
> > 
> > You should obtain your counter in the place where it be updated or exported.
> > If you update the same counter in several place, it seems to be as a bad 
> > code design which should be investigated.
> > 
> > I treat this as a feature that adds flexibility to the Ignite code.
> > 
> > > Moreover, we can create registry in one place and add some
> > > set of metrics to the registry in another place and we have many
> > > examples of it.
> > 
> > What's wrong with this approach?
> > I treat this as a feature, also.
> > 
> > > Of course, metrics API allows it and it is meaningful drawback from my 
> > > point view
> > 
> > The goal of Metric API was simplifying of metric creation.
> > Because, in the past years Ignite doesn't have a way to create meaningfull 
> > metric.
> > I know many cases where specific metrics was not implemented because of 
> > development complexity.
> > 
> > From my point of view, we shouldn't complicate process of creation of 
> > specific metric.
> > It will slow down creation of new metric therefore slow down product 
> > evolving.
> > 
> > > 2. Introduce MetricRegistryBuilder that returns immutable instance of 
> > > MetricRegistry
> > 
> > If we want immutable MetricRegistry we can create simple POJO objects with 
> > metric getters as follows
> > 
> > public class CacheMetric {
> > public LongMetric putsCount() { ... }
> > public LongMetric getsCount() { ... }
> > }
> > 
> > and export it using `AttributeWalker` approach I implement in System View 
> > contribution.
> > There is no need for a HashMap unified solution.
> > 
> > We can extend `AttibuteWalkerGenerator` to generate all required 
> > boilerplace code (for enabling/disabling metrics, etc.)
> > 
> > I want to bold my statement: We should keep metric creation as simple as we 
> > can.
> > Developer of specific metric shouldn't change 

Re: [DISCUSSION] Single point in API for changing cluster state.

[Alexei Scherbakov]

Sergey Antonov,

The states ACTIVE, INACTIVE, READ-ONLY look confusing.
Actually read-only cluster is active too.

I would suggest adding new property to Ignite configuration like
setActivationOptions(ActivationOption... options) which should be mutable
in runtime.

For control.sh should be the same options for activate command.

Also it would be useful to allow users wait for re-balance which could
happen after activation in read-only mode to achieve really idle grid.

So, available options list in my opinion: READ_ONLY, WAIT_FOR_REBALANCE.

This proposal also better regarding backward compatibility.

вт, 24 сент. 2019 г. в 20:35, Sergey Antonov :

> Maxim,
>
> > I think from the user point the INACTIVE -> READ-ONLY transition [1]
> should be allowed prior to adding a new `state` command [2] to avoid
> unnecessary error messages.
> Yes. I plan complete both tickets till the code freeze of 2.8 release.
>
> >   I also think we can avoid the word 'set` in this command.
> I don't think so. We already have command control.sh --state command for
> getting current state. I think we shouldn't "overload" commands in
> control.sh.
>
> > What about cluster deactivation confirmation? Will it be used for all the
> cluster state changes?
> Yes. I think we should confirm any cluster state transition.
>
> вт, 24 сент. 2019 г. в 19:07, Maxim Muzafarov :
>
> > Sergey,
> >
> > +1, I like your idea.
> >
> > I think from the user point the INACTIVE -> READ-ONLY transition [1]
> > should be allowed prior to adding a new `state` command [2] to avoid
> > unnecessary error messages. I also think we can avoid the word 'set`
> > in this command.
> >
> > Example: control.sh --state ACTIVE [--yes]
> >
> >
> > What about cluster deactivation confirmation? Will it be used for all
> > the cluster state changes?
> >   Deactivate cluster:
> > control.(sh|bat) --deactivate [--yes]
> >
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-11866
> > [2] https://issues.apache.org/jira/browse/IGNITE-12225
> >
> >
> >
> > On Tue, 24 Sep 2019 at 16:50, Sergey Antonov 
> > wrote:
> > >
> > > Andrey,
> > >
> > > > What are state transitions valid?
> > > Now all transitions are valid, except INACTIVE -> READ-ONLY. This
> > > transition will be fixed under [1]
> > >
> > > > Regarding state names, as I understand, all transitions are valid
> from
> > > any to any of 3 states.
> > > Yes, see my comment above.
> > >
> > > > But, regarding on console.sh command it is not obvious.
> > > Yes. It's one of points why we should have single command in
> control.sh.
> > >
> > > > What effect will --read-only-on and --read-only-off commands have if
> > > current state is INACTIVE ?
> > > --read-only-on - cluster will be activated in read-only mode
> > > --read-only-off - cluster will be activated. I.e --read-only-off will
> > have
> > > the same effect as --activate
> > > [1] https://issues.apache.org/jira/browse/IGNITE-11866
> > >
> > > вт, 24 сент. 2019 г. в 16:40, Andrey Mashenkov <
> > andrey.mashen...@gmail.com>:
> > >
> > > > Sergey,
> > > >
> > > > What are state transitions valid?
> > > > For now we have only 2 states (active and inactive) and possible
> > > > transitions are obvious Active <--> Inactive.
> > > >
> > > > Regarding state names, as I understand, all transitions are valid
> from
> > any
> > > > to any of 3 states.
> > > > But, regarding on console.sh command it is not obvious.
> > > > What effect will --read-only-on and --read-only-off commands have if
> > > > current state is INACTIVE ?
> > > >
> > > >
> > > > On Tue, Sep 24, 2019 at 4:23 PM Sergey Antonov <
> > antonovserge...@gmail.com>
> > > > wrote:
> > > >
> > > > > Also, I would add IGNITE-12225
> > > > >  ticket to 2.8
> > > > release
> > > > > scope.
> > > > >
> > > > > вт, 24 сент. 2019 г. в 16:18, Sergey Antonov <
> > antonovserge...@gmail.com
> > > > >:
> > > > >
> > > > > > Hi, Igniters!
> > > > > >
> > > > > > We have 3 cluster states at the moment: inactive, active,
> > read-only.
> > > > > >
> > > > > > For getting current cluster state and changing them IgniteCluster
> > has
> > > > > > methods:
> > > > > >
> > > > > >- boolean active(), void active(boolean active) - for cluster
> > > > > >activation/deactivation
> > > > > >- boolean readOnly(), void readOnly(boolean readOnly) - for
> > > > > >enabling/disabling read-only mode.
> > > > > >
> > > > > > Also we have control.sh commans for changing cluster state:
> > > > > >
> > > > > >- --activate
> > > > > >- --deactivate
> > > > > >- --read-only-on
> > > > > >- --read-only-off
> > > > > >
> > > > > > For me current API looks unuseful. My proposal:
> > > > > >
> > > > > >1. Create enum ClusterState with values ACTIVE, INACTIVE,
> > READ-ONLY.
> > > > > >2. Add methods to IgniteCluster:
> > > > > >   - ClusterState state() returns current cluster state
> > > > > >   - void state(ClusterState newState) changes cluster 

Re: [IEP-35] Metrics management in Ignite

[Andrey Gura]

Nikolay,

>> 1. There is no unified approach to adding metrics during development.

> Yes, we have it.
> Just call `MetricRegistry#longMetric` or other method and you got your 
> counter.

I talk about code structure, not API.

>> Now the logic is spread throughout the code base and there is now any
>> way to find proper place where particular metric could be added to the
>> registry

> You should obtain your counter in the place where it be updated or exported.

And again, I talk about code structure, not API.

>> Moreover, we can create registry in one place and add some
>> set of metrics to the registry in another place and we have many
>> examples of it.

> What's wrong with this approach?

Bad code organization. Redundant synchronization because there is no
real life cases for changing metric registry content  at runtime.

>> Of course, metrics API allows it and it is meaningful drawback from my point 
>> view

> I know many cases where specific metrics was not implemented because of 
> development complexity.

>  From my point of view, we shouldn't complicate process of creation of 
> specific metric.
> It will slow down creation of new metric therefore slow down product evolving.

My proposal target is simplification of metrics adding to the code.
There is no any complexity.

>> 2. Introduce MetricRegistryBuilder that returns immutable instance of 
>> MetricRegistry

> We can extend `AttibuteWalkerGenerator` to generate all required boilerplace 
> code (for enabling/disabling metrics, etc.)

I think it is redundant complexity. Code generation is is a
controversial approach. Metrics and system vies is not a case for code
generation. IMHO.

> I want to bold my statement: We should keep metric creation as simple as we 
> can.

Please, read my proposal again. Main point is code structure changing
in order to simplify adding new metrics for developer.

> Developer of specific metric shouldn't change dozens of static interfaces, 
> implementations, enable/disable methods.

Absolutely agree. But metric sources should have unified life-cycle methods.

> It should only update metric in the specific places in core code.

It is not related with my proposal and will true after implementation.

On Fri, Sep 20, 2019 at 1:03 PM Nikolay Izhikov  wrote:
>
> Hello, Andrey.
>
> Thanks for starting this discussion.
>
> > 1. There is no unified approach to adding metrics during development.
>
> Yes, we have it.
> Just call `MetricRegistry#longMetric` or other method and you got your 
> counter.
>
> > Now the logic is spread throughout the code base and there is now any
> > way to find proper place where particular metric could be added to the
> > registry
>
> You should obtain your counter in the place where it be updated or exported.
> If you update the same counter in several place, it seems to be as a bad code 
> design which should be investigated.
>
> I treat this as a feature that adds flexibility to the Ignite code.
>
> > Moreover, we can create registry in one place and add some
> > set of metrics to the registry in another place and we have many
> > examples of it.
>
> What's wrong with this approach?
> I treat this as a feature, also.
>
> > Of course, metrics API allows it and it is meaningful drawback from my 
> > point view
>
> The goal of Metric API was simplifying of metric creation.
> Because, in the past years Ignite doesn't have a way to create meaningfull 
> metric.
> I know many cases where specific metrics was not implemented because of 
> development complexity.
>
> From my point of view, we shouldn't complicate process of creation of 
> specific metric.
> It will slow down creation of new metric therefore slow down product evolving.
>
> > 2. Introduce MetricRegistryBuilder that returns immutable instance of 
> > MetricRegistry
>
> If we want immutable MetricRegistry we can create simple POJO objects with 
> metric getters as follows
>
> public class CacheMetric {
> public LongMetric putsCount() { ... }
> public LongMetric getsCount() { ... }
> }
>
> and export it using `AttributeWalker` approach I implement in System View 
> contribution.
> There is no need for a HashMap unified solution.
>
> We can extend `AttibuteWalkerGenerator` to generate all required boilerplace 
> code (for enabling/disabling metrics, etc.)
>
> I want to bold my statement: We should keep metric creation as simple as we 
> can.
> Developer of specific metric shouldn't change dozens of static interfaces, 
> implementations, enable/disable methods.
> It should only update metric in the specific places in core code.
>
>
> В Пн, 26/08/2019 в 20:36 +0300, Andrey Gura пишет:
> > Hi, Igniters!
> >
> > I'm working on some metrics related aspects and it seems that we have
> > missed some points regarding the metrics management. There are at
> > least two major problems from my point of view.
> >
> > Problem definition.
> > ---
> >
> > 1. There is no unified approach to adding metrics during 

Re: Spark examples on TC

[Petr Ivanov]

I can help with JDK8 enforcement — just give me the links to target suites when 
agreed.


> On 25 Sep 2019, at 01:53, Denis Magda  wrote:
> 
> Nikolay,
> 
> Thanks for the details. How about enforcing JDK 8 for the Spark integration
> in general and run its tests on the VM only? We'll do this until Spark
> community fully support JDK 11+.
> 
> As for Spark 2.4.0 module [1], how about releasing Ignite 2.7.7 with Spark
> related changes only? As long as Ignite modularization is a long process,
> we can follow your advice and create two more modules ignite_spark_24 and
> ignite_spark_25 . Plus Alexey is working on extra Spark improvements.
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-12054
> -
> Denis
> 
> 
> On Thu, Sep 19, 2019 at 11:58 PM Nikolay Izhikov 
> wrote:
> 
>> Hello, Denis.
>> 
>> Spark uses Scala 2.11 [1]
>> Scala 2.11 compatible with the jdk version described on [2].
>> 
>> Few notes from the link:
>> 
>>> We recommend using Java 8 for compiling Scala code
>>> JDK 11 compatibility notes
>>> As of Scala 2.13.0, 2.12.8 and 2.11.12, JDK 11 support is incomplete
>> 
>> I also got several issues with the Spark itself when trying to run it on
>> the Java 11.
>> Please, take a look at [3], [4]
>> 
>> All in all, with the spark 2.3 we should stay on java8 to get things
>> worked.
>> 
>> [1] http://spark.apache.org/docs/2.4.0/
>> [2] https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html
>> [3]
>> 
>> https://stackoverflow.com/questions/49961991/java-lang-illegalargumentexception-at-org-apache-xbean-asm5-classreader-initu?noredirect=1=1
>> [4]
>> 
>> https://stackoverflow.com/questions/51352591/spark-java-illegalargumentexception-at-org-apache-xbean-asm5-classreader
>> 
>> 
>> чт, 19 сент. 2019 г. в 21:39, Denis Magda :
>> 
>>> Nikolay,
>>> 
>>> What needs to be done to ensure Java 9++ works with Spark Examples? Do we
>>> need to create another Spark module or drop some scala version?
>>> 
>>> -
>>> Denis
>>> 
>>> 
>>> On Thu, Sep 19, 2019 at 9:23 AM Nikolay Izhikov 
>>> wrote:
>>> 
 Hello, Igniters.
 
 Examples suite on TC doesn't work correctly under java9+. [1], [2]
 
 The reason is the Spark examples.
 
 Spark 2.3(our current version) work with scala 2.11 and scala 2.11
>>> support
 of java9+ is "incomplete".
 
 Can we stick example suite to jdk8 only?
 
 [1]
 
>>> 
>> https://ci.ignite.apache.org/viewLog.html?buildId=4610507=IgniteTests24Java8_Examples
 [2]
 
>>> 
>> https://ci.ignite.apache.org/viewLog.html?buildId=4609013=IgniteTests24Java8_Examples
 
>>> 
>> 

Re: nodes are restarting when i try to drop a table created with persistence enabled

[Denis Mekhanikov]

I think, the issue is that Ignite can't recover from
IgniteOutOfMemory, even by removing data.
Shiva, did IgniteOutOfMemory occur for the first time when you did the
DROP TABLE, or before that?

Denis

ср, 25 сент. 2019 г. в 02:30, Denis Magda :
>
> Shiva,
>
> Does this issue still exist? Ignite Dev how do we debug this sort of thing?
>
> -
> Denis
>
>
> On Tue, Sep 17, 2019 at 7:22 AM Shiva Kumar  wrote:
>>
>> Hi dmagda,
>>
>> I am trying to drop the table which has around 10 million records and I am 
>> seeing "Out of memory in data region" error messages in Ignite logs and 
>> ignite node [Ignite pod on kubernetes] is restarting.
>> I have configured 3GB for default data region, 7GB for JVM and total 15GB 
>> for Ignite container and enabled native persistence.
>> Earlier I was in an impression that restart was caused by 
>> "SYSTEM_WORKER_BLOCKED" errors but now I am realized that  
>> "SYSTEM_WORKER_BLOCKED" is added to ignore failure list and the actual cause 
>> is " CRITICAL_ERROR " due to  "Out of memory in data region"
>>
>> This is the error messages in logs:
>>
>> ""[2019-09-17T08:25:35,054][ERROR][sys-#773][] JVM will be halted 
>> immediately due to the failure: [failureCtx=FailureContext 
>> [type=CRITICAL_ERROR, err=class o.a.i.i.mem.IgniteOutOfMemoryException: 
>> Failed to find a page for eviction [segmentCapacity=971652, loaded=381157, 
>> maxDirtyPages=285868, dirtyPages=381157, cpPages=0, pinnedInSegment=3, 
>> failedToPrepare=381155]
>> Out of memory in data region [name=Default_Region, initSize=500.0 MiB, 
>> maxSize=3.0 GiB, persistenceEnabled=true] Try the following:
>>   ^-- Increase maximum off-heap memory size (DataRegionConfiguration.maxSize)
>>   ^-- Enable Ignite persistence (DataRegionConfiguration.persistenceEnabled)
>>   ^-- Enable eviction or expiration policies]]
>>
>> Could you please help me on why drop table operation causing  "Out of memory 
>> in data region"? and how I can avoid it?
>>
>> We have a use case where application inserts records to many tables in 
>> Ignite simultaneously for some time period and other applications run a 
>> query on that time period data and update the dashboard. we need to delete 
>> the records inserted in the previous time period before inserting new 
>> records.
>>
>> even during delete from table operation, I have seen:
>>
>> "Critical system error detected. Will be handled accordingly to configured 
>> handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, 
>> super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], 
>> failureCtx=FailureContext [type=CRITICAL_ERROR, err=class 
>> o.a.i.IgniteException: Checkpoint read lock acquisition has been timed 
>> out.]] class org.apache.ignite.IgniteException: Checkpoint read lock 
>> acquisition has been timed out.|
>>
>>
>>
>> On Mon, Apr 29, 2019 at 12:17 PM Denis Magda  wrote:
>>>
>>> Hi Shiva,
>>>
>>> That was designed to prevent global cluster performance degradation or 
>>> other outages. Have you tried to apply my recommendation of turning of the 
>>> failure handler for this system threads?
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Sun, Apr 28, 2019 at 10:28 AM shivakumar  
>>> wrote:

 HI Denis,

 is there any specific reason for the blocking of critical thread, like CPU
 is full or Heap is full ?
 We are again and again hitting this issue.
 is there any other way to drop tables/cache ?
 This looks like a critical issue.

 regards,
 shiva



 --
 Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Dmitriy Pavlov]

Hi Igniters,

I suppose discussion is still at phase 0-Initializing
https://cwiki.apache.org/confluence/display/IGNITE/Release+Process

So it is probably no reason to discuss particular blockers. It would make
sense when the process of removal irrelevant tickets starts (phase 1.2) and
till phase 4-Release candidate building.

Sincerely,
Dmitriy Pavlov

вт, 24 сент. 2019 г. в 18:53, Anton Kalashnikov :

> Hello, Igniters.
>
> I want to notice one more blocker for release [1]. This bug can lead to
> some incorrect baseline default enabled flag calculation(more details in
> the ticket).
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12227
>
> --
> Best regards,
> Anton Kalashnikov
>
>
> 24.09.2019, 17:01, "Andrey Gura" :
> > Sergey,
> >
> > As I know, scope freeze is not announced yet.
> >
> > On Tue, Sep 24, 2019 at 4:41 PM Sergey Antonov
> >  wrote:
> >>  Hi, I would add to release scope my ticket [1].
> >>
> >>  Any objections?
> >>
> >>  [1] https://issues.apache.org/jira/browse/IGNITE-12225
> >>
> >>  вт, 24 сент. 2019 г. в 09:21, Nikolay Izhikov :
> >>
> >>  > > merge to master only fully finished features
> >>  >
> >>  > It's already true for Ignite master branch.
> >>  >
> >>  >
> >>  > В Вт, 24/09/2019 в 09:03 +0300, Alexey Zinoviev пишет:
> >>  > > The planned before 2_3 months release dates are good defender from
> >>  > > partially merged features, In my opinion
> >>  > >
> >>  > > Or we should have Master and dev branch separetely, and merge to
> master
> >>  > > only fully finished features
> >>  > >
> >>  > > пн, 23 сент. 2019 г., 20:27 Maxim Muzafarov :
> >>  > >
> >>  > > > Andrey,
> >>  > > >
> >>  > > > Agree with you. It can affect the user impression.
> >>  > > >
> >>  > > > Can you advise, how can we guarantee in our case when we
> complete with
> >>  > > > current partially merged features that someone will not partially
> >>  > > > merge the new one? Should we monitor the master branch commits
> for
> >>  > > > such purpose?
> >>  > > >
> >>  > > > On Mon, 23 Sep 2019 at 20:18, Andrey Gura 
> wrote:
> >>  > > > >
> >>  > > > > Maxim,
> >>  > > > >
> >>  > > > > > > From my point, if some components will not be ready by
> >>  > > > > > > previously discussed `scope freeze` date it is absolutely
> OK to
> >>  > > > > > > perform the next (e.g. 2.8.1, 2.8.2) releases.
> >>  > > > >
> >>  > > > > It is good approach if partial implemented features aren't
> merged to
> >>  > > > > master branch. Unfortunately this is not our case.
> >>  > > > >
> >>  > > > > I don't see any reasons to force new Apache Ignite release.
> Time is
> >>  > > > > not driver for release. If we want release Ignite periodically
> we
> >>  > must
> >>  > > > > significantly review the process. And most valuable change in
> this
> >>  > > > > process is feature branches that will not block new release by
> >>  > design.
> >>  > > > >
> >>  > > > > On Mon, Sep 23, 2019 at 8:12 PM Andrey Gura 
> >>  > wrote:
> >>  > > > > >
> >>  > > > > > > > From my point of view monitoring isn't ready for release.
> >>  > > > > > > Can you clarify, what exactly is not ready?
> >>  > > > > > > Can we track planned changes somehow?
> >>  > > > > >
> >>  > > > > > We have too many not resolved tickets under IEP-35 label
> [1]. Also
> >>  > it
> >>  > > > > > makes sense to do some usability testing: JMX beans
> interfaces,
> >>  > system
> >>  > > > > > views, etc.
> >>  > > > > >
> >>  > > > > >
> >>  > > > > > [1]
> >>  > https://issues.apache.org/jira/issues/?jql=labels%20%3D%20IEP-35
> >>  > > > > >
> >>  > > > > > On Mon, Sep 23, 2019 at 6:04 PM Nikolay Izhikov <
> >>  > nizhi...@apache.org>
> >>  > > >
> >>  > > > wrote:
> >>  > > > > > >
> >>  > > > > > > Hello, Andrey.
> >>  > > > > > >
> >>  > > > > > > > From my point of view monitoring isn't ready for release.
> >>  > > > > > >
> >>  > > > > > > Can you clarify, what exactly is not ready?
> >>  > > > > > > Can we track planned changes somehow?
> >>  > > > > > >
> >>  > > > > > >
> >>  > > > > > > В Пн, 23/09/2019 в 17:59 +0300, Andrey Gura пишет:
> >>  > > > > > > > Igniters,
> >>  > > > > > > >
> >>  > > > > > > > From my point of view monitoring isn't ready for
> release. So it
> >>  > > >
> >>  > > > would
> >>  > > > > > > > be great to return to this discussion later. It seems
> that
> >>  > > >
> >>  > > > beginning
> >>  > > > > > > > of November is good time for it.
> >>  > > > > > > >
> >>  > > > > > > > On Mon, Sep 23, 2019 at 5:37 PM Alexey Zinoviev <
> >>  > > >
> >>  > > > zaleslaw@gmail.com> wrote:
> >>  > > > > > > > >
> >>  > > > > > > > > Nikolay Izhikov, ok, let's arrange the talk in ASF
> slack
> >>  > between
> >>  > > >
> >>  > > > 16 and 19
> >>  > > > > > > > > MSK, is it possible?
> >>  > > > > > > > >
> >>  > > > > > > > > пн, 23 сент. 2019 г. в 17:35, Alexey Zinoviev <
> >>  > > >
> >>  > > > zaleslaw@gmail.com>:
> >>  > > > > > > > >
> >>  > > > > > > > > > Ok, I'll clarify the situation
> >>  > > > > > > > > >
> >>  > > > > > > > > > 

Re: A question

[Павлухин Иван]

Hi Maria,

Yes, it should work with your classes. It worth noting, that
"lastName" will be accessible in your queries, "SELECT lastName FROM
..." should work. Note that you should use simply "lastName" in SQL
but not "general.lastName".

By the way, usually it is better to write such questions to Ignite
users list u...@ignite.apache.org, there you will definitely get the
answer.

вт, 24 сент. 2019 г. в 20:57, Мария Помазкина :
>
> Hello!
> Qould you please tell me,
> can I get Embeddable field data by sql query?
> My field is (I need to get lastNamefrow sql query):
>
> @Table(name = "rzd_employee")
> public class EmployeeExtVO implements Serializable {
>
> ...
>
> @Embedded
> @QuerySqlField
> private GeneralAttrs general;
> ...
> }
>
> @Embeddable
> public class GeneralAttrs implements Serializable {
>
>@QuerySqlField
> private String lastName;



-- 
Best regards,
Ivan Pavlukhin