[jira] [Created] (IGNITE-12262) Implement an Apache Camel Data Streamer

2019-10-04 Thread Emmanouil Gkatziouras (Jira)
Emmanouil Gkatziouras created IGNITE-12262: -- Summary: Implement an Apache Camel Data Streamer Key: IGNITE-12262 URL: https://issues.apache.org/jira/browse/IGNITE-12262 Project: Ignite

Re: Replacing default work dir from tmp to current dir

2019-10-04 Thread Denis Magda
Ilya, thanks for the ticket. However, I would advise us to enforce "user.home" as the only one default instead of the proposed fallback mechanism. There is already a lot "if else if else if else" scenarios in Ignite. Let's focus on simplicity and stick to one possible option when it works. This

Re: Getting involved in Apache Ignite

2019-10-04 Thread Denis Magda
Emmanouil, Thanks for reaching us out! It's great to have you as a contributor to Ignite. More integrations with the cloud ecosystem is an invaluable addition to the project. Ping me or anybody else within the community if you have any challenges with Ignite internals. Please feel free to start

[jira] [Created] (IGNITE-12261) Issue with adding nested index dynamically

2019-10-04 Thread Hemambara (Jira)
Hemambara created IGNITE-12261: -- Summary: Issue with adding nested index dynamically Key: IGNITE-12261 URL: https://issues.apache.org/jira/browse/IGNITE-12261 Project: Ignite Issue Type: Bug

Re: Getting involved in Apache Ignite

2019-10-04 Thread Emmanouil Gkatziouras
Thanks! Much appreciated! Regards, *Emmanouil Gkatziouras* https://egkatzioura.com/ | https://www.linkedin.com/in/gkatziourasemmanouil/ https://github.com/gkatzioura On Fri, 4 Oct 2019 at 16:25, Ilya Kasnacheev wrote: > Hello! > > I have added you to contributors, now you can assign issues

Re: ApacheCon Europe 2019 talks which are relevant to Apache Ignite

2019-10-04 Thread Denis Magda
Nice! Alexey, I completely forgot that you are going as a speaker. Kseniya, could you help to update our events page with all the upcoming Ignite events? https://ignite.apache.org/events.html - Denis On Fri, Oct 4, 2019 at 10:10 AM Alexey Zinoviev wrote: > Dear @myrle the link on my talk

Re: ApacheCon Europe 2019 talks which are relevant to Apache Ignite

2019-10-04 Thread Myrle Krantz
Thank you Alexey, I've corrected it in the remaining mailings that I'm still sending out. I look forward to seeing you in Berlin! Best, Myrle On Fri, Oct 4, 2019 at 7:10 PM Alexey Zinoviev wrote: > Dear @myrle the link on my talk about ML is incorrect > This is a correct link > >

Re: ApacheCon Europe 2019 talks which are relevant to Apache Ignite

2019-10-04 Thread Alexey Zinoviev
Dear @myrle the link on my talk about ML is incorrect This is a correct link https://aceu19.apachecon.com/session/ensembles-ml-algorithms-and-distributed-online-machine-learning-apache-ignite-0 пт, 4 окт. 2019 г. в 19:59, Denis Magda : > Igniters, > > Is anybody planning to visit the event this

Re: Metric showing how many nodes may safely leave the cluster

2019-10-04 Thread Denis Magda
I'm for the proposal to add new JMX metrics and enhance the existing tooling. But I would encourage us to integrate this into the new metrics framework Nikolay has been working on. Otherwise, we will be deprecating these JMX metrics in a short time frame in favor of the new monitoring APIs. -

Re: [DISCUSSION][IEP-35] Replace RunningQueryManager with GridSystemViewManager

2019-10-04 Thread Ivan Pavlukhin
Nikolay, > As I understand, RunningQueryManager track query only for an export. Not quite. Also it is responsible for explicit query cancellation and running queries cancellation on node stop. I do not think that a view should be responsible for it. пт, 4 окт. 2019 г. в 18:17, Nikolay Izhikov :

Re: ApacheCon Europe 2019 talks which are relevant to Apache Ignite

2019-10-04 Thread Denis Magda
Igniters, Is anybody planning to visit the event this year? Unfortunately, I won't be able to make it in Berlin. However, the one we had in Las Vegas this fall was amazing. So, highly recommended. - Denis On Fri, Oct 4, 2019 at 9:45 AM wrote: > Dear Apache Ignite committers, > > In a little

ApacheCon Europe 2019 talks which are relevant to Apache Ignite

2019-10-04 Thread myrle
Dear Apache Ignite committers, In a little over 2 weeks time, ApacheCon Europe is taking place in Berlin. Join us from October 22 to 24 for an exciting program and lovely get-together of the Apache Community. We are also planning a hackathon.  If your project is interested in participating,

Re: How to free up space on disc after removing entries from IgniteCache with enabled PDS?

2019-10-04 Thread Alexey Goncharuk
Maxim, Having a cluster-wide lock for a cache does not improve availability of the solution. A user cannot defragment a cache if the cache is involved in a mission-critical operation, so having a lock on such a cache is equivalent to the whole cluster shutdown. We should decide between either a

Re: Metric showing how many nodes may safely leave the cluster

2019-10-04 Thread Alexey Goncharuk
I agree that we should have the ability to read any metric using simple Ignite tooling. I am not sure if visor.sh is a good fit - if I remember correctly, it will start a daemon node which will bump the topology version with all related consequences. I believe in the long term it will beneficial

Re: Metric showing how many nodes may safely leave the cluster

2019-10-04 Thread Ivan Rakov
Nikolay, Many users start to use Ignite with a small project without production-level monitoring. When proof-of-concept appears to be viable, they tend to expand Ignite usage by growing cluster and adding needed environment (including monitoring systems). Inability to find such basic thing as

Re: Replacing default work dir from tmp to current dir

2019-10-04 Thread Ilya Kasnacheev
Hello! I have created https://issues.apache.org/jira/browse/IGNITE-12260 Regards, -- Ilya Kasnacheev пт, 4 окт. 2019 г. в 10:16, Ivan Pavlukhin : > Interesting things about those LINQPad/JPad scenarios. Was not aware > of it. Still some doubts about applicability. It seems to me that JPad >

[jira] [Created] (IGNITE-12260) Fallback to {user.home}/ignite/work if {user.dir} is not writable

2019-10-04 Thread Ilya Kasnacheev (Jira)
Ilya Kasnacheev created IGNITE-12260: Summary: Fallback to {user.home}/ignite/work if {user.dir} is not writable Key: IGNITE-12260 URL: https://issues.apache.org/jira/browse/IGNITE-12260 Project:

Re: Getting involved in Apache Ignite

2019-10-04 Thread Ilya Kasnacheev
Hello! I have added you to contributors, now you can assign issues to yourself. Please familiarise yourself with https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute Feel free to create new issues or even IEPs! https://cwiki.apache.org/confluence/display/IGNITE/Active+Proposals

Re: [DISCUSSION][IEP-35] Replace RunningQueryManager with GridSystemViewManager

2019-10-04 Thread Nikolay Izhikov
Ivan. > RunningQueryManager is responsible for tracking running queries (and query > history) As I understand, RunningQueryManager track query only for an export. So we don't need explicit entity for that, we already have System Views. В Пт, 04/10/2019 в 17:40 +0300, Ivan Pavlukhin пишет: >

Re: Getting involved in Apache Ignite

2019-10-04 Thread Emmanouil Gkatziouras
Greetings! Thank you for your response. This is my JIRA login https://issues.apache.org/jira/secure/ViewProfile.jspa?name=gkatzioura Once you add me to the contributors should I create the tickets I described on the initial mail? Kind regards, Emmanouil *Emmanouil Gkatziouras*

Re: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)

2019-10-04 Thread Andrey Mashenkov
Yuriy, Just FYI we have a review checklist [1], coding guidelines [2]. To test a PR someone can use TeamCity [3] or TeamCityBot project [4]. The last way (using TCBot) makes test validation much easier and do not bother with flacky tests. Long story short you can trigger tests for the PR from

Re: [DISCUSSION][IEP-35] Replace RunningQueryManager with GridSystemViewManager

2019-10-04 Thread Ivan Pavlukhin
Nikolay, Thank you for sharing knowledge. > I think we should replace `RunningQueryManager` with the special SystemView > implementation. Not sure that I got the intention and abstraction here. For me a straightforward approach here is to keep RunningQueryManager as is and use a new API to

[jira] [Created] (IGNITE-12259) Create new module for support spring-5.2.X and spring-data-2.2.X

2019-10-04 Thread Surkov Aleksandr (Jira)
Surkov Aleksandr created IGNITE-12259: - Summary: Create new module for support spring-5.2.X and spring-data-2.2.X Key: IGNITE-12259 URL: https://issues.apache.org/jira/browse/IGNITE-12259

Re: Metric showing how many nodes may safely leave the cluster

2019-10-04 Thread Nikolay Izhikov
Ivan. > We shouldn't force users to configure external tools and write extra code for > basic things. Actually, I don't agree with you. Having external monitoring system for any production cluster is a *basic* thing. Can you, please, define "basic things"? > single method for the whole

Re: Metric showing how many nodes may safely leave the cluster

2019-10-04 Thread Ivan Rakov
Max, What if user simply don't have configured monitoring system? Knowing whether cluster will survive node shutdown is critical for any administrator that performs any manipulations with cluster topology. Essential information should be easily accessed. We shouldn't force users to configure

Re: Metric showing how many nodes may safely leave the cluster

2019-10-04 Thread Ilya Kasnacheev
Hello! That's a very useful metric which we already discussed in the past. This may be called "cluster backup factor" and "effective cache backup factor". You can look up other mentions by searching in maillist archives. Regards, -- Ilya Kasnacheev пт, 4 окт. 2019 г. в 16:17, Ivan Rakov : >

Re: Metric showing how many nodes may safely leave the cluster

2019-10-04 Thread Alex Plehanov
Ivan, there already exist metric CacheGroupMetricsMXBean#getMinimumNumberOfPartitionCopies, which shows the current redundancy level for the cache group. We can lose up to ( getMinimumNumberOfPartitionCopies-1) nodes without data loss in this cache group. пт, 4 окт. 2019 г. в 16:17, Ivan Rakov :

Re: Metric showing how many nodes may safely leave the cluster

2019-10-04 Thread Maxim Muzafarov
Ivan, 1. I think the rebalance cache metrics should be deprecated and removed (someday). Here is the [1] issue to do such things. 2. I think #isRebalanceInProgress can and should be calculated by an external monitoring system from local nodes based on #localMovingPartitionsCount > 0 (or the more

Metric showing how many nodes may safely leave the cluster

2019-10-04 Thread Ivan Rakov
Igniters, I've seen numerous requests to find out an easy way to check whether is it safe to turn off cluster node. As we know, in Ignite protection from sudden node shutdown is implemented through keeping several backup copies of each partition. However, this guarantee can be weakened for a

Re: Getting involved in Apache Ignite

2019-10-04 Thread Ilya Kasnacheev
Hello! Do you have an Apache JIRA login? Please create it if not exists, share with us so that you can be added to contributors. Regards, -- Ilya Kasnacheev пт, 4 окт. 2019 г. в 10:59, Emmanouil Gkatziouras : > Greetings, > > I am amazed by Apache Ignite and its features! > For my use case

Re: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)

2019-10-04 Thread Yuriy Shuliga
Andrew, I have corrected PR according to your notes. Please review. What will be the next steps in order to merge in? Y. чт, 3 жовт. 2019 о 17:47 Andrey Mashenkov пише: > Yuri, > > I've done with review. > No crime found, but trivial compatibility bug. > > On Thu, Oct 3, 2019 at 3:54 PM Yuriy

Re: [DISCUSSION][IEP-35] Replace RunningQueryManager with GridSystemViewManager

2019-10-04 Thread Nikolay Izhikov
Hello, Ivan. > 1. How system views are going to be exposed? Is there any difference > in comparison to other metrics? We have a `SystemViewExporterSpi`. Built-in implementations are `JmxSystemViewExporterSpi` and `SqlViewExporterSpi`. > 2. What should be done to adopt RunningQueryManager to

[jira] [Created] (IGNITE-12258) .NET: Add ContinuousQueryWithTransformer

2019-10-04 Thread Pavel Tupitsyn (Jira)
Pavel Tupitsyn created IGNITE-12258: --- Summary: .NET: Add ContinuousQueryWithTransformer Key: IGNITE-12258 URL: https://issues.apache.org/jira/browse/IGNITE-12258 Project: Ignite Issue

Re: How to free up space on disc after removing entries from IgniteCache with enabled PDS?

2019-10-04 Thread Maxim Muzafarov
Igniters, This thread seems to be endless, but we if some kind of cache group distributed write lock (exclusive for some of the internal Ignite process) will be introduced? I think it will help to solve a batch of problems, like: 1. defragmentation of all cache group partitions on the local node

Re: [DISCUSSION][IEP-35] Replace RunningQueryManager with GridSystemViewManager

2019-10-04 Thread Ivan Pavlukhin
Nikolay, I checked the IEP [1]. Now it is more clear for me about SystemView API. Follow-up questions: 1. How system views are going to be exposed? Is there any difference in comparison to other metrics? 2. What should be done to adopt RunningQueryManager to SystemView API? Also some bits for my

Re: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)

2019-10-04 Thread Ivan Pavlukhin
Yuriy, Thank you, fine with it. пт, 4 окт. 2019 г. в 11:01, Yuriy Shuliga : > > Ivan, > > Yes, your observation is correct. > > This behavior lasts from the very beginning when Lucene indexing was > implemented for distributed queries. > Implementation of the *limit* solves the problem of

Re: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)

2019-10-04 Thread Yuriy Shuliga
Ivan, Yes, your observation is correct. This behavior lasts from the very beginning when Lucene indexing was implemented for distributed queries. Implementation of the *limit* solves the problem of redundant response size. Without it *ALL* off the records are fetched each time; that is not good,

Re: Getting involved in Apache Ignite

2019-10-04 Thread Emmanouil Gkatziouras
Greetings, I am amazed by Apache Ignite and its features! For my use case integrating with Google Cloud Pub/Sub and Amazon SQS would help getting the most out of it. Since developing those streamers is something I would do in any case, I would like to get involved in your project and therefore

Re: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)

2019-10-04 Thread Ivan Pavlukhin
Yuriy, Am I getting it right that in your PR if we have a limit N than returned items (at most N) will not be strictly the most relevant ones? E.g. if one node returned N items faster than others but with not so good relevance? чт, 3 окт. 2019 г. в 17:47, Andrey Mashenkov : > > Yuri, > > I've

Re: New SQL execution engine

2019-10-04 Thread Ivan Pavlukhin
Nikolay, Guys updated IEP [1]. Could you please check it? Are there any missing parts needed at that stage? [1] https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine вт, 1 окт. 2019 г. в 12:19, Ivan Pavlukhin : > > Folks, > > I marked IEP-33 as obsolete. Also

Re: Replacing default work dir from tmp to current dir

2019-10-04 Thread Ivan Pavlukhin
Interesting things about those LINQPad/JPad scenarios. Was not aware of it. Still some doubts about applicability. It seems to me that JPad having work dir in "Program Files" have a lot of problems by itself, e.g. a user is not able to run basic file IO snippets with relative file paths. чт, 3