Re: Ignite as distributed file storage
Dmitriy, Pavel,
Everything that gets accepted into the project has to make sense. I agree
with Vladimir - we do not need more than one file system in Ignite. Given
the number of usage and questions we get about IGFS, I would question
whether Ignite needs a file system at all.
As community members we should drive the community towards improving the
project instead of advocating that no change will be rejected, no matter
what it is. In this case, I am not convinced this is a real problem for
users and why should Ignite even try to solve it.
Instead, if we must focus on large blobs, I would solve the problem of
supporting large blobs in regular Ignite caches, as I suggested before.
D.
On Wed, Aug 1, 2018 at 2:50 AM, Dmitriy Pavlov
wrote:
> Hi Vladimir,
>
> I think not accepting by community is possible only if PMC will veto
> change. I didn't find any reasons why not to do this change and why it can
> be vetoed..
>
> I would appreciate if you will become mentor of this change and will assist
> to Pavel or other community member to make this happen.
>
> To my mind, the Apache Way is not abot rejecting things, it is about
> sharing knowlege. If you will be able to share you experience to grow
> community it would be good donation.
>
> If you have any disagreements about this change, can we set up voice call
> where you will explain how to do this proposal as good as it is possible.
>
> Sincerely,
> Dmitriy Pavlov
>
> пт, 6 июл. 2018 г. в 10:35, Vladimir Ozerov :
>
> > Pavel,
> >
> > I do not think it is a good idea to delay discussions and decisions.
> > Because it puts your efforts at risk being not accepted by community in
> the
> > end. Our ultimate goal is not having as much features as possible, but to
> > have a consistent product which is easy to understand and use. Having
> both
> > IGFS and another one "not-IGFS" which is in fact the same IGFS but with
> > different name is not a good idea, because it would cause more harm than
> > value.
> >
> > Approaches which seems reasonable to me:
> > 1) Integrate your ideas into IGFS, which is really flexible in how to
> > process data and where to store it. PROXY mode is not a "crutch" as you
> > said, but a normal mode which was used in real deployments.
> > 2) Replace IGFS with your solution but with clear explanation how it is
> > better than IGFS and why we need to drop thousands lines of battle-tested
> > code with something new, what does virtually the same thing
> > 3) Just drop IGFS from the product, and do not implement any replacement
> at
> > all - personally, I am all for this decision.
> >
> > If you want I can guide you through IGFS architecture so that we better
> > understand what should be done to integrate your ideas into it.
> >
> > Lat, but not least - we need objective facts why proposed solution is
> > better in terms of performance - concrete use cases and performance
> numbers
> > (or at least estimations).
> >
> > On Fri, Jul 6, 2018 at 1:45 AM Pavel Kovalenko
> wrote:
> >
> > > Vladimir,
> > >
> > > I just want to add to my words, that we can implement BLOB storage and
> > > then, if community really wants it, we can adapt this storage to use as
> > > underlying file system in IGFS. But IGFS shouldn't be entry point for
> > BLOB
> > > storage. I think this conclusion can satisfy both of us.
> > >
> > > 2018-07-06 0:47 GMT+03:00 Pavel Kovalenko :
> > >
> > > > Vladimir,
> > > >
> > > > I didn't say that it stores data in on-heap, I said that it performs
> a
> > > lot
> > > > of operations with byte[] arrays in on-heap as I see in , which will
> > lead
> > > > to frequent GCs and unnecessary data copying.
> > > > "But the whole idea around mmap sounds like premature optimisation to
> > me"
> > > > - this is not premature optimisation, this is on of the key
> performance
> > > > features. E.g. Apache Kafka wouldn't be so fast and extremely
> > performant
> > > > without zero-copy.
> > > > If we can do better, why not just do it? Especially if it costs
> nothing
> > > > for us (This is OS level).
> > > > As I said in my first message, our end target is handling video and
> > > > streaming, copying every chunk of it to on-heap userspace then to
> > offheap
> > > > and then to disk is unacceptable.
> > > > You ask me to implement almost anything using just IGFS interface,
> why
> > we
> > > > need to do that? Proxy mode looks like crutch, to support replication
> > and
> > > > possibility to have some data in-memory I need to write again a lot
> of
> > > > stuff.
> > > > Let's finally leave IGFS alone and wait for IEP.
> > > >
> > > >
> > > > 2018-07-06 0:01 GMT+03:00 Vladimir Ozerov :
> > > >
> > > >> Pavel,
> > > >>
> > > >> IGFS doesn't enforce you to have block in heap. What you suggest can
> > be
> > > >> achieved with IGFS as follows:
> > > >> 1) Disable caching, so data cache is not used ("PROXY" mode)
> > > >> 2) Implement IgniteFileSystem interface which operates on abstract
> > > streams
> > > >>
> > > >> But the whole
Re: Removing "fabric" from Ignite binary package name
I vote to remove the fabric from the build in the easiest way possible. Can other Igniters comment? D. On Wed, Aug 1, 2018 at 12:46 PM, Petr Ivanov wrote: > My concern here is exactly about internal build processes — removing > fabric from the name of binary archive (with any way) will break lots of > them. > There will be no sacrifices, just lots of work for fixing build processes > (where we won’t be able to introduce changes proactively). > > Therefore only fabric removal implementation (quick with some legacy left > or full refactoring) is on the agenda. > And this matter should be jugged by the community: currently we have (if > our voices are equal) 1:1 with Anton about it. > > > > > > On 1 Aug 2018, at 22:28, Dmitriy Setrakyan > wrote: > > > > Let's focus on what is important here. Our users do not care about our > > internal build process.If we could remove the word fabric from the next > > release without any significant sacrifices in the build process or making > > it less maintainable, I suggest we do it. > > > > D. > > > > On Wed, Aug 1, 2018 at 12:24 PM, Petr Ivanov > wrote: > > > >> Simple way with some hack and legacy maintenance: accept patch as it is > >> implemented now. > >> Hard way: full assembly refactoring and hadoop rejection. > >> > >> Anyway, after this is merged to master — complete automation systems > >> revision (TeamCity for example) is required due to heavy hardcode of > >> “fabric” in such systems. > >> > >> > >>> On 1 Aug 2018, at 21:55, Dmitriy Setrakyan > >> wrote: > >>> > >>> OK, so what is the plan? How do we get rid of the fabric name? > >>> > >>> D. > >>> > >>> On Wed, Aug 1, 2018 at 2:21 AM, Anton Vinogradov > wrote: > >>> > Since you proposing patch to the community, you are the very man :) > > ср, 1 авг. 2018 г. в 12:16, Petr Ivanov : > > > You are convincing the wrong person. > > > > > > > >> On 1 Aug 2018, at 12:05, Anton Vinogradov wrote: > >> > >> Peter, > >> > >> We had a discussion about how to do this properly. > >> Proposed solution cannot be merged, since it makes code harder than > it > > was. > >> > >> The only case is to perform complete refactoring and get rid of all > >> postfixes and other weird stuff. > >> > >> For example > >> - > >> - > >> should be definetely removed from code. > >> > >> > >> ср, 1 авг. 2018 г. в 9:39, Peter Ivanov : > >> > >>> The task was ready long ago, but community failed to review and > merge > it > >>> ¯\_(ツ)_/¯ > >>> Not being a committer, my capabilities of introducing such changes > >> are > >>> limited. > >>> > >>> I will update code during this week and will pass for review once > again. > >>> > >>> > >>> On Wed, 1 Aug 2018 at 00:24, Dmitriy Setrakyan < > >> dsetrak...@apache.org > > > >>> wrote: > >>> > Yes, agree, fabric has to be removed. If it is done in 2.7, would > be > >>> great! > > On Tue, Jul 31, 2018 at 2:18 PM, Denis Magda > > wrote: > > > Peter, folks, > > > > It's weird, but we have been failing to introduce this minor > change > >>> since > > December. Can we get it done for 2.7 that is being discussed at > the > moment? > > Are there any technical issues that block you from merging the > > changes? > > > > -- > > Denis > > > > On Thu, Jun 7, 2018 at 10:03 PM Peter Ivanov < > mr.wei...@gmail.com> > wrote: > > > >> Ok, then I will update issue code and start preparation for > build > >> configuration changes. > >> > >> > >> On Thu, 7 Jun 2018 at 23:41, Denis Magda > wrote: > >> > > With which one — current implementation in issue? > >>> > >>> > >>> That's the answer to your question: > >>> > >>> 1. quickly fix all of them (can be solved by preliminary > preparations — > >>> searching for -fabric- usages in build configuration); > >>> 2. update all branches to master because otherwise old branch > will > >> stop > >>> building. > >>> > >>> > >>> -- > >>> Denis > >>> > >>> On Thu, Jun 7, 2018 at 1:12 PM, Petr Ivanov < > mr.wei...@gmail.com > >>> > > wrote: > >>> > > > On 7 Jun 2018, at 23:04, Denis Magda > >>> wrote: > > > > I'm fine with the suggested approach. > > With which one — current implementation in issue? > > > > However, not sure we need to update > > all the branches. Can't branch owners just pull the changes > >>> back > > from > > master if the plan to merge back later? >
Re: IP finder in tests
Hi Yakov, Currently TC agents defended by Docker virtual network, that's why we don't see intersection between several clusters, but in case of any step aside (running several suites on one agent, running several tests on one machine and so on) we will have problems and return back to this conversation. I'm voting for simplifying and speeding up testing process. It will also reduce the number of copy-paste in ton of tests, where Vm Ip Finder is used explicitly. As developer I'm confusing when I see in a test VmIpFinder and in other test Multicast without any reason or comment. If you care about test coverage of MulticastIpFinder you can pick several suites where number of starting/stopping is most frequent and leave multicast there, but in general it's not necessary to have it everywhere. 2018-08-02 0:54 GMT+03:00 Yakov Zhdanov : > It should be true, otherwise we would have nodes from all agents > intersecting. No? > > And multicast IP finder is the defailt one, so I would not reduce its test > volume. > > Yakov Zhdanov > www.gridgain.com > > 2018-08-02 0:32 GMT+03:00 Dmitriy Pavlov : > > > Hi Yakov, > > > > Regarding Each TC agent use own multicast: I'm not sure it is true, TC > > admins tried to do so, but not succeded. > > > > One more reason is speed of tests run. Why do we need to scan something > if > > we always will connect localhost. TC tests do not use multicast in almost > > every test. > > > > Sincerely, > > Dmitriy Pavlov > > > > чт, 2 авг. 2018 г. в 0:27, Yakov Zhdanov : > > > > > I disagree. Probably, no change required. Each TC agent use own > multicast > > > group so nodes do not intersect. If any of the test does not properly > > clean > > > up and leaves nodes running this dhould be flagged as test fail which > is > > > the case. > > > > > > Please provide strong reasons to start with this. > > > > > > --Yakov > > > > > >
Re: [Distributed SQL] Do we have a plan to implement QuadTree index?
Hello Alexey, It's not so difficult to implement new type of indexing of data, but if you want to reach performance in distributed environment you need to have strong knowledge of a data you're indexing and what kind of queries you want to execute. Should be this index in-memory only or you want to persist it? In case of persistence your index should fit our page memory model requirements. In both cases your index should be ready to work in concurrent environment. In general index can be implemented in two ways per-partition and per-node. Per-partition may be efficient if you have a lot of points (x,y) representing a big one, e.g. image. In this case it's required that all of these points will be in one partition that query e.g. makes images intersection will execute in one node. But if you have multiple images, your index will contain also another points from other object and will overload it. Per-node may be efficient if you have a lot of points (x,y) that are independent of each other, that you will use it as spatial e.g.. But in this case, I think K-d tree is preferable as it can be used in more wide way. Could you please share use cases you're trying to solve with QuadTree? With close to real data and examples of queries? Because now the question is too abstract and it's hard to understand how it should be implemented to reach good results. 2018-08-01 16:45 GMT+03:00 Alexey Zinoviev : > Hi, Igniters. > > Currently I'm working on different math stuff over the Apache Ignite and in > a few tasks I need to implement in memory something like this > > https://en.wikipedia.org/wiki/Quadtree > > I didn't find such index in Apache Ignite, but maybe it's under development > by somebody? > > Is it a difficult to add a new index type to our distributed SQL (from > point of view of different infrastructure issues and so on P.S I don't > worry the math stuff here because I've implemented it many times in > non-distributed version)? > > It will be great to hear any kind of your thoughts and maybe somebody could > help with implementation > > zaleslaw >
Re: IP finder in tests
It should be true, otherwise we would have nodes from all agents intersecting. No? And multicast IP finder is the defailt one, so I would not reduce its test volume. Yakov Zhdanov www.gridgain.com 2018-08-02 0:32 GMT+03:00 Dmitriy Pavlov : > Hi Yakov, > > Regarding Each TC agent use own multicast: I'm not sure it is true, TC > admins tried to do so, but not succeded. > > One more reason is speed of tests run. Why do we need to scan something if > we always will connect localhost. TC tests do not use multicast in almost > every test. > > Sincerely, > Dmitriy Pavlov > > чт, 2 авг. 2018 г. в 0:27, Yakov Zhdanov : > > > I disagree. Probably, no change required. Each TC agent use own multicast > > group so nodes do not intersect. If any of the test does not properly > clean > > up and leaves nodes running this dhould be flagged as test fail which is > > the case. > > > > Please provide strong reasons to start with this. > > > > --Yakov > > >
Re: IP finder in tests
Hi Yakov, Regarding Each TC agent use own multicast: I'm not sure it is true, TC admins tried to do so, but not succeded. One more reason is speed of tests run. Why do we need to scan something if we always will connect localhost. TC tests do not use multicast in almost every test. Sincerely, Dmitriy Pavlov чт, 2 авг. 2018 г. в 0:27, Yakov Zhdanov : > I disagree. Probably, no change required. Each TC agent use own multicast > group so nodes do not intersect. If any of the test does not properly clean > up and leaves nodes running this dhould be flagged as test fail which is > the case. > > Please provide strong reasons to start with this. > > --Yakov >
Re: IP finder in tests
I disagree. Probably, no change required. Each TC agent use own multicast group so nodes do not intersect. If any of the test does not properly clean up and leaves nodes running this dhould be flagged as test fail which is the case. Please provide strong reasons to start with this. --Yakov
RE: Adding experimental support for Intel Optane DC Persistent Memory
Hi Dmitriy, Do you mean our LLPL library? It has a license, please look here: https://github.com/pmem/llpl Regarding the changes made to Ignite, you may refer to the pull request here: https://github.com/apache/ignite/pull/4381 Thanks, Mulugeta -Original Message- From: Dmitriy Pavlov [mailto:dpavlov@gmail.com] Sent: Wednesday, August 1, 2018 10:49 AM To: dev@ignite.apache.org Subject: Re: Adding experimental support for Intel Optane DC Persistent Memory Hi Mulugeta Mammo, I've just noticed that repository, what you refer is full fork of Ignite. How can I see differences with original Ignite? One more thing, library which you're referencing seems to not contain license, at least github can not parse it. Apache product has limitations which libraries may be used (see https://www.apache.org/legal/resolved.html#category-a and https://www.apache.org/legal/resolved.html#category-b) Could you please comment if there is some legal risk? Sincerely, Dmitriy Pavlov ср, 1 авг. 2018 г. в 20:43, Dmitriy Pavlov : > Hi, > > This link works for me > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-26%3A+Adding+Ex > perimental+Support+for+Intel+Optane+DC+Persistent+Memory > > Sincerely, > Dmitriy Pavlov > > чт, 26 июл. 2018 г. в 15:31, Stanislav Lukyanov : > >> Ah, ok, it’s just the ‘.’ at the end of the link. Removed it and it’s >> fine. >> >> From: Stanislav Lukyanov >> Sent: 26 июля 2018 г. 15:12 >> To: dev@ignite.apache.org >> Subject: RE: Adding experimental support for Intel Optane DC >> Persistent Memory >> >> Hi, >> >> The link you’ve shared gives me 404. >> Perhaps you need to add a permission for everyone to access the page? >> >> Thanks, >> Stan >> >> From: Mammo, Mulugeta >> Sent: 26 июля 2018 г. 2:44 >> To: dev@ignite.apache.org >> Subject: Adding experimental support for Intel Optane DC Persistent >> Memory >> >> Hi, >> >> I have added a new proposal to support Intel Optane DC Persistent >> Memory for Ignite here: >> https://cwiki.apache.org/confluence/display/IGNITE/Adding+Experimenta >> l+Support+for+Intel+Optane+DC+Persistent+Memory >> . >> >> I'm looking forward to your feedback and collaboration on this. >> >> Thanks, >> Mulugeta >> >> >> >>
Re: Spark DataFrames With Cache Key and Value Objects
What’s perhaps not clear is that the PrunedFilteredScan trait which needs
to be implemented to allow for predicate pushdown does need to return
RDD:
https://spark.apache.org/docs/2.3.0/api/java/org/apache/spark/sql/sources/PrunedFilteredScan.html
IgniteSQLRelation implements this to perform Ignite-led Spark SQL
predication.
Stuart.
On 1 Aug 2018, at 20:17, Valentin Kulichenko
wrote:
Stuart,
>From API standpoint Dataframe is just a Dataset of Rows, so they have the
same set of methods.
You have a valid point, by it's valid for both Dataset and Dataframe in the
same way. If you provide a function that filter Rows in a Dataframe,
current integration would also not take advantage of Ignite indexing.
In your example you should rather do 'filter(col("age") = 20)' or
'filter("age = 20")' in which case we can do our optimizations. Again, I'm
sure that would work in exact same way with Dataframes and Datasets, we
just need to provide proper support for the latter.
-Val
On Wed, Aug 1, 2018 at 11:52 AM Stuart Macdonald wrote:
> Val,
>
> Happy to clarify my thoughts. Let’s take an example, say we have an Ignite
> cache of Person objects, so in Nikolay’s Ignite Spark SQL implementation you
> can currently obtain a DataFrame with a column called “age” because that’s
> been registered as a field in Ignite. Then you can do something like this:
>
> Dataset = spark.sql(“select * from Person where age = 20”)
>
> And Spark will extract the age predicate and pass it back to the Ignite
> Relation implementation to perform the predication. This is useful because
> Ignite can index the age column and optimise the query appropriately.
>
> Now what I’m talking about being able to do is:
>
> Dataset = spark.sql(“select _val from Person where age =
> 20”).as(ignite.encoder(Person.class))
>
> This would also push the predicate to Ignite but then additionally allow
> access to the actual cache value objects in order to perform further local
> Spark operations on data which hasn’t been — or can’t be — registered as a
> field.
>
> Of course even without Nikolay’s implementation you can just grab a
> Dataset of Person objects from Ignite and then do something like:
>
> dataset.filter(p -> p.age = 20)
>
> But this has no way of passing that predicate back to Ignite, because the
> filter doesn’t go through Catalyst (which is the Spark engine which
> performs pushdown). Even if you convert that dataset to a dataframe and do
> a select() there is no way of pushing down that predicate. Spark has to
> have obtained the schema and dataframe from Ignite’s SQL relation
> implementation in order for the Spark Catalyst implementation to perform
> predicate pushdown.
>
> I’m not saying that we need to use Ignite’s _key or _val columns in order
> for this to work (though I have no other proposals as to how to make this
> work), but I am saying that an API which provides a Dataset will not
> to the best of my knowledge allow for predicate pushdown to Ignite.
>
> I’m likewise keen to hear Nikolay’s point of view as he is obviously the
> expert.
>
> Thanks for your help so far.
>
> Stuart.
>
> On 1 Aug 2018, at 18:17, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> Stuart,
>
> I don't see a reason why it would work with DataFrames, but not with
> Datasets - they are pretty much the same thing. If you have any particular
> thoughts on this, please let us know.
>
> In any case, I would like to hear from Nikolay as he is an implementor of
> this functionality. Nikolay, please share your thoughts on my suggestion
> above.
>
> -Val
>
> On Wed, Aug 1, 2018 at 12:05 AM Stuart Macdonald
> wrote:
>
>> I believe suggested approach will not work with the Spark SQL
>> relational optimisations which perform predicate pushdown from Spark
>> to Ignite. For that to work we need both the key/val and the
>> relational fields in a dataframe schema.
>>
>> Stuart.
>>
>> > On 1 Aug 2018, at 04:23, Valentin Kulichenko <
>> valentin.kuliche...@gmail.com> wrote:
>> >
>> > I don't think there are exact plans to remove _key and _value fields as
>> > it's pretty hard considering the fact that many users use them and that
>> > they are deeply integrated into the product. However, we already had
>> > multiple usability and other issues due to their existence, and while
>> > fixing them we gradually get rid of _key/_val on public API. Hard to
>> tell
>> > when we will be able to completely deprecate/remove these fields, but we
>> > definitely should avoid building new features based on them.
>> >
>> > On top of that, I also don't like this approach because it doesn't seem
>> to
>> > be Spark-friendly to me. That's not how they typically create typed
>> > datasets (I already provided a documentation link [1] with examples
>> > earlier).
>> >
>> > From API standpoint, I think we should do the following:
>> > 1. Add 'IgniteSparkSession#createDataset(IgniteCache[K, V] cache):
>> > Dataset[(K, V)]' method that would create a dataset based on a cache.
>> >
Re: Removing "fabric" from Ignite binary package name
My concern here is exactly about internal build processes — removing fabric from the name of binary archive (with any way) will break lots of them. There will be no sacrifices, just lots of work for fixing build processes (where we won’t be able to introduce changes proactively). Therefore only fabric removal implementation (quick with some legacy left or full refactoring) is on the agenda. And this matter should be jugged by the community: currently we have (if our voices are equal) 1:1 with Anton about it. > On 1 Aug 2018, at 22:28, Dmitriy Setrakyan wrote: > > Let's focus on what is important here. Our users do not care about our > internal build process.If we could remove the word fabric from the next > release without any significant sacrifices in the build process or making > it less maintainable, I suggest we do it. > > D. > > On Wed, Aug 1, 2018 at 12:24 PM, Petr Ivanov wrote: > >> Simple way with some hack and legacy maintenance: accept patch as it is >> implemented now. >> Hard way: full assembly refactoring and hadoop rejection. >> >> Anyway, after this is merged to master — complete automation systems >> revision (TeamCity for example) is required due to heavy hardcode of >> “fabric” in such systems. >> >> >>> On 1 Aug 2018, at 21:55, Dmitriy Setrakyan >> wrote: >>> >>> OK, so what is the plan? How do we get rid of the fabric name? >>> >>> D. >>> >>> On Wed, Aug 1, 2018 at 2:21 AM, Anton Vinogradov wrote: >>> Since you proposing patch to the community, you are the very man :) ср, 1 авг. 2018 г. в 12:16, Petr Ivanov : > You are convincing the wrong person. > > > >> On 1 Aug 2018, at 12:05, Anton Vinogradov wrote: >> >> Peter, >> >> We had a discussion about how to do this properly. >> Proposed solution cannot be merged, since it makes code harder than it > was. >> >> The only case is to perform complete refactoring and get rid of all >> postfixes and other weird stuff. >> >> For example >> - >> - >> should be definetely removed from code. >> >> >> ср, 1 авг. 2018 г. в 9:39, Peter Ivanov : >> >>> The task was ready long ago, but community failed to review and merge it >>> ¯\_(ツ)_/¯ >>> Not being a committer, my capabilities of introducing such changes >> are >>> limited. >>> >>> I will update code during this week and will pass for review once again. >>> >>> >>> On Wed, 1 Aug 2018 at 00:24, Dmitriy Setrakyan < >> dsetrak...@apache.org > >>> wrote: >>> Yes, agree, fabric has to be removed. If it is done in 2.7, would be >>> great! On Tue, Jul 31, 2018 at 2:18 PM, Denis Magda > wrote: > Peter, folks, > > It's weird, but we have been failing to introduce this minor change >>> since > December. Can we get it done for 2.7 that is being discussed at the moment? > Are there any technical issues that block you from merging the > changes? > > -- > Denis > > On Thu, Jun 7, 2018 at 10:03 PM Peter Ivanov wrote: > >> Ok, then I will update issue code and start preparation for build >> configuration changes. >> >> >> On Thu, 7 Jun 2018 at 23:41, Denis Magda wrote: >> With which one — current implementation in issue? >>> >>> >>> That's the answer to your question: >>> >>> 1. quickly fix all of them (can be solved by preliminary preparations — >>> searching for -fabric- usages in build configuration); >>> 2. update all branches to master because otherwise old branch will >> stop >>> building. >>> >>> >>> -- >>> Denis >>> >>> On Thu, Jun 7, 2018 at 1:12 PM, Petr Ivanov >> > wrote: >>> > On 7 Jun 2018, at 23:04, Denis Magda >>> wrote: > > I'm fine with the suggested approach. With which one — current implementation in issue? > However, not sure we need to update > all the branches. Can't branch owners just pull the changes >>> back > from > master if the plan to merge back later? Of course, we as an initiative group of this issue should do nothing, >> it will lie on shoulders of developers. > > -- > Denis > > On Thu, Jun 7, 2018 at 12:57 PM, Petr Ivanov < mr.wei...@gmail.com> wrote: > >> Denis, >> >> >> The most
Re: Removing "fabric" from Ignite binary package name
Let's focus on what is important here. Our users do not care about our internal build process.If we could remove the word fabric from the next release without any significant sacrifices in the build process or making it less maintainable, I suggest we do it. D. On Wed, Aug 1, 2018 at 12:24 PM, Petr Ivanov wrote: > Simple way with some hack and legacy maintenance: accept patch as it is > implemented now. > Hard way: full assembly refactoring and hadoop rejection. > > Anyway, after this is merged to master — complete automation systems > revision (TeamCity for example) is required due to heavy hardcode of > “fabric” in such systems. > > > > On 1 Aug 2018, at 21:55, Dmitriy Setrakyan > wrote: > > > > OK, so what is the plan? How do we get rid of the fabric name? > > > > D. > > > > On Wed, Aug 1, 2018 at 2:21 AM, Anton Vinogradov wrote: > > > >> Since you proposing patch to the community, you are the very man :) > >> > >> ср, 1 авг. 2018 г. в 12:16, Petr Ivanov : > >> > >>> You are convincing the wrong person. > >>> > >>> > >>> > On 1 Aug 2018, at 12:05, Anton Vinogradov wrote: > > Peter, > > We had a discussion about how to do this properly. > Proposed solution cannot be merged, since it makes code harder than it > >>> was. > > The only case is to perform complete refactoring and get rid of all > postfixes and other weird stuff. > > For example > - > - > should be definetely removed from code. > > > ср, 1 авг. 2018 г. в 9:39, Peter Ivanov : > > > The task was ready long ago, but community failed to review and merge > >> it > > ¯\_(ツ)_/¯ > > Not being a committer, my capabilities of introducing such changes > are > > limited. > > > > I will update code during this week and will pass for review once > >> again. > > > > > > On Wed, 1 Aug 2018 at 00:24, Dmitriy Setrakyan < > dsetrak...@apache.org > >>> > > wrote: > > > >> Yes, agree, fabric has to be removed. If it is done in 2.7, would be > > great! > >> > >> On Tue, Jul 31, 2018 at 2:18 PM, Denis Magda > >>> wrote: > >> > >>> Peter, folks, > >>> > >>> It's weird, but we have been failing to introduce this minor change > > since > >>> December. Can we get it done for 2.7 that is being discussed at the > >> moment? > >>> Are there any technical issues that block you from merging the > >>> changes? > >>> > >>> -- > >>> Denis > >>> > >>> On Thu, Jun 7, 2018 at 10:03 PM Peter Ivanov > >> wrote: > >>> > Ok, then I will update issue code and start preparation for build > configuration changes. > > > On Thu, 7 Jun 2018 at 23:41, Denis Magda > >> wrote: > > >> > >> With which one — current implementation in issue? > > > > > > That's the answer to your question: > > > > 1. quickly fix all of them (can be solved by preliminary > >> preparations — > > searching for -fabric- usages in build configuration); > > 2. update all branches to master because otherwise old branch > >> will > stop > > building. > > > > > > -- > > Denis > > > > On Thu, Jun 7, 2018 at 1:12 PM, Petr Ivanov > > >>> wrote: > > > >> > >>> On 7 Jun 2018, at 23:04, Denis Magda > > wrote: > >>> > >>> I'm fine with the suggested approach. > >> > >> With which one — current implementation in issue? > >> > >> > >>> However, not sure we need to update > >>> all the branches. Can't branch owners just pull the changes > > back > >>> from > >>> master if the plan to merge back later? > >> > >> Of course, we as an initiative group of this issue should do > >> nothing, > it > >> will lie on shoulders of developers. > >> > >> > >>> > >>> -- > >>> Denis > >>> > >>> On Thu, Jun 7, 2018 at 12:57 PM, Petr Ivanov < > >> mr.wei...@gmail.com> > >> wrote: > >>> > Denis, > > > The most simple approach — repack and rearchive binary archive > >>> after > release build, however that would not resolve the problem > >> globally > > (and > will require fixing every build configuration we have on > >>> TeamCity). > Current approach implemented in task — creates already correct > folder > >> and > binary archive name, but old name (with -fabric-) is used in > >>> almost > >> every > build configuration too and merge code to master will require > >> to: > 1. quickly fix all of them (can be solved by preliminary >
Re: Removing "fabric" from Ignite binary package name
Simple way with some hack and legacy maintenance: accept patch as it is implemented now. Hard way: full assembly refactoring and hadoop rejection. Anyway, after this is merged to master — complete automation systems revision (TeamCity for example) is required due to heavy hardcode of “fabric” in such systems. > On 1 Aug 2018, at 21:55, Dmitriy Setrakyan wrote: > > OK, so what is the plan? How do we get rid of the fabric name? > > D. > > On Wed, Aug 1, 2018 at 2:21 AM, Anton Vinogradov wrote: > >> Since you proposing patch to the community, you are the very man :) >> >> ср, 1 авг. 2018 г. в 12:16, Petr Ivanov : >> >>> You are convincing the wrong person. >>> >>> >>> On 1 Aug 2018, at 12:05, Anton Vinogradov wrote: Peter, We had a discussion about how to do this properly. Proposed solution cannot be merged, since it makes code harder than it >>> was. The only case is to perform complete refactoring and get rid of all postfixes and other weird stuff. For example - - should be definetely removed from code. ср, 1 авг. 2018 г. в 9:39, Peter Ivanov : > The task was ready long ago, but community failed to review and merge >> it > ¯\_(ツ)_/¯ > Not being a committer, my capabilities of introducing such changes are > limited. > > I will update code during this week and will pass for review once >> again. > > > On Wed, 1 Aug 2018 at 00:24, Dmitriy Setrakyan >> > wrote: > >> Yes, agree, fabric has to be removed. If it is done in 2.7, would be > great! >> >> On Tue, Jul 31, 2018 at 2:18 PM, Denis Magda >>> wrote: >> >>> Peter, folks, >>> >>> It's weird, but we have been failing to introduce this minor change > since >>> December. Can we get it done for 2.7 that is being discussed at the >> moment? >>> Are there any technical issues that block you from merging the >>> changes? >>> >>> -- >>> Denis >>> >>> On Thu, Jun 7, 2018 at 10:03 PM Peter Ivanov >> wrote: >>> Ok, then I will update issue code and start preparation for build configuration changes. On Thu, 7 Jun 2018 at 23:41, Denis Magda >> wrote: >> >> With which one — current implementation in issue? > > > That's the answer to your question: > > 1. quickly fix all of them (can be solved by preliminary >> preparations — > searching for -fabric- usages in build configuration); > 2. update all branches to master because otherwise old branch >> will stop > building. > > > -- > Denis > > On Thu, Jun 7, 2018 at 1:12 PM, Petr Ivanov >>> wrote: > >> >>> On 7 Jun 2018, at 23:04, Denis Magda > wrote: >>> >>> I'm fine with the suggested approach. >> >> With which one — current implementation in issue? >> >> >>> However, not sure we need to update >>> all the branches. Can't branch owners just pull the changes > back >>> from >>> master if the plan to merge back later? >> >> Of course, we as an initiative group of this issue should do >> nothing, it >> will lie on shoulders of developers. >> >> >>> >>> -- >>> Denis >>> >>> On Thu, Jun 7, 2018 at 12:57 PM, Petr Ivanov < >> mr.wei...@gmail.com> >> wrote: >>> Denis, The most simple approach — repack and rearchive binary archive >>> after release build, however that would not resolve the problem >> globally > (and will require fixing every build configuration we have on >>> TeamCity). Current approach implemented in task — creates already correct folder >> and binary archive name, but old name (with -fabric-) is used in >>> almost >> every build configuration too and merge code to master will require >> to: 1. quickly fix all of them (can be solved by preliminary > preparations — searching for -fabric- usages in build configuration); 2. update all branches to master because otherwise old > branch will stop building. WDYT? > On 7 Jun 2018, at 22:42, Denis Magda >> wrote: > > Petr, > > Thanks for pulling up the conversation. > > I still prefer us not to complicate the things and just > remove > "fabric" > from the *package
Re: Spark DataFrames With Cache Key and Value Objects
Stuart,
>From API standpoint Dataframe is just a Dataset of Rows, so they have the
same set of methods.
You have a valid point, by it's valid for both Dataset and Dataframe in the
same way. If you provide a function that filter Rows in a Dataframe,
current integration would also not take advantage of Ignite indexing.
In your example you should rather do 'filter(col("age") = 20)' or
'filter("age = 20")' in which case we can do our optimizations. Again, I'm
sure that would work in exact same way with Dataframes and Datasets, we
just need to provide proper support for the latter.
-Val
On Wed, Aug 1, 2018 at 11:52 AM Stuart Macdonald wrote:
> Val,
>
> Happy to clarify my thoughts. Let’s take an example, say we have an Ignite
> cache of Person objects, so in Nikolay’s Ignite Spark SQL implementation you
> can currently obtain a DataFrame with a column called “age” because that’s
> been registered as a field in Ignite. Then you can do something like this:
>
> Dataset = spark.sql(“select * from Person where age = 20”)
>
> And Spark will extract the age predicate and pass it back to the Ignite
> Relation implementation to perform the predication. This is useful because
> Ignite can index the age column and optimise the query appropriately.
>
> Now what I’m talking about being able to do is:
>
> Dataset = spark.sql(“select _val from Person where age =
> 20”).as(ignite.encoder(Person.class))
>
> This would also push the predicate to Ignite but then additionally allow
> access to the actual cache value objects in order to perform further local
> Spark operations on data which hasn’t been — or can’t be — registered as a
> field.
>
> Of course even without Nikolay’s implementation you can just grab a
> Dataset of Person objects from Ignite and then do something like:
>
> dataset.filter(p -> p.age = 20)
>
> But this has no way of passing that predicate back to Ignite, because the
> filter doesn’t go through Catalyst (which is the Spark engine which
> performs pushdown). Even if you convert that dataset to a dataframe and do
> a select() there is no way of pushing down that predicate. Spark has to
> have obtained the schema and dataframe from Ignite’s SQL relation
> implementation in order for the Spark Catalyst implementation to perform
> predicate pushdown.
>
> I’m not saying that we need to use Ignite’s _key or _val columns in order
> for this to work (though I have no other proposals as to how to make this
> work), but I am saying that an API which provides a Dataset will not
> to the best of my knowledge allow for predicate pushdown to Ignite.
>
> I’m likewise keen to hear Nikolay’s point of view as he is obviously the
> expert.
>
> Thanks for your help so far.
>
> Stuart.
>
> On 1 Aug 2018, at 18:17, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> Stuart,
>
> I don't see a reason why it would work with DataFrames, but not with
> Datasets - they are pretty much the same thing. If you have any particular
> thoughts on this, please let us know.
>
> In any case, I would like to hear from Nikolay as he is an implementor of
> this functionality. Nikolay, please share your thoughts on my suggestion
> above.
>
> -Val
>
> On Wed, Aug 1, 2018 at 12:05 AM Stuart Macdonald
> wrote:
>
>> I believe suggested approach will not work with the Spark SQL
>> relational optimisations which perform predicate pushdown from Spark
>> to Ignite. For that to work we need both the key/val and the
>> relational fields in a dataframe schema.
>>
>> Stuart.
>>
>> > On 1 Aug 2018, at 04:23, Valentin Kulichenko <
>> valentin.kuliche...@gmail.com> wrote:
>> >
>> > I don't think there are exact plans to remove _key and _value fields as
>> > it's pretty hard considering the fact that many users use them and that
>> > they are deeply integrated into the product. However, we already had
>> > multiple usability and other issues due to their existence, and while
>> > fixing them we gradually get rid of _key/_val on public API. Hard to
>> tell
>> > when we will be able to completely deprecate/remove these fields, but we
>> > definitely should avoid building new features based on them.
>> >
>> > On top of that, I also don't like this approach because it doesn't seem
>> to
>> > be Spark-friendly to me. That's not how they typically create typed
>> > datasets (I already provided a documentation link [1] with examples
>> > earlier).
>> >
>> > From API standpoint, I think we should do the following:
>> > 1. Add 'IgniteSparkSession#createDataset(IgniteCache[K, V] cache):
>> > Dataset[(K, V)]' method that would create a dataset based on a cache.
>> > 2. (Scala only) Introduce 'IgniteCache.toDS()' that would do the same,
>> but
>> > via implicit conversions instead of SparkSession extension.
>> >
>> > On implementation level, we can use SqlQuery API (not SqlFieldQuery)
>> that
>> > is specifically designed to return key-value pairs instead of specific
>> > fields, while still providing all SQL capabilities.
>> >
>> > *Nikolay*, does this
Re: Removing "fabric" from Ignite binary package name
OK, so what is the plan? How do we get rid of the fabric name? D. On Wed, Aug 1, 2018 at 2:21 AM, Anton Vinogradov wrote: > Since you proposing patch to the community, you are the very man :) > > ср, 1 авг. 2018 г. в 12:16, Petr Ivanov : > > > You are convincing the wrong person. > > > > > > > > > On 1 Aug 2018, at 12:05, Anton Vinogradov wrote: > > > > > > Peter, > > > > > > We had a discussion about how to do this properly. > > > Proposed solution cannot be merged, since it makes code harder than it > > was. > > > > > > The only case is to perform complete refactoring and get rid of all > > > postfixes and other weird stuff. > > > > > > For example > > > - > > > - > > > should be definetely removed from code. > > > > > > > > > ср, 1 авг. 2018 г. в 9:39, Peter Ivanov : > > > > > >> The task was ready long ago, but community failed to review and merge > it > > >> ¯\_(ツ)_/¯ > > >> Not being a committer, my capabilities of introducing such changes are > > >> limited. > > >> > > >> I will update code during this week and will pass for review once > again. > > >> > > >> > > >> On Wed, 1 Aug 2018 at 00:24, Dmitriy Setrakyan > > > >> wrote: > > >> > > >>> Yes, agree, fabric has to be removed. If it is done in 2.7, would be > > >> great! > > >>> > > >>> On Tue, Jul 31, 2018 at 2:18 PM, Denis Magda > > wrote: > > >>> > > Peter, folks, > > > > It's weird, but we have been failing to introduce this minor change > > >> since > > December. Can we get it done for 2.7 that is being discussed at the > > >>> moment? > > Are there any technical issues that block you from merging the > > changes? > > > > -- > > Denis > > > > On Thu, Jun 7, 2018 at 10:03 PM Peter Ivanov > > >>> wrote: > > > > > Ok, then I will update issue code and start preparation for build > > > configuration changes. > > > > > > > > > On Thu, 7 Jun 2018 at 23:41, Denis Magda > wrote: > > > > > >>> > > >>> With which one — current implementation in issue? > > >> > > >> > > >> That's the answer to your question: > > >> > > >> 1. quickly fix all of them (can be solved by preliminary > > >>> preparations — > > >> searching for -fabric- usages in build configuration); > > >>2. update all branches to master because otherwise old branch > > >>> will > > > stop > > >> building. > > >> > > >> > > >> -- > > >> Denis > > >> > > >> On Thu, Jun 7, 2018 at 1:12 PM, Petr Ivanov > > wrote: > > >> > > >>> > > On 7 Jun 2018, at 23:04, Denis Magda > > >> wrote: > > > > I'm fine with the suggested approach. > > >>> > > >>> With which one — current implementation in issue? > > >>> > > >>> > > However, not sure we need to update > > all the branches. Can't branch owners just pull the changes > > >> back > > from > > master if the plan to merge back later? > > >>> > > >>> Of course, we as an initiative group of this issue should do > > >>> nothing, > > > it > > >>> will lie on shoulders of developers. > > >>> > > >>> > > > > -- > > Denis > > > > On Thu, Jun 7, 2018 at 12:57 PM, Petr Ivanov < > > >>> mr.wei...@gmail.com> > > >>> wrote: > > > > > Denis, > > > > > > > > > The most simple approach — repack and rearchive binary archive > > after > > > release build, however that would not resolve the problem > > >>> globally > > >> (and > > > will require fixing every build configuration we have on > > TeamCity). > > > Current approach implemented in task — creates already correct > > > folder > > >>> and > > > binary archive name, but old name (with -fabric-) is used in > > almost > > >>> every > > > build configuration too and merge code to master will require > > >>> to: > > > 1. quickly fix all of them (can be solved by preliminary > > >> preparations > > > — searching for -fabric- usages in build configuration); > > > 2. update all branches to master because otherwise old > > >> branch > > > will > > > stop building. > > > > > > WDYT? > > > > > > > > > > > >> On 7 Jun 2018, at 22:42, Denis Magda > > >>> wrote: > > >> > > >> Petr, > > >> > > >> Thanks for pulling up the conversation. > > >> > > >> I still prefer us not to complicate the things and just > > >> remove > > >> "fabric" > > >> from the *package name*. Use the easiest way possible. > > >> > > >> Personally, I don't care about Hadoop and would not suggest > > >> the > > >>> community > > >> wasting its time on it. So, just rename the suffixes/prefixes > > >>> of > > > the > > > build > > >>
Re: Spark DataFrames With Cache Key and Value Objects
Val, Happy to clarify my thoughts. Let’s take an example, say we have an Ignite cache of Person objects, so in Nikolay’s Ignite Spark SQL implementation you can currently obtain a DataFrame with a column called “age” because that’s been registered as a field in Ignite. Then you can do something like this: Dataset = spark.sql(“select * from Person where age = 20”) And Spark will extract the age predicate and pass it back to the Ignite Relation implementation to perform the predication. This is useful because Ignite can index the age column and optimise the query appropriately. Now what I’m talking about being able to do is: Dataset = spark.sql(“select _val from Person where age = 20”).as(ignite.encoder(Person.class)) This would also push the predicate to Ignite but then additionally allow access to the actual cache value objects in order to perform further local Spark operations on data which hasn’t been — or can’t be — registered as a field. Of course even without Nikolay’s implementation you can just grab a Dataset of Person objects from Ignite and then do something like: dataset.filter(p -> p.age = 20) But this has no way of passing that predicate back to Ignite, because the filter doesn’t go through Catalyst (which is the Spark engine which performs pushdown). Even if you convert that dataset to a dataframe and do a select() there is no way of pushing down that predicate. Spark has to have obtained the schema and dataframe from Ignite’s SQL relation implementation in order for the Spark Catalyst implementation to perform predicate pushdown. I’m not saying that we need to use Ignite’s _key or _val columns in order for this to work (though I have no other proposals as to how to make this work), but I am saying that an API which provides a Dataset will not to the best of my knowledge allow for predicate pushdown to Ignite. I’m likewise keen to hear Nikolay’s point of view as he is obviously the expert. Thanks for your help so far. Stuart. On 1 Aug 2018, at 18:17, Valentin Kulichenko wrote: Stuart, I don't see a reason why it would work with DataFrames, but not with Datasets - they are pretty much the same thing. If you have any particular thoughts on this, please let us know. In any case, I would like to hear from Nikolay as he is an implementor of this functionality. Nikolay, please share your thoughts on my suggestion above. -Val On Wed, Aug 1, 2018 at 12:05 AM Stuart Macdonald wrote: > I believe suggested approach will not work with the Spark SQL > relational optimisations which perform predicate pushdown from Spark > to Ignite. For that to work we need both the key/val and the > relational fields in a dataframe schema. > > Stuart. > > > On 1 Aug 2018, at 04:23, Valentin Kulichenko < > valentin.kuliche...@gmail.com> wrote: > > > > I don't think there are exact plans to remove _key and _value fields as > > it's pretty hard considering the fact that many users use them and that > > they are deeply integrated into the product. However, we already had > > multiple usability and other issues due to their existence, and while > > fixing them we gradually get rid of _key/_val on public API. Hard to tell > > when we will be able to completely deprecate/remove these fields, but we > > definitely should avoid building new features based on them. > > > > On top of that, I also don't like this approach because it doesn't seem > to > > be Spark-friendly to me. That's not how they typically create typed > > datasets (I already provided a documentation link [1] with examples > > earlier). > > > > From API standpoint, I think we should do the following: > > 1. Add 'IgniteSparkSession#createDataset(IgniteCache[K, V] cache): > > Dataset[(K, V)]' method that would create a dataset based on a cache. > > 2. (Scala only) Introduce 'IgniteCache.toDS()' that would do the same, > but > > via implicit conversions instead of SparkSession extension. > > > > On implementation level, we can use SqlQuery API (not SqlFieldQuery) that > > is specifically designed to return key-value pairs instead of specific > > fields, while still providing all SQL capabilities. > > > > *Nikolay*, does this makes sense to you? Is it feasible and how hard > would > > it be to implement? How much of the existing code can we reuse (I believe > > it should it be majority of it)? > > > > [1] > > > https://spark.apache.org/docs/2.3.1/sql-programming-guide.html#creating-datasets > > > > -Val > > > >> On Tue, Jul 31, 2018 at 2:03 PM Denis Magda wrote: > >> > >> Hello folks, > >> > >> The documentation goes with a small reference about _key and _val usage, > >> and only for Ignite SQL APIs (Java, Net, C++). I tried to clean up all > the > >> documentation code snippets. > >> > >> As for the GitHub examples, they require a major overhaul. Instead of > _key > >> and _val usage, we need to use SQL fields. Hopefully, someone will groom > >> the examples. > >> > >> Considering this, I wouldn't suggest us exposing _key and _val in other > >>
[GitHub] ignite pull request #4398: IGNITE-8697 Flink sink throws java.lang.IllegalAr...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4398 ---
[GitHub] ignite pull request #4474: Ignite 2.4.5 p2
GitHub user aealeksandrov opened a pull request: https://github.com/apache/ignite/pull/4474 Ignite 2.4.5 p2 For TC run. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-2.4.5-p2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4474.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4474 commit 915dd2966084d78f7b4f3d482e6bd25f860c1e23 Author: Alexey Goncharuk Date: 2018-01-31T08:22:26Z IGNITE-7569 Fixed index rebuild future - Fixes #3454. commit 8ea8609259039852ab0c26f26ac528c1ffae7c94 Author: Alexey Goncharuk Date: 2018-01-31T08:24:57Z IGNITE-7577 Fixing public API active flag on baseline changes - Fixes #3455. commit c8ce1f66e98b3174d771a3b801a2538499dc2c3d Author: Ivan Rakov Date: 2018-01-31T09:51:09Z IGNITE-7475 Improved VerifyBackupPartitionsTask to calculate partition hashes in parallel - Fixes #3407. Signed-off-by: Alexey Goncharuk commit 258ff4299da20122d7c387cb8579264035c93c18 Author: Alexey Goncharuk Date: 2018-01-31T13:52:24Z IGNITE-7573 Fixed full API tests to be compliant with baseline topology commit 254ed3a9c32d092702a0461509bf867cbd7cdee6 Author: Alexey Kuznetsov Date: 2018-02-01T08:22:53Z ignite-2.4.0 Update version. (cherry picked from commit 2e43749) commit c1a9c0a404d77fba08170bedf14844f87abe3028 Author: Alexey Goncharuk Date: 2018-02-01T10:17:28Z IGNITE-7569 Fixing index rebuild future commit e43799ce70cdbe03d9e206381d1d5138b820b075 Author: Alexey Goncharuk Date: 2018-02-01T13:39:17Z IGNITE-7520 Provide util-methods to get baseline from context - Fixes #3431. commit 8f5fc7cfb0624cf2048efad38dfff32f782116e8 Author: Sergey Chugunov Date: 2018-02-02T08:24:29Z IGNITE-7580 Fix compatibilityMode flag consistency This closes #3466 (cherry picked from commit 8f2045e) commit d3ddd50cb2b889173176b6c47c9ff61410e1d909 Author: Ilya Lantukh Date: 2018-02-07T10:33:28Z IGNITE-7514 Affinity assignment should be recalculated when primary node is not OWNER (cherry picked from commit faf50f1) commit d3745e9d0a3ff5a64fba494889b7e2605f3af6bb Author: Alexey Goncharuk Date: 2018-02-07T18:10:32Z IGNITE-7639 Fixed NPE commit f7c16855ba802d9d47048521aec7e14285e4a281 Author: Pavel Kovalenko Date: 2018-02-09T13:55:15Z IGNITE-7540 Prevent page memory metadata corruption during checkpoint and group destroying. - Fixes #3490. Signed-off-by: Alexey Goncharuk commit c92f167fc491078f02b9f94fe89edafc2902ebc2 Author: ilantukh Date: 2018-02-14T12:40:13Z Updated version in properties. commit 1ecf348dd429cf7861b414e0e5a7776b72dba281 Author: Sergey Chugunov Date: 2018-02-16T13:21:12Z IGNITE-7699 BinaryMetadata exchange should not be triggered if metadata was not updated - Fixes #3523. Signed-off-by: Alexey Goncharuk (cherry-picked from commit bcd3881) commit 2458bd08a5b501b3eeb5caf0ae6dcaa2bcccd915 Author: EdShangGG Date: 2018-02-16T13:29:49Z IGNITE-7676 Add affinity version to snapshot plugin stub - Fixes #3510. Signed-off-by: Alexey Goncharuk (cherry picked from commit b6d21fb) commit bfdcda7a2a6b5cf64f15ed169d2beb886f131fac Author: EdShangGG Date: 2018-02-12T16:36:30Z IGNITE-7626 Unify code in test which cleans up persistence directories - Fixes #3477. Signed-off-by: Alexey Goncharuk (cherry picked from commit a0997b9) commit 2e92e0094b270aa8489e66d94bfcf15eadabfb4f Author: EdShangGG Date: 2018-02-12T18:44:10Z IGNITE-7626 Unify code in test which clean up persistence directories - Fixes #3512. Signed-off-by: Alexey Goncharuk (cherry picked from commit 6f6f8dd) commit 3f86c127c78065999663a4fc4eaedb5e5d4bee1c Author: EdShangGG Date: 2018-02-12T18:26:31Z compilation fix commit 0b9322c566f9b464291854142ac02495bd1817e4 Author: gg-shq Date: 2018-02-07T11:28:04Z IGNITE-6917: Implemented SQL COPY command. commit c5e386ca96750213bddcd98d0af0c589fee476ca Author: gg-shq Date: 2018-02-07T15:31:27Z IGNITE-7586: Added COPY command into the JDBC example. This closes #3485 commit d8203e2d81f8fbf0f7fbe5e710c9908f2fcb8307 Author: shq Date: 2018-02-15T10:36:00Z IGNITE-7709: SQL COPY command: make sure file name is always quoted. This closes #3526. commit 1185993ee7cd83695388f698f18f95b43e15de06 Author: devozerov Date: 2018-02-15T11:00:42Z IGNITE-7714: SQL COPY command: fixed "Table not found" issue on the client node. commit 88c8bdcc0dc2fdf2b2b22562a6b30031e053f671 Author: devozerov Date: 2018-02-16T14:54:24Z IGNITE-7737: SQL COPY: renamed BUFFER_SIZE to PACKET_SIZE. This closes #3533. commit bc331f9de716c30d6f733e28821ab44da7ed0cf7 Author: Alexander
Re: Adding experimental support for Intel Optane DC Persistent Memory
Hi Mulugeta Mammo, I've just noticed that repository, what you refer is full fork of Ignite. How can I see differences with original Ignite? One more thing, library which you're referencing seems to not contain license, at least github can not parse it. Apache product has limitations which libraries may be used (see https://www.apache.org/legal/resolved.html#category-a and https://www.apache.org/legal/resolved.html#category-b) Could you please comment if there is some legal risk? Sincerely, Dmitriy Pavlov ср, 1 авг. 2018 г. в 20:43, Dmitriy Pavlov : > Hi, > > This link works for me > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-26%3A+Adding+Experimental+Support+for+Intel+Optane+DC+Persistent+Memory > > Sincerely, > Dmitriy Pavlov > > чт, 26 июл. 2018 г. в 15:31, Stanislav Lukyanov : > >> Ah, ok, it’s just the ‘.’ at the end of the link. Removed it and it’s >> fine. >> >> From: Stanislav Lukyanov >> Sent: 26 июля 2018 г. 15:12 >> To: dev@ignite.apache.org >> Subject: RE: Adding experimental support for Intel Optane DC Persistent >> Memory >> >> Hi, >> >> The link you’ve shared gives me 404. >> Perhaps you need to add a permission for everyone to access the page? >> >> Thanks, >> Stan >> >> From: Mammo, Mulugeta >> Sent: 26 июля 2018 г. 2:44 >> To: dev@ignite.apache.org >> Subject: Adding experimental support for Intel Optane DC Persistent Memory >> >> Hi, >> >> I have added a new proposal to support Intel Optane DC Persistent Memory >> for Ignite here: >> https://cwiki.apache.org/confluence/display/IGNITE/Adding+Experimental+Support+for+Intel+Optane+DC+Persistent+Memory >> . >> >> I'm looking forward to your feedback and collaboration on this. >> >> Thanks, >> Mulugeta >> >> >> >>
Re: Adding experimental support for Intel Optane DC Persistent Memory
Hi, This link works for me https://cwiki.apache.org/confluence/display/IGNITE/IEP-26%3A+Adding+Experimental+Support+for+Intel+Optane+DC+Persistent+Memory Sincerely, Dmitriy Pavlov чт, 26 июл. 2018 г. в 15:31, Stanislav Lukyanov : > Ah, ok, it’s just the ‘.’ at the end of the link. Removed it and it’s fine. > > From: Stanislav Lukyanov > Sent: 26 июля 2018 г. 15:12 > To: dev@ignite.apache.org > Subject: RE: Adding experimental support for Intel Optane DC Persistent > Memory > > Hi, > > The link you’ve shared gives me 404. > Perhaps you need to add a permission for everyone to access the page? > > Thanks, > Stan > > From: Mammo, Mulugeta > Sent: 26 июля 2018 г. 2:44 > To: dev@ignite.apache.org > Subject: Adding experimental support for Intel Optane DC Persistent Memory > > Hi, > > I have added a new proposal to support Intel Optane DC Persistent Memory > for Ignite here: > https://cwiki.apache.org/confluence/display/IGNITE/Adding+Experimental+Support+for+Intel+Optane+DC+Persistent+Memory > . > > I'm looking forward to your feedback and collaboration on this. > > Thanks, > Mulugeta > > > >
Re: [MTCGA]: new failures in builds [1570367] needs to be handled
Great, thanks for the update. ср, 1 авг. 2018 г. в 20:31, Pavel Pereslegin : > Hello, Dmitriy. > > Yes, these tests was previously muted in another test suite. > I re-muted them again. > > 2018-08-01 20:15 GMT+03:00 Dmitriy Pavlov : > > Hi Ed, Pavel, > > > > Do you know, why these test were failed were failed? > > > > Is it because of new suite? > > > > Sincerely, > > Dmitriy Pavlov > > > > ср, 1 авг. 2018 г. в 20:12, : > >> > >> Hi Ignite Developer, > >> > >> I am MTCGA.Bot, and I've detected some issue on TeamCity to be > addressed. > >> I hope you can help. > >> > >> *Recently contributed test failed in master > >> TxRollbackAsyncWithPersistenceTest.testMixedAsyncRollbackTypes > >> > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=351762253942201430=%3Cdefault%3E=testDetails > >> > >> *Recently contributed test failed in master > >> > AuthenticationProcessorNodeRestartTest.testConcurrentAddUpdateRemoveNodeRestartCoordinator > >> > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7272291508896143933=%3Cdefault%3E=testDetails > >> > >> *Recently contributed test failed in master > >> WalModeChangeAdvancedSelfTest.testServerRestartCoordinator > >> > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=5593183502826541277=%3Cdefault%3E=testDetails > >> Changes may led to failure were done by > >> - nsamelchev > >> > http://ci.ignite.apache.org/viewModification.html?modId=827132=false > >> - biryukovvitaliy92 > >> > http://ci.ignite.apache.org/viewModification.html?modId=827130=false > >> - ppozerov > >> > http://ci.ignite.apache.org/viewModification.html?modId=827122=false > >> - ppozerov > >> > http://ci.ignite.apache.org/viewModification.html?modId=827121=false > >> - dmitrievanthony > >> > http://ci.ignite.apache.org/viewModification.html?modId=827116=false > >> - eshangareev > >> > http://ci.ignite.apache.org/viewModification.html?modId=827112=false > >> - kaa.dev > >> > http://ci.ignite.apache.org/viewModification.html?modId=827106=false > >> - alkuznetsov.sb > >> > http://ci.ignite.apache.org/viewModification.html?modId=827101=false > >> - akuznetsov > >> > http://ci.ignite.apache.org/viewModification.html?modId=827091=false > >> - akuznetsov > >> > http://ci.ignite.apache.org/viewModification.html?modId=827089=false > >> - dmitriyff > >> > http://ci.ignite.apache.org/viewModification.html?modId=827084=false > >> - ppozerov > >> > http://ci.ignite.apache.org/viewModification.html?modId=827066=false > >> - vsisko > >> > http://ci.ignite.apache.org/viewModification.html?modId=827036=false > >> - akuznetsov > >> > http://ci.ignite.apache.org/viewModification.html?modId=827034=false > >> - verbalab > >> > http://ci.ignite.apache.org/viewModification.html?modId=827032=false > >> - verbalab > >> > http://ci.ignite.apache.org/viewModification.html?modId=827030=false > >> - verbalab > >> > http://ci.ignite.apache.org/viewModification.html?modId=827028=false > >> - verbalab > >> > http://ci.ignite.apache.org/viewModification.html?modId=827027=false > >> - verbalab > >> > http://ci.ignite.apache.org/viewModification.html?modId=827025=false > >> - verbalab > >> > http://ci.ignite.apache.org/viewModification.html?modId=827023=false > >> > >> - If your changes can led to this failure(s), please create > issue > >> with label MakeTeamCityGreenAgain and assign it to you. > >> -- If you have fix, please set ticket to PA state and write to > dev > >> list fix is ready > >> -- For case fix will require some time please mute test and set > >> label Muted_Test to issue > >> - If you know which change caused failure please contact change > >> author directly > >> - If you don't know which change caused failure please send > >> message to dev list to find out > >> Should you have any questions please contact dpav...@apache.org or > write > >> to dev.list > >> Best Regards, > >> MTCGA.Bot > >> Notification generated at Wed Aug 01 20:12:42 MSK 2018 >
Re: IP finder in tests
Hi Dmitriy, I guess this ticket relates to examples, but new proposal is related only to test. Community didn't make an agreement about examples change, so I suggest to create new ticket (and probably link both tickets) and PR. Sincerely, Dmitriy Pavlov ср, 1 авг. 2018 г. в 20:30, Dmitrii Ryabov : > We have ticket [1] for this change. > > [1] https://issues.apache.org/jira/browse/IGNITE-6826 > > 2018-08-01 15:58 GMT+03:00 Alexey Zinoviev : > > > Great, sometimes I made it manually in local tests. It will be great to > > change the situation. > > > > 2018-08-01 18:22 GMT+06:00 Stanislav Lukyanov : > > > > > +1. > > > > > > I don’t see why do we need to fallback to multicast for multi-JVM – > let’s > > > just set 127.0.0.1:47500..47509 by default, > > > it’ll be enough for most (if not all) tests. > > > > > > Stan > > > > > > From: Dmitriy Pavlov > > > Sent: 1 августа 2018 г. 14:21 > > > To: dev@ignite.apache.org > > > Subject: Re: IP finder in tests > > > > > > Hi Denis, > > > > > > Thank you for bringing the question here. > > > > > > I totally support this change. > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > ср, 1 авг. 2018 г. в 13:23, Denis Mekhanikov : > > > > > > > Igniters, > > > > > > > > Almost every test in Ignite project has the following pattern: shared > > > > *TcpDiscoveryVmIpFinder > > > > *is defined as a field of a test class, which is then used in > discovery > > > > configuration in *getConfiguration(...)* method. There are more than > > 700 > > > > test classes with this setup. > > > > > > > > But for some reason *TcpDiscoveryMulticastIpFinder *is used in tests > by > > > > default. I don't think, that it should be used in tests at all, since > > > > external nodes may accidentally affect test results. > > > > > > > > The only case, where it makes sense is multi JVM tests. Shared static > > IP > > > > finder is not applicable there, since nodes are run in different JVMs > > and > > > > cannot share the same IP finder object. > > > > > > > > I would like to change the default IP finder to a shared > > > > *TcpDiscoveryVmIpFinder. > > > > *In cases, when *GridAbstractTest#isMultiJvm() *returns *true*, we > > could > > > > fall back to multicast. > > > > > > > > What do you think? > > > > > > > > Denis > > > > > > > > > > > > >
Re: [MTCGA]: new failures in builds [1570367] needs to be handled
Hello, Dmitriy. Yes, these tests was previously muted in another test suite. I re-muted them again. 2018-08-01 20:15 GMT+03:00 Dmitriy Pavlov : > Hi Ed, Pavel, > > Do you know, why these test were failed were failed? > > Is it because of new suite? > > Sincerely, > Dmitriy Pavlov > > ср, 1 авг. 2018 г. в 20:12, : >> >> Hi Ignite Developer, >> >> I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. >> I hope you can help. >> >> *Recently contributed test failed in master >> TxRollbackAsyncWithPersistenceTest.testMixedAsyncRollbackTypes >> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=351762253942201430=%3Cdefault%3E=testDetails >> >> *Recently contributed test failed in master >> AuthenticationProcessorNodeRestartTest.testConcurrentAddUpdateRemoveNodeRestartCoordinator >> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7272291508896143933=%3Cdefault%3E=testDetails >> >> *Recently contributed test failed in master >> WalModeChangeAdvancedSelfTest.testServerRestartCoordinator >> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=5593183502826541277=%3Cdefault%3E=testDetails >> Changes may led to failure were done by >> - nsamelchev >> http://ci.ignite.apache.org/viewModification.html?modId=827132=false >> - biryukovvitaliy92 >> http://ci.ignite.apache.org/viewModification.html?modId=827130=false >> - ppozerov >> http://ci.ignite.apache.org/viewModification.html?modId=827122=false >> - ppozerov >> http://ci.ignite.apache.org/viewModification.html?modId=827121=false >> - dmitrievanthony >> http://ci.ignite.apache.org/viewModification.html?modId=827116=false >> - eshangareev >> http://ci.ignite.apache.org/viewModification.html?modId=827112=false >> - kaa.dev >> http://ci.ignite.apache.org/viewModification.html?modId=827106=false >> - alkuznetsov.sb >> http://ci.ignite.apache.org/viewModification.html?modId=827101=false >> - akuznetsov >> http://ci.ignite.apache.org/viewModification.html?modId=827091=false >> - akuznetsov >> http://ci.ignite.apache.org/viewModification.html?modId=827089=false >> - dmitriyff >> http://ci.ignite.apache.org/viewModification.html?modId=827084=false >> - ppozerov >> http://ci.ignite.apache.org/viewModification.html?modId=827066=false >> - vsisko >> http://ci.ignite.apache.org/viewModification.html?modId=827036=false >> - akuznetsov >> http://ci.ignite.apache.org/viewModification.html?modId=827034=false >> - verbalab >> http://ci.ignite.apache.org/viewModification.html?modId=827032=false >> - verbalab >> http://ci.ignite.apache.org/viewModification.html?modId=827030=false >> - verbalab >> http://ci.ignite.apache.org/viewModification.html?modId=827028=false >> - verbalab >> http://ci.ignite.apache.org/viewModification.html?modId=827027=false >> - verbalab >> http://ci.ignite.apache.org/viewModification.html?modId=827025=false >> - verbalab >> http://ci.ignite.apache.org/viewModification.html?modId=827023=false >> >> - If your changes can led to this failure(s), please create issue >> with label MakeTeamCityGreenAgain and assign it to you. >> -- If you have fix, please set ticket to PA state and write to dev >> list fix is ready >> -- For case fix will require some time please mute test and set >> label Muted_Test to issue >> - If you know which change caused failure please contact change >> author directly >> - If you don't know which change caused failure please send >> message to dev list to find out >> Should you have any questions please contact dpav...@apache.org or write >> to dev.list >> Best Regards, >> MTCGA.Bot >> Notification generated at Wed Aug 01 20:12:42 MSK 2018
Re: IP finder in tests
We have ticket [1] for this change. [1] https://issues.apache.org/jira/browse/IGNITE-6826 2018-08-01 15:58 GMT+03:00 Alexey Zinoviev : > Great, sometimes I made it manually in local tests. It will be great to > change the situation. > > 2018-08-01 18:22 GMT+06:00 Stanislav Lukyanov : > > > +1. > > > > I don’t see why do we need to fallback to multicast for multi-JVM – let’s > > just set 127.0.0.1:47500..47509 by default, > > it’ll be enough for most (if not all) tests. > > > > Stan > > > > From: Dmitriy Pavlov > > Sent: 1 августа 2018 г. 14:21 > > To: dev@ignite.apache.org > > Subject: Re: IP finder in tests > > > > Hi Denis, > > > > Thank you for bringing the question here. > > > > I totally support this change. > > > > Sincerely, > > Dmitriy Pavlov > > > > ср, 1 авг. 2018 г. в 13:23, Denis Mekhanikov : > > > > > Igniters, > > > > > > Almost every test in Ignite project has the following pattern: shared > > > *TcpDiscoveryVmIpFinder > > > *is defined as a field of a test class, which is then used in discovery > > > configuration in *getConfiguration(...)* method. There are more than > 700 > > > test classes with this setup. > > > > > > But for some reason *TcpDiscoveryMulticastIpFinder *is used in tests by > > > default. I don't think, that it should be used in tests at all, since > > > external nodes may accidentally affect test results. > > > > > > The only case, where it makes sense is multi JVM tests. Shared static > IP > > > finder is not applicable there, since nodes are run in different JVMs > and > > > cannot share the same IP finder object. > > > > > > I would like to change the default IP finder to a shared > > > *TcpDiscoveryVmIpFinder. > > > *In cases, when *GridAbstractTest#isMultiJvm() *returns *true*, we > could > > > fall back to multicast. > > > > > > What do you think? > > > > > > Denis > > > > > > > >
Re: Spark DataFrames With Cache Key and Value Objects
Stuart, I don't see a reason why it would work with DataFrames, but not with Datasets - they are pretty much the same thing. If you have any particular thoughts on this, please let us know. In any case, I would like to hear from Nikolay as he is an implementor of this functionality. Nikolay, please share your thoughts on my suggestion above. -Val On Wed, Aug 1, 2018 at 12:05 AM Stuart Macdonald wrote: > I believe suggested approach will not work with the Spark SQL > relational optimisations which perform predicate pushdown from Spark > to Ignite. For that to work we need both the key/val and the > relational fields in a dataframe schema. > > Stuart. > > > On 1 Aug 2018, at 04:23, Valentin Kulichenko < > valentin.kuliche...@gmail.com> wrote: > > > > I don't think there are exact plans to remove _key and _value fields as > > it's pretty hard considering the fact that many users use them and that > > they are deeply integrated into the product. However, we already had > > multiple usability and other issues due to their existence, and while > > fixing them we gradually get rid of _key/_val on public API. Hard to tell > > when we will be able to completely deprecate/remove these fields, but we > > definitely should avoid building new features based on them. > > > > On top of that, I also don't like this approach because it doesn't seem > to > > be Spark-friendly to me. That's not how they typically create typed > > datasets (I already provided a documentation link [1] with examples > > earlier). > > > > From API standpoint, I think we should do the following: > > 1. Add 'IgniteSparkSession#createDataset(IgniteCache[K, V] cache): > > Dataset[(K, V)]' method that would create a dataset based on a cache. > > 2. (Scala only) Introduce 'IgniteCache.toDS()' that would do the same, > but > > via implicit conversions instead of SparkSession extension. > > > > On implementation level, we can use SqlQuery API (not SqlFieldQuery) that > > is specifically designed to return key-value pairs instead of specific > > fields, while still providing all SQL capabilities. > > > > *Nikolay*, does this makes sense to you? Is it feasible and how hard > would > > it be to implement? How much of the existing code can we reuse (I believe > > it should it be majority of it)? > > > > [1] > > > https://spark.apache.org/docs/2.3.1/sql-programming-guide.html#creating-datasets > > > > -Val > > > >> On Tue, Jul 31, 2018 at 2:03 PM Denis Magda wrote: > >> > >> Hello folks, > >> > >> The documentation goes with a small reference about _key and _val usage, > >> and only for Ignite SQL APIs (Java, Net, C++). I tried to clean up all > the > >> documentation code snippets. > >> > >> As for the GitHub examples, they require a major overhaul. Instead of > _key > >> and _val usage, we need to use SQL fields. Hopefully, someone will groom > >> the examples. > >> > >> Considering this, I wouldn't suggest us exposing _key and _val in other > >> places like Spark. Are there any alternatives to this approach? > >> > >> -- > >> Denis > >> > >> > >> > >> On Tue, Jul 31, 2018 at 2:49 AM Nikolay Izhikov > >> wrote: > >> > >>> Hello, Igniters. > >>> > >>> Valentin, > >>> > We never recommend to use these fields > >>> > >>> Actually, we did: > >>> > >>>* Documentation [1]. Please, see "Predefined Fields" section. > >>>* Java Example [2] > >>>* DotNet Example [3] > >>>* Scala Example [4] > >>> > ...hopefully will be removed altogether one day > >>> > >>> This is new for me. > >>> > >>> Do we have specific plans for it? > >>> > >>> [1] https://apacheignite-sql.readme.io/docs/schema-and-indexes > >>> [2] > >>> > >> > https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/sql/SqlDmlExample.java#L88 > >>> [3] > >>> > >> > https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/examples/Apache.Ignite.Examples/Sql/SqlDmlExample.cs#L91 > >>> [4] > >>> > >> > https://github.com/apache/ignite/blob/master/examples/src/main/scala/org/apache/ignite/scalar/examples/ScalarCachePopularNumbersExample.scala#L124 > >>> > >>> В Пт, 27/07/2018 в 15:22 -0700, Valentin Kulichenko пишет: > Stuart, > > _key and _val fields is quite a dirty hack that was added years ago > and > >>> is > virtually never used now. We never recommend to use these fields and I > would definitely avoid building new features based on them. > > Having said that, I'm not arguing the use case, but we need better > implementation approach here. I suggest we think it over and come back > >> to > this next week :) I'm sure Nikolay will also chime in and share his > thoughts. > > -Val > > On Fri, Jul 27, 2018 at 12:39 PM Stuart Macdonald > >>> wrote: > > > If your predicates and joins are expressed in Spark SQL, you cannot > > currently optimise those and also gain access to the key/val objects. > >>> If > > you went without
Re: [MTCGA]: new failures in builds [1570367] needs to be handled
Hi Ed, Pavel, Do you know, why these test were failed were failed? Is it because of new suite? Sincerely, Dmitriy Pavlov ср, 1 авг. 2018 г. в 20:12, : > Hi Ignite Developer, > > I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. > I hope you can help. > > *Recently contributed test failed in master > TxRollbackAsyncWithPersistenceTest.testMixedAsyncRollbackTypes > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=351762253942201430=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > AuthenticationProcessorNodeRestartTest.testConcurrentAddUpdateRemoveNodeRestartCoordinator > > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7272291508896143933=%3Cdefault%3E=testDetails > > *Recently contributed test failed in master > WalModeChangeAdvancedSelfTest.testServerRestartCoordinator > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=5593183502826541277=%3Cdefault%3E=testDetails > Changes may led to failure were done by > - nsamelchev > http://ci.ignite.apache.org/viewModification.html?modId=827132=false > - biryukovvitaliy92 > http://ci.ignite.apache.org/viewModification.html?modId=827130=false > - ppozerov > http://ci.ignite.apache.org/viewModification.html?modId=827122=false > - ppozerov > http://ci.ignite.apache.org/viewModification.html?modId=827121=false > - dmitrievanthony > http://ci.ignite.apache.org/viewModification.html?modId=827116=false > - eshangareev > http://ci.ignite.apache.org/viewModification.html?modId=827112=false > - kaa.dev > http://ci.ignite.apache.org/viewModification.html?modId=827106=false > - alkuznetsov.sb > http://ci.ignite.apache.org/viewModification.html?modId=827101=false > - akuznetsov > http://ci.ignite.apache.org/viewModification.html?modId=827091=false > - akuznetsov > http://ci.ignite.apache.org/viewModification.html?modId=827089=false > - dmitriyff > http://ci.ignite.apache.org/viewModification.html?modId=827084=false > - ppozerov > http://ci.ignite.apache.org/viewModification.html?modId=827066=false > - vsisko > http://ci.ignite.apache.org/viewModification.html?modId=827036=false > - akuznetsov > http://ci.ignite.apache.org/viewModification.html?modId=827034=false > - verbalab > http://ci.ignite.apache.org/viewModification.html?modId=827032=false > - verbalab > http://ci.ignite.apache.org/viewModification.html?modId=827030=false > - verbalab > http://ci.ignite.apache.org/viewModification.html?modId=827028=false > - verbalab > http://ci.ignite.apache.org/viewModification.html?modId=827027=false > - verbalab > http://ci.ignite.apache.org/viewModification.html?modId=827025=false > - verbalab > http://ci.ignite.apache.org/viewModification.html?modId=827023=false > > - If your changes can led to this failure(s), please create issue > with label MakeTeamCityGreenAgain and assign it to you. > -- If you have fix, please set ticket to PA state and write to dev > list fix is ready > -- For case fix will require some time please mute test and set > label Muted_Test to issue > - If you know which change caused failure please contact change > author directly > - If you don't know which change caused failure please send > message to dev list to find out > Should you have any questions please contact dpav...@apache.org or write > to dev.list > Best Regards, > MTCGA.Bot > Notification generated at Wed Aug 01 20:12:42 MSK 2018 >
[MTCGA]: new failures in builds [1570367] needs to be handled
Hi Ignite Developer, I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I hope you can help. *Recently contributed test failed in master TxRollbackAsyncWithPersistenceTest.testMixedAsyncRollbackTypes https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=351762253942201430=%3Cdefault%3E=testDetails *Recently contributed test failed in master AuthenticationProcessorNodeRestartTest.testConcurrentAddUpdateRemoveNodeRestartCoordinator https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7272291508896143933=%3Cdefault%3E=testDetails *Recently contributed test failed in master WalModeChangeAdvancedSelfTest.testServerRestartCoordinator https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=5593183502826541277=%3Cdefault%3E=testDetails Changes may led to failure were done by - nsamelchev http://ci.ignite.apache.org/viewModification.html?modId=827132=false - biryukovvitaliy92 http://ci.ignite.apache.org/viewModification.html?modId=827130=false - ppozerov http://ci.ignite.apache.org/viewModification.html?modId=827122=false - ppozerov http://ci.ignite.apache.org/viewModification.html?modId=827121=false - dmitrievanthony http://ci.ignite.apache.org/viewModification.html?modId=827116=false - eshangareev http://ci.ignite.apache.org/viewModification.html?modId=827112=false - kaa.dev http://ci.ignite.apache.org/viewModification.html?modId=827106=false - alkuznetsov.sb http://ci.ignite.apache.org/viewModification.html?modId=827101=false - akuznetsov http://ci.ignite.apache.org/viewModification.html?modId=827091=false - akuznetsov http://ci.ignite.apache.org/viewModification.html?modId=827089=false - dmitriyff http://ci.ignite.apache.org/viewModification.html?modId=827084=false - ppozerov http://ci.ignite.apache.org/viewModification.html?modId=827066=false - vsisko http://ci.ignite.apache.org/viewModification.html?modId=827036=false - akuznetsov http://ci.ignite.apache.org/viewModification.html?modId=827034=false - verbalab http://ci.ignite.apache.org/viewModification.html?modId=827032=false - verbalab http://ci.ignite.apache.org/viewModification.html?modId=827030=false - verbalab http://ci.ignite.apache.org/viewModification.html?modId=827028=false - verbalab http://ci.ignite.apache.org/viewModification.html?modId=827027=false - verbalab http://ci.ignite.apache.org/viewModification.html?modId=827025=false - verbalab http://ci.ignite.apache.org/viewModification.html?modId=827023=false - If your changes can led to this failure(s), please create issue with label MakeTeamCityGreenAgain and assign it to you. -- If you have fix, please set ticket to PA state and write to dev list fix is ready -- For case fix will require some time please mute test and set label Muted_Test to issue - If you know which change caused failure please contact change author directly - If you don't know which change caused failure please send message to dev list to find out Should you have any questions please contact dpav...@apache.org or write to dev.list Best Regards, MTCGA.Bot Notification generated at Wed Aug 01 20:12:42 MSK 2018
Re: [MTCGA]: new failures in builds [1564654] needs to be handled
Hi, Failure came from same test ( https://lists.apache.org/thread.html/de7c65383e3cd6c7de1667a1b7c28059761e5ac2c38206b018d90e7c@%3Cdev.ignite.apache.org%3E ) but from other suite, and already dicsussed. I will merge https://issues.apache.org/jira/browse/IGNITE-9157 once it is ready. Sincerely, ср, 1 авг. 2018 г. в 18:57, : > Hi Ignite Developer, > > I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. > I hope you can help. > > *New test failure in master > SlowHistoricalRebalanceSmallHistoryTest.testReservation > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=1100469653784618964=%3Cdefault%3E=testDetails > Changes may led to failure were done by > - plehanov.alex > http://ci.ignite.apache.org/viewModification.html?modId=827017=false > - biryukovvitaliy92 > http://ci.ignite.apache.org/viewModification.html?modId=827010=false > - vsisko > http://ci.ignite.apache.org/viewModification.html?modId=826941=false > - akuznetsov > http://ci.ignite.apache.org/viewModification.html?modId=826936=false > - vsisko > http://ci.ignite.apache.org/viewModification.html?modId=826932=false > - dmitriyff > http://ci.ignite.apache.org/viewModification.html?modId=826926=false > > - If your changes can led to this failure(s), please create issue > with label MakeTeamCityGreenAgain and assign it to you. > -- If you have fix, please set ticket to PA state and write to dev > list fix is ready > -- For case fix will require some time please mute test and set > label Muted_Test to issue > - If you know which change caused failure please contact change > author directly > - If you don't know which change caused failure please send > message to dev list to find out > Should you have any questions please contact dpav...@apache.org or write > to dev.list > Best Regards, > MTCGA.Bot > Notification generated at Wed Aug 01 18:57:40 MSK 2018 >
[MTCGA]: new failures in builds [1564654] needs to be handled
Hi Ignite Developer, I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I hope you can help. *New test failure in master SlowHistoricalRebalanceSmallHistoryTest.testReservation https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=1100469653784618964=%3Cdefault%3E=testDetails Changes may led to failure were done by - plehanov.alex http://ci.ignite.apache.org/viewModification.html?modId=827017=false - biryukovvitaliy92 http://ci.ignite.apache.org/viewModification.html?modId=827010=false - vsisko http://ci.ignite.apache.org/viewModification.html?modId=826941=false - akuznetsov http://ci.ignite.apache.org/viewModification.html?modId=826936=false - vsisko http://ci.ignite.apache.org/viewModification.html?modId=826932=false - dmitriyff http://ci.ignite.apache.org/viewModification.html?modId=826926=false - If your changes can led to this failure(s), please create issue with label MakeTeamCityGreenAgain and assign it to you. -- If you have fix, please set ticket to PA state and write to dev list fix is ready -- For case fix will require some time please mute test and set label Muted_Test to issue - If you know which change caused failure please contact change author directly - If you don't know which change caused failure please send message to dev list to find out Should you have any questions please contact dpav...@apache.org or write to dev.list Best Regards, MTCGA.Bot Notification generated at Wed Aug 01 18:57:40 MSK 2018
[GitHub] ignite pull request #4473: Ignite 2.4.8
GitHub user slukyano opened a pull request: https://github.com/apache/ignite/pull/4473 Ignite 2.4.8 You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-2.4.8 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4473.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4473 commit f9ce2c07046a1a413ea69ef5d32f91af70c06863 Author: Dmitriy Govorukhin Date: 2018-03-02T12:11:50Z IGNITE-7865 Supported serializerVersion method for WAL manager - Fixes #3594. Signed-off-by: Alexey Goncharuk (cherry picked from commit 932692e) commit e863e5af37a9ab3bc8c431de467a58ca683eb2ef Author: Alexey Goncharuk Date: 2018-03-05T09:01:50Z Merge branch 'ignite-2.4' of https://git-wip-us.apache.org/repos/asf/ignite into ignite-2.4.3 commit 176d98bbf33864e07976e01568e825c5f9c59b0a Author: Dmitriy Govorukhin Date: 2018-02-20T08:49:45Z IGNITE-7747 Exception should not be throw if segments not found for getAndReserveWalFiles. Stopped iteration if next segment not found - Fixes #3539. Signed-off-by: Alexey Goncharuk (cherry picked from commit b0359c7) commit f0ca3bfbf6a223110818b59f7a4d5be99e4e02b2 Author: Dmitriy Govorukhin Date: 2018-02-21T16:52:58Z IGNITE-7747 Exception should not be throw if segments not found for getAndReserveWalFiles. Stopped iteration if next segment not found. Additional fix. - Fixes #3539. (cherry picked from commit 5c2b7c8) commit e9321a2436f2f81c55cbd6a6163a11f487280186 Author: Alexey Kuznetsov Date: 2018-03-02T08:15:17Z IGNITE-7612 Web Console: Fixed ObjectId type. (cherry picked from commit eaf094b) commit 6a1869d30434a1607ab853723d7b306efdf84bb0 Author: Ilya Borisov Date: 2018-03-02T09:03:37Z IGNITE-7462 Web Console: Fixed templates. (cherry picked from commit a84c900) commit 98b5c559826858e8a496cdc8293b1d69329c607e Author: Alexey Kuznetsov Date: 2018-03-02T10:49:01Z IGNITE-7712 Web Console: SQL lazy mode enabled by default. (cherry picked from commit b3f9c15) commit b2e5c717162a0988399e63a7b0be88d5341bc7af Author: Ilya Borisov Date: 2018-03-05T07:49:51Z IGNITE-7064 Web Console: Fixed wrong syntax E2E tests. (cherry picked from commit 2fd5696) commit 17ecb70922a6706c7a219cd02000acbbb55d120b Author: Ilya Borisov Date: 2018-03-06T08:45:56Z IGNITE-7810 Use Roboto as sans-serif font in Bootstrap variables. (cherry picked from commit 16ed6b5) commit 6631c2cae9ffbd1a2b4f527477eddb5023fbae14 Author: Ilya Borisov Date: 2018-03-06T09:29:53Z IGNITE-5916 Web console: Fixed incorrect default value for sqlIndexMaxInlineSize in CacheConfiguration. (cherry picked from commit 7b9526b) commit 2892637ac95159c520df1f6583f87384534b5b82 Author: Alexey Kuznetsov Date: 2018-03-07T11:33:28Z IGNITE-7880 Web Console: Return enum values from SQL queries. (cherry picked from commit a8170f7) commit 4597818f2ea02c0f58e3072913ca6c21a6c46558 Author: Ivan Rakov Date: 2018-03-13T17:27:20Z IGNITE-7751 only fix of CP buffer overflow with enabled throttling is ported commit 1b187209411e54905e2ff219e10c605a1614ed57 Author: Anton Kalashnikov Date: 2018-03-12T09:53:36Z IGNITE-7869 Dynamic start cache by stored cache data (cherry picked from commit 4ee181c) commit bb202c0dcafc26c6e884eabc0e17daf1ac3d018f Author: Eduard Shangareev Date: 2018-03-14T13:48:23Z IGNITE-7947 Not all OWNING partitions saved in PartitionAllocationMap during checkpoint - Fixes #3632. commit a735dbf4901da738daa83a77baeb173c3d882c57 Author: Alexander Kalinin Date: 2018-03-16T10:05:33Z IGNITE-6390 Web-console: Clusters sorted alphabetically. (cherry picked from commit 6d85177) commit 12c2af68558eec24c57590932ff26fdf1d08f03b Author: Alexander Kalinin Date: 2018-03-16T10:13:41Z IGNITE-7929 Web Console: Fixed popover width. (cherry picked from commit 5b40243) commit 3a12c0798dfa2360c28cc19e9b0ddce2ee44c89b Author: Alexander Kalinin Date: 2018-03-16T10:25:59Z IGNITE-7970 Web Console: Fixed typo in notification test. (cherry picked from commit 094f6c3) commit eb57f986b45400a970d1e297dba63028579d2118 Author: Ilya Lantukh Date: 2018-03-16T11:25:13Z IGNITE-7890 Fixed error handing on corrupted storage during node startup commit 166042800bfa03090672d2de53cd2d599ced2dbf Author: Ilya Lantukh Date: 2018-03-16T13:54:23Z Fixed compilation of tests. commit 43fbc95d18e792680b949965c1759ca590cdd19d Author: Alexander Kalinin Date: 2018-03-14T04:23:29Z IGNITE-7895 Web Console: Replaced PhantomJS with headless Chrome. Cleanup dev dependencies. Added docker image to run tests on TeamCity.
[GitHub] ignite pull request #4177: IGNITE-8179: Fluky test.
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4177 ---
Re: [MTCGA]: new failures in builds [1564650] needs to be handled
This failure is known and happened because of missed maxSize of DataRegion. All such problems will be fixed in https://issues.apache.org/jira/browse/IGNITE-9157 2018-08-01 18:12 GMT+03:00 : > Hi Ignite Developer, > > I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. > I hope you can help. > > *New test failure in master > SlowHistoricalRebalanceSmallHistoryTest.testReservation > https://ci.ignite.apache.org/project.html?projectId= > IgniteTests24Java8=5380168528216195188=% > 3Cdefault%3E=testDetails > Changes may led to failure were done by > - plehanov.alex http://ci.ignite.apache.org/ > viewModification.html?modId=827017=false > - biryukovvitaliy92 http://ci.ignite.apache.org/ > viewModification.html?modId=827010=false > - vsisko http://ci.ignite.apache.org/viewModification.html?modId= > 826941=false > - akuznetsov http://ci.ignite.apache.org/ > viewModification.html?modId=826936=false > - vsisko http://ci.ignite.apache.org/viewModification.html?modId= > 826932=false > - dmitriyff http://ci.ignite.apache.org/ > viewModification.html?modId=826926=false > > - If your changes can led to this failure(s), please create issue > with label MakeTeamCityGreenAgain and assign it to you. > -- If you have fix, please set ticket to PA state and write to dev > list fix is ready > -- For case fix will require some time please mute test and set > label Muted_Test to issue > - If you know which change caused failure please contact change > author directly > - If you don't know which change caused failure please send > message to dev list to find out > Should you have any questions please contact dpav...@apache.org or write > to dev.list > Best Regards, > MTCGA.Bot > Notification generated at Wed Aug 01 18:12:40 MSK 2018 >
[MTCGA]: new failures in builds [1564650] needs to be handled
Hi Ignite Developer, I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed. I hope you can help. *New test failure in master SlowHistoricalRebalanceSmallHistoryTest.testReservation https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=5380168528216195188=%3Cdefault%3E=testDetails Changes may led to failure were done by - plehanov.alex http://ci.ignite.apache.org/viewModification.html?modId=827017=false - biryukovvitaliy92 http://ci.ignite.apache.org/viewModification.html?modId=827010=false - vsisko http://ci.ignite.apache.org/viewModification.html?modId=826941=false - akuznetsov http://ci.ignite.apache.org/viewModification.html?modId=826936=false - vsisko http://ci.ignite.apache.org/viewModification.html?modId=826932=false - dmitriyff http://ci.ignite.apache.org/viewModification.html?modId=826926=false - If your changes can led to this failure(s), please create issue with label MakeTeamCityGreenAgain and assign it to you. -- If you have fix, please set ticket to PA state and write to dev list fix is ready -- For case fix will require some time please mute test and set label Muted_Test to issue - If you know which change caused failure please contact change author directly - If you don't know which change caused failure please send message to dev list to find out Should you have any questions please contact dpav...@apache.org or write to dev.list Best Regards, MTCGA.Bot Notification generated at Wed Aug 01 18:12:40 MSK 2018
[GitHub] ignite pull request #4472: IGNITE-9148
GitHub user xtern opened a pull request: https://github.com/apache/ignite/pull/4472 IGNITE-9148 You can merge this pull request into a Git repository by running: $ git pull https://github.com/xtern/ignite IGNITE-9148 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4472.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4472 commit a7dc5e3d098ceb08774da8775955397b039c5852 Author: pereslegin-pa Date: 2018-08-01T14:57:53Z IGNITE-9148 Split Cache 3 suite into two. ---
[GitHub] ignite pull request #4471: IGNITE-9159 Basic 2 TC configuration is halted by...
GitHub user EdShangGG opened a pull request: https://github.com/apache/ignite/pull/4471 IGNITE-9159 Basic 2 TC configuration is halted by failure handler You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9159 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4471.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4471 commit 7178b22eb4596c5603878bd113a5162de0656447 Author: EdShangGG Date: 2018-08-01T14:45:36Z IGNITE-9159 Basic 2 TC configuration is halted by failure handler ---
[jira] [Created] (IGNITE-9159) Basic 2 TC configuration is halted by failure handler
Eduard Shangareev created IGNITE-9159:
-
Summary: Basic 2 TC configuration is halted by failure handler
Key: IGNITE-9159
URL: https://issues.apache.org/jira/browse/IGNITE-9159
Project: Ignite
Issue Type: Bug
Reporter: Eduard Shangareev
{code}
[01:19:21][org.apache.ignite:ignite-core] [2018-07-13
22:19:21,713][ERROR][exchange-worker-#3043%messaging.IgniteMessagingConfigVariationFullApiTest3%][IgniteTestResources]
JVM will be halted immediately due to the failure: [failureCtx=FailureContext
[type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteCheckedException: Failed
to send message (node may have left the grid or TCP connection cannot be
established due to firewall issues) [node=TcpDiscoveryNode
[id=b8f7b201-ab85-4a14-bcf0-2b397532, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3,
lastExchangeTime=1531520358330, loc=false, ver=2.7.0#20180713-sha1:6a8a2ff8,
isClient=false], topic=TOPIC_CACHE, msg=GridDhtPartitionsSingleMessage
[parts=null, partCntrs=null, partsSizes=null, partHistCntrs=null, err=null,
client=true, compress=true, finishMsg=null,
super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0],
discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=0fe68fcc-eca6-4f40-9f16-c6181fa0, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1531520358330, loc=false, ver=2.7.0#20180713-sha1:6a8a2ff8,
isClient=false], topVer=10, nodeId8=7b265a8e, msg=Node left: TcpDiscoveryNode
[id=0fe68fcc-eca6-4f40-9f16-c6181fa0, addrs=[127.0.0.1],
sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1531520358330, loc=false, ver=2.7.0#20180713-sha1:6a8a2ff8,
isClient=false], type=NODE_LEFT, tstamp=1531520361620], nodeId=0fe68fcc,
evt=NODE_LEFT], lastVer=GridCacheVersion [topVer=0, order=1531520357089,
nodeOrder=0], super=GridCacheMessage [msgId=238, depInfo=null, err=null,
skipPrepare=false]]], policy=2]]]
[01:19:22][org.apache.ignite:ignite-core] Process exited with code 130
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [ML] Machine Learning Pipeline Improvement
Sure, https://issues.apache.org/jira/browse/IGNITE-9158. Regards, Yury -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
[jira] [Created] (IGNITE-9158) [ML] Pipeline
Yury Babak created IGNITE-9158: -- Summary: [ML] Pipeline Key: IGNITE-9158 URL: https://issues.apache.org/jira/browse/IGNITE-9158 Project: Ignite Issue Type: New Feature Components: ml Reporter: Yury Babak Assignee: Aleksey Zinoviev Fix For: 2.7 We want to implement our own pipeline for ML operations. More details in [dev-list|http://apache-ignite-developers.2346864.n4.nabble.com/ML-Machine-Learning-Pipeline-Improvement-tt32772.html] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ignite pull request #4470: IGNITE-9157 Optimize data regions memory usage in...
GitHub user Jokser opened a pull request: https://github.com/apache/ignite/pull/4470 IGNITE-9157 Optimize data regions memory usage in tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9157 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4470.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4470 commit 2cd9ec6a68bc88af93c0f0890f9aa566d8f08c8c Author: Pavel Kovalenko Date: 2018-08-01T13:58:30Z IGNITE-9157 Fixed data region sizes. commit 0b6f4d0c21b2e76ea708548b0fd4302851369481 Author: Pavel Kovalenko Date: 2018-08-01T14:10:06Z IGNITE-9157 Fixed data region sizes. ---
[jira] [Created] (IGNITE-9157) Optimize memory usage of data regions in tests
Pavel Kovalenko created IGNITE-9157: --- Summary: Optimize memory usage of data regions in tests Key: IGNITE-9157 URL: https://issues.apache.org/jira/browse/IGNITE-9157 Project: Ignite Issue Type: Bug Components: cache Affects Versions: 2.6 Reporter: Pavel Kovalenko Assignee: Pavel Kovalenko Fix For: 2.7 If we use persistence in tests and do not explicitly set the max size of a data region, by default it will be 20% of available RAM on a host. This can lead to memory over-usage and sometimes JVMs, where such tests are running, will be killed by Linux OOM killer. We should find all tests where data region max size has forgotten and set this value explicitly to minimal possible value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[Distributed SQL] Do we have a plan to implement QuadTree index?
Hi, Igniters. Currently I'm working on different math stuff over the Apache Ignite and in a few tasks I need to implement in memory something like this https://en.wikipedia.org/wiki/Quadtree I didn't find such index in Apache Ignite, but maybe it's under development by somebody? Is it a difficult to add a new index type to our distributed SQL (from point of view of different infrastructure issues and so on P.S I don't worry the math stuff here because I've implemented it many times in non-distributed version)? It will be great to hear any kind of your thoughts and maybe somebody could help with implementation zaleslaw
[jira] [Created] (IGNITE-9156) SQL: Impossible to specify equality condition on sub-select field
Denis Mekhanikov created IGNITE-9156:
Summary: SQL: Impossible to specify equality condition on
sub-select field
Key: IGNITE-9156
URL: https://issues.apache.org/jira/browse/IGNITE-9156
Project: Ignite
Issue Type: Bug
Components: sql
Affects Versions: 2.6
Reporter: Denis Mekhanikov
Execution of the following sequence of SQL statements leads to an exception:
{code:sql}
CREATE TABLE PEOPLE (id int PRIMARY KEY, name varchar);
INSERT INTO PEOPLE(id, name) VALUES(1, 'Mike');
SELECT * FROM (SELECT * FROM People p) tmp WHERE tmp.name='Mike';{code}
Exception:
{noformat}
java.lang.ClassCastException: org.h2.table.TableView cannot be cast to
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table
at
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.extractPartitionFromEquality(GridSqlQuerySplitter.java:2340)
~[classes/:?]
at
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.extractPartition(GridSqlQuerySplitter.java:2272)
~[classes/:?]
at
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.derivePartitionsFromQuery(GridSqlQuerySplitter.java:2254)
~[classes/:?]
at
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.splitSelect(GridSqlQuerySplitter.java:1539)
~[classes/:?]
at
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.splitQueryModel(GridSqlQuerySplitter.java:1227)
~[classes/:?]
at
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.splitQuery(GridSqlQuerySplitter.java:306)
~[classes/:?]
at
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.split(GridSqlQuerySplitter.java:224)
~[classes/:?]
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.split(IgniteH2Indexing.java:1991)
~[classes/:?]
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.parseAndSplit(IgniteH2Indexing.java:1953)
~[classes/:?]
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.querySqlFields(IgniteH2Indexing.java:1702)
~[classes/:?]
at
org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:2107)
~[classes/:?]
at
org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:2102)
~[classes/:?]
at
org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:36)
~[classes/:?]
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:2650)
~[classes/:?]
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:2116)
~[classes/:?]
... 12 more{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: IP finder in tests
Great, sometimes I made it manually in local tests. It will be great to change the situation. 2018-08-01 18:22 GMT+06:00 Stanislav Lukyanov : > +1. > > I don’t see why do we need to fallback to multicast for multi-JVM – let’s > just set 127.0.0.1:47500..47509 by default, > it’ll be enough for most (if not all) tests. > > Stan > > From: Dmitriy Pavlov > Sent: 1 августа 2018 г. 14:21 > To: dev@ignite.apache.org > Subject: Re: IP finder in tests > > Hi Denis, > > Thank you for bringing the question here. > > I totally support this change. > > Sincerely, > Dmitriy Pavlov > > ср, 1 авг. 2018 г. в 13:23, Denis Mekhanikov : > > > Igniters, > > > > Almost every test in Ignite project has the following pattern: shared > > *TcpDiscoveryVmIpFinder > > *is defined as a field of a test class, which is then used in discovery > > configuration in *getConfiguration(...)* method. There are more than 700 > > test classes with this setup. > > > > But for some reason *TcpDiscoveryMulticastIpFinder *is used in tests by > > default. I don't think, that it should be used in tests at all, since > > external nodes may accidentally affect test results. > > > > The only case, where it makes sense is multi JVM tests. Shared static IP > > finder is not applicable there, since nodes are run in different JVMs and > > cannot share the same IP finder object. > > > > I would like to change the default IP finder to a shared > > *TcpDiscoveryVmIpFinder. > > *In cases, when *GridAbstractTest#isMultiJvm() *returns *true*, we could > > fall back to multicast. > > > > What do you think? > > > > Denis > > > >
[GitHub] ignite pull request #4469: IGNITE-8680: One Hot Encoder (including String In...
GitHub user zaleslaw opened a pull request: https://github.com/apache/ignite/pull/4469 IGNITE-8680: One Hot Encoder (including String Indexing) You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-8680 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4469.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4469 commit 0f39821e5fb140d93639b8c2ccdca487454c3093 Author: zaleslaw Date: 2018-08-01T12:49:19Z IGNITE-8680: Added One-Hot Encoder commit 99ebfacab93233f85dcb87d2092e64c6c58f7372 Author: zaleslaw Date: 2018-08-01T12:53:41Z IGNITE-8680: Fixed formatting commit 706262dae33929a8dc22498e943c66199937a328 Author: zaleslaw Date: 2018-08-01T12:54:20Z IGNITE-8680: Fixed imports ---
[GitHub] ignite pull request #4468: Ignite 2.4.5.b4
GitHub user aealeksandrov opened a pull request: https://github.com/apache/ignite/pull/4468 Ignite 2.4.5.b4 For ignite-2.4.5.b4 TC run. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-2.4.5.b4 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4468.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4468 commit 915dd2966084d78f7b4f3d482e6bd25f860c1e23 Author: Alexey Goncharuk Date: 2018-01-31T08:22:26Z IGNITE-7569 Fixed index rebuild future - Fixes #3454. commit 8ea8609259039852ab0c26f26ac528c1ffae7c94 Author: Alexey Goncharuk Date: 2018-01-31T08:24:57Z IGNITE-7577 Fixing public API active flag on baseline changes - Fixes #3455. commit c8ce1f66e98b3174d771a3b801a2538499dc2c3d Author: Ivan Rakov Date: 2018-01-31T09:51:09Z IGNITE-7475 Improved VerifyBackupPartitionsTask to calculate partition hashes in parallel - Fixes #3407. Signed-off-by: Alexey Goncharuk commit 258ff4299da20122d7c387cb8579264035c93c18 Author: Alexey Goncharuk Date: 2018-01-31T13:52:24Z IGNITE-7573 Fixed full API tests to be compliant with baseline topology commit 254ed3a9c32d092702a0461509bf867cbd7cdee6 Author: Alexey Kuznetsov Date: 2018-02-01T08:22:53Z ignite-2.4.0 Update version. (cherry picked from commit 2e43749) commit c1a9c0a404d77fba08170bedf14844f87abe3028 Author: Alexey Goncharuk Date: 2018-02-01T10:17:28Z IGNITE-7569 Fixing index rebuild future commit e43799ce70cdbe03d9e206381d1d5138b820b075 Author: Alexey Goncharuk Date: 2018-02-01T13:39:17Z IGNITE-7520 Provide util-methods to get baseline from context - Fixes #3431. commit 8f5fc7cfb0624cf2048efad38dfff32f782116e8 Author: Sergey Chugunov Date: 2018-02-02T08:24:29Z IGNITE-7580 Fix compatibilityMode flag consistency This closes #3466 (cherry picked from commit 8f2045e) commit d3ddd50cb2b889173176b6c47c9ff61410e1d909 Author: Ilya Lantukh Date: 2018-02-07T10:33:28Z IGNITE-7514 Affinity assignment should be recalculated when primary node is not OWNER (cherry picked from commit faf50f1) commit d3745e9d0a3ff5a64fba494889b7e2605f3af6bb Author: Alexey Goncharuk Date: 2018-02-07T18:10:32Z IGNITE-7639 Fixed NPE commit f7c16855ba802d9d47048521aec7e14285e4a281 Author: Pavel Kovalenko Date: 2018-02-09T13:55:15Z IGNITE-7540 Prevent page memory metadata corruption during checkpoint and group destroying. - Fixes #3490. Signed-off-by: Alexey Goncharuk commit c92f167fc491078f02b9f94fe89edafc2902ebc2 Author: ilantukh Date: 2018-02-14T12:40:13Z Updated version in properties. commit 1ecf348dd429cf7861b414e0e5a7776b72dba281 Author: Sergey Chugunov Date: 2018-02-16T13:21:12Z IGNITE-7699 BinaryMetadata exchange should not be triggered if metadata was not updated - Fixes #3523. Signed-off-by: Alexey Goncharuk (cherry-picked from commit bcd3881) commit 2458bd08a5b501b3eeb5caf0ae6dcaa2bcccd915 Author: EdShangGG Date: 2018-02-16T13:29:49Z IGNITE-7676 Add affinity version to snapshot plugin stub - Fixes #3510. Signed-off-by: Alexey Goncharuk (cherry picked from commit b6d21fb) commit bfdcda7a2a6b5cf64f15ed169d2beb886f131fac Author: EdShangGG Date: 2018-02-12T16:36:30Z IGNITE-7626 Unify code in test which cleans up persistence directories - Fixes #3477. Signed-off-by: Alexey Goncharuk (cherry picked from commit a0997b9) commit 2e92e0094b270aa8489e66d94bfcf15eadabfb4f Author: EdShangGG Date: 2018-02-12T18:44:10Z IGNITE-7626 Unify code in test which clean up persistence directories - Fixes #3512. Signed-off-by: Alexey Goncharuk (cherry picked from commit 6f6f8dd) commit 3f86c127c78065999663a4fc4eaedb5e5d4bee1c Author: EdShangGG Date: 2018-02-12T18:26:31Z compilation fix commit 0b9322c566f9b464291854142ac02495bd1817e4 Author: gg-shq Date: 2018-02-07T11:28:04Z IGNITE-6917: Implemented SQL COPY command. commit c5e386ca96750213bddcd98d0af0c589fee476ca Author: gg-shq Date: 2018-02-07T15:31:27Z IGNITE-7586: Added COPY command into the JDBC example. This closes #3485 commit d8203e2d81f8fbf0f7fbe5e710c9908f2fcb8307 Author: shq Date: 2018-02-15T10:36:00Z IGNITE-7709: SQL COPY command: make sure file name is always quoted. This closes #3526. commit 1185993ee7cd83695388f698f18f95b43e15de06 Author: devozerov Date: 2018-02-15T11:00:42Z IGNITE-7714: SQL COPY command: fixed "Table not found" issue on the client node. commit 88c8bdcc0dc2fdf2b2b22562a6b30031e053f671 Author: devozerov Date: 2018-02-16T14:54:24Z IGNITE-7737: SQL COPY: renamed BUFFER_SIZE to PACKET_SIZE. This closes #3533. commit bc331f9de716c30d6f733e28821ab44da7ed0cf7
RE: IP finder in tests
+1. I don’t see why do we need to fallback to multicast for multi-JVM – let’s just set 127.0.0.1:47500..47509 by default, it’ll be enough for most (if not all) tests. Stan From: Dmitriy Pavlov Sent: 1 августа 2018 г. 14:21 To: dev@ignite.apache.org Subject: Re: IP finder in tests Hi Denis, Thank you for bringing the question here. I totally support this change. Sincerely, Dmitriy Pavlov ср, 1 авг. 2018 г. в 13:23, Denis Mekhanikov : > Igniters, > > Almost every test in Ignite project has the following pattern: shared > *TcpDiscoveryVmIpFinder > *is defined as a field of a test class, which is then used in discovery > configuration in *getConfiguration(...)* method. There are more than 700 > test classes with this setup. > > But for some reason *TcpDiscoveryMulticastIpFinder *is used in tests by > default. I don't think, that it should be used in tests at all, since > external nodes may accidentally affect test results. > > The only case, where it makes sense is multi JVM tests. Shared static IP > finder is not applicable there, since nodes are run in different JVMs and > cannot share the same IP finder object. > > I would like to change the default IP finder to a shared > *TcpDiscoveryVmIpFinder. > *In cases, when *GridAbstractTest#isMultiJvm() *returns *true*, we could > fall back to multicast. > > What do you think? > > Denis >
Re: IP finder in tests
Hi Denis, I also definitely support this change. As an engineer who tightly working with MakeTeamCityGreenAgain activity, I notice several test suites hanging related with MulticastIpFinder infinite loops on datagram receiving. Here is last recorded fail - https://ci.ignite.apache.org/viewLog.html?buildId=1544659=buildResultsDiv=IgniteTests24Java8_BinaryObjectsSimpleMapperBasic . Changing default ipFinder in tests to VM will help with TeamCity stability. 2018-08-01 14:20 GMT+03:00 Dmitriy Pavlov : > Hi Denis, > > Thank you for bringing the question here. > > I totally support this change. > > Sincerely, > Dmitriy Pavlov > > ср, 1 авг. 2018 г. в 13:23, Denis Mekhanikov : > > > Igniters, > > > > Almost every test in Ignite project has the following pattern: shared > > *TcpDiscoveryVmIpFinder > > *is defined as a field of a test class, which is then used in discovery > > configuration in *getConfiguration(...)* method. There are more than 700 > > test classes with this setup. > > > > But for some reason *TcpDiscoveryMulticastIpFinder *is used in tests by > > default. I don't think, that it should be used in tests at all, since > > external nodes may accidentally affect test results. > > > > The only case, where it makes sense is multi JVM tests. Shared static IP > > finder is not applicable there, since nodes are run in different JVMs and > > cannot share the same IP finder object. > > > > I would like to change the default IP finder to a shared > > *TcpDiscoveryVmIpFinder. > > *In cases, when *GridAbstractTest#isMultiJvm() *returns *true*, we could > > fall back to multicast. > > > > What do you think? > > > > Denis > > >
[GitHub] ignite pull request #4467: IGNITE-9155 : ExchangeFuture will not complete wi...
GitHub user ilantukh opened a pull request: https://github.com/apache/ignite/pull/4467 IGNITE-9155 : ExchangeFuture will not complete with Exception on state change failure. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9155 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4467.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4467 commit 490266417bdbd97c43ed74341a20f9b8fc2d1a77 Author: Ilya Lantukh Date: 2018-06-09T14:59:38Z IGNITE-9155 : ExchangeFuture will not complete with Exception on state change failure. ---
[jira] [Created] (IGNITE-9155) Exception during cluster state change terminates ExchangeWorker
Ilya Lantukh created IGNITE-9155: Summary: Exception during cluster state change terminates ExchangeWorker Key: IGNITE-9155 URL: https://issues.apache.org/jira/browse/IGNITE-9155 Project: Ignite Issue Type: Bug Reporter: Ilya Lantukh After IGNITE-8311 we throw an exception in ExchangeFuture instead swallowing it. ClusterStateChangeProcessor has it's own exception handling mechanism, which doesn't require ExchangeWorker termination (and leaving node in broken state). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: IP finder in tests
Hi Denis, Thank you for bringing the question here. I totally support this change. Sincerely, Dmitriy Pavlov ср, 1 авг. 2018 г. в 13:23, Denis Mekhanikov : > Igniters, > > Almost every test in Ignite project has the following pattern: shared > *TcpDiscoveryVmIpFinder > *is defined as a field of a test class, which is then used in discovery > configuration in *getConfiguration(...)* method. There are more than 700 > test classes with this setup. > > But for some reason *TcpDiscoveryMulticastIpFinder *is used in tests by > default. I don't think, that it should be used in tests at all, since > external nodes may accidentally affect test results. > > The only case, where it makes sense is multi JVM tests. Shared static IP > finder is not applicable there, since nodes are run in different JVMs and > cannot share the same IP finder object. > > I would like to change the default IP finder to a shared > *TcpDiscoveryVmIpFinder. > *In cases, when *GridAbstractTest#isMultiJvm() *returns *true*, we could > fall back to multicast. > > What do you think? > > Denis >
Re: IP finder in tests
Hello! I think it's a sensible decision that is long overdue. Regards, -- Ilya Kasnacheev 2018-08-01 13:22 GMT+03:00 Denis Mekhanikov : > Igniters, > > Almost every test in Ignite project has the following pattern: shared > *TcpDiscoveryVmIpFinder > *is defined as a field of a test class, which is then used in discovery > configuration in *getConfiguration(...)* method. There are more than 700 > test classes with this setup. > > But for some reason *TcpDiscoveryMulticastIpFinder *is used in tests by > default. I don't think, that it should be used in tests at all, since > external nodes may accidentally affect test results. > > The only case, where it makes sense is multi JVM tests. Shared static IP > finder is not applicable there, since nodes are run in different JVMs and > cannot share the same IP finder object. > > I would like to change the default IP finder to a shared > *TcpDiscoveryVmIpFinder. > *In cases, when *GridAbstractTest#isMultiJvm() *returns *true*, we could > fall back to multicast. > > What do you think? > > Denis >
[GitHub] ignite pull request #4454: IGNITE-8939 Catch and proper propagate transactio...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4454 ---
Re: Assign JIRA tickets to someone else
Setting Assignee doesn't mean that the ticket is reserved for that particular person. If anyone wants to participate in that ticket he can always ask about that in JIRA or on mailing list. In fact, this is how we deal with tickets that got stuck on a contributor for a long time. On Wed, Aug 1, 2018 at 1:28 PM Dmitriy Pavlov wrote: > Hi Vladimir, > > please see Code of conduct principles > > https://community.apache.org/newbiefaq.html#NewbieFAQ-IsthereaCodeofConductforApacheprojects > ? > I would be happy to have a chat with you about Apache and open source > principles. > > I encourage all Igniters to carefully read thread with ASF fellows replies: > > https://lists.apache.org/thread.html/c6013b99940de32aae831a0b76e8fd53febe5040e9e0d67abb4f62a5@%3Cdev.community.apache.org%3E > (I > can't say better). > > We, as community, has a huge potential of growing. I beleive if someone > strongly wants to assign ticket directly , that means other community > members can not complete this ticket, - This problems do matter, and it is > deeper than mechanical action in JIRA. > > I would like that 'domain expert' share their knowlege instead completing > tickets. Tickets completion is investing into product codebase. > > Let's invest in community so every contributor will became 'domain expert'. > > That is my point, and I hope it make sense to you and for every Igniter. > > Sincerely, > Dmitriy Pavlov > > > ср, 1 авг. 2018 г. в 13:11, Vladimir Ozerov : > > > Dmitry, > > > > "If something is not happened on mailing list it is not happened" is just > > not true. Mailing list is only one of communication channels. I do not > see > > any reason why I need to discuss assignee on a dev list .This is just > spam > > for 99% of members. > > > > Again - lets focus on things that matter, rather than on bureaucracy. I > > thought that we overgrown those old days in incubator when we thought > that > > the only extreme is possible in ASF - to log your every breath on the dev > > list. . > > > > On Wed, Aug 1, 2018 at 12:32 PM Dmitriy Pavlov > > wrote: > > > > > Hi Vladimir, > > > > > > I think I understand your point. I've heard negative feedback. Which is > > why > > > I would like to speak it up, I hope other Igniters who disagree with > this > > > assignment will be feeling free to unassign ticket. I would like that > > noone > > > was trying to adapt commercial company experience to the Comminuty. > > > > > > It's a pity that I need to continiously remind some of Igniters about > > > principle: > > > "If something is not happened on mailing list it is not happened." > > > > > > In you case you can decide who will do ticket on dev.list - (it is > > > perfectly ok if only you two of you will communicate) - or tell your > > fellow > > > to pick up ticket. > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > ср, 1 авг. 2018 г. в 10:58, Vladimir Ozerov : > > > > > > > In general there is nothing wrong with assigning a ticket to someone > if > > > you > > > > know that this would not cause negative reaction from assignee side. > > The > > > > thing is that community has a lot of members who knows each other in > > > person > > > > and in close contact for years. So there is nothing wrong if I > assign a > > > > ticket to my fellow. Especially if ticket is complex or critically > > > > important for the product. These is no "project management" in such > an > > > > action. > > > > > > > > On the other hand, we should not overuse this practice, and set > > specific > > > > assignee only if there is a string reason for this. > > > > > > > > All in all, remember that community is alive thing built of real > > people. > > > > The more rules you set without a clear reason, the more uncomfortable > > it > > > is > > > > to be in such a community. > > > > > > > > On Wed, Aug 1, 2018 at 10:30 AM Maxim Muzafarov > > > > wrote: > > > > > > > > > Vyacheslav, > > > > > > > > > > Thank you! This seems exactly what I was asking about. > > > > > > > > > > > > > > > On Wed, 1 Aug 2018 at 10:18 Vyacheslav Daradur < > daradu...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi, Maxim, > > > > > > > > > > > > There is information about project components maintainers [1], > but > > > I'm > > > > > > not sure if it is actual. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers > > > > > > On Wed, Aug 1, 2018 at 9:33 AM Maxim Muzafarov < > maxmu...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > Folks, > > > > > > > > > > > > > > I don't think we need additional notification for catching > > > attention > > > > of > > > > > > > some community > > > > > > > experts. Such notifications can be overused by other > > participants. > > > > > > > Developer mail list > > > > > > > the best place we have for any discussions. > > > > > > > > > > > > > > What I really would like to have is the list of community >
[GitHub] ignite pull request #4466: IGNITE-9144 client and daemon nodes are supported
GitHub user ezagumennov opened a pull request: https://github.com/apache/ignite/pull/4466 IGNITE-9144 client and daemon nodes are supported You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9144 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4466.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4466 commit 827640ef107d1fc19fccd769e61fb4be880203e3 Author: ezagumennov Date: 2018-08-01T10:31:27Z IGNITE-9144 client and daemon nodes are supported ---
Re: Assign JIRA tickets to someone else
Hi Vladimir, please see Code of conduct principles https://community.apache.org/newbiefaq.html#NewbieFAQ-IsthereaCodeofConductforApacheprojects? I would be happy to have a chat with you about Apache and open source principles. I encourage all Igniters to carefully read thread with ASF fellows replies: https://lists.apache.org/thread.html/c6013b99940de32aae831a0b76e8fd53febe5040e9e0d67abb4f62a5@%3Cdev.community.apache.org%3E (I can't say better). We, as community, has a huge potential of growing. I beleive if someone strongly wants to assign ticket directly , that means other community members can not complete this ticket, - This problems do matter, and it is deeper than mechanical action in JIRA. I would like that 'domain expert' share their knowlege instead completing tickets. Tickets completion is investing into product codebase. Let's invest in community so every contributor will became 'domain expert'. That is my point, and I hope it make sense to you and for every Igniter. Sincerely, Dmitriy Pavlov ср, 1 авг. 2018 г. в 13:11, Vladimir Ozerov : > Dmitry, > > "If something is not happened on mailing list it is not happened" is just > not true. Mailing list is only one of communication channels. I do not see > any reason why I need to discuss assignee on a dev list .This is just spam > for 99% of members. > > Again - lets focus on things that matter, rather than on bureaucracy. I > thought that we overgrown those old days in incubator when we thought that > the only extreme is possible in ASF - to log your every breath on the dev > list. . > > On Wed, Aug 1, 2018 at 12:32 PM Dmitriy Pavlov > wrote: > > > Hi Vladimir, > > > > I think I understand your point. I've heard negative feedback. Which is > why > > I would like to speak it up, I hope other Igniters who disagree with this > > assignment will be feeling free to unassign ticket. I would like that > noone > > was trying to adapt commercial company experience to the Comminuty. > > > > It's a pity that I need to continiously remind some of Igniters about > > principle: > > "If something is not happened on mailing list it is not happened." > > > > In you case you can decide who will do ticket on dev.list - (it is > > perfectly ok if only you two of you will communicate) - or tell your > fellow > > to pick up ticket. > > > > Sincerely, > > Dmitriy Pavlov > > > > ср, 1 авг. 2018 г. в 10:58, Vladimir Ozerov : > > > > > In general there is nothing wrong with assigning a ticket to someone if > > you > > > know that this would not cause negative reaction from assignee side. > The > > > thing is that community has a lot of members who knows each other in > > person > > > and in close contact for years. So there is nothing wrong if I assign a > > > ticket to my fellow. Especially if ticket is complex or critically > > > important for the product. These is no "project management" in such an > > > action. > > > > > > On the other hand, we should not overuse this practice, and set > specific > > > assignee only if there is a string reason for this. > > > > > > All in all, remember that community is alive thing built of real > people. > > > The more rules you set without a clear reason, the more uncomfortable > it > > is > > > to be in such a community. > > > > > > On Wed, Aug 1, 2018 at 10:30 AM Maxim Muzafarov > > > wrote: > > > > > > > Vyacheslav, > > > > > > > > Thank you! This seems exactly what I was asking about. > > > > > > > > > > > > On Wed, 1 Aug 2018 at 10:18 Vyacheslav Daradur > > > > wrote: > > > > > > > > > Hi, Maxim, > > > > > > > > > > There is information about project components maintainers [1], but > > I'm > > > > > not sure if it is actual. > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers > > > > > On Wed, Aug 1, 2018 at 9:33 AM Maxim Muzafarov > > > > > wrote: > > > > > > > > > > > > Folks, > > > > > > > > > > > > I don't think we need additional notification for catching > > attention > > > of > > > > > > some community > > > > > > experts. Such notifications can be overused by other > participants. > > > > > > Developer mail list > > > > > > the best place we have for any discussions. > > > > > > > > > > > > What I really would like to have is the list of community > > > > members\experts > > > > > > and their zones > > > > > > of Ignite project code responsibilities. May be such list already > > > > exists, > > > > > > does it? > > > > > > > > > > > > As for me, I think we should pose right the complexity of the > issue > > > at > > > > > the > > > > > > moment > > > > > > of their creation. Ask expert to provide some high level > > > implementation > > > > > > details and > > > > > > keep it unassingned, so anyone can take it. > > > > > > > > > > > > On Wed, 1 Aug 2018 at 01:09 Nikolay Izhikov > > > > > wrote: > > > > > > > > > > > > > How about mention desired expert with some predefined message? > > > > > >
[GitHub] ignite pull request #4463: GG-14025: Fix deleted entry version.
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4463 ---
IP finder in tests
Igniters, Almost every test in Ignite project has the following pattern: shared *TcpDiscoveryVmIpFinder *is defined as a field of a test class, which is then used in discovery configuration in *getConfiguration(...)* method. There are more than 700 test classes with this setup. But for some reason *TcpDiscoveryMulticastIpFinder *is used in tests by default. I don't think, that it should be used in tests at all, since external nodes may accidentally affect test results. The only case, where it makes sense is multi JVM tests. Shared static IP finder is not applicable there, since nodes are run in different JVMs and cannot share the same IP finder object. I would like to change the default IP finder to a shared *TcpDiscoveryVmIpFinder. *In cases, when *GridAbstractTest#isMultiJvm() *returns *true*, we could fall back to multicast. What do you think? Denis
[jira] [Created] (IGNITE-9154) Incorrect new version is used during conflict resolution in ATOMIC cache
Vladimir Ozerov created IGNITE-9154:
---
Summary: Incorrect new version is used during conflict resolution
in ATOMIC cache
Key: IGNITE-9154
URL: https://issues.apache.org/jira/browse/IGNITE-9154
Project: Ignite
Issue Type: Bug
Components: cache
Affects Versions: 2.6
Reporter: Vladimir Ozerov
Assignee: Andrew Mashenkov
Fix For: 2.7
Currently atomic cache update routine in {{GridCacheMapEntry}}. We use
different versions in case of update and remove - {{newVer}} and {{updateVer}}.
Correct version for both cases is {{updateVer}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ignite pull request #4465: Ignite 9144
Github user ezagumennov closed the pull request at: https://github.com/apache/ignite/pull/4465 ---
Re: Assign JIRA tickets to someone else
Dmitry, "If something is not happened on mailing list it is not happened" is just not true. Mailing list is only one of communication channels. I do not see any reason why I need to discuss assignee on a dev list .This is just spam for 99% of members. Again - lets focus on things that matter, rather than on bureaucracy. I thought that we overgrown those old days in incubator when we thought that the only extreme is possible in ASF - to log your every breath on the dev list. . On Wed, Aug 1, 2018 at 12:32 PM Dmitriy Pavlov wrote: > Hi Vladimir, > > I think I understand your point. I've heard negative feedback. Which is why > I would like to speak it up, I hope other Igniters who disagree with this > assignment will be feeling free to unassign ticket. I would like that noone > was trying to adapt commercial company experience to the Comminuty. > > It's a pity that I need to continiously remind some of Igniters about > principle: > "If something is not happened on mailing list it is not happened." > > In you case you can decide who will do ticket on dev.list - (it is > perfectly ok if only you two of you will communicate) - or tell your fellow > to pick up ticket. > > Sincerely, > Dmitriy Pavlov > > ср, 1 авг. 2018 г. в 10:58, Vladimir Ozerov : > > > In general there is nothing wrong with assigning a ticket to someone if > you > > know that this would not cause negative reaction from assignee side. The > > thing is that community has a lot of members who knows each other in > person > > and in close contact for years. So there is nothing wrong if I assign a > > ticket to my fellow. Especially if ticket is complex or critically > > important for the product. These is no "project management" in such an > > action. > > > > On the other hand, we should not overuse this practice, and set specific > > assignee only if there is a string reason for this. > > > > All in all, remember that community is alive thing built of real people. > > The more rules you set without a clear reason, the more uncomfortable it > is > > to be in such a community. > > > > On Wed, Aug 1, 2018 at 10:30 AM Maxim Muzafarov > > wrote: > > > > > Vyacheslav, > > > > > > Thank you! This seems exactly what I was asking about. > > > > > > > > > On Wed, 1 Aug 2018 at 10:18 Vyacheslav Daradur > > > wrote: > > > > > > > Hi, Maxim, > > > > > > > > There is information about project components maintainers [1], but > I'm > > > > not sure if it is actual. > > > > > > > > [1] > > > > > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers > > > > On Wed, Aug 1, 2018 at 9:33 AM Maxim Muzafarov > > > wrote: > > > > > > > > > > Folks, > > > > > > > > > > I don't think we need additional notification for catching > attention > > of > > > > > some community > > > > > experts. Such notifications can be overused by other participants. > > > > > Developer mail list > > > > > the best place we have for any discussions. > > > > > > > > > > What I really would like to have is the list of community > > > members\experts > > > > > and their zones > > > > > of Ignite project code responsibilities. May be such list already > > > exists, > > > > > does it? > > > > > > > > > > As for me, I think we should pose right the complexity of the issue > > at > > > > the > > > > > moment > > > > > of their creation. Ask expert to provide some high level > > implementation > > > > > details and > > > > > keep it unassingned, so anyone can take it. > > > > > > > > > > On Wed, 1 Aug 2018 at 01:09 Nikolay Izhikov > > > wrote: > > > > > > > > > > > How about mention desired expert with some predefined message? > > > > > > So expert can setup simple email filter and know about tickets > that > > > > have > > > > > > to be done. > > > > > > > > > > > > Example: > > > > > > > > > > > > "[~dpavlov] TFY" - ticket for you. > > > > > > > > > > > > В Вт, 31/07/2018 в 14:04 -0700, Dmitriy Setrakyan пишет: > > > > > > > Dmitriy, > > > > > > > > > > > > > > I would agree with everything you are saying. There are some > > > tickets, > > > > > > > however, that cannot be done by just anyone and preferably have > > to > > > be > > > > > > > looked at by certain domain experts. In those cases only it > would > > > be > > > > OK > > > > > > to > > > > > > > assign a ticket explicitly in my view. For all other cases we > > > should > > > > > > allow > > > > > > > the community pick up a ticket they want to work on. > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > On Tue, Jul 31, 2018 at 10:34 AM, Dmitriy Pavlov < > > > > dpavlov@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Igniters, > > > > > > > > > > > > > > > > Ignite is complex project and sometimes we know > > > > > > > > 1. who can solve the issue > > > > > > > > 2. and we would like that particular contributor would take > an > > > > issue. > > > > > > > > > > > > > > > > In the same time "in an open source community where >
[jira] [Created] (IGNITE-9153) Accessing cache from transaction on client node, where it was not accessed yet throws an exception
Evgenii Zhuravlev created IGNITE-9153: - Summary: Accessing cache from transaction on client node, where it was not accessed yet throws an exception Key: IGNITE-9153 URL: https://issues.apache.org/jira/browse/IGNITE-9153 Project: Ignite Issue Type: Improvement Reporter: Evgenii Zhuravlev Attachments: ClientCacheTransactionsTest.java Exception message: Cannot start/stop cache within lock or transaction. Reproducer is attached: ClientCacheTransactionsTest.java -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Apache Ignite tickets with empty description & dev list references
Hi Dmitriy, Yakov, thank you for your support. I think if I have an odd hour I will create wiki describing how to create good ticket. Igniters, BTW I found one more way to refer to dev list threads (in addition to nabble): https://lists.apache.org/list.html?dev@ignite.apache.org Sincerely, Dmitriy Pavlov ср, 1 авг. 2018 г. в 12:01, Yakov Zhdanov : > Agree with Dmitry P. > > Guys! Even if you file a ticket on yourself please spend additional 5 mins > to add minimal description to provide idea on the intended changes for the > rest of our community or those who will have to deal with your changes in > future. > > --Yakov >
Re: Orphaned, duplicate, and main-class tests!
Hi Ilya, could you please actualize this PR. TC Bot can now detect newly contributed tests' failures, so I think it is best point to apply you change. Sincerely, Dmitriy Pavlov пт, 25 мая 2018 г. в 18:16, Eduard Shangareev : > Igniters, > > While making review I checked next main-method tests: > > org.apache.ignite.loadtests.mapper.GridContinuousMapperLoadTest1 > org.apache.ignite.loadtests.mapper.GridContinuousMapperLoadTest2 > > And I have found that they are totally outdated! > They use config which was changed a long time ago. > And use localPeek with parameters which don't make sense now. > > So, I suggest to delete them. > > If there wouldn't be any objection I will do it myself. > > > > > On Tue, May 22, 2018 at 6:59 PM, Ilya Kasnacheev < > ilya.kasnach...@gmail.com> > wrote: > > > Hello, Igniters! > > > > One moment more of your time. One, we seem to have a consensus now that > > tests should be added to suites, but commented out. They should be > > uncommented out later, for which numerous tickets will be created. This > way > > we can tackle. > > > > Another issue sprang up, just now I have discovered an 'ignored-tests' > > module. My proposal thus is to: > > - Move tests from this suite to relevant suites, comment them out. > > - Kill this module (with fire). > > > > Would be glad to her your input, > > > > > > > > -- > > Ilya Kasnacheev > > > > 2018-05-03 20:03 GMT+03:00 Ilya Kasnacheev : > > > > > Hello Dmitry, igniters! > > > > > > Still, the policy of removal of unused tests is not clear to me. > > > > > > We have roughly three groups of such tests: > > > - Odd ancient main class tests. I think we can remove those. > > > - JVM features/quirks tests (some are main class, some are JUnit tests. > > > Reside in package jvmtest. Should we remove these? > > > - JUnit "load" tests. Should we kill all of these? I'm asking since > > you've > > > commited such test recently. I think you wanted it to linger. And yet, > > > what's our policy? How do I determine whether it's safe to nuke a > "load" > > > test not in any suite? Or just tuck them in a fake TestSuite and keep? > > > > > > Regards, > > > > > > -- > > > Ilya Kasnacheev > > > > > > 2018-04-24 17:54 GMT+03:00 Dmitry Pavlov : > > > > > >> I agree with Yakov here. If nobody responds here we can consider we > have > > >> lazy consensus on removal of tests. > > >> > > >> I'm going to review PRs from Ilya. > > >> > > >> вт, 24 апр. 2018 г. в 6:11, Yakov Zhdanov : > > >> > > >> > Alexey Goncharuk, Vladimir Ozerov, what do you think about these > > tests? > > >> > > > >> > I believe they were created as a part of variuos optimization and > > >> profiling > > >> > activities. I also think we can remove them since nobody cares about > > >> them > > >> > for too long. > > >> > > > >> > Thoughts? > > >> > > > >> > Yakov Zhdanov > > >> > > > >> > ср, 18 апр. 2018 г., 16:42 Ilya Kasnacheev < > ilya.kasnach...@gmail.com > > >: > > >> > > > >> > > Hello! > > >> > > > > >> > > I've decided to return to this task after a break. > > >> > > > > >> > > Can you please tell me why do we have main-class tests? Such as > > >> > > > > >> > > GridBasicPerformanceTest.class, > > >> > > GridBenchmarkCacheGetLoadTest.class, > > >> > > GridBoundedConcurrentLinkedHashSetLoadTest.class, > > >> > > GridCacheDataStructuresLoadTest.class, > > >> > > GridCacheReplicatedPreloadUndeploysTest.class, > > >> > > GridCacheLoadTest.class, > > >> > > GridCacheMultiNodeDataStructureTest.class, > > >> > > GridCapacityLoadTest.class, > > >> > > GridContinuousOperationsLoadTest.class, > > >> > > GridFactoryVmShutdownTest.class, > > >> > > GridFutureListenPerformanceTest.class, > > >> > > GridFutureQueueTest.class, > > >> > > GridGcTimeoutTest.class, > > >> > > GridJobExecutionSingleNodeLoadTest.class, > > >> > > GridJobExecutionSingleNodeSemaphoreLoadTest.class, > > >> > > GridJobLoadTest.class, > > >> > > GridMergeSortLoadTest.class, > > >> > > GridNioBenchmarkTest.class, > > >> > > GridThreadPriorityTest.class, > > >> > > GridSystemCurrentTimeMillisTest.class, > > >> > > BlockingQueueTest.class, > > >> > > MultipleFileIOTest.class, > > >> > > GridSingleExecutionTest.class > > >> > > > > >> > > > > >> > > If nobody wants them, how about we delete them in master branch? > > Start > > >> > > afresh? > > >> > > > > >> > > -- > > >> > > Ilya Kasnacheev > > >> > > > > >> > > 2018-02-13 17:02 GMT+03:00 Ilya Kasnacheev < > > ilya.kasnach...@gmail.com > > >> >: > > >> > > > > >> > > > Anton, > > >> > > > > > >> > > > >Tests should be attached to appropriate suites > > >> > > > > > >> > > > This I can do > > >> > > > > > >> > > > > and muted if necessary, Issues should be created on each mute. > > >> > > > > > >> > > > This is roughly a week of work. I can't spare that right now. I > > >> doubt > > >> > > > anyone can. > > >> > > > > > >> > > > Can we approach this by smaller steps? > > >> > > > > > >> > > >
[GitHub] ignite pull request #4465: Ignite 9144
GitHub user ezagumennov opened a pull request: https://github.com/apache/ignite/pull/4465 Ignite 9144 You can merge this pull request into a Git repository by running: $ git pull https://github.com/gridgain/apache-ignite ignite-9144 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4465.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4465 commit 1e537d4a11ea86a5003b50a2287dd1d1924c5f20 Author: ezagumennov Date: 2018-06-15T10:48:55Z IGNITE-8738 log "coordinator changed" message in every node commit 11ef506fc65e205971d13f35ad54506eb5143c97 Author: Aleksei Scherbakov Date: 2018-06-18T13:56:24Z Merge branch 'master' of https://github.com/apache/ignite into ignite-8738 commit 30b6fc93dccd94ba9c951914c5361a0336167a19 Author: Aleksei Scherbakov Date: 2018-06-18T16:19:52Z IGNITE-8738 Improve coordinator change information. commit 149dd3d286cd16aadd45108a34b8c29e22c56853 Author: Aleksei Scherbakov Date: 2018-06-18T16:23:59Z IGNITE-8738 Improve coordinator change information. commit ce37135487909143b838db8c7eb2e0de6966e80e Author: ezagumennov Date: 2018-06-27T11:14:32Z IGNITE-8738 improve coordinator change message commit 37bfb4e2a1c8a7d3064f5d25e24ba69ebd8455e3 Author: ezagumennov Date: 2018-07-09T07:16:15Z IGNITE-8738 oldestServerNode instead of oldestAliveServerNode commit fc303e24666c7cb0430f241f53bdccd2c1f69f11 Author: ezagumennov Date: 2018-08-01T09:46:10Z IGNITE-9144 client and daemon modes are supported ---
Re: Ignite as distributed file storage
Hi Vladimir,
I think not accepting by community is possible only if PMC will veto
change. I didn't find any reasons why not to do this change and why it can
be vetoed..
I would appreciate if you will become mentor of this change and will assist
to Pavel or other community member to make this happen.
To my mind, the Apache Way is not abot rejecting things, it is about
sharing knowlege. If you will be able to share you experience to grow
community it would be good donation.
If you have any disagreements about this change, can we set up voice call
where you will explain how to do this proposal as good as it is possible.
Sincerely,
Dmitriy Pavlov
пт, 6 июл. 2018 г. в 10:35, Vladimir Ozerov :
> Pavel,
>
> I do not think it is a good idea to delay discussions and decisions.
> Because it puts your efforts at risk being not accepted by community in the
> end. Our ultimate goal is not having as much features as possible, but to
> have a consistent product which is easy to understand and use. Having both
> IGFS and another one "not-IGFS" which is in fact the same IGFS but with
> different name is not a good idea, because it would cause more harm than
> value.
>
> Approaches which seems reasonable to me:
> 1) Integrate your ideas into IGFS, which is really flexible in how to
> process data and where to store it. PROXY mode is not a "crutch" as you
> said, but a normal mode which was used in real deployments.
> 2) Replace IGFS with your solution but with clear explanation how it is
> better than IGFS and why we need to drop thousands lines of battle-tested
> code with something new, what does virtually the same thing
> 3) Just drop IGFS from the product, and do not implement any replacement at
> all - personally, I am all for this decision.
>
> If you want I can guide you through IGFS architecture so that we better
> understand what should be done to integrate your ideas into it.
>
> Lat, but not least - we need objective facts why proposed solution is
> better in terms of performance - concrete use cases and performance numbers
> (or at least estimations).
>
> On Fri, Jul 6, 2018 at 1:45 AM Pavel Kovalenko wrote:
>
> > Vladimir,
> >
> > I just want to add to my words, that we can implement BLOB storage and
> > then, if community really wants it, we can adapt this storage to use as
> > underlying file system in IGFS. But IGFS shouldn't be entry point for
> BLOB
> > storage. I think this conclusion can satisfy both of us.
> >
> > 2018-07-06 0:47 GMT+03:00 Pavel Kovalenko :
> >
> > > Vladimir,
> > >
> > > I didn't say that it stores data in on-heap, I said that it performs a
> > lot
> > > of operations with byte[] arrays in on-heap as I see in , which will
> lead
> > > to frequent GCs and unnecessary data copying.
> > > "But the whole idea around mmap sounds like premature optimisation to
> me"
> > > - this is not premature optimisation, this is on of the key performance
> > > features. E.g. Apache Kafka wouldn't be so fast and extremely
> performant
> > > without zero-copy.
> > > If we can do better, why not just do it? Especially if it costs nothing
> > > for us (This is OS level).
> > > As I said in my first message, our end target is handling video and
> > > streaming, copying every chunk of it to on-heap userspace then to
> offheap
> > > and then to disk is unacceptable.
> > > You ask me to implement almost anything using just IGFS interface, why
> we
> > > need to do that? Proxy mode looks like crutch, to support replication
> and
> > > possibility to have some data in-memory I need to write again a lot of
> > > stuff.
> > > Let's finally leave IGFS alone and wait for IEP.
> > >
> > >
> > > 2018-07-06 0:01 GMT+03:00 Vladimir Ozerov :
> > >
> > >> Pavel,
> > >>
> > >> IGFS doesn't enforce you to have block in heap. What you suggest can
> be
> > >> achieved with IGFS as follows:
> > >> 1) Disable caching, so data cache is not used ("PROXY" mode)
> > >> 2) Implement IgniteFileSystem interface which operates on abstract
> > streams
> > >>
> > >> But the whole idea around mmap sounds like premature optimisation to
> > me. I
> > >> conducted a number of experiments with IGFS on large Hadoop workload.
> > Even
> > >> with old AI 1.x architecture, where everything was stored onheap, I
> > never
> > >> had an issue with GC. The key point is that IGFS operates on large
> > (64Kb)
> > >> data blocks, so even with 100Gb full of these blocks you will have
> > >> relatively small number of objects and normal GC pauses. Additional
> > memory
> > >> copying is not an issue either in most workloads in distributed
> systems,
> > >> because most of the time is spent on IO and internal synchronization
> > >> anyway.
> > >>
> > >> Do you have specific scenario when you observed long GC pauses with GC
> > or
> > >> serious performance degradation with IGFS?
> > >>
> > >> Even if we agree that mmap usage is a critical piece, all we need is
> to
> > >> implement a single IGFS interface.
> > >>
> > >> On Thu, Jul 5, 2018 at 10:44
Re: Assign JIRA tickets to someone else
Hi Vladimir, I think I understand your point. I've heard negative feedback. Which is why I would like to speak it up, I hope other Igniters who disagree with this assignment will be feeling free to unassign ticket. I would like that noone was trying to adapt commercial company experience to the Comminuty. It's a pity that I need to continiously remind some of Igniters about principle: "If something is not happened on mailing list it is not happened." In you case you can decide who will do ticket on dev.list - (it is perfectly ok if only you two of you will communicate) - or tell your fellow to pick up ticket. Sincerely, Dmitriy Pavlov ср, 1 авг. 2018 г. в 10:58, Vladimir Ozerov : > In general there is nothing wrong with assigning a ticket to someone if you > know that this would not cause negative reaction from assignee side. The > thing is that community has a lot of members who knows each other in person > and in close contact for years. So there is nothing wrong if I assign a > ticket to my fellow. Especially if ticket is complex or critically > important for the product. These is no "project management" in such an > action. > > On the other hand, we should not overuse this practice, and set specific > assignee only if there is a string reason for this. > > All in all, remember that community is alive thing built of real people. > The more rules you set without a clear reason, the more uncomfortable it is > to be in such a community. > > On Wed, Aug 1, 2018 at 10:30 AM Maxim Muzafarov > wrote: > > > Vyacheslav, > > > > Thank you! This seems exactly what I was asking about. > > > > > > On Wed, 1 Aug 2018 at 10:18 Vyacheslav Daradur > > wrote: > > > > > Hi, Maxim, > > > > > > There is information about project components maintainers [1], but I'm > > > not sure if it is actual. > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers > > > On Wed, Aug 1, 2018 at 9:33 AM Maxim Muzafarov > > wrote: > > > > > > > > Folks, > > > > > > > > I don't think we need additional notification for catching attention > of > > > > some community > > > > experts. Such notifications can be overused by other participants. > > > > Developer mail list > > > > the best place we have for any discussions. > > > > > > > > What I really would like to have is the list of community > > members\experts > > > > and their zones > > > > of Ignite project code responsibilities. May be such list already > > exists, > > > > does it? > > > > > > > > As for me, I think we should pose right the complexity of the issue > at > > > the > > > > moment > > > > of their creation. Ask expert to provide some high level > implementation > > > > details and > > > > keep it unassingned, so anyone can take it. > > > > > > > > On Wed, 1 Aug 2018 at 01:09 Nikolay Izhikov > > wrote: > > > > > > > > > How about mention desired expert with some predefined message? > > > > > So expert can setup simple email filter and know about tickets that > > > have > > > > > to be done. > > > > > > > > > > Example: > > > > > > > > > > "[~dpavlov] TFY" - ticket for you. > > > > > > > > > > В Вт, 31/07/2018 в 14:04 -0700, Dmitriy Setrakyan пишет: > > > > > > Dmitriy, > > > > > > > > > > > > I would agree with everything you are saying. There are some > > tickets, > > > > > > however, that cannot be done by just anyone and preferably have > to > > be > > > > > > looked at by certain domain experts. In those cases only it would > > be > > > OK > > > > > to > > > > > > assign a ticket explicitly in my view. For all other cases we > > should > > > > > allow > > > > > > the community pick up a ticket they want to work on. > > > > > > > > > > > > D. > > > > > > > > > > > > On Tue, Jul 31, 2018 at 10:34 AM, Dmitriy Pavlov < > > > dpavlov@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Hi Igniters, > > > > > > > > > > > > > > Ignite is complex project and sometimes we know > > > > > > > 1. who can solve the issue > > > > > > > 2. and we would like that particular contributor would take an > > > issue. > > > > > > > > > > > > > > In the same time "in an open source community where everything > is > > > > > > > distributed, asynchronous and volunteer work we should leave > the > > > issues > > > > > > > open to anyone until someone volunteers to work on it." > > > > > > > > > > > > > > It seems it is not prohibited by Apache to assign it directly, > in > > > the > > > > > same > > > > > > > time it is better for community grown to avoid it: > > > > > > > "Assigning things to people seems the role of a project manager > > > that > > > > > has > > > > > > > some sort of power over the managed team. Speaking from > > experience > > > I > > > > > do not > > > > > > > take it nicely when someone that uses the project I am a > > volunteer > > > at > > > > > > > (which I might now know ) "assigns work" to me." > > > > > > > > > > > > > > Please find Apache mentors' comments on this: > >
Re: find bugs code check
Hi Maxim, Evgeniy,
Thank you for researching and finding these issues.
Who could create tickets (probably, newbie) so newcomers can pick up and
fix it?
Sincerely,
Dmitriy Pavlov
ср, 1 авг. 2018 г. в 11:36, Maxim Muzafarov :
> Hello,
>
> I can be wrong, but looks like -
>
> `GridIoManager` not a bug, we are checking isEmpty here.
> `GridCacheLockImpl` not a bug, variable is volatile.
>
> Suppose, null-check not too critial for these classes, but can be done.
>
> `GridTuple6` - must be fixed.
> `GridClientJdkMarshaller` - should be fixed.
>
> On Tue, 31 Jul 2018 at 16:44 Евгений Станиловский
> wrote:
>
> > hi, igniters.
> > I take a little time to analyze findbugs
> http://findbugs.sourceforge.net/
> > output from ignite-core module.
> > There is summary of suspicious messages:
> >
> > GridIoManager
> >
> > sync on conc map:
> >
> > synchronized (map) {
> > rmv = map.remove(set.nodeId(), set);
> > }
> > ---
> >
> > GridCacheLockImpl
> >
> > read without sync monitor
> >
> > final int getPermits() {
> > return getState();
> > }
> >
> > final synchronized void setPermits(int permits) {
> > setState(permits);
> > }
> >
> > ---
> >
> > GridDhtPartitionFullMap
> >
> > add null check
> >
> > @Override public boolean equals(Object o) {
> > if (this == o)
> > return true;
> >
> > GridDhtPartitionFullMap other = (GridDhtPartitionFullMap)o;
> >
> > return other.nodeId.equals(nodeId) && other.updateSeq == updateSeq;
> > }
> >
> > GridDhtPartitionMap
> >
> > add null check
> >
> > @Override public boolean equals(Object o) {
> > if (this == o)
> > return true;
> >
> > GridDhtPartitionMap other = (GridDhtPartitionMap)o;
> >
> > return other.nodeId.equals(nodeId) && other.updateSeq == updateSeq;
> > }
> >
> > add null check
> >
> > GridNearOptimisticTxPrepareFuture
> >
> > @Override public boolean equals(Object o) {
> > MappingKey that = (MappingKey) o;
> >
> > return nearEntries == that.nearEntries && nodeId.equals(that.nodeId);
> > }
> >
> > -
> >
> > copy-paste:
> > public class GridTuple6
> >
> > @Override public boolean equals(Object o) {
> > if (this == o)
> > return true;
> >
> > if (!(o instanceof GridTuple5))
> > return false;
> >
> >
> > ---
> >
> > not closing stream:
> > public class GridClientJdkMarshaller implements GridClientMarshaller {
> > /** ID. */
> > public static final byte ID = 2;
> >
> > /** {@inheritDoc} */
> > @Override public ByteBuffer marshal(Object obj, int off) throws
> > IOException {
> > GridByteArrayOutputStream bOut = new GridByteArrayOutputStream();
> >
> > ObjectOutput out = new ObjectOutputStream(bOut);
> > plz take a look on it, thanks !
>
> --
> --
> Maxim Muzafarov
>
Re: async operation is not fair async
Igniters, I've re-read this thread in brief. As far as I know this change is not coming from some company, so this change will be at least good for healthy community building. And I didn't find any obstacles why not to implement approach with new mode .withFairAsync() for cases user is completely aware of consequences. It could be not public API to avoid anyone will use it. It could be used,e.g. in integrations and by qualified users to gain as much throutghput as it is possible. So I would like to be an sponsor here. If anyone or Dmitriy G. will contribute this change, I will merge it. I hope PMCs are agree here and will not veto this change. Sincerely, Dmitriy Pavlov чт, 24 мая 2018 г. в 15:13, Yakov Zhdanov : > Alexey Goncharuk, I remember we started working on async connection > establishment. This should fix latency issue related to network which I > believe gives the most contribution to overall latency. Mapping logic and > other stuff can be ignored as it can very rarely be an issue at least on > stable tolopogies. What is the status with async connections? That would > really be a huge improvement! > > Also please remember that uncontrolled async operations may lead to OOME, > therefore at some point when there are too many uncompleted async > operations newly invoked async operations should become synchronous, i.e. > we should return completed future, ignoring the fact that user expected us > to be async. > > I would like to have very strong reasons to start reapproaching this. > > --Yakov >
Re: Removing "fabric" from Ignite binary package name
Since you proposing patch to the community, you are the very man :) ср, 1 авг. 2018 г. в 12:16, Petr Ivanov : > You are convincing the wrong person. > > > > > On 1 Aug 2018, at 12:05, Anton Vinogradov wrote: > > > > Peter, > > > > We had a discussion about how to do this properly. > > Proposed solution cannot be merged, since it makes code harder than it > was. > > > > The only case is to perform complete refactoring and get rid of all > > postfixes and other weird stuff. > > > > For example > > - > > - > > should be definetely removed from code. > > > > > > ср, 1 авг. 2018 г. в 9:39, Peter Ivanov : > > > >> The task was ready long ago, but community failed to review and merge it > >> ¯\_(ツ)_/¯ > >> Not being a committer, my capabilities of introducing such changes are > >> limited. > >> > >> I will update code during this week and will pass for review once again. > >> > >> > >> On Wed, 1 Aug 2018 at 00:24, Dmitriy Setrakyan > >> wrote: > >> > >>> Yes, agree, fabric has to be removed. If it is done in 2.7, would be > >> great! > >>> > >>> On Tue, Jul 31, 2018 at 2:18 PM, Denis Magda > wrote: > >>> > Peter, folks, > > It's weird, but we have been failing to introduce this minor change > >> since > December. Can we get it done for 2.7 that is being discussed at the > >>> moment? > Are there any technical issues that block you from merging the > changes? > > -- > Denis > > On Thu, Jun 7, 2018 at 10:03 PM Peter Ivanov > >>> wrote: > > > Ok, then I will update issue code and start preparation for build > > configuration changes. > > > > > > On Thu, 7 Jun 2018 at 23:41, Denis Magda wrote: > > > >>> > >>> With which one — current implementation in issue? > >> > >> > >> That's the answer to your question: > >> > >> 1. quickly fix all of them (can be solved by preliminary > >>> preparations — > >> searching for -fabric- usages in build configuration); > >>2. update all branches to master because otherwise old branch > >>> will > > stop > >> building. > >> > >> > >> -- > >> Denis > >> > >> On Thu, Jun 7, 2018 at 1:12 PM, Petr Ivanov > wrote: > >> > >>> > On 7 Jun 2018, at 23:04, Denis Magda > >> wrote: > > I'm fine with the suggested approach. > >>> > >>> With which one — current implementation in issue? > >>> > >>> > However, not sure we need to update > all the branches. Can't branch owners just pull the changes > >> back > from > master if the plan to merge back later? > >>> > >>> Of course, we as an initiative group of this issue should do > >>> nothing, > > it > >>> will lie on shoulders of developers. > >>> > >>> > > -- > Denis > > On Thu, Jun 7, 2018 at 12:57 PM, Petr Ivanov < > >>> mr.wei...@gmail.com> > >>> wrote: > > > Denis, > > > > > > The most simple approach — repack and rearchive binary archive > after > > release build, however that would not resolve the problem > >>> globally > >> (and > > will require fixing every build configuration we have on > TeamCity). > > Current approach implemented in task — creates already correct > > folder > >>> and > > binary archive name, but old name (with -fabric-) is used in > almost > >>> every > > build configuration too and merge code to master will require > >>> to: > > 1. quickly fix all of them (can be solved by preliminary > >> preparations > > — searching for -fabric- usages in build configuration); > > 2. update all branches to master because otherwise old > >> branch > > will > > stop building. > > > > WDYT? > > > > > > > >> On 7 Jun 2018, at 22:42, Denis Magda > >>> wrote: > >> > >> Petr, > >> > >> Thanks for pulling up the conversation. > >> > >> I still prefer us not to complicate the things and just > >> remove > >> "fabric" > >> from the *package name*. Use the easiest way possible. > >> > >> Personally, I don't care about Hadoop and would not suggest > >> the > >>> community > >> wasting its time on it. So, just rename the suffixes/prefixes > >>> of > > the > > build > >> files the way you like to address Anton's concerns. > >> > >> -- > >> Denis > >> > >> > >> On Thu, Jun 7, 2018 at 1:49 AM, Petr Ivanov < > >>> mr.wei...@gmail.com > > > >>> wrote: > >> > >>> Igniters, > >>> > >>> > >>> Lets define once again what should be done in this [1] task? > >>> If current implementation is good, than I’ll update it to > >>> master >
Re: Removing "fabric" from Ignite binary package name
You are convincing the wrong person. > On 1 Aug 2018, at 12:05, Anton Vinogradov wrote: > > Peter, > > We had a discussion about how to do this properly. > Proposed solution cannot be merged, since it makes code harder than it was. > > The only case is to perform complete refactoring and get rid of all > postfixes and other weird stuff. > > For example > - > - > should be definetely removed from code. > > > ср, 1 авг. 2018 г. в 9:39, Peter Ivanov : > >> The task was ready long ago, but community failed to review and merge it >> ¯\_(ツ)_/¯ >> Not being a committer, my capabilities of introducing such changes are >> limited. >> >> I will update code during this week and will pass for review once again. >> >> >> On Wed, 1 Aug 2018 at 00:24, Dmitriy Setrakyan >> wrote: >> >>> Yes, agree, fabric has to be removed. If it is done in 2.7, would be >> great! >>> >>> On Tue, Jul 31, 2018 at 2:18 PM, Denis Magda wrote: >>> Peter, folks, It's weird, but we have been failing to introduce this minor change >> since December. Can we get it done for 2.7 that is being discussed at the >>> moment? Are there any technical issues that block you from merging the changes? -- Denis On Thu, Jun 7, 2018 at 10:03 PM Peter Ivanov >>> wrote: > Ok, then I will update issue code and start preparation for build > configuration changes. > > > On Thu, 7 Jun 2018 at 23:41, Denis Magda wrote: > >>> >>> With which one — current implementation in issue? >> >> >> That's the answer to your question: >> >> 1. quickly fix all of them (can be solved by preliminary >>> preparations — >> searching for -fabric- usages in build configuration); >>2. update all branches to master because otherwise old branch >>> will > stop >> building. >> >> >> -- >> Denis >> >> On Thu, Jun 7, 2018 at 1:12 PM, Petr Ivanov wrote: >> >>> On 7 Jun 2018, at 23:04, Denis Magda >> wrote: I'm fine with the suggested approach. >>> >>> With which one — current implementation in issue? >>> >>> However, not sure we need to update all the branches. Can't branch owners just pull the changes >> back from master if the plan to merge back later? >>> >>> Of course, we as an initiative group of this issue should do >>> nothing, > it >>> will lie on shoulders of developers. >>> >>> -- Denis On Thu, Jun 7, 2018 at 12:57 PM, Petr Ivanov < >>> mr.wei...@gmail.com> >>> wrote: > Denis, > > > The most simple approach — repack and rearchive binary archive after > release build, however that would not resolve the problem >>> globally >> (and > will require fixing every build configuration we have on TeamCity). > Current approach implemented in task — creates already correct > folder >>> and > binary archive name, but old name (with -fabric-) is used in almost >>> every > build configuration too and merge code to master will require >>> to: > 1. quickly fix all of them (can be solved by preliminary >> preparations > — searching for -fabric- usages in build configuration); > 2. update all branches to master because otherwise old >> branch > will > stop building. > > WDYT? > > > >> On 7 Jun 2018, at 22:42, Denis Magda >>> wrote: >> >> Petr, >> >> Thanks for pulling up the conversation. >> >> I still prefer us not to complicate the things and just >> remove >> "fabric" >> from the *package name*. Use the easiest way possible. >> >> Personally, I don't care about Hadoop and would not suggest >> the >>> community >> wasting its time on it. So, just rename the suffixes/prefixes >>> of > the > build >> files the way you like to address Anton's concerns. >> >> -- >> Denis >> >> >> On Thu, Jun 7, 2018 at 1:49 AM, Petr Ivanov < >>> mr.wei...@gmail.com > >>> wrote: >> >>> Igniters, >>> >>> >>> Lets define once again what should be done in this [1] task? >>> If current implementation is good, than I’ll update it to >>> master > and > pass >>> for review. >>> >>> Yet, there is other part of the task which concerns our >> build > server >>> — I >>> assume that almost all our build configurations will fail >> due >>> to >> name >>> change and there is no simple way of updating configurations other >>> then >>> merge task to master and start fixing
Re: Removing "fabric" from Ignite binary package name
Peter, We had a discussion about how to do this properly. Proposed solution cannot be merged, since it makes code harder than it was. The only case is to perform complete refactoring and get rid of all postfixes and other weird stuff. For example - - should be definetely removed from code. ср, 1 авг. 2018 г. в 9:39, Peter Ivanov : > The task was ready long ago, but community failed to review and merge it > ¯\_(ツ)_/¯ > Not being a committer, my capabilities of introducing such changes are > limited. > > I will update code during this week and will pass for review once again. > > > On Wed, 1 Aug 2018 at 00:24, Dmitriy Setrakyan > wrote: > > > Yes, agree, fabric has to be removed. If it is done in 2.7, would be > great! > > > > On Tue, Jul 31, 2018 at 2:18 PM, Denis Magda wrote: > > > > > Peter, folks, > > > > > > It's weird, but we have been failing to introduce this minor change > since > > > December. Can we get it done for 2.7 that is being discussed at the > > moment? > > > Are there any technical issues that block you from merging the changes? > > > > > > -- > > > Denis > > > > > > On Thu, Jun 7, 2018 at 10:03 PM Peter Ivanov > > wrote: > > > > > > > Ok, then I will update issue code and start preparation for build > > > > configuration changes. > > > > > > > > > > > > On Thu, 7 Jun 2018 at 23:41, Denis Magda wrote: > > > > > > > > > > > > > > > > With which one — current implementation in issue? > > > > > > > > > > > > > > > That's the answer to your question: > > > > > > > > > > 1. quickly fix all of them (can be solved by preliminary > > preparations — > > > > > searching for -fabric- usages in build configuration); > > > > > 2. update all branches to master because otherwise old branch > > will > > > > stop > > > > > building. > > > > > > > > > > > > > > > -- > > > > > Denis > > > > > > > > > > On Thu, Jun 7, 2018 at 1:12 PM, Petr Ivanov > > > wrote: > > > > > > > > > > > > > > > > > > On 7 Jun 2018, at 23:04, Denis Magda > wrote: > > > > > > > > > > > > > > I'm fine with the suggested approach. > > > > > > > > > > > > With which one — current implementation in issue? > > > > > > > > > > > > > > > > > > > However, not sure we need to update > > > > > > > all the branches. Can't branch owners just pull the changes > back > > > from > > > > > > > master if the plan to merge back later? > > > > > > > > > > > > Of course, we as an initiative group of this issue should do > > nothing, > > > > it > > > > > > will lie on shoulders of developers. > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Denis > > > > > > > > > > > > > > On Thu, Jun 7, 2018 at 12:57 PM, Petr Ivanov < > > mr.wei...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > >> Denis, > > > > > > >> > > > > > > >> > > > > > > >> The most simple approach — repack and rearchive binary archive > > > after > > > > > > >> release build, however that would not resolve the problem > > globally > > > > > (and > > > > > > >> will require fixing every build configuration we have on > > > TeamCity). > > > > > > >> Current approach implemented in task — creates already correct > > > > folder > > > > > > and > > > > > > >> binary archive name, but old name (with -fabric-) is used in > > > almost > > > > > > every > > > > > > >> build configuration too and merge code to master will require > > to: > > > > > > >>1. quickly fix all of them (can be solved by preliminary > > > > > preparations > > > > > > >> — searching for -fabric- usages in build configuration); > > > > > > >>2. update all branches to master because otherwise old > branch > > > > will > > > > > > >> stop building. > > > > > > >> > > > > > > >> WDYT? > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >>> On 7 Jun 2018, at 22:42, Denis Magda > > wrote: > > > > > > >>> > > > > > > >>> Petr, > > > > > > >>> > > > > > > >>> Thanks for pulling up the conversation. > > > > > > >>> > > > > > > >>> I still prefer us not to complicate the things and just > remove > > > > > "fabric" > > > > > > >>> from the *package name*. Use the easiest way possible. > > > > > > >>> > > > > > > >>> Personally, I don't care about Hadoop and would not suggest > the > > > > > > community > > > > > > >>> wasting its time on it. So, just rename the suffixes/prefixes > > of > > > > the > > > > > > >> build > > > > > > >>> files the way you like to address Anton's concerns. > > > > > > >>> > > > > > > >>> -- > > > > > > >>> Denis > > > > > > >>> > > > > > > >>> > > > > > > >>> On Thu, Jun 7, 2018 at 1:49 AM, Petr Ivanov < > > mr.wei...@gmail.com > > > > > > > > > > wrote: > > > > > > >>> > > > > > > Igniters, > > > > > > > > > > > > > > > > > > Lets define once again what should be done in this [1] task? > > > > > > If current implementation is good, than I’ll update it to > > master > > > > and > > > > > > >> pass > > > > > > for review. > > > > > > > > > > > > Yet, there is other part of the task which concerns our > build > >
Re: Apache Ignite tickets with empty description & dev list references
Agree with Dmitry P. Guys! Even if you file a ticket on yourself please spend additional 5 mins to add minimal description to provide idea on the intended changes for the rest of our community or those who will have to deal with your changes in future. --Yakov
Re: find bugs code check
Hello,
I can be wrong, but looks like -
`GridIoManager` not a bug, we are checking isEmpty here.
`GridCacheLockImpl` not a bug, variable is volatile.
Suppose, null-check not too critial for these classes, but can be done.
`GridTuple6` - must be fixed.
`GridClientJdkMarshaller` - should be fixed.
On Tue, 31 Jul 2018 at 16:44 Евгений Станиловский
wrote:
> hi, igniters.
> I take a little time to analyze findbugs http://findbugs.sourceforge.net/
> output from ignite-core module.
> There is summary of suspicious messages:
>
> GridIoManager
>
> sync on conc map:
>
> synchronized (map) {
> rmv = map.remove(set.nodeId(), set);
> }
> ---
>
> GridCacheLockImpl
>
> read without sync monitor
>
> final int getPermits() {
> return getState();
> }
>
> final synchronized void setPermits(int permits) {
> setState(permits);
> }
>
> ---
>
> GridDhtPartitionFullMap
>
> add null check
>
> @Override public boolean equals(Object o) {
> if (this == o)
> return true;
>
> GridDhtPartitionFullMap other = (GridDhtPartitionFullMap)o;
>
> return other.nodeId.equals(nodeId) && other.updateSeq == updateSeq;
> }
>
> GridDhtPartitionMap
>
> add null check
>
> @Override public boolean equals(Object o) {
> if (this == o)
> return true;
>
> GridDhtPartitionMap other = (GridDhtPartitionMap)o;
>
> return other.nodeId.equals(nodeId) && other.updateSeq == updateSeq;
> }
>
> add null check
>
> GridNearOptimisticTxPrepareFuture
>
> @Override public boolean equals(Object o) {
> MappingKey that = (MappingKey) o;
>
> return nearEntries == that.nearEntries && nodeId.equals(that.nodeId);
> }
>
> -
>
> copy-paste:
> public class GridTuple6
>
> @Override public boolean equals(Object o) {
> if (this == o)
> return true;
>
> if (!(o instanceof GridTuple5))
> return false;
>
>
> ---
>
> not closing stream:
> public class GridClientJdkMarshaller implements GridClientMarshaller {
> /** ID. */
> public static final byte ID = 2;
>
> /** {@inheritDoc} */
> @Override public ByteBuffer marshal(Object obj, int off) throws
> IOException {
> GridByteArrayOutputStream bOut = new GridByteArrayOutputStream();
>
> ObjectOutput out = new ObjectOutputStream(bOut);
> plz take a look on it, thanks !
--
--
Maxim Muzafarov
[jira] [Created] (IGNITE-9152) Web console: add a checkbox 'include in generated project'
Pavel Konstantinov created IGNITE-9152: -- Summary: Web console: add a checkbox 'include in generated project' Key: IGNITE-9152 URL: https://issues.apache.org/jira/browse/IGNITE-9152 Project: Ignite Issue Type: Improvement Components: wizards Reporter: Pavel Konstantinov -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IGNITE-9151) Web console: incorrect caches list in 'Import from DB' modal
Pavel Konstantinov created IGNITE-9151: -- Summary: Web console: incorrect caches list in 'Import from DB' modal Key: IGNITE-9151 URL: https://issues.apache.org/jira/browse/IGNITE-9151 Project: Ignite Issue Type: Bug Components: wizards Reporter: Pavel Konstantinov -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Assign JIRA tickets to someone else
In general there is nothing wrong with assigning a ticket to someone if you know that this would not cause negative reaction from assignee side. The thing is that community has a lot of members who knows each other in person and in close contact for years. So there is nothing wrong if I assign a ticket to my fellow. Especially if ticket is complex or critically important for the product. These is no "project management" in such an action. On the other hand, we should not overuse this practice, and set specific assignee only if there is a string reason for this. All in all, remember that community is alive thing built of real people. The more rules you set without a clear reason, the more uncomfortable it is to be in such a community. On Wed, Aug 1, 2018 at 10:30 AM Maxim Muzafarov wrote: > Vyacheslav, > > Thank you! This seems exactly what I was asking about. > > > On Wed, 1 Aug 2018 at 10:18 Vyacheslav Daradur > wrote: > > > Hi, Maxim, > > > > There is information about project components maintainers [1], but I'm > > not sure if it is actual. > > > > [1] > > > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers > > On Wed, Aug 1, 2018 at 9:33 AM Maxim Muzafarov > wrote: > > > > > > Folks, > > > > > > I don't think we need additional notification for catching attention of > > > some community > > > experts. Such notifications can be overused by other participants. > > > Developer mail list > > > the best place we have for any discussions. > > > > > > What I really would like to have is the list of community > members\experts > > > and their zones > > > of Ignite project code responsibilities. May be such list already > exists, > > > does it? > > > > > > As for me, I think we should pose right the complexity of the issue at > > the > > > moment > > > of their creation. Ask expert to provide some high level implementation > > > details and > > > keep it unassingned, so anyone can take it. > > > > > > On Wed, 1 Aug 2018 at 01:09 Nikolay Izhikov > wrote: > > > > > > > How about mention desired expert with some predefined message? > > > > So expert can setup simple email filter and know about tickets that > > have > > > > to be done. > > > > > > > > Example: > > > > > > > > "[~dpavlov] TFY" - ticket for you. > > > > > > > > В Вт, 31/07/2018 в 14:04 -0700, Dmitriy Setrakyan пишет: > > > > > Dmitriy, > > > > > > > > > > I would agree with everything you are saying. There are some > tickets, > > > > > however, that cannot be done by just anyone and preferably have to > be > > > > > looked at by certain domain experts. In those cases only it would > be > > OK > > > > to > > > > > assign a ticket explicitly in my view. For all other cases we > should > > > > allow > > > > > the community pick up a ticket they want to work on. > > > > > > > > > > D. > > > > > > > > > > On Tue, Jul 31, 2018 at 10:34 AM, Dmitriy Pavlov < > > dpavlov@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi Igniters, > > > > > > > > > > > > Ignite is complex project and sometimes we know > > > > > > 1. who can solve the issue > > > > > > 2. and we would like that particular contributor would take an > > issue. > > > > > > > > > > > > In the same time "in an open source community where everything is > > > > > > distributed, asynchronous and volunteer work we should leave the > > issues > > > > > > open to anyone until someone volunteers to work on it." > > > > > > > > > > > > It seems it is not prohibited by Apache to assign it directly, in > > the > > > > same > > > > > > time it is better for community grown to avoid it: > > > > > > "Assigning things to people seems the role of a project manager > > that > > > > has > > > > > > some sort of power over the managed team. Speaking from > experience > > I > > > > do not > > > > > > take it nicely when someone that uses the project I am a > volunteer > > at > > > > > > (which I might now know ) "assigns work" to me." > > > > > > > > > > > > Please find Apache mentors' comments on this: > > > > > > > > https://lists.apache.org/thread.html/c6013b99940de32aae831a0b76e8fd > > > > > > 53febe5040e9e0d67abb4f62a5@%3Cdev.community.apache.org%3E > > > > > > > > > > > > > > > > > > Please share you vision. > > > > > > > > > > > > Sincerely, > > > > > > Dmitriy Pavlov > > > > > > > > > > > > -- > > > -- > > > Maxim Muzafarov > > > > > > > > -- > > Best Regards, Vyacheslav D. > > > -- > -- > Maxim Muzafarov >
[GitHub] ignite pull request #4363: IGNITE-7752: Update Ignite KafkaStreamer to use n...
Github user asfgit closed the pull request at: https://github.com/apache/ignite/pull/4363 ---
[GitHub] ignite pull request #4464: IGNITE-8158 Wrap the method afterTestsStopped() c...
GitHub user zzzadruga opened a pull request: https://github.com/apache/ignite/pull/4464 IGNITE-8158 Wrap the method afterTestsStopped() call with try/catch b⦠â¦lock You can merge this pull request into a Git repository by running: $ git pull https://github.com/zzzadruga/ignite IGNITE-8158 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/ignite/pull/4464.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4464 commit 67adeb02efb776446075d1939cb3e08afbf36616 Author: zzzadruga Date: 2018-07-31T16:19:53Z IGNITE-8158 Wrap the method afterTestsStopped() call with try/catch block ---
Re: Assign JIRA tickets to someone else
Vyacheslav, Thank you! This seems exactly what I was asking about. On Wed, 1 Aug 2018 at 10:18 Vyacheslav Daradur wrote: > Hi, Maxim, > > There is information about project components maintainers [1], but I'm > not sure if it is actual. > > [1] > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers > On Wed, Aug 1, 2018 at 9:33 AM Maxim Muzafarov wrote: > > > > Folks, > > > > I don't think we need additional notification for catching attention of > > some community > > experts. Such notifications can be overused by other participants. > > Developer mail list > > the best place we have for any discussions. > > > > What I really would like to have is the list of community members\experts > > and their zones > > of Ignite project code responsibilities. May be such list already exists, > > does it? > > > > As for me, I think we should pose right the complexity of the issue at > the > > moment > > of their creation. Ask expert to provide some high level implementation > > details and > > keep it unassingned, so anyone can take it. > > > > On Wed, 1 Aug 2018 at 01:09 Nikolay Izhikov wrote: > > > > > How about mention desired expert with some predefined message? > > > So expert can setup simple email filter and know about tickets that > have > > > to be done. > > > > > > Example: > > > > > > "[~dpavlov] TFY" - ticket for you. > > > > > > В Вт, 31/07/2018 в 14:04 -0700, Dmitriy Setrakyan пишет: > > > > Dmitriy, > > > > > > > > I would agree with everything you are saying. There are some tickets, > > > > however, that cannot be done by just anyone and preferably have to be > > > > looked at by certain domain experts. In those cases only it would be > OK > > > to > > > > assign a ticket explicitly in my view. For all other cases we should > > > allow > > > > the community pick up a ticket they want to work on. > > > > > > > > D. > > > > > > > > On Tue, Jul 31, 2018 at 10:34 AM, Dmitriy Pavlov < > dpavlov@gmail.com> > > > > wrote: > > > > > > > > > Hi Igniters, > > > > > > > > > > Ignite is complex project and sometimes we know > > > > > 1. who can solve the issue > > > > > 2. and we would like that particular contributor would take an > issue. > > > > > > > > > > In the same time "in an open source community where everything is > > > > > distributed, asynchronous and volunteer work we should leave the > issues > > > > > open to anyone until someone volunteers to work on it." > > > > > > > > > > It seems it is not prohibited by Apache to assign it directly, in > the > > > same > > > > > time it is better for community grown to avoid it: > > > > > "Assigning things to people seems the role of a project manager > that > > > has > > > > > some sort of power over the managed team. Speaking from experience > I > > > do not > > > > > take it nicely when someone that uses the project I am a volunteer > at > > > > > (which I might now know ) "assigns work" to me." > > > > > > > > > > Please find Apache mentors' comments on this: > > > > > > https://lists.apache.org/thread.html/c6013b99940de32aae831a0b76e8fd > > > > > 53febe5040e9e0d67abb4f62a5@%3Cdev.community.apache.org%3E > > > > > > > > > > > > > > > Please share you vision. > > > > > > > > > > Sincerely, > > > > > Dmitriy Pavlov > > > > > > > > > -- > > -- > > Maxim Muzafarov > > > > -- > Best Regards, Vyacheslav D. > -- -- Maxim Muzafarov
Re: Assign JIRA tickets to someone else
Hi, Maxim, There is information about project components maintainers [1], but I'm not sure if it is actual. [1] https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers On Wed, Aug 1, 2018 at 9:33 AM Maxim Muzafarov wrote: > > Folks, > > I don't think we need additional notification for catching attention of > some community > experts. Such notifications can be overused by other participants. > Developer mail list > the best place we have for any discussions. > > What I really would like to have is the list of community members\experts > and their zones > of Ignite project code responsibilities. May be such list already exists, > does it? > > As for me, I think we should pose right the complexity of the issue at the > moment > of their creation. Ask expert to provide some high level implementation > details and > keep it unassingned, so anyone can take it. > > On Wed, 1 Aug 2018 at 01:09 Nikolay Izhikov wrote: > > > How about mention desired expert with some predefined message? > > So expert can setup simple email filter and know about tickets that have > > to be done. > > > > Example: > > > > "[~dpavlov] TFY" - ticket for you. > > > > В Вт, 31/07/2018 в 14:04 -0700, Dmitriy Setrakyan пишет: > > > Dmitriy, > > > > > > I would agree with everything you are saying. There are some tickets, > > > however, that cannot be done by just anyone and preferably have to be > > > looked at by certain domain experts. In those cases only it would be OK > > to > > > assign a ticket explicitly in my view. For all other cases we should > > allow > > > the community pick up a ticket they want to work on. > > > > > > D. > > > > > > On Tue, Jul 31, 2018 at 10:34 AM, Dmitriy Pavlov > > > wrote: > > > > > > > Hi Igniters, > > > > > > > > Ignite is complex project and sometimes we know > > > > 1. who can solve the issue > > > > 2. and we would like that particular contributor would take an issue. > > > > > > > > In the same time "in an open source community where everything is > > > > distributed, asynchronous and volunteer work we should leave the issues > > > > open to anyone until someone volunteers to work on it." > > > > > > > > It seems it is not prohibited by Apache to assign it directly, in the > > same > > > > time it is better for community grown to avoid it: > > > > "Assigning things to people seems the role of a project manager that > > has > > > > some sort of power over the managed team. Speaking from experience I > > do not > > > > take it nicely when someone that uses the project I am a volunteer at > > > > (which I might now know ) "assigns work" to me." > > > > > > > > Please find Apache mentors' comments on this: > > > > https://lists.apache.org/thread.html/c6013b99940de32aae831a0b76e8fd > > > > 53febe5040e9e0d67abb4f62a5@%3Cdev.community.apache.org%3E > > > > > > > > > > > > Please share you vision. > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > -- > -- > Maxim Muzafarov -- Best Regards, Vyacheslav D.
Re: Spark DataFrames With Cache Key and Value Objects
I believe suggested approach will not work with the Spark SQL relational optimisations which perform predicate pushdown from Spark to Ignite. For that to work we need both the key/val and the relational fields in a dataframe schema. Stuart. > On 1 Aug 2018, at 04:23, Valentin Kulichenko > wrote: > > I don't think there are exact plans to remove _key and _value fields as > it's pretty hard considering the fact that many users use them and that > they are deeply integrated into the product. However, we already had > multiple usability and other issues due to their existence, and while > fixing them we gradually get rid of _key/_val on public API. Hard to tell > when we will be able to completely deprecate/remove these fields, but we > definitely should avoid building new features based on them. > > On top of that, I also don't like this approach because it doesn't seem to > be Spark-friendly to me. That's not how they typically create typed > datasets (I already provided a documentation link [1] with examples > earlier). > > From API standpoint, I think we should do the following: > 1. Add 'IgniteSparkSession#createDataset(IgniteCache[K, V] cache): > Dataset[(K, V)]' method that would create a dataset based on a cache. > 2. (Scala only) Introduce 'IgniteCache.toDS()' that would do the same, but > via implicit conversions instead of SparkSession extension. > > On implementation level, we can use SqlQuery API (not SqlFieldQuery) that > is specifically designed to return key-value pairs instead of specific > fields, while still providing all SQL capabilities. > > *Nikolay*, does this makes sense to you? Is it feasible and how hard would > it be to implement? How much of the existing code can we reuse (I believe > it should it be majority of it)? > > [1] > https://spark.apache.org/docs/2.3.1/sql-programming-guide.html#creating-datasets > > -Val > >> On Tue, Jul 31, 2018 at 2:03 PM Denis Magda wrote: >> >> Hello folks, >> >> The documentation goes with a small reference about _key and _val usage, >> and only for Ignite SQL APIs (Java, Net, C++). I tried to clean up all the >> documentation code snippets. >> >> As for the GitHub examples, they require a major overhaul. Instead of _key >> and _val usage, we need to use SQL fields. Hopefully, someone will groom >> the examples. >> >> Considering this, I wouldn't suggest us exposing _key and _val in other >> places like Spark. Are there any alternatives to this approach? >> >> -- >> Denis >> >> >> >> On Tue, Jul 31, 2018 at 2:49 AM Nikolay Izhikov >> wrote: >> >>> Hello, Igniters. >>> >>> Valentin, >>> We never recommend to use these fields >>> >>> Actually, we did: >>> >>>* Documentation [1]. Please, see "Predefined Fields" section. >>>* Java Example [2] >>>* DotNet Example [3] >>>* Scala Example [4] >>> ...hopefully will be removed altogether one day >>> >>> This is new for me. >>> >>> Do we have specific plans for it? >>> >>> [1] https://apacheignite-sql.readme.io/docs/schema-and-indexes >>> [2] >>> >> https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/sql/SqlDmlExample.java#L88 >>> [3] >>> >> https://github.com/apache/ignite/blob/master/modules/platforms/dotnet/examples/Apache.Ignite.Examples/Sql/SqlDmlExample.cs#L91 >>> [4] >>> >> https://github.com/apache/ignite/blob/master/examples/src/main/scala/org/apache/ignite/scalar/examples/ScalarCachePopularNumbersExample.scala#L124 >>> >>> В Пт, 27/07/2018 в 15:22 -0700, Valentin Kulichenko пишет: Stuart, _key and _val fields is quite a dirty hack that was added years ago and >>> is virtually never used now. We never recommend to use these fields and I would definitely avoid building new features based on them. Having said that, I'm not arguing the use case, but we need better implementation approach here. I suggest we think it over and come back >> to this next week :) I'm sure Nikolay will also chime in and share his thoughts. -Val On Fri, Jul 27, 2018 at 12:39 PM Stuart Macdonald >>> wrote: > If your predicates and joins are expressed in Spark SQL, you cannot > currently optimise those and also gain access to the key/val objects. >>> If > you went without the Ignite Spark SQL optimisations and expressed >> your > query in Ignite SQL, you still need to use the _key/_val columns. The > Ignite documentation has this specific example using the _val column >>> (right > at the end): > https://apacheignite-fs.readme.io/docs/ignitecontext-igniterdd > > Stuart. > > On 27 Jul 2018, at 20:05, Valentin Kulichenko < > valentin.kuliche...@gmail.com> > wrote: > > Well, the second approach would use the optimizations, no? > > -Val > > > On Fri, Jul 27, 2018 at 11:49 AM Stuart Macdonald >> > wrote: > > Val, > > > Yes you can already get access to the cache objects
Re: Removing "fabric" from Ignite binary package name
The task was ready long ago, but community failed to review and merge it ¯\_(ツ)_/¯ Not being a committer, my capabilities of introducing such changes are limited. I will update code during this week and will pass for review once again. On Wed, 1 Aug 2018 at 00:24, Dmitriy Setrakyan wrote: > Yes, agree, fabric has to be removed. If it is done in 2.7, would be great! > > On Tue, Jul 31, 2018 at 2:18 PM, Denis Magda wrote: > > > Peter, folks, > > > > It's weird, but we have been failing to introduce this minor change since > > December. Can we get it done for 2.7 that is being discussed at the > moment? > > Are there any technical issues that block you from merging the changes? > > > > -- > > Denis > > > > On Thu, Jun 7, 2018 at 10:03 PM Peter Ivanov > wrote: > > > > > Ok, then I will update issue code and start preparation for build > > > configuration changes. > > > > > > > > > On Thu, 7 Jun 2018 at 23:41, Denis Magda wrote: > > > > > > > > > > > > > With which one — current implementation in issue? > > > > > > > > > > > > That's the answer to your question: > > > > > > > > 1. quickly fix all of them (can be solved by preliminary > preparations — > > > > searching for -fabric- usages in build configuration); > > > > 2. update all branches to master because otherwise old branch > will > > > stop > > > > building. > > > > > > > > > > > > -- > > > > Denis > > > > > > > > On Thu, Jun 7, 2018 at 1:12 PM, Petr Ivanov > > wrote: > > > > > > > > > > > > > > > On 7 Jun 2018, at 23:04, Denis Magda wrote: > > > > > > > > > > > > I'm fine with the suggested approach. > > > > > > > > > > With which one — current implementation in issue? > > > > > > > > > > > > > > > > However, not sure we need to update > > > > > > all the branches. Can't branch owners just pull the changes back > > from > > > > > > master if the plan to merge back later? > > > > > > > > > > Of course, we as an initiative group of this issue should do > nothing, > > > it > > > > > will lie on shoulders of developers. > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Denis > > > > > > > > > > > > On Thu, Jun 7, 2018 at 12:57 PM, Petr Ivanov < > mr.wei...@gmail.com> > > > > > wrote: > > > > > > > > > > > >> Denis, > > > > > >> > > > > > >> > > > > > >> The most simple approach — repack and rearchive binary archive > > after > > > > > >> release build, however that would not resolve the problem > globally > > > > (and > > > > > >> will require fixing every build configuration we have on > > TeamCity). > > > > > >> Current approach implemented in task — creates already correct > > > folder > > > > > and > > > > > >> binary archive name, but old name (with -fabric-) is used in > > almost > > > > > every > > > > > >> build configuration too and merge code to master will require > to: > > > > > >>1. quickly fix all of them (can be solved by preliminary > > > > preparations > > > > > >> — searching for -fabric- usages in build configuration); > > > > > >>2. update all branches to master because otherwise old branch > > > will > > > > > >> stop building. > > > > > >> > > > > > >> WDYT? > > > > > >> > > > > > >> > > > > > >> > > > > > >>> On 7 Jun 2018, at 22:42, Denis Magda > wrote: > > > > > >>> > > > > > >>> Petr, > > > > > >>> > > > > > >>> Thanks for pulling up the conversation. > > > > > >>> > > > > > >>> I still prefer us not to complicate the things and just remove > > > > "fabric" > > > > > >>> from the *package name*. Use the easiest way possible. > > > > > >>> > > > > > >>> Personally, I don't care about Hadoop and would not suggest the > > > > > community > > > > > >>> wasting its time on it. So, just rename the suffixes/prefixes > of > > > the > > > > > >> build > > > > > >>> files the way you like to address Anton's concerns. > > > > > >>> > > > > > >>> -- > > > > > >>> Denis > > > > > >>> > > > > > >>> > > > > > >>> On Thu, Jun 7, 2018 at 1:49 AM, Petr Ivanov < > mr.wei...@gmail.com > > > > > > > > wrote: > > > > > >>> > > > > > Igniters, > > > > > > > > > > > > > > > Lets define once again what should be done in this [1] task? > > > > > If current implementation is good, than I’ll update it to > master > > > and > > > > > >> pass > > > > > for review. > > > > > > > > > > Yet, there is other part of the task which concerns our build > > > server > > > > > — I > > > > > assume that almost all our build configurations will fail due > to > > > > name > > > > > change and there is no simple way of updating configurations > > other > > > > > then > > > > > merge task to master and start fixing failing builds. > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-7251 > > > > > > > > > > > > > > > > > > > > > On 10 Feb 2018, at 01:56, Denis Magda > > wrote: > > > > > > > > > > > >> I don't think we necessarily need to remove 'fabric' word > from > > > > every > > > > > file > > > > > >> in
Re: Assign JIRA tickets to someone else
Folks, I don't think we need additional notification for catching attention of some community experts. Such notifications can be overused by other participants. Developer mail list the best place we have for any discussions. What I really would like to have is the list of community members\experts and their zones of Ignite project code responsibilities. May be such list already exists, does it? As for me, I think we should pose right the complexity of the issue at the moment of their creation. Ask expert to provide some high level implementation details and keep it unassingned, so anyone can take it. On Wed, 1 Aug 2018 at 01:09 Nikolay Izhikov wrote: > How about mention desired expert with some predefined message? > So expert can setup simple email filter and know about tickets that have > to be done. > > Example: > > "[~dpavlov] TFY" - ticket for you. > > В Вт, 31/07/2018 в 14:04 -0700, Dmitriy Setrakyan пишет: > > Dmitriy, > > > > I would agree with everything you are saying. There are some tickets, > > however, that cannot be done by just anyone and preferably have to be > > looked at by certain domain experts. In those cases only it would be OK > to > > assign a ticket explicitly in my view. For all other cases we should > allow > > the community pick up a ticket they want to work on. > > > > D. > > > > On Tue, Jul 31, 2018 at 10:34 AM, Dmitriy Pavlov > > wrote: > > > > > Hi Igniters, > > > > > > Ignite is complex project and sometimes we know > > > 1. who can solve the issue > > > 2. and we would like that particular contributor would take an issue. > > > > > > In the same time "in an open source community where everything is > > > distributed, asynchronous and volunteer work we should leave the issues > > > open to anyone until someone volunteers to work on it." > > > > > > It seems it is not prohibited by Apache to assign it directly, in the > same > > > time it is better for community grown to avoid it: > > > "Assigning things to people seems the role of a project manager that > has > > > some sort of power over the managed team. Speaking from experience I > do not > > > take it nicely when someone that uses the project I am a volunteer at > > > (which I might now know ) "assigns work" to me." > > > > > > Please find Apache mentors' comments on this: > > > https://lists.apache.org/thread.html/c6013b99940de32aae831a0b76e8fd > > > 53febe5040e9e0d67abb4f62a5@%3Cdev.community.apache.org%3E > > > > > > > > > Please share you vision. > > > > > > Sincerely, > > > Dmitriy Pavlov > > > -- -- Maxim Muzafarov
