Re: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)
Yuriy, Thank you for providing details! Quite interesting. Yes, we already have support of distributed limit and merging sorted subresults for SQL queries. E.g. ReduceIndexSorted and MergeStreamIterator are used for merging sorted streams. Could you please also clarify about score/relevance? Is it provided by Lucene engine for each query result? I am thinking how to do sorted merge properly in this case. ср, 25 сент. 2019 г. в 18:56, Yuriy Shuliga : > > Ivan, > > Thank you for interesting question! > > Text searches (or full text searches) are mostly human-oriented. And the > point of user's interest is topmost part of response. > Then user can read it, evaluate and use the given records for further > purposes. > > Particularly in our case, we use Ignite for operations with financial data, > and there lots of text stuff like assets names, fin. instruments, companies > etc. > In order to operate with this quickly and reliably, users used to work with > text search, type-ahead completions, suggestions. > > For this purposes we are indexing particular string data in separate caches. > > Sorting capabilities and response size limitations are very important > there. As our API have to provide most relevant information in view of > limited size. > > Now let me comment some Ignite/Lucene perspective. > Actually Ignite queries and Lucene returns *TopDocs.scoresDocs *already > sorted by *score *(relevance). So most relevant documents are on the top. > And currently distributed queries responses from different nodes are merged > into final query cursor queue in arbitrary way. > So in fact we already have the score order ruined here. Also Ignite > requests all possible documents from Lucene that is redundant and not good > for performance. > > I'm implementing *limit* parameter to be part of *TextQuery *and have to > notice that we still have to add sorting for text queries processing in > order to have applicable results. > > *Limit* parameter itself should improve the part of issues from above, but > definitely, sorting by document score at least should be implemented along > with limit. > > This is a pretty short commentary if you still have any questions, please > ask, do not hesitate) > > BR, > Yuriy Shuliha > > чт, 19 вер. 2019 о 11:38 Павлухин Иван пише: > > > Yuriy, > > > > Greatly appreciate your interest. > > > > Could you please elaborate a little bit about sorting? What tasks does > > it help to solve and how? It would be great to provide an example. > > > > ср, 18 сент. 2019 г. в 09:39, Alexei Scherbakov < > > alexey.scherbak...@gmail.com>: > > > > > > Denis, > > > > > > I like the idea of throwing an exception for enabled text queries on > > > persistent caches. > > > > > > Also I'm fine with proposed limit for unsorted searches. > > > > > > Yury, please proceed with ticket creation. > > > > > > вт, 17 сент. 2019 г., 22:06 Denis Magda : > > > > > > > Igniters, > > > > > > > > I see nothing wrong with Yury's proposal in regards full-text search > > API > > > > evolution as long as Yury is ready to push it forward. > > > > > > > > As for the in-memory mode only, it makes total sense for in-memory data > > > > grid deployments when Ignite caches data of an underlying DB like > > Postgres. > > > > As part of the changes, I would simply throw an exception (by default) > > if > > > > the one attempts to use text indices with the native persistence > > enabled. > > > > If the person is ready to live with that limitation that an explicit > > > > configuration change is needed to come around the exception. > > > > > > > > Thoughts? > > > > > > > > > > > > - > > > > Denis > > > > > > > > > > > > On Tue, Sep 17, 2019 at 7:44 AM Yuriy Shuliga > > wrote: > > > > > > > > > Hello to all again, > > > > > > > > > > Thank you for important comments and notes given below! > > > > > > > > > > Let me answer and continue the discussion. > > > > > > > > > > (I) Overall needs in Lucene indexing > > > > > > > > > > Alexei has referenced to > > > > > https://issues.apache.org/jira/browse/IGNITE-5371 where > > > > > absence of index persistence was declared as an obstacle to further > > > > > development. > > > > > > > > > > a) This tic
Re: A question
Hi Maria,
Yes, it should work with your classes. It worth noting, that
"lastName" will be accessible in your queries, "SELECT lastName FROM
..." should work. Note that you should use simply "lastName" in SQL
but not "general.lastName".
By the way, usually it is better to write such questions to Ignite
users list u...@ignite.apache.org, there you will definitely get the
answer.
вт, 24 сент. 2019 г. в 20:57, Мария Помазкина :
>
> Hello!
> Qould you please tell me,
> can I get Embeddable field data by sql query?
> My field is (I need to get lastNamefrow sql query):
>
> @Table(name = "rzd_employee")
> public class EmployeeExtVO implements Serializable {
>
> ...
>
> @Embedded
> @QuerySqlField
> private GeneralAttrs general;
> ...
> }
>
> @Embeddable
> public class GeneralAttrs implements Serializable {
>
>@QuerySqlField
> private String lastName;
--
Best regards,
Ivan Pavlukhin
Re: Non-blocking PME Phase One (Node fail)
Anton, folks, Out of curiosity. Do we have some measurements of PME execution in an environment similar to some real-world one? In my mind some kind of "profile" showing durations of different PME phases with an indication where we are "blocked" would be ideal. ср, 18 сент. 2019 г. в 13:27, Anton Vinogradov : > > Alexey, > > >> Can you describe the complete protocol changes first > The current goal is to find a better way. > I had at least 5 scenarios discarded because of finding corner cases > (Thanks to Alexey Scherbakov, Aleksei Plekhanov and Nikita Amelchev). > That's why I explained what we able to improve and why I think it works. > > >> we need to remove this property, not add new logic that relies on it. > Agree. > > >> How are you going to synchronize this? > Thanks for providing this case, seems it discards #1 + #2.2 case and #2.1 > still possible with some optimizations. > > "Zombie eating transactions" case can be theoretically solved, I think. > As I said before we may perform "Local switch" in case affinity was not > changed (except failed mode miss) other cases require regular PME. > In this case, new-primary is an ex-backup and we expect that old-primary > will try to update new-primary as a backup. > New primary will handle operations as a backup until it notified it's a > primary now. > Operations from ex-primary will be discarded or remapped once new-primary > notified it became the primary. > > Discarding is a big improvement, > remapping is a huge improvement, > there is no 100% warranty that ex-primary will try to update new-primary as > a backup. > > A lot of corner cases here. > So, seems minimized sync is a better solution. > > Finally, according to your and Alexey Scherbakov's comments, the better > case is just to improve PME to wait for less, at least now. > Seems, we have to wait for (or cancel, I vote for this case - any > objections?) operations related to the failed primaries and wait for > recovery finish (which is fast). > In case affinity was changed and backup-primary switch (not related to the > failed primaries) required between the owners or even rebalance required, > we should just ignore this and perform only "Recovery PME". > Regular PME should happen later (if necessary), it can be even delayed (if > necessary). > > Sounds good? > > On Wed, Sep 18, 2019 at 11:46 AM Alexey Goncharuk < > alexey.goncha...@gmail.com> wrote: > > > Anton, > > > > I looked through the presentation and the ticket but did not find any new > > protocol description that you are going to implement. Can you describe the > > complete protocol changes first (to save time for both you and during the > > later review)? > > > > Some questions that I have in mind: > > * It looks like that for "Local Switch" optimization you assume that node > > failure happens immediately for the whole cluster. This is not true - some > > nodes may "see" a node A failure, while others still consider it alive. > > Moreover, node A may not know yet that it is about to be failed and process > > requests correctly. This may happen, for example, due to a long GC pause on > > node A. In this case, node A resumes it's execution and proceeds to work as > > a primary (it has not received a segmentation event yet), node B also did > > not receive the A FAILED event yet. Node C received the event, ran the > > "Local switch" and assumed a primary role, node D also received the A > > FAILED event and switched to the new primary. Transactions from nodes B and > > D will be processed simultaneously on different primaries. How are you > > going to synchronize this? > > * IGNITE_MAX_COMPLETED_TX_COUNT is fragile and we need to remove this > > property, not add new logic that relies on it. There is no way a user can > > calculate this property or adjust it in runtime. If a user decreases this > > property below a safe value, we will get inconsistent update counters and > > cluster desync. Besides, IGNITE_MAX_COMPLETED_TX_COUNT is quite a large > > value and will push HWM forward very quickly, much faster than during > > regular updates (you will have to increment it for each partition) > > > > ср, 18 сент. 2019 г. в 10:53, Anton Vinogradov : > > > > > Igniters, > > > > > > Recently we had a discussion devoted to the non-blocking PME. > > > We agreed that the most important case is a blocking on node failure and > > it > > > can be splitted to: > > > > > > 1) Affected partition’s operations latency will be increased by node > > > failure detection duration. > > > So, some operations may be freezed for 10+ seconds at real clusters just > > > waiting for a failed primary response. > > > In other words, some operations will be blocked even before blocking PME > > > started. > > > > > > The good news here that "bigger cluster decrease blocked operations > > > percent". > > > > > > Bad news that these operations may block non-affected operations at > > > - customers code (single_thread/striped pool usage) > > > - multikey operations
Re: Text queries/indexes (GridLuceneIndex, @QueryTextFiled)
Yuriy, Greatly appreciate your interest. Could you please elaborate a little bit about sorting? What tasks does it help to solve and how? It would be great to provide an example. ср, 18 сент. 2019 г. в 09:39, Alexei Scherbakov : > > Denis, > > I like the idea of throwing an exception for enabled text queries on > persistent caches. > > Also I'm fine with proposed limit for unsorted searches. > > Yury, please proceed with ticket creation. > > вт, 17 сент. 2019 г., 22:06 Denis Magda : > > > Igniters, > > > > I see nothing wrong with Yury's proposal in regards full-text search API > > evolution as long as Yury is ready to push it forward. > > > > As for the in-memory mode only, it makes total sense for in-memory data > > grid deployments when Ignite caches data of an underlying DB like Postgres. > > As part of the changes, I would simply throw an exception (by default) if > > the one attempts to use text indices with the native persistence enabled. > > If the person is ready to live with that limitation that an explicit > > configuration change is needed to come around the exception. > > > > Thoughts? > > > > > > - > > Denis > > > > > > On Tue, Sep 17, 2019 at 7:44 AM Yuriy Shuliga wrote: > > > > > Hello to all again, > > > > > > Thank you for important comments and notes given below! > > > > > > Let me answer and continue the discussion. > > > > > > (I) Overall needs in Lucene indexing > > > > > > Alexei has referenced to > > > https://issues.apache.org/jira/browse/IGNITE-5371 where > > > absence of index persistence was declared as an obstacle to further > > > development. > > > > > > a) This ticket is already closed as not valid.b) There are definite needs > > > (and in our project as well) in just in-memory indexing of selected data. > > > We intend to use search capabilities for fetching limited amount of > > records > > > that should be used in type-ahead search / suggestions. > > > Not all of the data will be indexed and the are no need in Lucene index > > to > > > be persistence. Hope this is a wide pattern of text-search usage. > > > > > > (II) Necessary fixes in current implementation. > > > > > > a) Implementation of correct *limit *(*offset* seems to be not required > > in > > > text-search tasks for now) > > > I have investigated the data flow for distributed text queries. it was > > > simple test prefix query, like 'name'*='ene*'* > > > For now each server-node returns all response records to the client-node > > > and it may contain ~thousands, ~hundred thousands records. > > > Event if we need only first 10-100. Again, all the results are added to > > > queue in GridCacheQueryFutureAdapter in arbitrary order by pages. > > > I did not find here any means to deliver deterministic result. > > > So implementing limit as part of query and (GridCacheQueryRequest) will > > not > > > change the nature of response but will limit load on nodes and > > networking. > > > > > > Can we consider to open a ticket for this? > > > > > > (III) Further extension of Lucene API exposition to Ignite > > > > > > a) Sorting > > > The solution for this could be: > > > - Make entities comparable > > > - Add custom comparator to entity > > > - Add annotations to mark sorted fields for Lucene indexing > > > - Use comparators when merging responses or reducing to desired limit on > > > client node. > > > Will require full result set to be loaded into memory. Though can be used > > > for relatively small limits. > > > BR, > > > Yuriy Shuliha > > > > > > пт, 30 серп. 2019 о 10:37 Alexei Scherbakov < > > alexey.scherbak...@gmail.com> > > > пише: > > > > > > > Yuriy, > > > > > > > > Note what one of major blockers for text queries is [1] which makes > > > lucene > > > > indexes unusable with persistence and main reason for discontinuation. > > > > Probably it's should be addressed first to make text queries a valid > > > > product feature. > > > > > > > > Distributed sorting and advanved querying is indeed not a trivial task. > > > > Some kind of merging must be implemented on query originating node. > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-5371 > > > > > > > > чт, 29 авг. 2019 г. в 23:38, Denis Magda : > > > > > > > > > Yuriy, > > > > > > > > > > If you are ready to take over the full-text search indexes then > > please > > > go > > > > > ahead. The primary reason why the community wants to discontinue them > > > > first > > > > > (and, probable, resurrect later) are the limitations listed by Andrey > > > and > > > > > minimal support from the community end. > > > > > > > > > > - > > > > > Denis > > > > > > > > > > > > > > > On Thu, Aug 29, 2019 at 1:29 PM Andrey Mashenkov < > > > > > andrey.mashen...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi Yuriy, > > > > > > > > > > > > Unfortunatelly, there is a plan to discontinue TextQueries in > > Ignite > > > > [1]. > > > > > > Motivation here is text indexes are not persistent, not > > transactional > > > > and > > > > > > can't be user together
Re: Non-blocking PME discussion, ASF Slack, September 10, 13.00 (MSK)
Anton, Thank you very much for performing the discussion! I found it pretty much useful. And I really like such kind of collaboration and hope that we will go forward in that direction. Looking forward for meeting minutes and slides. вт, 10 сент. 2019 г. в 10:24, Alexey Goncharuk : > > Anton, > > I will not be able to attend the meeting in person, so I will test this new > communication channel from the other side. Let's see how well decisions and > discussion is translated to the dev list :) > > пн, 9 сент. 2019 г. в 13:52, Anton Vinogradov : > > > Alexey, > > thanks for pointing this. > > > > I've prepared the presentation with explanation how it works now and with a > > set of proposals. > > So, I'm not going to have a chat with somebody tomorrow. > > I'll, as usual, will start a video call and share the screen with the > > presentation, like a regular webinar. > > Sounds good? > > > > BTW, I'll be at GG office, you may join me via the internet or in the same > > room. > > > > On Mon, Sep 9, 2019 at 1:34 PM Alexey Goncharuk < > > alexey.goncha...@gmail.com> > > wrote: > > > > > Hello Anton, > > > > > > I doubt that Slack channel is a good way to discuss/make decisions on > > such > > > an important topic. Changes in PME are usually the hard ones, and slack > > is > > > kind of a synchronous communication channel - with no known context it > > will > > > be hard to give appropriate feedback. Do you have something to make > > > Igniters familiar with beforehand? > > > > > > пн, 9 сент. 2019 г. в 09:01, Anton Vinogradov : > > > > > > > Igniters, > > > > > > > > As far as you may know, we have blocking PME now. > > > > It means that the upcoming operation's latency may be increased or > > > already > > > > started operations can be canceled. > > > > > > > > I've done some homework and found it's possible to have almost > > > non-blocking > > > > PME (fully non-blocked in some cases). > > > > I'd like to discuss found with the community before starting the > > > > implementation. > > > > > > > > Let's have a first community-wide discussion using the ASF Slack. > > > > I've created the channel [1] and going to explain found tomorrow > > > (September > > > > 10) 13.00 (MSK). > > > > Disclaimer: We'll use the Russian language this time. > > > > > > > > [1] https://the-asf.slack.com/messages/CMTJBREKD > > > > > > > > > -- Best regards, Ivan Pavlukhin
Re: New Сommitter: Maxim Muzafarov
Maxim, my congratulations! 2019-08-29 7:25 GMT+11:00, Andrey Kuznetsov : > Great news! Congratulations! > > ср, 28 авг. 2019 г., 18:28 Alex Plehanov : > >> Maxim, congratulations! >> >> ср, 28 авг. 2019 г. в 18:13, Nikita Amelchev : >> >> > My congratulations, Maxim! >> > >> > ср, 28 авг. 2019 г. в 18:11, Dmitriy Pavlov : >> > > >> > > Dear community, >> > > >> > > The Project Management Committee (PMC) for Apache Ignite has invited >> > Maxim >> > > Muzafarov to become a committer and we are pleased to announce that >> > > he >> > has >> > > accepted. >> > > >> > > PMC recognizes Maxim's efforts in developing file transfer for >> > rebalancing, >> > > removal of WAL applying steps from Partition Map Exchange, finding >> > > and >> > > fixing non-trivial issues with the product, like mem leaks, and in >> > setting >> > > up code inspections and maintaining this process. >> > > >> > > Being a committer enables easier contribution to the project since >> there >> > is >> > > no need to go via the patch submission process. This should enable >> better >> > > productivity. >> > > >> > > Best regards, >> > > Dmitriy Pavlov >> > > on behalf of PMC, Apache Ignite >> > >> > >> > >> > -- >> > Best wishes, >> > Amelchev Nikita >> > >> > -- Best regards, Ivan Pavlukhin
Re: Making Ignite Collaboration 100% Open and Transparent
+ Meeting minutes 2019-08-28 22:29 GMT+11:00, Alexey Zinoviev : > I am totally support the idea with the planned and widely announced Hangout > meeting between commiters and contributers and posting the link to the > dev-list with the special Topic Name and short agenda. Maybe, the recorded > video could be added to the YouTube (or to another platform) to share with > the community members. > > пн, 26 авг. 2019 г. в 22:23, Amit Chavan : > >> Hi Denis, >> >> I really like the initiative for transparency and collaboration. Are >> there >> any plans to help get new contributors up to speed with the project where >> they can contribute effectively? Sometimes it can be intimidating to >> start >> on a large project without some help or advice. Maybe assigning a mentor >> or >> an existing committer can be useful or some slack channel people can ping >> on. I am starting new on the project and have picked out newbie ticket >> from >> the Jira board. As I make myself familiar with the code base maybe its >> good >> to have some direction on which area should I focus more on etc. >> >> Thanks, >> Amit >> >> On Mon, Aug 26, 2019 at 11:54 AM Denis Magda wrote: >> >> > Folks, let me share more details on why Anton started the conversation >> > about Ignite Slack: >> > >> > >> http://apache-ignite-developers.2346864.n4.nabble.com/The-ASF-Slack-td43233.html >> > >> > Recently, a group of GridGain and Sberbank committers of Ignite has met >> to >> > discuss how to make our community more transparent and open. Anton and >> > I >> > took part in that meeting. The primary problems we see in regards to >> > transparency and openness are as follows: >> > >> >- A lot of discussions on the dev list abrupts suddenly and it's >> unclear >> >whether a discussion is abandoned or something else is going on in >> > the >> >background with a task/bug/improvement. In many cases, we tend to >> > fall >> > back >> >to faster communication ways like instant messaging, calls, or >> > face-to-face >> >meetings that are not visible to the rest of the community. Emails >> (dev >> >list) are the right communication channel but not for all of the >> stages. >> >- Change reviews seem to be in a chaotic state. Sometimes it takes >> many >> >rounds for a committer to urge another committer to do a review. In >> many >> >cases, the other committer might be simply overwhelmed with regular >> > tasks >> >imposed by an employer. It will be good to come up with some public >> >tracking approach that will help us all to see who and when will be >> > able to >> >review certain changes and make them to Ignite. >> > >> > To address the problems we want to propose the following: >> > >> >- Keep using dev list the way you do today. No changes need to be >> > done >> >here. >> >- Introduce Ignite Slack for instant messaging across all the >> community >> >members who are obviously employed by different companies. Ignite >> > PMC >> > will >> >be managing channels for various topics. Go to Slack when email (dev >> > list) >> >conversation is no longer effective, the way we do daily, no need to >> >complicate our lives just because we work on an open-source project >> >together. >> >- Two or more committers need to talk verbally? Go ahead and >> > schedule >> a >> >meeting with Google Hangouts or another tool. Send an invite to the >> dev >> >list for those who'd like to join and listen or share opinion. Want >> > to >> > talk >> >in your native language? Go ahead and put a disclaimer that a >> > conversation >> >will be in Chinese, Russia, French, whatever. Simple and open. >> >- Don't know how soon you'll be able to review some changes and, >> > thus, >> >ignoring other committers requests? GridGain and Sberbank are ready >> > to >> >propose a solution here. Both vendors use an approach to cooperate >> > between >> >GridGain and Sberbank committers. Now we'd like to make it fully >> > open >> > and >> >adjust for community needs if required. >> > >> > Thoughts, suggestions? I think we'll schedule a community meeting to >> finish >> > the conversation or discuss any cornerstone points. But start with your >> > questions first. >> > >> > - >> > Denis >> > >> > -- Best regards, Ivan Pavlukhin
Re: Replacing default work dir from tmp to current dir
Ilya,
2 points:
1. It is a good point that a directory name "work" in arbitrary place
can cause a lot of confusion.
2. As far as I got, default directory is not in e.g. /home/username
but in one pointed by "user.dir" system property which is a directory
where a java process started (if property was not overridden).
2019-08-26 1:59 GMT+11:00, Ilya Kasnacheev :
> Hello!
>
> I am really worried by the fact that previously we had /tmp/ignite and in
> it work/, whereas now we're going to write to /home/username/work
>
> I doubt that users can attribute work/ directory in their home to Ignite,
> especially when it is used as library by something else.
>
> Is there a chance we could move this work dir to /home/username/ignite with
> work/ (and possibly logs/) dir in it? WDYT?
>
> We could even auto-create README.TXT in this /home/username/ignite/ to
> describe that it's Apache Ignite work directory and how to change it.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 12 авг. 2019 г. в 19:02, Denis Magda :
>
>> +1 for the user.dir as a default one.
>>
>> Denis
>>
>> On Monday, August 12, 2019, Dmitriy Pavlov wrote:
>>
>> > +1 to user home directory. A number of open source products create
>> > their
>> > dirs there. For me, it is a kind of expected behavior.
>> >
>> > Ivan mentioned an important point: binary meta & marshaller. We should
>> > update documentation and stop require PDS dir setup, but require home
>> setup
>> > (for older versions of Ignite, it is relevant anyway).
>> >
>> > пн, 12 авг. 2019 г. в 18:49, Pavel Tupitsyn :
>> >
>> > > Hi Ivan,
>> > >
>> > > > fail Ignite node in case neither IGNITE_HOME
>> > > nor IgniteConfiguration#igniteWorkDir is set
>> > > I strongly disagree, this is bad usability.
>> > > Ignition.start() should work without any extra configuration as is it
>> > right
>> > > now.
>> > >
>> > > Let's come up with reasonable defaults instead, user dir sounds good
>> > > to
>> > me.
>> > >
>> > > On Mon, Aug 12, 2019 at 6:45 PM Stephen Darlington <
>> > > stephen.darling...@gridgain.com> wrote:
>> > >
>> > > > Yes, when data is a stake, fail early is the absolutely the right
>> thing
>> > > to
>> > > > do.
>> > > >
>> > > > Regards,
>> > > > Stephen
>> > > >
>> > > > > On 12 Aug 2019, at 16:37, Ivan Rakov
>> wrote:
>> > > > >
>> > > > > Hi Anton,
>> > > > >
>> > > > > Actually, the issue is even more unpleasant.
>> > > > >
>> > > > > Official Ignite documentation says that it's possible to
>> > > > > configure
>> > path
>> > > > where your persistence files will be stored:
>> > > > https://apacheignite.readme.io/docs/distributed-persistent-store
>> > > > > However, even if you have set all path options (storage, WAL, WAL
>> > > > archive), Ignite will still store crucial metadata in resolved work
>> > > > directory (java.io.tmpdir by default). Example is binary metadata
>> > files,
>> > > > absence of which can make your data unavailable.
>> > > > >
>> > > > > I propose to fail Ignite node in case neither IGNITE_HOME nor
>> > > > IgniteConfiguration#igniteWorkDir is set. It's better to let user
>> know
>> > > > about missing configuration options during startup than let OS
>> corrupt
>> > > > storage by cleaning temp dirs.
>> > > > >
>> > > > > Thoughts?
>> > > > >
>> > > > > Best Regards,
>> > > > > Ivan Rakov
>> > > > >
>> > > > > On 12.08.2019 18:10, Anton Kalashnikov wrote:
>> > > > >> Hello, Igniters.
>> > > > >>
>> > > > >> Currently, in the case, when work directory wasn't set by user
>> > ignite
>> > > > can resolve it to tmp directory which leads to some problem - tmp
>> > > directory
>> > > > can be cleared at some unexpected moment by operation system and
>> > > different
>> > > > types of critical data would be lost(ex. binary_meta, persistance
>> > data).
>> > > > >>
>> > > > >> Looks like it is not expected behaviour and maybe it is better
>> > instead
>> > > > of tmp directory use the current working directory("user.dir")? Or
>> any
>> > > > other idea?
>> > > > >>
>> > > > >> A little more details you can find in the ticket -
>> > > > https://issues.apache.org/jira/browse/IGNITE-12057
>> > > > >> --
>> > > > >> Best regards,
>> > > > >> Anton Kalashnikov
>> > > > >>
>> > > >
>> > > >
>> > > >
>> > >
>> >
>>
>>
>> --
>> -
>> Denis
>>
>
--
Best regards,
Ivan Pavlukhin
Re: Metastore disappears in Docker on restarts
Denis, Actually binary metadata is not stored in a metastore (however there were a discussion about moving it). User scenario is valid. A meaningful exception and documentation are good. And also I can think about revisiting a needed configuration for persistence in containerized environments. It might be perfectly sane that user does not want to configure IGNITE_HOME and work directory pointing to a persistent volume. I think that there should a single option to point all persistent stuff an appropriate directory (other options to redirect WAL could be also provided). With a such option a new user will have less chances to misconfigure. 2019-08-19 22:18 GMT+11:00, Denis Magda : > Dmitry G. and Igniters, > > Seems there is another issue with the metastore that is related to a Docker > environment: > https://stackoverflow.com/questions/57529702/ignite-persisting-a-set-cannot-find-metadata-for-object-with-compact-footer/57537838#57537838 > > In short, this exception is generated on a restart (more details are on > SO): > > Cannot find metadata for object with compact footer: 2097659979 > > > I see two ways to troubleshoot this configuration and usability issue: > >1. To provide a self-explanatory exception so that the users can fix the >configuration error easily, like "... This exception can happen if >IGNITE_WORK dir points out to a folder that can be or remounted on > restarts" >2. Create a special documentation section explaining how to set >IGNITE_WORK dir for virtualized environments. > > Does this sound right? Any better, probably automatic solution? > - > Denis > -- Best regards, Ivan Pavlukhin
Re: SQL query timeout: in progress or abandoned
Hi Saikat, I left a comment in JIRA ticket [1]. Also, I invited Andrey to help with a further review. Andrey, could you please step in and continue the review? Unfortunately, for a couple of weeks I have limited access to my computer and cannot do a review in a timely manner. [1] https://issues.apache.org/jira/browse/IGNITE-7285 2019-08-19 7:24 GMT+11:00, Saikat Maitra : > Hi Ivan, > > I have updated the PR and made changes in IgniteH2Indexing for query > timeout so that default query timeout get used during query execution. > > Please take a look and let me know if this change looks good. > > I will update tests if the approach looks good. > > PR https://github.com/apache/ignite/pull/6490 > > Regards, > > Saikat > > On Sat, Aug 17, 2019 at 8:30 PM Saikat Maitra > wrote: > >> Hi Ivan, Denis >> >> Thank you for your feedback, I am looking into the changes needed for >> this >> issue. >> >> I am also looking into these configurations parameters >> https://apacheignite.readme.io/v2.2/docs/configuration-parameters to see >> if there are similar attributes being used in SqlFieldsQuery and >> SqlQuery. >> >> >> Regards, >> >> Saikat >> >> >> >> On Thu, Aug 15, 2019 at 6:13 AM Павлухин Иван >> wrote: >> >>> Saikat, Denis, >>> >>> I left comments in the ticket [1]. >>> >>> [1] https://issues.apache.org/jira/browse/IGNITE-7285 >>> >>> вт, 13 авг. 2019 г. в 21:53, Denis Magda : >>> > >>> > Hi Saikat, >>> > >>> > Thanks for a quick turnaround! Ivan, could you please step in and do a >>> > review? >>> > >>> > - >>> > Denis >>> > >>> > >>> > On Sun, Aug 11, 2019 at 6:26 AM Saikat Maitra >>> > >>> > wrote: >>> > >>> > > Hi Denis, Ivan >>> > > >>> > > As discussed I have updated the PR and incorporated review comments. >>> > > >>> > > https://github.com/apache/ignite/pull/6490/files >>> > > >>> > > Please take a look and share your feedback. >>> > > >>> > > Regard, >>> > > Saikat >>> > > >>> > > >>> > > >>> > > On Sat, Aug 10, 2019 at 5:51 PM Saikat Maitra < >>> saikat.mai...@gmail.com> >>> > > wrote: >>> > > >>> > > > Hello Denis, Ivan >>> > > > >>> > > > Yes, I can take up the changes for IGNITE-7825. >>> > > > >>> > > > I had a doubt on the usage of the Default Query Timeout. >>> > > > >>> > > > I had raised the PR in an assumption that Default Query Timeout >>> will only >>> > > > be used if user had not provided Cache Query Timeout >>> > > > >>> > > > https://github.com/apache/ignite/pull/6490/files >>> > > > >>> > > > I wanted to discuss if it is correct intended usage of Default >>> > > > Query >>> > > > Timeout or should we reconsider? >>> > > > >>> > > > Regards, >>> > > > Saikat >>> > > > >>> > > > >>> > > > >>> > > > On Fri, Aug 9, 2019 at 12:11 PM Denis Magda >>> wrote: >>> > > > >>> > > >> Ivan, thanks for sharing this discussion. Let's use it for our >>> > > >> conversation. >>> > > >> >>> > > >> - >>> > > >> Denis >>> > > >> >>> > > >> >>> > > >> On Thu, Aug 8, 2019 at 11:15 PM Павлухин Иван >>> > > >> >> > >>> > > >> wrote: >>> > > >> >>> > > >> > Just for the protocol. There was an original dev-list >>> > > >> > discussion >>> [1]. >>> > > >> > Added a link to the ticket as well. >>> > > >> > >>> > > >> > [1] >>> > > >> > >>> > > >> >>> > > >>> http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-7285-Add-default-query-timeout-td41828.html >>> > > >> > >>> > > >> > пт, 9 авг. 2019 г. в 01:22, Denis Magda : >>> > > >> > > >>> > > >> > > Hey Saikat, >>> > > >> > > >>> > > >> > > Are you still working on this ticket? >>> > > >> > > https://issues.apache.org/jira/browse/IGNITE-7285 >>> > > >> > > >>> > > >> > > Seems that's the last API that doesn't support timeouts - >>> > > >> > > JDBC >>> and >>> > > >> ODBC >>> > > >> > > drivers already go with it. >>> > > >> > > >>> > > >> > > If you don't have time to complete the changes then someone >>> else >>> > > from >>> > > >> the >>> > > >> > > community can take over. We see a lot of demand for this API >>> and >>> > > here >>> > > >> is >>> > > >> > > one example: >>> > > >> > > >>> > > >> > >>> > > >> >>> > > >>> https://stackoverflow.com/questions/57275301/how-to-set-a-query-timeout-for-apache-ignite-cache >>> > > >> > > >>> > > >> > > - >>> > > >> > > Denis >>> > > >> > >>> > > >> > >>> > > >> > >>> > > >> > -- >>> > > >> > Best regards, >>> > > >> > Ivan Pavlukhin >>> > > >> > >>> > > >> >>> > > > >>> > > >>> >>> >>> >>> -- >>> Best regards, >>> Ivan Pavlukhin >>> >> > -- Best regards, Ivan Pavlukhin
Re: Re[4]: Apache Ignite 2.7.6 (Time, Scope, and Release manager)
Dmitriy, I cherry-picked fix for https://issues.apache.org/jira/browse/IGNITE-12068 to 2.7.6. Bot visa seems good comparing to 2.7.6 but there are some reported blockers. It would be great if someone can double-check it. If something is wrong you may consider a revert. пт, 16 авг. 2019 г. в 18:22, Dmitriy Pavlov : > > Hi Igniters, > > I also suggest adding newly created critical bug into scope > https://issues.apache.org/jira/browse/IGNITE-12081 (already assigned since > it was discussed before in private communication). > > It is not a regression but can cause data corruption, so I'd rather wait > for the fix. > > Eduard Shangareev, Stanilovsky Evgeny I'm going to assemble the first RC > during the weekend, so I would be grateful if fixes can be cherry-piked to > the branch today. Also, since narrow time limit, please trigger Run All > after each commit in 2.7.6. This will allow to immediately locate which > commit caused test failure. > > Sincerely, > Dmitriy Pavlov > > пт, 16 авг. 2019 г. в 13:33, Dmitriy Pavlov : > > > Hi Ivan, > > > > yes, sure. > > > > If it is cherry-picked without conflicts, just push it. > > > > If it contains conflicts it is better to push to 2.7.6-based branch and > > start additional run-all, example > > https://github.com/apache/ignite/pull/6781 -it is PR for ignite-9562, > > base: 2.7.6 > > > > Sincerely, > > Dmitriy Pavlov > > > > пт, 16 авг. 2019 г. в 12:28, Павлухин Иван : > > > >> Dmitriy, > >> > >> We accomplished with https://issues.apache.org/jira/browse/IGNITE-12068 > >> > >> Should I cherry-pick it to release branch? > >> > >> пт, 16 авг. 2019 г. в 11:28, Dmitriy Pavlov : > >> > > >> > Hi Igniters, > >> > > >> > Yesterday was a planned date of code freeze. > >> > > >> > So the scope (related both to features and to bugs) is final, we're > >> > entering to stabilization phase. Nothing can be included into scope > >> besides > >> > blockers approved by the release manager. > >> > https://cwiki.apache.org/confluence/display/IGNITE/Release+Process > >> > <https://cwiki.apache.org/confluence/display/IGNITE/Release+Process> > >> > > >> > Unresolved tickets are: > >> > IGNITE-12071 Eduard Shangareev > >> > IGNITE-12068 Ivan Pavlukhin > >> > IGNITE-9562 Eduard Shangareev > >> > IGNITE-12057 Anton Kalashnikov > >> > IGNITE-12061 Stanilovsky Evgeny > >> > > >> > Sincerely, > >> > Dmitriy Pavlov > >> > > >> > чт, 15 авг. 2019 г. в 20:12, Zhenya Stanilovsky : > >> > > >> > > Dmitriy, review passed, plz proceed. > >> > > thanks ! > >> > > > >> > > > I removed https://issues.apache.org/jira/browse/IGNITE-12032 until > >> > > > nobody > >> > > > volunteer to complete the task. > >> > > > > >> > > > Evgeniy, please keep me updated about IGNITE-12061. > >> > > > > >> > > > чт, 15 авг. 2019 г. в 16:56, Zhenya Stanilovsky > >> > > > >> > > >> : > >> > > > > >> > > >> yep, i`l try to call someone who probably familiar with this > >> problem. > >> > > >> > >> > > >> > > >> > > >> > > >> > > >> >Ok, I've removed Spark from the scope. > >> > > >> > > >> > > >> >What about the unassigned issue related to ODBC? It is not > >> looking like > >> > > >> >somebody will fix it before release. > >> > > >> > > >> > > >> >Evgeniy Stanilovsky, what about > >> > > >> >https://issues.apache.org/jira/browse/IGNITE-12061 > >> > > >> >< > >> https://issues.apache.org/jira/browse/IGNITE-12061?src=confmacro > > >> > > >> >do you have someone to review the fix? > >> > > >> > > >> > > >> >чт, 15 авг. 2019 г. в 16:05, Denis Magda < dma...@apache.org >: > >> > > >> > > >> > > >> >> Dmitry, > >> > > >> >> > >> > > >> >> It would be better for us to put off the Spark upgrade until > >> 2.8 as > >> > > >> Nikolay > >> > > >> >> suggested
Re: Re[4]: Apache Ignite 2.7.6 (Time, Scope, and Release manager)
gt; > > Hi, > > >> >> > > > > > > > > >> >> > > > > > > Thanks to everyone, who participated in the discussion. > > >> >> > > > > > > > > >> >> > > > > > > I would like to wait also fix for > > >> >> > > > > > > https://issues.apache.org/jira/browse/IGNITE-12057 > > >> >> > > > > > > > > >> >> > > > > > > and discussion results for that issue: > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > >> >> > > > >> >> > > >> > > https://lists.apache.org/thread.html/498a14b3971950a45ef57f45cc23d2438ce1afba000b586e230927bf@%3Cdev.ignite.apache.org%3E > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > Since some issues are still open, we'll probably move > > >> some > > >> >> dates > > >> >> > > > > > > >> >> > > > > forward, > > >> >> > > > > > > but it seems we managed to discuss scope before scope > > >> freeze. > > >> >> > > > > > > > > >> >> > > > > > > I consider now the scope is frozen - no new feature can > > >> be > > >> >> added > > >> >> > > to the > > >> >> > > > > > > scope of 2.7.6. We're entering the rampdown phase. > > >> >> > > > > > > > > >> >> > > > > > > Sincerely, > > >> >> > > > > > > Dmitriy Pavlov > > >> >> > > > > > > > > >> >> > > > > > > See > > >> >> > > > > >> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process > > >> >> > > > > > > for > > >> >> > > > > > > more details. > > >> >> > > > > > > > > >> >> > > > > > > пн, 12 авг. 2019 г. в 16:37, Zhenya Stanilovsky > > >> >> > > > > > > < arzamas...@mail.ru.invalid > > >> >> > > > > > > > : > > >> >> > > > > > > > Hi al,, i also suggest to append [1], cause it could > > >> produce > > >> >> > > > > > > > CorruptedTreeException in some scenario. > > >> >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > [1] > > https://issues.apache.org/jira/browse/IGNITE-12061 > > >> >> > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > Hi all, > > >> >> > > > > > > > > Can we include > > >> >> > > https://issues.apache.org/jira/browse/IGNITE-12060 ? > > >> >> > > > > > > > > > >> >> > > > > > > > This is > > >> >> > > > > > > > > a > > https://issues.apache.org/jira/browse/IGNITE-11953 > > >> >> > > > > > > > > > > >> >> > > > > > > > > On Fri, Aug 9, 2019 at 10:04 AM Nikolay Izhikov < > > >> >> > > > > > > >> >> > > > > nizhi...@apache.org > > >> >> > > > > > > > > > >> >> > > > > > > > wrote: > > >> >> > > > > > > > > > > >> >> > > > > > > > > > Hello, Deni. > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > Nickolay, could you check if that's a quick > > >> upgrade? > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > Yes,
Re: Re[2]: Apache Ignite 2.7.6 (Time, Scope, and Release manager)
> > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/498a14b3971950a45ef57f45cc23d2438ce1afba000b586e230927bf@%3Cdev.ignite.apache.org%3E > > > > > > > > > > > > > > > > > > > > > Since some issues are still open, we'll probably move some dates > > > > > > > > > > forward, > > > > > > > but it seems we managed to discuss scope before scope freeze. > > > > > > > > > > > > > > I consider now the scope is frozen - no new feature can be added > > > to the > > > > > > > scope of 2.7.6. We're entering the rampdown phase. > > > > > > > > > > > > > > Sincerely, > > > > > > > Dmitriy Pavlov > > > > > > > > > > > > > > See > > > https://cwiki.apache.org/confluence/display/IGNITE/Release+Process > > > > > > > for > > > > > > > more details. > > > > > > > > > > > > > > пн, 12 авг. 2019 г. в 16:37, Zhenya Stanilovsky > > > > > > > > > > > > > > : > > > > > > > > Hi al,, i also suggest to append [1], cause it could produce > > > > > > > > CorruptedTreeException in some scenario. > > > > > > > > > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12061 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > Can we include > > > https://issues.apache.org/jira/browse/IGNITE-12060 ? > > > > > > > > > > > > > > > > This is > > > > > > > > > a https://issues.apache.org/jira/browse/IGNITE-11953 > > > > > > > > > > > > > > > > > > On Fri, Aug 9, 2019 at 10:04 AM Nikolay Izhikov < > > > > > > > > > > nizhi...@apache.org > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hello, Deni. > > > > > > > > > > > > > > > > > > > > > Nickolay, could you check if that's a quick upgrade? > > > > > > > > > > > > > > > > > > > > Yes, I will take a look. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > В Чт, 08/08/2019 в 11:08 -0700, Denis Magda пишет: > > > > > > > > > > > Dmitry, > > > > > > > > > > > > > > > > > > > > > > Please add this BTree corruption fix to the scope: > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-11953 > > > > > > > > > > > > > > > > > > > > > > Plus, I would upgrade our Spark integration to version > > 2.4 > > > as > > > > > > > > > > long > > > > > > > as > > > > > > > > 2.3 > > > > > > > > > > > goes with limitations reported by our users: > > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-12054 > > > > > > > > > > > > > > > > > > > > > > Nickolay, could you check if that's a quick upgrade? > > > > > > > > > > > > > > > > > > > > > > - > > > > > > > > > > > Denis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 8, 2019 at 10:40 AM Dmitriy Pavlov < > > > > > > > > > > > > > > dpav...@apache.org > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Hi Ivan, Ilya, Igniters, > > > > > > > > > > > > > > > > > > > > > > > > &g
Re: SQL query timeout: in progress or abandoned
Saikat, Denis, I left comments in the ticket [1]. [1] https://issues.apache.org/jira/browse/IGNITE-7285 вт, 13 авг. 2019 г. в 21:53, Denis Magda : > > Hi Saikat, > > Thanks for a quick turnaround! Ivan, could you please step in and do a > review? > > - > Denis > > > On Sun, Aug 11, 2019 at 6:26 AM Saikat Maitra > wrote: > > > Hi Denis, Ivan > > > > As discussed I have updated the PR and incorporated review comments. > > > > https://github.com/apache/ignite/pull/6490/files > > > > Please take a look and share your feedback. > > > > Regard, > > Saikat > > > > > > > > On Sat, Aug 10, 2019 at 5:51 PM Saikat Maitra > > wrote: > > > > > Hello Denis, Ivan > > > > > > Yes, I can take up the changes for IGNITE-7825. > > > > > > I had a doubt on the usage of the Default Query Timeout. > > > > > > I had raised the PR in an assumption that Default Query Timeout will only > > > be used if user had not provided Cache Query Timeout > > > > > > https://github.com/apache/ignite/pull/6490/files > > > > > > I wanted to discuss if it is correct intended usage of Default Query > > > Timeout or should we reconsider? > > > > > > Regards, > > > Saikat > > > > > > > > > > > > On Fri, Aug 9, 2019 at 12:11 PM Denis Magda wrote: > > > > > >> Ivan, thanks for sharing this discussion. Let's use it for our > > >> conversation. > > >> > > >> - > > >> Denis > > >> > > >> > > >> On Thu, Aug 8, 2019 at 11:15 PM Павлухин Иван > > >> wrote: > > >> > > >> > Just for the protocol. There was an original dev-list discussion [1]. > > >> > Added a link to the ticket as well. > > >> > > > >> > [1] > > >> > > > >> > > http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-7285-Add-default-query-timeout-td41828.html > > >> > > > >> > пт, 9 авг. 2019 г. в 01:22, Denis Magda : > > >> > > > > >> > > Hey Saikat, > > >> > > > > >> > > Are you still working on this ticket? > > >> > > https://issues.apache.org/jira/browse/IGNITE-7285 > > >> > > > > >> > > Seems that's the last API that doesn't support timeouts - JDBC and > > >> ODBC > > >> > > drivers already go with it. > > >> > > > > >> > > If you don't have time to complete the changes then someone else > > from > > >> the > > >> > > community can take over. We see a lot of demand for this API and > > here > > >> is > > >> > > one example: > > >> > > > > >> > > > >> > > https://stackoverflow.com/questions/57275301/how-to-set-a-query-timeout-for-apache-ignite-cache > > >> > > > > >> > > - > > >> > > Denis > > >> > > > >> > > > >> > > > >> > -- > > >> > Best regards, > > >> > Ivan Pavlukhin > > >> > > > >> > > > > > -- Best regards, Ivan Pavlukhin
Re: IGNITE-7285 Add default query timeout
Just to keep history connected. The discussion continued in http://apache-ignite-developers.2346864.n4.nabble.com/SQL-query-timeout-in-progress-or-abandoned-td42964.html вт, 18 июн. 2019 г. в 12:22, Павлухин Иван : > > Hi Saikat, > > Thank you for driving it. I left my comments [1]. > > [1] https://issues.apache.org/jira/browse/IGNITE-7285 > > сб, 15 июн. 2019 г. в 20:48, Saikat Maitra : > > > > Hi Ivan, > > > > Thank you for your email. I have updated the PR to use default query > > timeout. > > > > Please take a look. > > > > https://github.com/apache/ignite/pull/6490/files > > > > Regards > > Saikat > > > > On Tue, May 7, 2019 at 12:30 AM Ivan Pavlukhina wrote: > > > > > Hi Saikat, > > > > > > It is good that we agreed how property in question should be configured. > > > But I worry about the following. If the PR merged it will not contain a > > > user value yet because an introduced property is not used. Consequently we > > > must start using that property before a next AI release. Just one thing to > > > keep in mind. > > > > > > Sent from my iPhone > > > > > > > On 4 May 2019, at 05:56, Saikat Maitra wrote: > > > > > > > > Hi Ivan, > > > > > > > > Thank you for reviewing the PR. I have updated the PR. Please review and > > > > share your feedback. > > > > > > > > I was thinking of making a separate PR for using the defaultQueryTimeout > > > > property in query execution flow. > > > > > > > > Regards, > > > > Saikat > > > > > > > >> On Wed, May 1, 2019 at 1:04 AM Павлухин Иван > > > wrote: > > > >> > > > >> Andrey K., > > > >> > > > >>> I think we should develop some kind of "Queries" options on Ignite > > > >>> configuration. > > > >> > > > >> Quite a reasonable idea. We already have couple of query-related > > > >> properties in IgniteConfiguration and we can move them (in a > > > >> compatible way) to a query properties sub-aggregate. I think it is > > > >> better to raise a separate topic for that. > > > >> > > > >> ср, 1 мая 2019 г. в 09:00, Павлухин Иван : > > > >>> > > > >>> Hi Saikat, > > > >>> > > > >>> I left a couple of comment on GitHub [1]. > > > >>> > > > >>> Perhaps I am missing it but what are the plans for making a default > > > >>> query timeout workable by using an introduced property in query > > > >>> execution flow? > > > >>> > > > >>> [1] https://github.com/apache/ignite/pull/6490 > > > >>> > > > >>> вт, 30 апр. 2019 г. в 02:50, Saikat Maitra : > > > >>>> > > > >>>> Hi Ivan, > > > >>>> > > > >>>> Yes, I checked this CacheQuery default value > > > >>>> > > > >> > > > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/CacheQuery.java#L200 > > > >>>> > > > >>>> Also, Andrew recommended the same in review feedback. > > > >>>> > > > >>>> https://github.com/apache/ignite/pull/6490#discussion_r277730394 > > > >>>> > > > >>>> Regards, > > > >>>> Saikat > > > >>>> > > > >>>> > > > >>>> On Mon, Apr 29, 2019 at 3:18 AM Павлухин Иван > > > >> wrote: > > > >>>> > > > >>>>> Hi Saikat, > > > >>>>> > > > >>>>> It a compatibility with previous versions the reason for an > > > >> indefinite > > > >>>>> timeout by default? > > > >>>>> > > > >>>>> сб, 27 апр. 2019 г. в 16:58, Saikat Maitra > > >>> : > > > >>>>>> > > > >>>>>> Hi Alexey, Ivan, Andrew > > > >>>>>> > > > >>>>>> I think we can provide an option to configure defaultQueryOption at > > > >>>>>> IgniteConfiguration and set the default value = 0 to imply if not > > > >> set it >
Re: Thin client: transactions support
Hi Alex, Could you please elaborate about thin client protocol versioning. As I see 1.5.0 is supposed to be a version supporting transactions. And we already have a version 1.4.0 with affinity awareness support. I forgot, does Java thin client support affinity awareness? Will it work properly if it does not? ср, 14 авг. 2019 г. в 13:59, Alex Plehanov : > > Hi Igniters, > > Finally, all dependent tickets are resolved and I've completed the > implementation of thin client transactions support. The patch [1] includes > server-side implementation and java thin client-side implementation. > Changes to thin client protocol and top-level view of implementation also > described in IEP [2]. > Can anyone review the patch? > > [1]: https://issues.apache.org/jira/browse/IGNITE-9410 > [2]: > https://cwiki.apache.org/confluence/display/IGNITE/IEP-34+Thin+client%3A+transactions+support > > пн, 27 мая 2019 г. в 13:27, Alex Plehanov : > > > Ivan, > > > > Yes, .NET client has such capability. Pavel Tupitsyn already mentions it > > in this thread. As far as I understand, in .NET client implementation to > > dispatch responses dedicated thread is used. > > In a draft implementation of IGNITE-11685 I've used another approach: each > > request thread can read a response (if lock is acquired by this thread > > successfully) and complete a future of its own request or another threads > > request. > > > > пн, 27 мая 2019 г. в 13:01, Павлухин Иван : > > > >> Alex, > >> > >> I am quite curious about async implementations from other clients. Is > >> there any design document describing such implementations? Does .NET > >> client have such capability? > >> > >> Actually, I forgot to finish my previous message. One of my concerns > >> is that a concurrent response dispatch does not sound as a trivial > >> thing. So, I would like to understand if we already have a good > >> approach for that. If not then I suppose it worth a discussion. > >> > >> пн, 27 мая 2019 г. в 12:51, Alex Plehanov : > >> > > >> > Hi Ivan. > >> > > >> > Thin client transactions support is not only for java thin client. There > >> > are other clients, some of them already work in async mode. > >> > Ticket IGNITE-11685 already has draft implementation too, but now it's > >> > based on some changes to java thin client which were made by > >> "transaction > >> > support" implementation. I think this ticket will be ready in a couple > >> of > >> > days after "transaction support" will be merged. And both patches will > >> be > >> > included in the same release. > >> > > >> > пн, 27 мая 2019 г. в 11:57, Павлухин Иван : > >> > > >> > > Hi Alex, > >> > > > >> > > Regarding a problem with possible deadlock when two concurrent > >> > > transactions from the same client are trying to lock the same key and > >> > > an issue [1]. It seems to me that without fixing the issue [1] a > >> > > client transactions feature is not practical. Everyone who uses a > >> > > client from multiple threads can face a deadlock which is impossible > >> > > to deal with. Or am I missing something here? > >> > > > >> > > One workaround I can imagine is failing a transactions execution from > >> > > concurrent threads for a first time. > >> > > > >> > > [1] https://issues.apache.org/jira/browse/IGNITE-11685 > >> > > > >> > > вт, 21 мая 2019 г. в 19:05, Alex Plehanov : > >> > > > > >> > > > Guys, > >> > > > > >> > > > I've updated the IEP [1]. Please have a look. > >> > > > > >> > > > [1] > >> > > > > >> > > > >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-34+Thin+client%3A+transactions+support > >> > > > > >> > > > > >> > > > вт, 21 мая 2019 г., 14:19 Alex Plehanov : > >> > > > > >> > > > > Ivan, > >> > > > > > >> > > > > Yes, I have plans to do that (at least for java thin client). > >> Something > >> > > > > like new class "ClientTransactionConfiguration" inside > >> > > > > "ClientConfiguration". > >> > > > > > >> > > > > вт, 21 мая 20
Re: Re[2]: Asynchronous registration of binary metadata
Denis, Several clarifying questions: 1. Do you have an idea why metadata registration takes so long? So poor disks? So many data to write? A contention with disk writes by other subsystems? 2. Do we need a persistent metadata for in-memory caches? Or is it so accidentally? Generally, I think that it is possible to move metadata saving operations out of discovery thread without loosing required consistency/integrity. As Alex mentioned using metastore looks like a better solution. Do we really need a fast fix here? (Are we talking about fast fix?) ср, 14 авг. 2019 г. в 11:45, Zhenya Stanilovsky : > > Alexey, but in this case customer need to be informed, that whole (for > example 1 node) cluster crash (power off) could lead to partial data > unavailability. > And may be further index corruption. > 1. Why your meta takes a substantial size? may be context leaking ? > 2. Could meta be compressed ? > > > >Среда, 14 августа 2019, 11:22 +03:00 от Alexei Scherbakov > >: > > > >Denis Mekhanikov, > > > >Currently metadata are fsync'ed on write. This might be the case of > >slow-downs in case of metadata burst writes. > >I think removing fsync could help to mitigate performance issues with > >current implementation until proper solution will be implemented: moving > >metadata to metastore. > > > > > >вт, 13 авг. 2019 г. в 17:09, Denis Mekhanikov < dmekhani...@gmail.com >: > > > >> I would also like to mention, that marshaller mappings are written to disk > >> even if persistence is disabled. > >> So, this issue affects purely in-memory clusters as well. > >> > >> Denis > >> > >> > On 13 Aug 2019, at 17:06, Denis Mekhanikov < dmekhani...@gmail.com > > >> wrote: > >> > > >> > Hi! > >> > > >> > When persistence is enabled, binary metadata is written to disk upon > >> registration. Currently it happens in the discovery thread, which makes > >> processing of related messages very slow. > >> > There are cases, when a lot of nodes and slow disks can make every > >> binary type be registered for several minutes. Plus it blocks processing of > >> other messages. > >> > > >> > I propose starting a separate thread that will be responsible for > >> writing binary metadata to disk. So, binary type registration will be > >> considered finished before information about it will is written to disks on > >> all nodes. > >> > > >> > The main concern here is data consistency in cases when a node > >> acknowledges type registration and then fails before writing the metadata > >> to disk. > >> > I see two parts of this issue: > >> > Nodes will have different metadata after restarting. > >> > If we write some data into a persisted cache and shut down nodes faster > >> than a new binary type is written to disk, then after a restart we won’t > >> have a binary type to work with. > >> > > >> > The first case is similar to a situation, when one node fails, and after > >> that a new type is registered in the cluster. This issue is resolved by the > >> discovery data exchange. All nodes receive information about all binary > >> types in the initial discovery messages sent by other nodes. So, once you > >> restart a node, it will receive information, that it failed to finish > >> writing to disk, from other nodes. > >> > If all nodes shut down before finishing writing the metadata to disk, > >> then after a restart the type will be considered unregistered, so another > >> registration will be required. > >> > > >> > The second case is a bit more complicated. But it can be resolved by > >> making the discovery threads on every node create a future, that will be > >> completed when writing to disk is finished. So, every node will have such > >> future, that will reflect the current state of persisting the metadata to > >> disk. > >> > After that, if some operation needs this binary type, it will need to > >> wait on that future until flushing to disk is finished. > >> > This way discovery threads won’t be blocked, but other threads, that > >> actually need this type, will be. > >> > > >> > Please let me know what you think about that. > >> > > >> > Denis > >> > >> > > > >-- > > > >Best regards, > >Alexei Scherbakov > > > -- > Zhenya Stanilovsky -- Best regards, Ivan Pavlukhin
Re: SecurityTestSuite as a separate test suite at TC
Denis, Perhaps with javassist we can make classes dynamically without use of surefire-fork-count parameters. We already use javassist in ConfigVariationsTestSuiteBuilder#makeTestClass, but for a different purpose. P.S. I did not check it yet. пт, 9 авг. 2019 г. в 14:41, Denis Garus : > > Excuse me! I was wrong. > I try to find that parameter on Step 4: Run test suite. > One more time, thank you! > > пт, 9 авг. 2019 г. в 14:05, Petr Ivanov : > > > Why do you think I did not use it? > > > > > > On 9 Aug 2019, at 13:25, Denis Garus wrote: > > > > Great! > > Could you please add surefire-fork-count-1 to additional Maven command > > line parameters? > > It's crucial. > > > > Thank you! > > > > пт, 9 авг. 2019 г. в 12:42, Petr Ivanov : > > > >> Done [1] > >> > >> > >> [1] https://ci.ignite.apache.org/viewLog.html?buildId=4482200 > >> > >> On 9 Aug 2019, at 12:02, Denis Garus wrote: > >> > >> Sure! I created the task [1]. > >> > >> Thank you! > >> > >> 1. https://issues.apache.org/jira/browse/IGNITE-12055 > >> > >> пт, 9 авг. 2019 г. в 11:38, Petr Ivanov : > >> > >>> Hi, Denis! > >>> > >>> > >>> Could file a ticket with description, please? > >>> > >>> On 9 Aug 2019, at 11:35, Denis Garus wrote: > >>> > >>> Thanks all for the feedback! > >>> > >>> I think no one is against of proposal. > >>> > >>> Petr, could you please assist with wit separation of SecurityTestSuite? > >>> > >>> чт, 8 авг. 2019 г. в 14:43, Denis Garus : > >>> > >>>> Hello, Ivan! > >>>> > >>>> >> Could you please provide more details why do we need to run these > >>>> tests in forked JVM? > >>>> > >>>> Surefite documentation [1] says: > >>>> If forkCount=0, it's impossible to use the system class loader or a > >>>> plain old Java classpath; we have to use an isolated class loader. > >>>> > >>>> When using isolated class loader will cause compiler error: > >>>> package org.apache.ignite.lang does not exist > >>>> > >>>> We cannot compile the TestIgniteCallable class. > >>>> > >>>> 1. > >>>> https://maven.apache.org/surefire/maven-surefire-plugin/examples/class-loading.html > >>>> > >>>> чт, 8 авг. 2019 г. в 09:44, Павлухин Иван : > >>>> > >>>>> Denis, > >>>>> > >>>>> Could you please provide more details why do we need to run these > >>>>> tests in forked JVM? > >>>>> > >>>>> Still, having separate security suite on TC sounds not bad. > >>>>> > >>>>> ср, 7 авг. 2019 г. в 09:35, Vyacheslav Daradur : > >>>>> > > >>>>> > Hi Denis. > >>>>> > > >>>>> > I think it is fine to extract security tests in a separate build > >>>>> plan on TC. > >>>>> > > >>>>> > BTW, if you are going to write a lot of Sandbox's tests pay attention > >>>>> > to 'extdata' module and an approach of P2P tests > >>>>> > (IgniteP2PSelfTestSuite) - this may help you to avoid Maven's > >>>>> > classloading issues. > >>>>> > > >>>>> > On Tue, Aug 6, 2019 at 3:25 PM Denis Garus > >>>>> wrote: > >>>>> > > > >>>>> > > Hello Igniters! > >>>>> > > > >>>>> > > I made the test DoPrivelegedOnRemoteNodeTest[1] > >>>>> (SecurityTestSuite) for the > >>>>> > > task "Sandbox for user-defined code" [2] > >>>>> > > that uses p2p deploy like the test > >>>>> > > ServiceHotRedeploymentViaDeploymentSpiTest [3] from > >>>>> > > IgniteServiceGridTestSuite. > >>>>> > > That test requires additional Maven command line parameter -P > >>>>> > > surefire-fork-count-1. > >>>>> > > The suite Basic 1 contains the SecurityTestSuite and many other > >>>>> test suites > >>>>> > > at TeamCity that do not need that additional Maven parameter. > >>>>> > > I suggest extracting SecurityTestSuite as a separate test suite to > >>>>> define > >>>>> > > additional Maven command line parameter for it. > >>>>> > > > >>>>> > > WDYT? > >>>>> > > > >>>>> > > > >>>>> > > 1. https://github.com/apache/ignite/pull/6707 > >>>>> > > 2. https://issues.apache.org/jira/browse/IGNITE-11410 > >>>>> > > 3. > >>>>> > > > >>>>> https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/processors/service/ServiceHotRedeploymentViaDeploymentSpiTest.java > >>>>> > > >>>>> > > >>>>> > > >>>>> > -- > >>>>> > Best Regards, Vyacheslav D. > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Best regards, > >>>>> Ivan Pavlukhin > >>>>> > >>>> > >>> > >> > > -- Best regards, Ivan Pavlukhin
Re: Replacing NodeFilter functionality with label approach
Folks, I feel that the picture still is not clear for the subject. Pavel K., could you please highlight problems related to user code in NodeFilter except classloading? Nikolay, could you please provide some examples when node filtering cannot be solved with label/regexp approach? чт, 8 авг. 2019 г. в 15:39, Pavel Tupitsyn : > > I agree that attribute-based filtering is enough. > > We should get rid of predicates in configuration as much as possible: > they introduce a lot of complexity for other platforms (.NET), among other > things mentioned above. > > On Thu, Aug 8, 2019 at 2:04 PM Pavel Kovalenko wrote: > > > Ivan, > > > > > And there is also one idea (I am not fan of it but still). Can we use > > > some kind of scripting for nodes filtering? In that case node filter > > > is represented by script string, e.g. javascript. > > > > I guess it can lead to the same situation as in Java NodeFilter's. We can't > > control what happens in a filter in this case. > > We can consider regex as an option instead of just labels. It's still > > string and can be validated on correctness during node start. > > But we still don't have any real examples that require more flexibility > > than labels have. > > > > вт, 6 авг. 2019 г. в 14:46, Павлухин Иван : > > > > > Alexey, > > > > > > It seems that a problem has a solution with using 2 attributes or 2 > > > labels. Is not it more clear than using custom code? > > > > > > Folks, > > > > > > > I don't think we should take "hard to implement" as an argument in this > > > discussion :) > > > Did not fully get the point. KISS principle is not true anymore? Or is > > > this discussion somehow special? I believe that every flexibility > > > handle should be critically justified. Would be great to justify > > > NodeFilter flexibility. > > > > > > > Filters based of hostname or ip address. > > > Is it a good idea to use IP address for node filtering? IP can be > > > changed for a node with persistence, does it mean that not relevant > > > data (according to a filter) should be cleared, does it work now? > > > > > > And there is also one idea (I am not fan of it but still). Can we use > > > some kind of scripting for nodes filtering? In that case node filter > > > is represented by script string, e.g. javascript. > > > > > > вт, 6 авг. 2019 г. в 12:22, Alexey Kukushkin > >: > > > > > > > > Pavel, > > > > > > > > Just a real example you asked for: we have a user attribute "ROLE_DC", > > > > which is a comma separated list like "wfe_a, as_a, db_a" (server role > > and > > > > data center designator) and we have node filters to deploy services and > > > > start caches on servers with specific role (like WFE) and sometimes > > > > specific role and DC (like WFE_A). The node filter splits the list and > > > uses > > > > a regular expression to match each segment. > > > > > > > > If you replace generic node filter with a user attribute filter then we > > > > still can achieve what we need by creating 3 user attributes (ROLE_DC, > > > > ROLE and DC) but we lose flexibility since now adding a new data center > > > > requires updating all cache and service configurations. With regex > > > matching > > > > we do not need to update the configurations since we still match all > > the > > > > roles in the new DC. > > > > > > > > So we would have a solution with user attributes filter but I we lose > > > some > > > > flexibility. > > > > > > > > -- > > > > Best regards, > > > > Alexey > > > > > > > > > > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > > > -- Best regards, Ivan Pavlukhin
Re: Issue with the TC disk space
Folks, I asked about TC infrastructure problems before and did not get answers. Does any of you know how to fix problems of that sort? Should not the process be transparent for a community members? пн, 12 авг. 2019 г. в 11:00, Dmitriy Pavlov : > > I also see some issues with 2.7.6 testing. Hopefully, it would be fixed > soon since it is a blocker for any activity for 2.7.6 > > пн, 12 авг. 2019 г. в 10:58, Nikolay Izhikov : > > > Hello, Igniters. > > > > Looks like TC has the issues with the free disk space [1] > > > > Can someone take a look? > > > > > > ``` > > [19:13:00]Free disk space requirement > > [19:13:00][Free disk space requirement] Removing files to meet 3 GB of > > free disk space required for directory /opt/buildagent/temp (only 122.05 MB > > is free now). > > [19:13:00][Free disk space requirement] Free disk space requirement of 3 > > GB is met for directory /opt/buildagent/work/69588afcb2ab3382 (106.14 GB is > > free) > > [19:13:00][Free disk space requirement] Free disk space requirement of 3 > > GB could not be met for directory /opt/buildagent/temp (only 123.32 MB is > > free) > > [19:13:00]Resolving artifact dependencies (4s) > > [19:13:01][Resolving artifact dependencies] Destination directory > > [/opt/buildagent/work/69588afcb2ab3382] cleaned > > [19:13:01][Resolving artifact dependencies] Downloading files from > Tests 2.4+ (Java 8/9/10/11) / ~Build Apache Ignite~, build #1084597 [id > > 4482201]> for pattern [ignite.zip!** => ./] > > [19:13:01][Resolving artifact dependencies] Downloading 1 artifact > > [19:13:04][Resolving artifact dependencies] Failed to resolve artifact > > dependency > build #1084597 [id 4482201]>: Failed to download file > > 'ignite.zip': No space left on device > > (jetbrains.buildServer.artifacts.impl.SourcePathAwareResolvingFailedException) > > [19:13:04]Failed to resolve artifacts from > 8/9/10/11) / ~Build Apache Ignite~, build #1084597 [id 4482201]> > > ``` > > > > [1] > > https://ci.ignite.apache.org/viewLog.html?tab=buildLog=tree=debug=all=4483694&_focus=60 > > -- Best regards, Ivan Pavlukhin
Re: Coding guidelines. Useless JavaDoc comments.
Maxim, > I'd prefer to leave the current situation with Javadoc as it is and just to > ask to apply the patch [1][2]. Can you? :-) You caught me. I left my comments for that PR .Pavel and all, > I think that API of our core internal things like PageMemory, WAL, all > existing managers and processors should be as well documented as possible. No doubts here. > Documentation should be a result of a proper code review. I suppose it does not mean that documentation should be written after a review. I suppose it means that we should not have a poor documentation *after* a review. Overall, Pavel's last message conforms well with my opinion on the subject. чт, 8 авг. 2019 г. в 18:34, Pavel Kovalenko : > > I can agree that some part of javadocs we have is useless. It relates to > DTOs, getters/setters without side-effects, short self-descriptive methods. > In an ideal world, proper modularization of architecture, leading to > KISS/SOLID/DRY/etc. principles, writing self-documented code should result > in javadocs disappearing, as they become not needed. > We live in a not ideal world. We don't have good architecture and can't > always lead to mentioned principles, because we need sometimes sacrifice > readability for optimization, fixing a critical bug, etc. > I think that API of our core internal things like PageMemory, WAL, all > existing managers and processors should be as well documented as possible. > If a developer uses some module / manager / processor without looking > inside, reading the only description of public methods, it's a good sign > that this part of the functionality is well documented. > Internal implementation should be also clear for a developer who likes to > make a change inside it. Every optimization, avoiding race-condition, not > obvious thing and especially crutch must be documented as detailed as > possible. > Documentation should be a result of a proper code review. If a reviewer has > questions regarding any code line it should be either refactored to make > this thing obvious or well documented. > If a class or method is self-documented and obvious there is no need to > document it anyway. > if each person takes the code review as seriously as possible, > documentation and code will be better automatically. > Mandatory documentation in places where it's really not needed looks like a > burden. A developer will avoid write it properly everywhere or do it "just > for check" and this will influence on documentation with the negative side. > Flexible approach with mandatory / optional javadocs with good code review > will result in readability improvement overall. > > > чт, 8 авг. 2019 г. в 17:52, Maxim Muzafarov : > > > Ivan, > > > > It is not a problem to check Javadocs at the moment code syle checking > > performed, but do we really need this? And the human-factor you > > mentioned above is also related to the `self-descriptive` names. I > > assume, that someone now is desiring to use single-letter variables > > and single-letter class names to save space an time. We will always > > have such an opinion race. > > > > I'd prefer to leave the current situation with Javadoc as it is and > > just to ask to apply the patch [1][2]. Can you? :-) > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12051 > > [2] https://github.com/apache/ignite/pull/6760 > > > > On Thu, 8 Aug 2019 at 17:24, Павлухин Иван wrote: > > > > > > Maxim, > > > > > > My main concern here is a human factor. Humans are usually not so good > > > in keeping documentation always in sync with a code. Examples from an > > > actual PR: > > > /** > > > * @param nodeId The remote node id. > > > * @param channel The channel to notify listeners with. > > > */ > > > private void onChannelOpened0(UUID nodeId, GridIoMessage initMsg, > > > Channel channel) > > > > > > First, there is a mismatch between number of parameters in javadoc and > > > code. Second, e.g. nodeId name can be made self-descriptive rmtNodeId > > > name. > > > > > > Mandatory javadocs do not imply good javadocs. Good javadocs do not > > > imply javadocs for every method and field. For me, mandatory and good > > > javadocs are like communism. Sounds quite nice in theory, but not > > > feasible in practice. > > > > > > чт, 8 авг. 2019 г. в 16:55, Denis Garus : > > > > > > > > Thank you, Maxim. > > > > > > > > >>On what we are trying to save by making Javadoc optional? > > > > > > > > Space and time. > > > > > > > > I doubt tha
Re: [EXTERNAL] Re: Replace or Put after PutAsync causes Ignite to hang
Ilya, Pavel,
Do we a have a proposal how to fix the root cause of the problem?
Should we a have a ticket for it?
ср, 7 авг. 2019 г. в 17:48, Ilya Kasnacheev :
>
> Hello!
>
> I think we should definitely stop running futures out of striped pool,
> while holding any cache logs (stripe thread counts as one).
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 7 авг. 2019 г. в 17:20, Pavel Tupitsyn :
>
> > Yes, this can be done purely on .NET side, which is an option that I
> > consider.
> > However, the root problem is on Java side, and I believe that we should fix
> > the root problem.
> >
> > > violate some of Ignite assumptions: that we never run user code from
> > certain thread pools
> > We actually do run user code from Ignite thread pools:
> >
> > cache.getAsync(1).listen(fut ->
> > System.out.println("Get operation completed [value=" + fut.get() +
> > ']'));
> >
> > `println` here is executed on the striped pool. This is stated in the
> > docs that I linked above.
> >
> > Users have to be aware of this and they have to be very careful with
> > every future listener. IMO, this is a tricky gotcha and a bad
> > usability.
> >
> >
> > Thoughts?
> >
> >
> > On Wed, Aug 7, 2019 at 12:22 PM Ilya Kasnacheev > >
> > wrote:
> >
> > > Hello!
> > >
> > > + dev@
> > >
> > > I think the current behavior, where .Net callbacks may be run from
> > striped
> > > pool, violate some of Ignite assumptions: that we never run user code
> > from
> > > certain thread pools (like sys-stripe) and that we try to limit options
> > of
> > > running user-supplied code from our internals.
> > >
> > > I think that future versions of .Net integration should remove the
> > ability
> > > of async callbacks to be called from non-user threads, even if it can
> > lead
> > > to performance degradation in some cases. I suggest removing this mode,
> > if
> > > possible, while keeping only the safe one, where internal threads are not
> > > waiting upon completion of user code.
> > >
> > > In this case my issue IGNITE-12033 could be used to track this work.
> > >
> > > WDYT?
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > ср, 7 авг. 2019 г. в 01:47, Pavel Tupitsyn :
> > >
> > >> Sorry guys, I've completely missed this thread, and the topic is very
> > >> important.
> > >>
> > >> First, a simple fix for the given example. Add the following on the
> > first
> > >> line of Main:
> > >> SynchronizationContext.SetSynchronizationContext(new
> > >> ThreadPoolSynchronizationContext());
> > >>
> > >> And put the ThreadPoolSynchronizationContext class somewhere:
> > >> class ThreadPoolSynchronizationContext : SynchronizationContext
> > >> {
> > >> // No-op.
> > >> }
> > >>
> > >>
> > >> Now, detailed explanation. The problem exists forever in Ignite and is
> > >> mentioned in the docs briefly [1].
> > >> Also mentioned in .NET docs (I've updated them a bit) [2].
> > >>
> > >> Breakdown:
> > >> * Ignite (Java side) runs async callbacks (continuations) on system
> > >> threads, and those threads have limitations (you should not call Ignite
> > >> APIs from them in general)
> > >> * Ignite.NET wraps async operations into native .NET Tasks
> > >> * Usually `await ...` call in .NET will continue execution on the
> > >> original Thread (simply put, actually it is more complex), so Ignite
> > system
> > >> thread issue is avoided
> > >> * However, Console applications have no `SynchronizationContext`, so the
> > >> continuation can't be dispatched to original thread, and is executed on
> > >> current (Ignite) thread
> > >> * Setting custom SynchronizationContext fixes the issue: all async
> > >> continuations will be dispatched to .NET thread pool and never executed
> > on
> > >> Ignite threads
> > >>
> > >> However, dispatching callbacks to a different thread causes performance
> > >> hit, and Ignite favors performance over usability right now.
> > >> So it is up to the user to configure desired behavior.
> > >>
> > >> Let me know if you need more details.
> > >>
> > >> Thanks
> > >>
> > >> [1] https://apacheignite.readme.io/docs/async-support
> > >> [2] https://apacheignite-net.readme.io/docs/asynchronous-support
> > >>
> > >> On Thu, Aug 1, 2019 at 3:41 PM Ilya Kasnacheev <
> > ilya.kasnach...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hello!
> > >>>
> > >>> I have filed a ticket about this issue so it won't get lost.
> > >>> https://issues.apache.org/jira/browse/IGNITE-12033
> > >>>
> > >>> Regards,
> > >>> --
> > >>> Ilya Kasnacheev
> > >>>
> > >>>
> > >>> чт, 2 мая 2019 г. в 10:53, Barney Pippin <
> > james.pri...@uk.bnpparibas.com
> > >>> >:
> > >>>
> > Thanks for the response Ilya. Did you get a chance to look at this
> > Pavel?
> > Thanks.
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
> >
> > >>>
> >
--
Best regards,
Ivan Pavlukhin
Re: SQL query timeout: in progress or abandoned
Just for the protocol. There was an original dev-list discussion [1]. Added a link to the ticket as well. [1] http://apache-ignite-developers.2346864.n4.nabble.com/IGNITE-7285-Add-default-query-timeout-td41828.html пт, 9 авг. 2019 г. в 01:22, Denis Magda : > > Hey Saikat, > > Are you still working on this ticket? > https://issues.apache.org/jira/browse/IGNITE-7285 > > Seems that's the last API that doesn't support timeouts - JDBC and ODBC > drivers already go with it. > > If you don't have time to complete the changes then someone else from the > community can take over. We see a lot of demand for this API and here is > one example: > https://stackoverflow.com/questions/57275301/how-to-set-a-query-timeout-for-apache-ignite-cache > > - > Denis -- Best regards, Ivan Pavlukhin
Re: Apache Ignite 2.7.6 (Time, Scope, and Release manager)
> What's the scope for this release? Same question. On the other hand an idea of 2.7.6 release attracts me because having a practice of doing frequent minor releases can help us to build reliable and predictable release rails. чт, 8 авг. 2019 г. в 15:09, Ilya Kasnacheev : > > Hello! > > What's the scope for this release? > > Regards, > -- > Ilya Kasnacheev > > > чт, 8 авг. 2019 г. в 15:07, Dmitriy Pavlov : > > > Hi Apache Ignite Developers, > > > > > > > > We seem to be on the same page about 2.8 release, but we’ve started new > > practice - minor releases, the first release was 2.7.5. Unfortunately, > > there is a couple of issues still not fixed in that release, so I would > > like to propose to prepare one more minor release for Apache Ignite. > > > > > > > > I propose my candidacy to be release manager once again. > > > > > > > > I could (of course with help from Peter Ivanov) do some additional efforts > > to complete and improve process doc > > https://cwiki.apache.org/confluence/display/IGNITE/Release+Process > > > > > > > > I propose (most optimistic) dates for release stages: > > > > Scope Freeze: August 12, 2019 > > > > Code Freeze: August 15, 2019 > > > > Voting Date: August 22, 2019 > > > > Release Date: August 27, 2019 > > > > > > WDYT? > > > > > > If nobody minds, I will create branch 2.7.6 based on 2.7.5 and set up in > > the TC Bot during the weekend. > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > -- Best regards, Ivan Pavlukhin
Re: Coding guidelines. Useless JavaDoc comments.
Maxim,
My main concern here is a human factor. Humans are usually not so good
in keeping documentation always in sync with a code. Examples from an
actual PR:
/**
* @param nodeId The remote node id.
* @param channel The channel to notify listeners with.
*/
private void onChannelOpened0(UUID nodeId, GridIoMessage initMsg,
Channel channel)
First, there is a mismatch between number of parameters in javadoc and
code. Second, e.g. nodeId name can be made self-descriptive rmtNodeId
name.
Mandatory javadocs do not imply good javadocs. Good javadocs do not
imply javadocs for every method and field. For me, mandatory and good
javadocs are like communism. Sounds quite nice in theory, but not
feasible in practice.
чт, 8 авг. 2019 г. в 16:55, Denis Garus :
>
> Thank you, Maxim.
>
> >>On what we are trying to save by making Javadoc optional?
>
> Space and time.
>
> I doubt that we need a verbal description of what private method make.
> Why we need it if we just can read the code?
>
> Bright examples:
>
> /**
> * @param helper Helper to attach to kernal context.
> */
> private void addHelper(Object helper) {
> ctx.addHelper(helper);
> }
>
> /**
> * Gets "on" or "off" string for given boolean value.
> *
> * @param b Boolean value to convert.
> * @return Result string.
> */
> private String onOff(boolean b) {
> return b ? "on" : "off";
> }
>
> You have to support both code of method and their JavaDoc description, but
> it doesn't get any sake.
>
> >>Let's just help them to read the source code.
>
> But at the same time, public method IgniteKernal#start doesn't have any
> description except enumeration of attributes with apparent notes.
> Maybe to give it a verbal description needs one or two pages of the screen.
> Does it make sense?
>
> чт, 8 авг. 2019 г. в 15:41, Maxim Muzafarov :
>
> > Igniters,
> >
> > I'm -1 with making Javadoc optional (except tests).
> >
> > Here is my patch [1] with PR [2] which fixes all the Javadoc comments
> > for the IgniteKernal class mentioned above. Please, take a look. The
> > review is very appreciated.
> >
> > On what we are trying to save by making Javadoc optional? In my humble
> > opinion - Javadoc comments are mostly concern users who debug the
> > Ignite when they have faced with some unhandled exception or related
> > to the community members with an average involvement (produces a few
> > small patches during the year) but not to the experienced one. Let's
> > just help them to read the source code.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-12051
> > [2] https://github.com/apache/ignite/pull/6760
> >
> >
> >
> > On Wed, 7 Aug 2019 at 13:54, Andrey Kuznetsov wrote:
> > >
> > > +1 for making javadoc comments optional.
> > >
> > > - Empty and tautological comments are kind of garbage that reduce
> > > readability.
> > > - It's better to leave the entity undocumented, than write
> > > unexpressive/misleading comment.
> > > - Even classes may not require javadocs, e.g. simple DTOs.
> > >
> > > ср, 7 авг. 2019 г., 13:39 Denis Garus :
> > >
> > > > Thx for feedback!
> > > >
> > > > >> we have to write proper javadoc for all production classes,
> > including
> > > > internal.
> > > >
> > > > Nikolay, I cannot agree with it.
> > > >
> > > > What should be the best comment for the next fields?
> > > > /** */
> > > > private static final long serialVersionUID = 0L;
> > > > or
> > > > /** */
> > > > @LoggerResource
> > > > private IgniteLogger log;
> > > >
> > > > There are more than 8000 lines of /** */ only at the ignite-core
> > module (do
> > > > not include tests).
> > > >
> > > > Any comments will be redundant and just noise. Obvious comment learns
> > > > readers skip all comments.
> > > >
> > > >
> > > > >> identical distance/padding/margin between fields in a class - is
> > really
> > > > cool
> > > >
> > > > Vyacheslav, but we have a blank line between fields. Why is one blank
> > line
> > > > not enough?
> > > >
> > > > ср, 7 авг. 2019 г. в 12:58, Павлухин Иван :
> > > >
> > > > > Hi,
> > > > >
> > > > > Denis, thank you for starting this discussion!
> > > > >
> > > > &
Re: SecurityTestSuite as a separate test suite at TC
Denis, Could you please provide more details why do we need to run these tests in forked JVM? Still, having separate security suite on TC sounds not bad. ср, 7 авг. 2019 г. в 09:35, Vyacheslav Daradur : > > Hi Denis. > > I think it is fine to extract security tests in a separate build plan on TC. > > BTW, if you are going to write a lot of Sandbox's tests pay attention > to 'extdata' module and an approach of P2P tests > (IgniteP2PSelfTestSuite) - this may help you to avoid Maven's > classloading issues. > > On Tue, Aug 6, 2019 at 3:25 PM Denis Garus wrote: > > > > Hello Igniters! > > > > I made the test DoPrivelegedOnRemoteNodeTest[1] (SecurityTestSuite) for the > > task "Sandbox for user-defined code" [2] > > that uses p2p deploy like the test > > ServiceHotRedeploymentViaDeploymentSpiTest [3] from > > IgniteServiceGridTestSuite. > > That test requires additional Maven command line parameter -P > > surefire-fork-count-1. > > The suite Basic 1 contains the SecurityTestSuite and many other test suites > > at TeamCity that do not need that additional Maven parameter. > > I suggest extracting SecurityTestSuite as a separate test suite to define > > additional Maven command line parameter for it. > > > > WDYT? > > > > > > 1. https://github.com/apache/ignite/pull/6707 > > 2. https://issues.apache.org/jira/browse/IGNITE-11410 > > 3. > > https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/processors/service/ServiceHotRedeploymentViaDeploymentSpiTest.java > > > > -- > Best Regards, Vyacheslav D. -- Best regards, Ivan Pavlukhin
Re: Partition loss event
Ilya, I am not sure that enabling subset of events will fit all needs. If we would like to do so then it might be good idea to make it clear in API that certain events are enabled by default (e.g. put them into separate class). Also we should be careful with backward compatibility, some kind of events look more like debug stuff, e.g. EVT_CACHE_REBALANCE_OBJECT_LOADED. Is it relevant for historical rebalance? What does it mean for rebalancing whole partition files? How user is going to use it? And a care stil should be put to performance, aforementioned EVT_CACHE_REBALANCE_OBJECT_LOADED can introduce non-negliable impact, cannot it? Also, we can employ some soft means like printing a warning if a listener is registered for disabled event. And a last point about code and javadocs. There is a line "Note that certain events are required for Ignite's internal operations and such events will still be generated". Perhaps we can provide a list of such events in docs or do not expose them to a user listeners. And a bit of mess. IgniteConfiguration#getIncludeEventTypes claims "Note that by default all events in Ignite are disabled". While EventType says "Note that by default all events in Ignite are enabled and therefore generated and stored by whatever event storage SPI is configured". чт, 1 авг. 2019 г. в 17:42, Ilya Kasnacheev : > > Dear fellows! > > I think we have a problem: when events were introduced, we were talking > about high-bandwdith events which may overflow your nodes if you > accidentally turn them on. > > However, now we have a bunch of low-bandwidth events, such as: > EVT_CHECKPOINT_SAVED > EVT_CHECKPOINT_LOADED > EVT_CHECKPOINT_REMOVED > EVT_NODE_JOINED > EVT_NODE_LEFT > EVT_NODE_FAILED > EVT_NODE_SEGMENTED > EVT_CACHE_REBALANCE_STARTED > EVT_CACHE_REBALANCE_STOPPED > EVT_CACHE_REBALANCE_PART_LOADED > EVT_CACHE_REBALANCE_PART_UNLOADED > EVT_CACHE_REBALANCE_OBJECT_LOADED > EVT_CACHE_REBALANCE_OBJECT_UNLOADED > EVT_CACHE_REBALANCE_PART_DATA_LOST > EVT_CACHE_REBALANCE_PART_SUPPLIED > EVT_CACHE_REBALANCE_PART_MISSED > EVT_CLIENT_NODE_DISCONNECTED > EVT_CLIENT_NODE_RECONNECTED > EVT_WAL_SEGMENT_ARCHIVED > EVT_WAL_SEGMENT_COMPACTED > EVT_CLUSTER_ACTIVATED > EVT_CLUSTER_DEACTIVATED > EVT_PAGE_REPLACEMENT_STARTED > > I suggest we enable these events by default, since I fail to see how this > may ever cause problems, but it will definitely decrease confusion > surrounding events. > > WDYT? > > Regards, > -- > Ilya Kasnacheev > > > чт, 1 авг. 2019 г. в 15:18, balazspeterfi : > > > Hi Alexandr, > > > > Thanks, that was the missing part. It would be nice to mention it in the > > docs I guess as it's quite easy to miss it. > > > > Regards, > > Balazs > > > > > > > > -- > > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > > -- Best regards, Ivan Pavlukhin
Re: Coding guidelines. Useless JavaDoc comments.
Hi, Denis, thank you for starting this discussion! My opinion here is that having a good javadoc for every class and method is not feasible in the real world. I am quite curious to see a non-trivial project which follows it. Also, all comments and javadocs are prone to become misleading when code changes (human factor). In my experience good method and variable names and clear code organization often are more helpful than javadocs. ср, 7 авг. 2019 г. в 12:49, Vyacheslav Daradur : > > I agree that useless comments look weird in the codebase. > > But, identical distance/padding/margin between fields in a class - is > really cool, and helps read the class very fast. > > On Wed, Aug 7, 2019 at 12:26 PM Nikolay Izhikov wrote: > > > > Hello, Denis. > > > > Thanks for starting this discussion. > > > > I think we have to write proper javadoc for all production classes, > > including internal. > > We should fix useless javadoc you provide. > > We should not accept patches without good javadocs. > > > > As for the tests, seems, we can make javadoc optional. > > > > What do you think? > > > > > > В Ср, 07/08/2019 в 11:41 +0300, Denis Garus пишет: > > > Igniters! > > > > > > I think it's time to change coding guidelines in part of JavaDoc comments > > > [1]: > > > > > Every method, field or initializer public, private or protected in > > > > > > top-level, > > > > > inner or anonymous type should have at least minimal Javadoc comments > > > > > > including > > > > > description and description of parameters using @param, @return and > > > > > > @throws Javadoc tags, > > > > > where applicable. > > > > > > Let's look at JavaDoc comments in the IgniteKernal class: > > > > > > Why? > > > > > > /** */ - 15 matches. > > > > > > What can you get new from these comments? > > > > > > /** Periodic starvation check interval. */ > > > private static final long PERIODIC_STARVATION_CHECK_FREQ = 1000 * 30; > > > > > > /** Long jvm pause detector. */ > > > private LongJVMPauseDetector longJVMPauseDetector; > > > > > > /** Scheduler. */ > > > private IgniteScheduler scheduler; > > > > > > /** Stop guard. */ > > > private final AtomicBoolean stopGuard = new AtomicBoolean(); > > > > > > /** > > > * @param cfg Configuration to use. > > > * @param utilityCachePool Utility cache pool. > > > * @param execSvc Executor service. > > > * @param sysExecSvc System executor service. > > > * @param stripedExecSvc Striped executor. > > > * @param p2pExecSvc P2P executor service. > > > * @param mgmtExecSvc Management executor service. > > > * @param igfsExecSvc IGFS executor service. > > > * @param dataStreamExecSvc data stream executor service. > > > * @param restExecSvc Reset executor service. > > > * @param affExecSvc Affinity executor service. > > > * @param idxExecSvc Indexing executor service. > > > * @param callbackExecSvc Callback executor service. > > > * @param qryExecSvc Query executor service. > > > * @param schemaExecSvc Schema executor service. > > > * @param customExecSvcs Custom named executors. > > > * @param errHnd Error handler to use for notification about startup > > > problems. > > > * @param workerRegistry Worker registry. > > > * @param hnd Default uncaught exception handler used by thread pools. > > > * @throws IgniteCheckedException Thrown in case of any errors. > > > */ > > > public void start( > > > final IgniteConfiguration cfg, > > > ExecutorService utilityCachePool, > > > final ExecutorService execSvc, > > > final ExecutorService svcExecSvc, > > > final ExecutorService sysExecSvc, > > > final StripedExecutor stripedExecSvc, > > > ExecutorService p2pExecSvc, > > > ExecutorService mgmtExecSvc, > > > ExecutorService igfsExecSvc, > > > StripedExecutor dataStreamExecSvc, > > > ExecutorService restExecSvc, > > > ExecutorService affExecSvc, > > > @Nullable ExecutorService idxExecSvc, > > > IgniteStripedThreadPoolExecutor callbackExecSvc, > > > ExecutorService qryExecSvc, > > > ExecutorService schemaExecSvc, > > > @Nullable final Map > > > customExecSvcs, > > > GridAbsClosure errHnd, > > > WorkersRegistry workerRegistry, > > > Thread.UncaughtExceptionHandler hnd, > > > TimeBag startTimer > > > ) > > > > > > These comments look ugly and useless, and that is the main class of core. > > > Why do they need us? > > > Let us change the coding guidelines in part of JavaDoc comments: > > > Every method public API should have at least minimal Javadoc comments, > > > including description and description of parameters using @param, @return, > > > and @throws Javadoc tags, > > > where applicable. > > > For internal API, JavaDoc comments should be optional. It's up to a > > > contributor or reviewer. > > > > > > What are you think? > > > > > > 1. >
Re: New Ignite PMC Member: Ilya Kasnacheev
Ilya, my congratulations! вт, 6 авг. 2019 г. в 23:12, Dmitriy Pavlov : > > Ilya, congratulations! > > вт, 6 авг. 2019 г. в 22:58, Denis Magda : > > > The Project Management Committee (PMC) for Apache Ignite > > has invited Ilya Kasnacheev to become a committer and we are pleased > > to announce that he has accepted. > > > > Ilya is one of the most active and valuable Ignite committers who is not > > only contributing source-code changes but also takes an active stance on > > Ignite adoption - he constantly helps Ignite users to design, troubleshoot > > and optimize their Ignite deployments. > > > > Being a PMC member enables assistance with the management and to guide the > > direction of the project. > > > > Please join me in congratulating Ilya with this new role! > > > > > > - > > Denis > > -- Best regards, Ivan Pavlukhin
Re: [DISCUSSION][IEP-35] Metrics configuration
Andrey, It seems that screenshot was rejected, I do not see it. вт, 6 авг. 2019 г. в 15:07, Andrey Gura : > > Good example of proper buckets configuration. Such configuration is > suitable for may cases. See attached screenshot (I hope it will not be > reject by mail system or forum). > > > On Tue, Aug 6, 2019 at 2:45 PM Andrey Gura wrote: > > > > > What do you mean by "exponential bounds"? > > > > Something like this if we talk about latency in ms for example: 5, 10, > > 25, 50, 100, 200, 500, ... > > > > > Thanks, for the feedback, appreciate you ownesty. > > > > Nothing personal. It is just about functionality from user's stand point. > > > > > What is your proposal? > > > How metrics configuration should work? > > > > My proposal is simple: just drop this change. We don't need the > > configuration. Metric owner (developer) defines buckets' bounds for > > each particular case (it could be done uniformly or exponentially, it > > depends on metric and problem definition). > > > > On Mon, Aug 5, 2019 at 6:36 PM Nikolay Izhikov wrote: > > > > > > Hello, Andrey. > > > > > > > Not necessary if we have exponential bounds' values for histograms. > > > > > > What do you mean by "exponential bounds"? > > > > > > > Anyway, in current solution it looks ugly and not usable. > > > > > > Thanks, for the feedback, appreciate you ownesty. > > > > > > > No. But we should admit that this is bad decision and do not include > > > > this change to the code base. > > > > > > What is your proposal? > > > How metrics configuration should work? > > > > > > > Yes. But it still will not give enough accuracy. > > > > > > Enough for what? > > > > > > В Пн, 05/08/2019 в 18:29 +0300, Andrey Gura пишет: > > > > > > - metric configuration is node local (not cluster wide). > > > > > This issue is easy to solve on the user-side and in Ignite core. > > > > > > > > It's imaginary simplicity. The first, you need some additional > > > > automation on user-side in order to configure all nodes of the > > > > cluster. The second, new nodes can join to the cluster and > > > > configuration will be different on new node and on other nodes of the > > > > cluster. This leads to complication whole functionality. Anyway, I > > > > don't like such simplified solution because at the moment it brings > > > > more problems than value. > > > > > > > > > The easiest solution was implemented. > > > > > Do we want to make it more complex right now :)? > > > > > > > > No. But we should admit that this is bad decision and do not include > > > > this change to the code base. > > > > > > > > > The reason it exists in PR - we already have this parameter in > > > > > DataStorageConfiguration#getMetricsSubIntervalCount > > > > > > > > I believe this method should be deprecated and removed in major release. > > > > > > > > > I think the user should be able to configure buckets for histogram > > > > > and rateTimeInterval for hitrate. > > > > > > > > Not necessary if we have exponential bounds' values for histograms. > > > > Anyway, in current solution it looks ugly and not usable. > > > > > > > > > Ignite has dozens of use-cases and deployment modes, seems, > > > > > we can't cover it all with the single predefined > > > > > buckets/rateTimeInterval set. > > > > > > > > Yes. But it still will not give enough accuracy. > > > > > > > > On Mon, Aug 5, 2019 at 5:25 PM Nikolay Izhikov > > > > wrote: > > > > > > > > > > Hello, Andrey. > > > > > > > > > > > - metric configuration is node local (not cluster wide). > > > > > > > > > > This issue is easy to solve on the user-side and in Ignite core. > > > > > > > > > > > - metric configuration doesn't survive node restart. > > > > > > > > > > We decide to go with the simplest solution, for now. > > > > > The easiest solution was implemented. > > > > > Do we want to make it more complex right now :)? > > > > > > > > > > > - User shouldn't configure hit rate metrics at runtime in most > > > > > > cases. > > > > > > > > > > I agree with you - the size of the counters array looks odd as a > > > > > configuration parameter. > > > > > The reason it exists in PR - we already have this parameter in > > > > > DataStorageConfiguration#getMetricsSubIntervalCount > > > > > > > > > > > - May be it is enough for user to have histograms with > > > > > > pre-configured buckets > > > > > > So I think we should drop this change and idea about runtime > > > > > > histrogram and hit rate configuration. > > > > > > > > > > I think the user should be able to configure buckets for histogram > > > > > and rateTimeInterval for hitrate. > > > > > > > > > > Ignite has dozens of use-cases and deployment modes, seems, > > > > > we can't cover it all with the single predefined > > > > > buckets/rateTimeInterval set. > > > > > > > > > > В Пн, 05/08/2019 в 16:59 +0300, Andrey Gura пишет: > > > > > > Igniters, > > > > > > > > > > > > I've took a look to the PR and I want follow up this discussion > > > > > > again. > > > > > > > > > > > >
Re: Replacing NodeFilter functionality with label approach
Alexey, It seems that a problem has a solution with using 2 attributes or 2 labels. Is not it more clear than using custom code? Folks, > I don't think we should take "hard to implement" as an argument in this > discussion :) Did not fully get the point. KISS principle is not true anymore? Or is this discussion somehow special? I believe that every flexibility handle should be critically justified. Would be great to justify NodeFilter flexibility. > Filters based of hostname or ip address. Is it a good idea to use IP address for node filtering? IP can be changed for a node with persistence, does it mean that not relevant data (according to a filter) should be cleared, does it work now? And there is also one idea (I am not fan of it but still). Can we use some kind of scripting for nodes filtering? In that case node filter is represented by script string, e.g. javascript. вт, 6 авг. 2019 г. в 12:22, Alexey Kukushkin : > > Pavel, > > Just a real example you asked for: we have a user attribute "ROLE_DC", > which is a comma separated list like "wfe_a, as_a, db_a" (server role and > data center designator) and we have node filters to deploy services and > start caches on servers with specific role (like WFE) and sometimes > specific role and DC (like WFE_A). The node filter splits the list and uses > a regular expression to match each segment. > > If you replace generic node filter with a user attribute filter then we > still can achieve what we need by creating 3 user attributes (ROLE_DC, > ROLE and DC) but we lose flexibility since now adding a new data center > requires updating all cache and service configurations. With regex matching > we do not need to update the configurations since we still match all the > roles in the new DC. > > So we would have a solution with user attributes filter but I we lose some > flexibility. > > -- > Best regards, > Alexey -- Best regards, Ivan Pavlukhin
Re: Replacing NodeFilter functionality with label approach
Hi Nikolay, Could you please elaborate how will NodeAttributeFilter behave? Do you mean specifying attribute name and value for exact comparison inside? Without any dynamic (user) code involved? Also I it is quite interesting for me what flexibility you are thinking about? I think that the topic is quite important and it would be great to collect use cases and describe best practices in documentation. пн, 5 авг. 2019 г. в 15:38, Nikolay Izhikov : > > Hello, Pavel. > > I think we shouldn't reduce flexibility of NodeFilter. > So, I am -1 to remove it in Ignite 3. > > Instead, Ignite can provide NodeAttributeFilter implementation of NodeFilter. > Seems, it will resolve all described issues. > > > В Чт, 01/08/2019 в 19:33 +0300, Ilya Kasnacheev пишет: > > Hello! > > > > I think this is a good idea. We already had problems with ClusterGroups > > that won't recompute until PME, or which become invalid after PME. Relying > > on string labels would fix all that. > > > > I can think of a node filter which can't be replaced with label filter > > (e.g. the one checking for presence of some partition) but generally that's > > a Bad Idea. > > > > Regards, -- Best regards, Ivan Pavlukhin
Re: [MTCGA]: new failures in builds [4394336] needs to be handled
Does not work from my home internet. =( пт, 26 июл. 2019 г. в 19:23, Alexey Zinoviev : > > Thank you so much! > > пт, 26 июл. 2019 г. в 21:15, Dmitriy Pavlov : > > > No, you and no one can not be banned because of this :). > > > > TC is accessible for me, but I'm not too far from its server now. I will > > double-check from home later. > > > > пт, 26 июл. 2019 г. в 18:58, Alexey Zinoviev : > > > > > Can somebody say me: could I be banned from TC after commit reverting. > > > The https://ci.ignite.apache.org/ became inaccessible in a few seconds > > > after Dmitry Pavlov reverting my commit. > > > > > > Maybe it's kind of paranoid mode, but... > > > > > > > > > пт, 26 июл. 2019 г. в 19:16, Alexey Zinoviev : > > > > > > > Absolutely, at this moment ML visa is not includes the new Checkstyle > > > > checker (but it includes licences and javadocs) I support that common > > > > things like checkstyle and licences should be separated from local visa > > > for > > > > different modules and should be run every time > > > > > > > > Thanks for reverting, Dmitry, I'll create new PR correctly and check it > > > > via common approach > > > > > > > > пт, 26 июл. 2019 г. в 19:11, Dmitriy Pavlov : > > > > > > > >> Hi Maxim, > > > >> > > > >> It may be reasonable, but probably we should start a separate topic. > > > IMO, > > > >> some Igniters (sad, but true) may have spam-filter for TC Bot messages > > > >> > > > >> Sincerely, > > > >> Dmitriy Pavlov > > > >> > > > >> пт, 26 июл. 2019 г. в 17:09, Maxim Muzafarov : > > > >> > > > >> > Folks, > > > >> > > > > >> > I've checked some build associated with PRs related to ML and it > > seems > > > >> > to me that the Run::ML suite [1] does not include the checkstyle > > > >> > suite in its workflow. It's a bit strange for me to add checkstyle, > > > >> > licenses headers etc. things to each aggregate suite configuration > > > >> > that we want to use. As its related to the code directly the general > > > >> > question here is - should we make our build procedure more intuitive > > > >> > and turn on checkstyle profile for the Apache Ignite Build suite? > > > >> > I think the answer is - yes. > > > >> > > > > >> > [1] > > > >> > > > > >> > > > > > https://ci.ignite.apache.org/viewLog.html?buildId=4381029=IgniteTests24Java8_RunMl > > > >> > > > > >> > On Fri, 26 Jul 2019 at 16:54, Alexey Zinoviev < > > zaleslaw@gmail.com > > > > > > > >> > wrote: > > > >> > > > > > >> > > Hi, Igniters, many thanks for the update on my PR, I didn't merge > > > for > > > >> a 3 > > > >> > > months and doesn't know that rules were changed > > > >> > > Please, revert my commit, I will update my PR according CheckStyle > > > job > > > >> > > > > > >> > > Please, tell me, is CheckStyle bot recommendations and changing of > > > PR > > > >> > name > > > >> > > (with ticket name addition) is enough to finish this issue? > > > >> > > > > > >> > > Thanks a lot for the clarification > > > >> > > > > > >> > > пт, 26 июл. 2019 г. в 18:28, Dmitriy Pavlov : > > > >> > > > > > >> > > > +1 to revert. Some day we should learn this process. Maybe this > > > day > > > >> is > > > >> > > > today. > > > >> > > > > > > >> > > > пт, 26 июл. 2019 г. в 10:00, Nikolay Izhikov < > > nizhi...@apache.org > > > >: > > > >> > > > > > > >> > > >> +1 to revert. > > > >> > > >> > > > >> > > >> В Пт, 26/07/2019 в 09:48 +0300, Павлухин Иван пишет: > > > >> > > >> > Alexey, > > > >> > > >> > > > > >> > > >> > Actually the commit [1] is very suspicious: > > > >> > > >> > 1. Commit message "[ML] Hyper
Re: [MTCGA]: new failures in builds [4394336] needs to be handled
Alexey, Actually the commit [1] is very suspicious: 1. Commit message "[ML] Hyper-parameter tuning via Genetic Algorithm (#6713)" does not refer to a ticket. 2. Is there a ticket? Consequently it is not easy to understand what was done and check ticket according to regular flow (review, TC run). 3. I skimmed through changes and found several code style violations quite soon. Should we revert the commit [1] and apply the changes according to our conventions [2]? [1] https://github.com/apache/ignite/commit/63fbcbf849640edf140047a5111a58f480c95294 [2] https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute чт, 25 июл. 2019 г. в 21:26, : > > Hi Igniters, > > I've detected some new issue on TeamCity to be handled. You are more than > welcomed to help. > > If your changes can lead to this failure(s): We're grateful that you were a > volunteer to make the contribution to this project, but things change and you > may no longer be able to finalize your contribution. > Could you respond to this email and indicate if you wish to continue and fix > test failures or step down and some committer may revert you commit. > > *New Trusted Suite failure in master [Check Code Style] > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_CheckCodeStyle=%3Cdefault%3E=buildTypeStatusDiv > Changes may lead to failure were done by > - zaleslaw@gmail.com > https://ci.ignite.apache.org/viewModification.html?modId=888540 > > - Here's a reminder of what contributors were agreed to do > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute > - Should you have any questions please contact dev@ignite.apache.org > > Best Regards, > Apache Ignite TeamCity Bot > https://github.com/apache/ignite-teamcity-bot > Notification generated at 21:26:38 25-07-2019 -- Best regards, Ivan Pavlukhin
Re: Clean up of our PRs and IEPs before 2019
Maxim, Quite a nice idea. Could we go even further? Add a comment to each 1-2 year old PR asking if the author could close it (most likely with help of some automation). As I know GitHub sends emails with PR comments to authors. чт, 25 июл. 2019 г. в 13:05, Dmitriy Pavlov : > > Folks, please close not needed PRs. > > I don't have contact with Pyatkov & dkarachentsev. Folks, please step in. > Also, feel free to reopen PRs if you still want change to be merged. > > чт, 25 июл. 2019 г. в 12:39, Maxim Muzafarov : > > > Folks, > > > > Can we contact with some members manually and ask them to close unused > > PRs? Most of the users are active community members, so I think they > > will respond quite fast. > > > > I've briefly checked GitHub: > > > > dkarachentsev - 62 opened PRs > > ilantukh - 58 opened PRs > > dgovorukhin - 44 opened PRs > > mcherkasov - 23 opened PRs > > ascherbakoff - 22 opened PRs > > vldpyatkov - 21 opened PRs > > > > On Thu, 25 Jul 2019 at 12:28, Dmitriy Pavlov wrote: > > > > > > Hi Alexey, > > > > > > second need it to check all open PRs from community members for fixes, > > > which could be merged to Ignite codebase. > > > > > > Which is why I'm not so sure that we should automatically close. I ask > > > everyone to close their PRs, and I manually double-check PRs remained > > > opened. > > > > > > The third need is to automatically tests all opened PRs and provide visas > > > to every PR we have. In case we have PRs with 0 blockers we should take > > it > > > into review process. No all newcomers aware of TC Bot, so I would like to > > > automate this process as much as possible. > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > чт, 25 июл. 2019 г. в 12:22, Alexey Zinoviev : > > > > > > > The long period totally reduces the discontent and outrage of community > > > > members (if you reduce to 2-6 weeks it could be intersected with human > > > > events of most part of contributors like vacation, birthdays, wedding, > > spam > > > > filters and etc.), believe me (I have the same experience as I > > mentioned) > > > > > > > > From the other hand, what the real reason to reduce it to the shorter > > > > period? Bot needs? Robot needs? > > > > Robot could wait, I hope:) > > > > > > > > чт, 25 июл. 2019 г. в 14:08, Павлухин Иван : > > > > > > > > > Alexey, > > > > > > > > > > Yep, I imagined a similar procedure in my mind. Just curious, why do > > > > > you think that a period before actions are taken should be so long > > > > > (3-6 months)?. > > > > > > > > > > чт, 25 июл. 2019 г. в 11:55, Alexey Zinoviev > >: > > > > > > > > > > > > Dear Igniters, I have one suggestion > > > > > > > > > > > > If a most of commiters will support idea of automatic "cleaning", > > we > > > > > should > > > > > > provide next options > > > > > > > > > > > >- declare a long period for putting labels or leaving comments > > for > > > > > >useful PRs from their authors (about 3-6 months) > > > > > >- generate notifications for all authors of all PRs with > > > > clarification > > > > > >of our goals > > > > > >- every month reminder in dev-list and via e-mail to each PR's > > > > author > > > > > > > > > > > > The best way, of course, the closing by our hands in each module > > and > > > > area > > > > > > with tags "obsolete" or something else. > > > > > > > > > > > > P.S. I was in the same situation in Open Street Map community and > > the > > > > > > principles for automated cleaning were the same like suggested by > > > > myself > > > > > > above > > > > > > > > > > > > I hope that we will be careful with our community > > > > > > > > > > > > чт, 25 июл. 2019 г. в 13:23, Dmitriy Pavlov : > > > > > > > > > > > > > Nikolay, committer could after setting up a link between GH & > > Apache > > > > > > > accounts. > > > > > > > https://gitbox
Re: Clean up of our PRs and IEPs before 2019
Alexey, Yep, I imagined a similar procedure in my mind. Just curious, why do you think that a period before actions are taken should be so long (3-6 months)?. чт, 25 июл. 2019 г. в 11:55, Alexey Zinoviev : > > Dear Igniters, I have one suggestion > > If a most of commiters will support idea of automatic "cleaning", we should > provide next options > >- declare a long period for putting labels or leaving comments for >useful PRs from their authors (about 3-6 months) >- generate notifications for all authors of all PRs with clarification >of our goals >- every month reminder in dev-list and via e-mail to each PR's author > > The best way, of course, the closing by our hands in each module and area > with tags "obsolete" or something else. > > P.S. I was in the same situation in Open Street Map community and the > principles for automated cleaning were the same like suggested by myself > above > > I hope that we will be careful with our community > > чт, 25 июл. 2019 г. в 13:23, Dmitriy Pavlov : > > > Nikolay, committer could after setting up a link between GH & Apache > > accounts. > > https://gitbox.apache.org/setup/ > > > > чт, 25 июл. 2019 г. в 11:17, Nikolay Izhikov : > > > > > Yes. > > > > > > Do someone have permission to close my(or any other contributor) PR to > > > apache/ignite? > > > > > > В Чт, 25/07/2019 в 11:05 +0300, Павлухин Иван пишет: > > > > NIkolay, > > > > > > > > Do you mean technical ability? > > > > > > > > чт, 25 июл. 2019 г. в 10:33, Nikolay Izhikov : > > > > > > > > > > Hello, Ivan. > > > > > > > > > > Do we have the ability to close PRs from other contributors? > > > > > > > > > > В Чт, 25/07/2019 в 09:12 +0300, Павлухин Иван пишет: > > > > > > Igniters, > > > > > > > > > > > > I would like to resume a discussion about PRs cleanup. > > Additionally > > > > > > to concerns provided earlier some TC Bot operations are slowed down > > > > > > due to a huge amount of open PRs. > > > > > > > > > > > > As time has passed, I ask you all again to share an opinion about > > > > > > centralized cleanup of obsolete PRs. Also, a precise criteria to > > > > > > consider PR as obsolete is a subject for dicsussion as well. > > > > > > > > > > > > чт, 13 дек. 2018 г. в 11:55, Petr Ivanov : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 11 Dec 2018, at 10:10, Nikolay Izhikov > > > > > wrote: > > > > > > > > > > > > > > > > Hello, Ivan. > > > > > > > > > > > > > > > > Personally, I keep my PR's clear. > > > > > > > > So, I don't have dozens of opened PR. > > > > > > > > > > > > > > > > But, I don't support Dmitriy proposal for several reasons: > > > > > > > > > > > > > > > > 1. We introduce some new, not required, level of bureaucracy. > > > > > > > > From my experience - not required bureaucracy is a BAD thing. > > > > > > > > > > > > > > > > 2. We spread our work pattern to whole community. > > > > > > > > I believe there are many patterns of dealing with *YOUR OWN* > > PRs. > > > > > > > > Some of them can lead to dozens of opened PRs to master. > > > > > > > > Whats wrong with it? > > > > > > > > > > > > > > > > 3. I dont' see any issues with many opened PRs. > > > > > > > > What problem we trying to solve? > > > > > > > > > > > > > > But I see. > > > > > > > Lots of opened PRs (and obsolete branches as well) consumes huge > > > amount of data and time when TC performs changes detect operations (every > > > minute, BTW). > > > > > > > Also, IMO, ORDER is not an unnecessary level of bureaucracy, but > > > part of the project development workflow in area of cleaning up and > > keeping > > > everything fresh and actual. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 4. Closi
Re: Clean up of our PRs and IEPs before 2019
NIkolay, Do you mean technical ability? чт, 25 июл. 2019 г. в 10:33, Nikolay Izhikov : > > Hello, Ivan. > > Do we have the ability to close PRs from other contributors? > > В Чт, 25/07/2019 в 09:12 +0300, Павлухин Иван пишет: > > Igniters, > > > > I would like to resume a discussion about PRs cleanup. Additionally > > to concerns provided earlier some TC Bot operations are slowed down > > due to a huge amount of open PRs. > > > > As time has passed, I ask you all again to share an opinion about > > centralized cleanup of obsolete PRs. Also, a precise criteria to > > consider PR as obsolete is a subject for dicsussion as well. > > > > чт, 13 дек. 2018 г. в 11:55, Petr Ivanov : > > > > > > > > > > > > > On 11 Dec 2018, at 10:10, Nikolay Izhikov wrote: > > > > > > > > Hello, Ivan. > > > > > > > > Personally, I keep my PR's clear. > > > > So, I don't have dozens of opened PR. > > > > > > > > But, I don't support Dmitriy proposal for several reasons: > > > > > > > > 1. We introduce some new, not required, level of bureaucracy. > > > > From my experience - not required bureaucracy is a BAD thing. > > > > > > > > 2. We spread our work pattern to whole community. > > > > I believe there are many patterns of dealing with *YOUR OWN* PRs. > > > > Some of them can lead to dozens of opened PRs to master. > > > > Whats wrong with it? > > > > > > > > 3. I dont' see any issues with many opened PRs. > > > > What problem we trying to solve? > > > > > > But I see. > > > Lots of opened PRs (and obsolete branches as well) consumes huge amount > > > of data and time when TC performs changes detect operations (every > > > minute, BTW). > > > Also, IMO, ORDER is not an unnecessary level of bureaucracy, but part of > > > the project development workflow in area of cleaning up and keeping > > > everything fresh and actual. > > > > > > > > > > > > > > 4. Closing abanodned PRs doesn't force anybody to review the rest. > > > > Instead of ordering something to one way or another, let's solve real > > > > problem: > > > > > > > > - help the community doing PR review. > > > > - fixing failing tests. > > > > - introducing new code inspections to make our code base clear. > > > > - making Ignite improvements > > > > > > > > 5. I don't see how our numbers differs from other Apache projects > > > > > > > > Apache Kafka - 533 PR opened. > > > > Apache Spark - 484 PR opened. > > > > Apache Flink - 430 PR opened. > > > > > > > > В Вт, 11/12/2018 в 09:24 +0300, Pavel Tupitsyn пишет: > > > > > Agree with Dmitriy. > > > > > > > > > > We use GitHub PRs in our workflow, therefore we should keep them in > > > > > order. > > > > > > > > > > We can close PRs that refer to closed tickets, this can be done with a > > > > > simple script. > > > > > > > > > > On Tue, Dec 11, 2018 at 9:15 AM Павлухин Иван > > > > > wrote: > > > > > > > > > > > Nikolay, > > > > > > > > > > > > I must say that when I first saw 1K+ open PRs my first thought was > > > > > > that something was wrong with a review process. In my mind in not > > > > > > very > > > > > > big project open PR list can reflect very well the real work in > > > > > > progress. For bigger projects things become more complicated. > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > Do you have some cleanup automation in mind? Immediately I think > > > > > > that > > > > > > it is fully safe to close all PRs that were not touched more than a > > > > > > year. > > > > > > пн, 10 дек. 2018 г. в 20:01, Dmitriy Pavlov : > > > > > > > > > > > > > > The main concern is related to chances that newcomer will have to > > > > > > > obtain > > > > > > > > > > > > a > > > > > > > review support from the community. > > > > > > > > > > > > > > Actually, a lot of people d
Re: Clean up of our PRs and IEPs before 2019
Igniters, I would like to resume a discussion about PRs cleanup. Additionally to concerns provided earlier some TC Bot operations are slowed down due to a huge amount of open PRs. As time has passed, I ask you all again to share an opinion about centralized cleanup of obsolete PRs. Also, a precise criteria to consider PR as obsolete is a subject for dicsussion as well. чт, 13 дек. 2018 г. в 11:55, Petr Ivanov : > > > > > On 11 Dec 2018, at 10:10, Nikolay Izhikov wrote: > > > > Hello, Ivan. > > > > Personally, I keep my PR's clear. > > So, I don't have dozens of opened PR. > > > > But, I don't support Dmitriy proposal for several reasons: > > > > 1. We introduce some new, not required, level of bureaucracy. > > From my experience - not required bureaucracy is a BAD thing. > > > > 2. We spread our work pattern to whole community. > > I believe there are many patterns of dealing with *YOUR OWN* PRs. > > Some of them can lead to dozens of opened PRs to master. > > Whats wrong with it? > > > > 3. I dont' see any issues with many opened PRs. > > What problem we trying to solve? > > But I see. > Lots of opened PRs (and obsolete branches as well) consumes huge amount of > data and time when TC performs changes detect operations (every minute, BTW). > Also, IMO, ORDER is not an unnecessary level of bureaucracy, but part of the > project development workflow in area of cleaning up and keeping everything > fresh and actual. > > > > > > 4. Closing abanodned PRs doesn't force anybody to review the rest. > > Instead of ordering something to one way or another, let's solve real > > problem: > > > > - help the community doing PR review. > > - fixing failing tests. > > - introducing new code inspections to make our code base clear. > > - making Ignite improvements > > > > 5. I don't see how our numbers differs from other Apache projects > > > > Apache Kafka - 533 PR opened. > > Apache Spark - 484 PR opened. > > Apache Flink - 430 PR opened. > > > > В Вт, 11/12/2018 в 09:24 +0300, Pavel Tupitsyn пишет: > >> Agree with Dmitriy. > >> > >> We use GitHub PRs in our workflow, therefore we should keep them in order. > >> > >> We can close PRs that refer to closed tickets, this can be done with a > >> simple script. > >> > >> On Tue, Dec 11, 2018 at 9:15 AM Павлухин Иван wrote: > >> > >>> Nikolay, > >>> > >>> I must say that when I first saw 1K+ open PRs my first thought was > >>> that something was wrong with a review process. In my mind in not very > >>> big project open PR list can reflect very well the real work in > >>> progress. For bigger projects things become more complicated. > >>> > >>> Dmitriy, > >>> > >>> Do you have some cleanup automation in mind? Immediately I think that > >>> it is fully safe to close all PRs that were not touched more than a > >>> year. > >>> пн, 10 дек. 2018 г. в 20:01, Dmitriy Pavlov : > >>>> > >>>> The main concern is related to chances that newcomer will have to obtain > >>> > >>> a > >>>> review support from the community. > >>>> > >>>> Actually, a lot of people doing their best to provide a feedback to > >>>> newcomers, and count of issues still in PA state goes down (84 is a > >>>> relatively small count of issues in PA state). But 1428 PRs may imply we > >>>> don't review here, as we have tons of incomplete PRs. Actually, most of > >>>> these PRs were merged (but not using ./apply-pull-request.sh script, but > >>>> manually, without reference to PRs). > >>>> > >>>> Another benefit of revising this list, if there are any changes which > >>>> were not accomplished with a proper ticket with PA status, we will > >>> > >>> identify > >>>> a number of additional contributions to be applied to the codebase. > >>>> > >>>> > >>>> пн, 10 дек. 2018 г. в 19:53, Nikolay Izhikov : > >>>> > >>>>> Hello, Dmitriy. > >>>>> > >>>>> What, exactly concerns newcomers? > >>>>> What is wrong with opened PR? > >>>>> How project will benefit from closed PR? > >>>>> > >>>>>> The same proposal is related to IEP statuses. If you were involved > >
Re: [TC] Move "Queries (Binary Objects Simple Mapper)" job to nightly
Also there are more similar candidates: * "Binary Objects (Simple Mapper Basic)" [1] corresponds to "Basic 1" [2] * "Binary Objects (Simple Mapper Cache Full API)" [3] -- "Cache (Full API)" [4] * "Binary Objects (Simple Mapper Compute Grid)" [5] -- "Compute (Grid)" [6] I think they could be moved to nightly as well. Also renaming sounds as a good idea because current names are misleading. [1] https://ci.ignite.apache.org/admin/editBuildParams.html?id=buildType:IgniteTests24Java8_BinaryObjectsSimpleMapperBasic [2] https://ci.ignite.apache.org/admin/editBuildParams.html?id=buildType:IgniteTests24Java8_Basic1 [3] https://ci.ignite.apache.org/admin/editBuildParams.html?id=buildType:IgniteTests24Java8_BinaryObjectsSimpleMapperCacheFullApi [4] https://ci.ignite.apache.org/admin/editBuildParams.html?id=buildType:IgniteTests24Java8_CacheFullApi [5] https://ci.ignite.apache.org/admin/editBuildParams.html?id=buildType:IgniteTests24Java8_BinaryObjectsSimpleMapperComputeGrid [6] https://ci.ignite.apache.org/admin/editBuildParams.html?id=buildType:IgniteTests24Java8_ComputeGrid пн, 22 июл. 2019 г. в 14:57, Dmitriy Pavlov : > > +1 for moving from RunAll to RunAllNighlty > > пн, 22 июл. 2019 г. в 12:21, Павлухин Иван : > > > Igniters, > > > > As you know Ignite RunAll on TC takes significant resources. I noticed > > that build job "Queries (Binary Objects Simple Mapper)" [1] actually > > duplicates "Queries 1" [2] job and the same test set using simple name > > mapper for binary objects. I suppose that we can exclude it from daily > > RunAll and move to nightly. > > > > What do you think? > > > > [1] > > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BinaryObjectsSimpleMapperQueries?branch=%3Cdefault%3E=overview > > [2] > > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Queries1?branch=%3Cdefault%3E=overview > > > > -- > > Best regards, > > Ivan Pavlukhin > > -- Best regards, Ivan Pavlukhin
[TC] Move "Queries (Binary Objects Simple Mapper)" job to nightly
Igniters, As you know Ignite RunAll on TC takes significant resources. I noticed that build job "Queries (Binary Objects Simple Mapper)" [1] actually duplicates "Queries 1" [2] job and the same test set using simple name mapper for binary objects. I suppose that we can exclude it from daily RunAll and move to nightly. What do you think? [1] https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BinaryObjectsSimpleMapperQueries?branch=%3Cdefault%3E=overview [2] https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Queries1?branch=%3Cdefault%3E=overview -- Best regards, Ivan Pavlukhin
Re: [DISCUSSION] Ignite 3.0 and to be removed list
Alex, I already added a couple of items to wishlist [1]. Yes, I agree that the process should be iterative. But I am confused on what stage we are in a current interation? I suppose that Denis is going to present a list of removal candidates which we as developers agreed on. And should not we have that list already available somewhere as a document? Now I see an infromation scattered in this thread and the wishlist [1]. And it is not easy to me to realize where we are now. [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist чт, 18 июл. 2019 г. в 18:14, Alexey Goncharuk : > > Ivan, > > The list is not final, we can still discuss and add more points to be > cleaned in 3.0. The more clear and understandable the API will be, the > better. This thread was intended to draft the removal scope for 3.0 and to > understand which portions will be definitely removed. > > > ср, 17 июл. 2019 г. в 15:26, Павлухин Иван : > > > Also, I did not quite get the point about JSR107 (JCache). From time > > to time I see on user-list threads where Ignite is used along with > > Spring annotation-based cache integration. I suppose it requires > > JCache interfaces. What is crucially wrong with supporting it? > > > > ср, 17 июл. 2019 г. в 15:19, Павлухин Иван : > > > > > > Folks, > > > > > > Sorry if I am repeating something. I checked a page [1] and have not > > > found several items. > > > 1. I thought that there was an agreement of dropping OLD service grid, > > > was not it? > > > 2. Also IndexingSpi seems to me as a candidate for removal. > > > > > > Should I add those items to the page? Or is there another page > > > containing items to be removed that we agreed on? > > > > > > [1] > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist > > > > > > ср, 17 июл. 2019 г. в 02:00, Denis Magda : > > > > > > > > Alex, Igniters, sorry for a delay. Got swamped with other duties. > > > > > > > > Does it wait till the next week? I'll make sure to dedicate some time > > for > > > > that. Or if we'd like to run faster then I'll appreciate if someone > > else > > > > steps in and prepares a list this week. I'll help to review and > > solidify it. > > > > > > > > - > > > > Denis > > > > > > > > > > > > On Tue, Jul 16, 2019 at 7:58 AM Alexey Goncharuk < > > alexey.goncha...@gmail.com> > > > > wrote: > > > > > > > > > Denis, > > > > > > > > > > Are we ready to present the list to the user list? > > > > > > > > > > вт, 2 июл. 2019 г. в 00:27, Denis Magda : > > > > > > > > > > > I wouldn't kick off dozens of voting discussions. Instead, the > > content on > > > > > > the wiki page needs to be cleaned and rearranged. This will make > > the > > > > > > content readable and comprehensible. I can do that. > > > > > > > > > > > > Next, let's ask the user community for an opinion. After reviewing > > and > > > > > > incorporating the latter we can do one more dev list discussion > > with the > > > > > > last call for opinions. Next, will be the voting time. If there is > > a > > > > > > feature someone from the dev list is against of removing, then we > > can > > > > > start > > > > > > a separate vote for it later. But let's get into those cases first. > > > > > > > > > > > > - > > > > > > Denis > > > > > > > > > > > > > > > > > > On Mon, Jul 1, 2019 at 1:47 AM Dmitriy Pavlov > > > > > wrote: > > > > > > > > > > > > > I propose each removal should have separated formal vote thread > > with > > > > > > > consensus approval (since it is code modification). > > > > > > > > > > > > > > This means a single binding objection with justification is a > > blocker > > > > > for > > > > > > > removal. > > > > > > > > > > > > > > We need separation to let community members pick up an > > interesting > > > > > topic > > > > > > > from email subject. Not all members reading carefully each post > > in > > > > >
Re: Read Repair (ex. Consistency Check) - review request #2
Anton, No doubts, just a lack of attention from my side. Thank you for clarifying. Regarding MVCC, it's Beta state is mentioned in documentation [1] (a warning badge in the beginning). And I think it is a really good idea to employ some means of delivering some experimental stuff. It can be somehow similar to feature toggling widely used today. I can imagine a following process: 1. A non-trivial feature development is started and all related stuff is marked as an experimental in a way transaparent for everybody. 2. The development continues and the feature can be merged to master by chunks when some parts are ready. 3. It is possible to have such feature in releases in beta/experiment state to gather a feedback from users. 4. Once the feature is completely ready (including documentation) the experiment mark is removed. [1] https://apacheignite.readme.io/docs/multiversion-concurrency-control пт, 19 июл. 2019 г. в 08:31, Anton Vinogradov : > > Ivan, > > My code is always production-ready, any doubts? :) > > Anyway, that's a good question. > It's a good case to mark feature experimental somehow while it's roadmap > not finished. > Was MVCC marked as experimental? > API can be marked as Experimental [1] since java 9, but we're not able to > use this annotation for now. > > Anyway again, seems it depends on the release date will RR be marked as > experimental or not. > In case we'll finish roadmap before 2.8 release there will be no need for > such mark. > So, let's assume it's production-ready until another state is not confirmed > :) > > [1] https://docs.oracle.com/javase/9/docs/api/jdk/jfr/Experimental.html > > On Thu, Jul 18, 2019 at 4:56 PM Павлухин Иван wrote: > > > Folks, > > > > Sorry, I was not very attentive. Could you please clarify whether (or > > not) this feature is currently in an experiment state? And a couple of > > obvious things: > > 1. If it is experimental it should be clear for a user (e.g. from javadoc). > > 2. If not all limitations should be described in javadocs and > > documentation. > > > > ср, 17 июл. 2019 г. в 14:59, Вячеслав Коптилин : > > > > > > Hello Anton, > > > > > > It's not possible, currently, to fix atomic caches. > > > > You may only check the consistency. *And it's better than nothing*, I > > > > think. > > > > > > Fair enough. > > > > > > >> 3. IgniteConsistencyViolationException is absolutely useless. It does > > not > > > > >> provide any information about the issue and possible way to fix it. > > > > It means that some keys from your get operation are broken. > > > > IgniteConsistencyViolationException CAN be extended with a list of > > broken > > > > keys in the future. > > > > > > I think it SHOULD be extended with additional fields/methods in the same > > > way as CacheConsistencyViolationEvent > > > > > > >> Well, near caches are widely used and fully transactional, so I think > > it > > > > >> makes sense to support the feature for near caches too. > > > > As I told before, it will be nice to implement this in the future, but > > we > > > > have more important tasks for now. > > > > > > I do not insist that it must be done right now. > > > > > > > > > > >> For instance, I would like to see all these limitations on the IEP > > page > > > > as > > > > >> JIRA tickets. Perhaps, it would be good to create an epic/umbrella > > > > ticket > > > > >> in order to track all activities related to `Read Repair` feature. > > > > Let's do this at merge day to be sure useless issues will not be > > created. > > > > > > > > > > I am just trying to say that it is a good time to create tickets in order > > > to track all of that otherwise the chance that all these > > > limitations/improvements will not be addressed is very high. > > > > > > Thanks, > > > S. > > > > > > вт, 16 июл. 2019 г. в 09:07, Anton Vinogradov : > > > > > > > Svala, > > > > > > > > >> Could you please take a look at PR: > > > > Going to review today, thanks for attaching the bot visa. > > > > > > > > >> 1. Should I consider that my cluster is broken? There is no answer! > > The > > > > >> false-positive result is possible. > > > > That's a question about atomic nature. > > > > It's not impossible to lock atomic entry to perform the check
Re: Read Repair (ex. Consistency Check) - review request #2
Folks, Sorry, I was not very attentive. Could you please clarify whether (or not) this feature is currently in an experiment state? And a couple of obvious things: 1. If it is experimental it should be clear for a user (e.g. from javadoc). 2. If not all limitations should be described in javadocs and documentation. ср, 17 июл. 2019 г. в 14:59, Вячеслав Коптилин : > > Hello Anton, > > It's not possible, currently, to fix atomic caches. > > You may only check the consistency. *And it's better than nothing*, I > > think. > > Fair enough. > > >> 3. IgniteConsistencyViolationException is absolutely useless. It does not > > >> provide any information about the issue and possible way to fix it. > > It means that some keys from your get operation are broken. > > IgniteConsistencyViolationException CAN be extended with a list of broken > > keys in the future. > > I think it SHOULD be extended with additional fields/methods in the same > way as CacheConsistencyViolationEvent > > >> Well, near caches are widely used and fully transactional, so I think it > > >> makes sense to support the feature for near caches too. > > As I told before, it will be nice to implement this in the future, but we > > have more important tasks for now. > > I do not insist that it must be done right now. > > > > >> For instance, I would like to see all these limitations on the IEP page > > as > > >> JIRA tickets. Perhaps, it would be good to create an epic/umbrella > > ticket > > >> in order to track all activities related to `Read Repair` feature. > > Let's do this at merge day to be sure useless issues will not be created. > > > > I am just trying to say that it is a good time to create tickets in order > to track all of that otherwise the chance that all these > limitations/improvements will not be addressed is very high. > > Thanks, > S. > > вт, 16 июл. 2019 г. в 09:07, Anton Vinogradov : > > > Svala, > > > > >> Could you please take a look at PR: > > Going to review today, thanks for attaching the bot visa. > > > > >> 1. Should I consider that my cluster is broken? There is no answer! The > > >> false-positive result is possible. > > That's a question about atomic nature. > > It's not impossible to lock atomic entry to perform the check. > > You should perform some attempts, it's your decision how many. > > By default, atomic RR performs 3 attempts, you may increase this by setting > > IGNITE_NEAR_GET_MAX_REMAPS or by just performing additional gets. > > > > >> 2. What should be done here in order to check/resolve the issue? > > Perhaps, I > > >> should restart a node (which one?), restart the whole cluster, put a new > > >> value... > > It's not possible, currently, to fix atomic caches. > > You may only check the consistency. And it's better than nothing, I think. > > We should find a way how to fix atomic consistency first. > > A possible strategy is to use ЕntryProcessor which will replace all owner's > > values with "latest" and do nothing in case newest (than latest) value > > found (opposite to preloading approach). > > > > >> 3. IgniteConsistencyViolationException is absolutely useless. It does > > not > > >> provide any information about the issue and possible way to fix it. > > It means that some keys from your get operation are broken. > > IgniteConsistencyViolationException CAN be extended with a list of broken > > keys in the future. > > > > >> It seems that transactional caches are covered much better. > > Correct. > > Tx caches consistency is more important that atomic consistency, that's why > > it was implemented first. > > BTW, AFAIK, atomics also were not fixed at 10078 [1]. > > > > >> Well, near caches are widely used and fully transactional, so I think it > > >> makes sense to support the feature for near caches too. > > As I told before, it will be nice to implement this in the future, but we > > have more important tasks for now. > > The main goal was to cover tx caches, to be able to fix them in case of the > > real problem at production. > > > > Summarizing the roadmap, > > My goal now is to finish the tx case, now we have an issue with false > > positive consistency violation [2]. > > Also, we're going to update Jepsen tests [3] with RR to ensure tx caches > > fixed. > > Next main goal is to use RR at TC checks [4], help with this issue are > > appreciated. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10078 > > [2] https://issues.apache.org/jira/browse/IGNITE-11973 > > [3] https://issues.apache.org/jira/browse/IGNITE-11972 > > [4] https://issues.apache.org/jira/browse/IGNITE-11971 > > > > > > On Mon, Jul 15, 2019 at 4:51 PM Dmitriy Pavlov wrote: > > > > > Ok, thank you > > > > > > пн, 15 июл. 2019 г., 16:46 Nikolay Izhikov : > > > > > > > I did the review. > > > > > > > > пн, 15 июля 2019 г., 16:15 Dmitriy Pavlov : > > > > > > > > > Igniters, who did a review of > > > > > https://issues.apache.org/jira/browse/IGNITE-10663 before the merge? > > > > I've > > > > > checked both PR
Re: Removal of TC Run Configuration PdsIndexingWalRecovery
FYI, Build configuration PdsIndexingWalRecovery was removed. ср, 17 июл. 2019 г. в 14:31, Павлухин Иван : > > Yet another candidate > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_QueriesOom?branch=pull%2F4611%2Fhead=overview > > ср, 17 июл. 2019 г. в 14:06, Павлухин Иван : > > > > Here is a closed PR [1]. Let's drop that run-config [2]. > > > > [1] https://github.com/apache/ignite/pull/4239/files > > [2] > > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PdsIndexingWalRecovery?branch=%3Cdefault%3E=overview > > > > чт, 21 мар. 2019 г. в 17:53, Petr Ivanov : > > > > > > > > According to git log, test was removed (or renamed) somewhere here [1], > > > but there is no hard evidence. > > > AFAIK, that is only possible if merging branch where this file did not > > > exist with branch where it was added first time — new files do not staged > > > automatically. > > > > > > > > > [1] > > > https://github.com/apache/ignite/commit/2ebe7b3e0d2d9671feffe44319d5b2dc743e5837 > > > > > > > On 21 Mar 2019, at 17:03, Dmitriy Pavlov wrote: > > > > > > > > Hi Igniters, > > > > > > > > TC Run config > > > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PdsIndexingWalRecovery_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv > > > > > > > > > > > > references to non-existent suite class > > > > IgnitePdsWithIndexingWalAndRecoveryTestSuite > > > > > > > > Please share information if you know how it is named now. > > > > > > > > Otherwise, I suggest deleting run-config after 3 days. > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > > > > -- > > Best regards, > > Ivan Pavlukhin > > > > -- > Best regards, > Ivan Pavlukhin -- Best regards, Ivan Pavlukhin
Re: [DISCUSSION] Ignite 3.0 and to be removed list
Also, I did not quite get the point about JSR107 (JCache). From time to time I see on user-list threads where Ignite is used along with Spring annotation-based cache integration. I suppose it requires JCache interfaces. What is crucially wrong with supporting it? ср, 17 июл. 2019 г. в 15:19, Павлухин Иван : > > Folks, > > Sorry if I am repeating something. I checked a page [1] and have not > found several items. > 1. I thought that there was an agreement of dropping OLD service grid, > was not it? > 2. Also IndexingSpi seems to me as a candidate for removal. > > Should I add those items to the page? Or is there another page > containing items to be removed that we agreed on? > > [1] > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist > > ср, 17 июл. 2019 г. в 02:00, Denis Magda : > > > > Alex, Igniters, sorry for a delay. Got swamped with other duties. > > > > Does it wait till the next week? I'll make sure to dedicate some time for > > that. Or if we'd like to run faster then I'll appreciate if someone else > > steps in and prepares a list this week. I'll help to review and solidify it. > > > > - > > Denis > > > > > > On Tue, Jul 16, 2019 at 7:58 AM Alexey Goncharuk > > > > wrote: > > > > > Denis, > > > > > > Are we ready to present the list to the user list? > > > > > > вт, 2 июл. 2019 г. в 00:27, Denis Magda : > > > > > > > I wouldn't kick off dozens of voting discussions. Instead, the content > > > > on > > > > the wiki page needs to be cleaned and rearranged. This will make the > > > > content readable and comprehensible. I can do that. > > > > > > > > Next, let's ask the user community for an opinion. After reviewing and > > > > incorporating the latter we can do one more dev list discussion with the > > > > last call for opinions. Next, will be the voting time. If there is a > > > > feature someone from the dev list is against of removing, then we can > > > start > > > > a separate vote for it later. But let's get into those cases first. > > > > > > > > - > > > > Denis > > > > > > > > > > > > On Mon, Jul 1, 2019 at 1:47 AM Dmitriy Pavlov > > > wrote: > > > > > > > > > I propose each removal should have separated formal vote thread with > > > > > consensus approval (since it is code modification). > > > > > > > > > > This means a single binding objection with justification is a blocker > > > for > > > > > removal. > > > > > > > > > > We need separation to let community members pick up an interesting > > > topic > > > > > from email subject. Not all members reading carefully each post in > > > > > mile-long threads. > > > > > > > > > > пн, 1 июл. 2019 г. в 11:17, Anton Vinogradov : > > > > > > > > > > > +1 to email survey with following types of votes > > > > > > - silence (agree with all proposed removals) > > > > > > - we have to keep XXX because ... > > > > > > > > > > > > As a result, will gain lists > > > > > > "to be removed" - no one objected > > > > > > "can be removed" - single objection > > > > > > "should be kept" - multi objections > > > > > > > > > > > > Denis or Dmitry Pavlov, could you please lead this thread? > > > > > > > > > > > > On Sat, Jun 29, 2019 at 12:27 AM Denis Magda > > > > wrote: > > > > > > > > > > > > > Alex, > > > > > > > > > > > > > > I would do an email survey to hear an opinion of why someone > > > > believes a > > > > > > > feature A has to stay. It makes sense to ask about the APIs to be > > > > > removed > > > > > > > as well as integrations to go out of community support [1] in the > > > > same > > > > > > > thread. > > > > > > > > > > > > > > Has everyone expressed an opinion? If yes, I can go ahead and > > > format > > > > > the > > > > > > > wishlist page and make it structured for the user thread. > > > > > > > > > > > > > > [1] > > > > > > >
Re: [DISCUSSION] Ignite 3.0 and to be removed list
Folks, Sorry if I am repeating something. I checked a page [1] and have not found several items. 1. I thought that there was an agreement of dropping OLD service grid, was not it? 2. Also IndexingSpi seems to me as a candidate for removal. Should I add those items to the page? Or is there another page containing items to be removed that we agreed on? [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist ср, 17 июл. 2019 г. в 02:00, Denis Magda : > > Alex, Igniters, sorry for a delay. Got swamped with other duties. > > Does it wait till the next week? I'll make sure to dedicate some time for > that. Or if we'd like to run faster then I'll appreciate if someone else > steps in and prepares a list this week. I'll help to review and solidify it. > > - > Denis > > > On Tue, Jul 16, 2019 at 7:58 AM Alexey Goncharuk > wrote: > > > Denis, > > > > Are we ready to present the list to the user list? > > > > вт, 2 июл. 2019 г. в 00:27, Denis Magda : > > > > > I wouldn't kick off dozens of voting discussions. Instead, the content on > > > the wiki page needs to be cleaned and rearranged. This will make the > > > content readable and comprehensible. I can do that. > > > > > > Next, let's ask the user community for an opinion. After reviewing and > > > incorporating the latter we can do one more dev list discussion with the > > > last call for opinions. Next, will be the voting time. If there is a > > > feature someone from the dev list is against of removing, then we can > > start > > > a separate vote for it later. But let's get into those cases first. > > > > > > - > > > Denis > > > > > > > > > On Mon, Jul 1, 2019 at 1:47 AM Dmitriy Pavlov > > wrote: > > > > > > > I propose each removal should have separated formal vote thread with > > > > consensus approval (since it is code modification). > > > > > > > > This means a single binding objection with justification is a blocker > > for > > > > removal. > > > > > > > > We need separation to let community members pick up an interesting > > topic > > > > from email subject. Not all members reading carefully each post in > > > > mile-long threads. > > > > > > > > пн, 1 июл. 2019 г. в 11:17, Anton Vinogradov : > > > > > > > > > +1 to email survey with following types of votes > > > > > - silence (agree with all proposed removals) > > > > > - we have to keep XXX because ... > > > > > > > > > > As a result, will gain lists > > > > > "to be removed" - no one objected > > > > > "can be removed" - single objection > > > > > "should be kept" - multi objections > > > > > > > > > > Denis or Dmitry Pavlov, could you please lead this thread? > > > > > > > > > > On Sat, Jun 29, 2019 at 12:27 AM Denis Magda > > > wrote: > > > > > > > > > > > Alex, > > > > > > > > > > > > I would do an email survey to hear an opinion of why someone > > > believes a > > > > > > feature A has to stay. It makes sense to ask about the APIs to be > > > > removed > > > > > > as well as integrations to go out of community support [1] in the > > > same > > > > > > thread. > > > > > > > > > > > > Has everyone expressed an opinion? If yes, I can go ahead and > > format > > > > the > > > > > > wishlist page and make it structured for the user thread. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-Modularization-td42486.html > > > > > > - > > > > > > Denis > > > > > > > > > > > > > > > > > > On Fri, Jun 28, 2019 at 8:54 AM Alexey Goncharuk < > > > > > > alexey.goncha...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Anton, good point. > > > > > > > > > > > > > > Do you have any idea how we can keep track of the voting? Should > > we > > > > > > launch > > > > > > > a google survey or survey monkey? Voting by email? > > > > > > > > > > > > > > пт, 28 июн. 2019 г. в 11:24, Anton Vinogradov : > > > > > > > > > > > > > > > Alexey, > > > > > > > > > > > > > > > > Thank's for keeping an eye on page updates. > > > > > > > > Near Caches is not a bad feature, but it should be used with > > > > caution. > > > > > > > > At least we have to explain how it works on readme.io, why and > > > > when > > > > > it > > > > > > > > should be used because usage can drop the performance instead > > of > > > > > > > increasing > > > > > > > > it. > > > > > > > > > > > > > > > > Anyway, I added near caches because I never heard someone used > > > them > > > > > > > > meaningfully, not like a silver bullet. > > > > > > > > So, that's just a proposal :) > > > > > > > > > > > > > > > > Also, I'd like to propose to have some voting about full list > > > later > > > > > to > > > > > > > gain > > > > > > > > "must be removed", "can be removed" and "should be kept" lists. > > > > > > > > > > > > > > > > On Thu, Jun 27, 2019 at 1:03 PM Alexey Goncharuk < > > > > > > > > alexey.goncha...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Anton, > > > > > > > > > > > > > > > > > > I would like to
Re: Removal of TC Run Configuration PdsIndexingWalRecovery
Yet another candidate https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_QueriesOom?branch=pull%2F4611%2Fhead=overview ср, 17 июл. 2019 г. в 14:06, Павлухин Иван : > > Here is a closed PR [1]. Let's drop that run-config [2]. > > [1] https://github.com/apache/ignite/pull/4239/files > [2] > https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PdsIndexingWalRecovery?branch=%3Cdefault%3E=overview > > чт, 21 мар. 2019 г. в 17:53, Petr Ivanov : > > > > > According to git log, test was removed (or renamed) somewhere here [1], but > > there is no hard evidence. > > AFAIK, that is only possible if merging branch where this file did not > > exist with branch where it was added first time — new files do not staged > > automatically. > > > > > > [1] > > https://github.com/apache/ignite/commit/2ebe7b3e0d2d9671feffe44319d5b2dc743e5837 > > > > > On 21 Mar 2019, at 17:03, Dmitriy Pavlov wrote: > > > > > > Hi Igniters, > > > > > > TC Run config > > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PdsIndexingWalRecovery_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv > > > > > > > > > references to non-existent suite class > > > IgnitePdsWithIndexingWalAndRecoveryTestSuite > > > > > > Please share information if you know how it is named now. > > > > > > Otherwise, I suggest deleting run-config after 3 days. > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > -- > Best regards, > Ivan Pavlukhin -- Best regards, Ivan Pavlukhin
Re: Removal of TC Run Configuration PdsIndexingWalRecovery
Here is a closed PR [1]. Let's drop that run-config [2]. [1] https://github.com/apache/ignite/pull/4239/files [2] https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PdsIndexingWalRecovery?branch=%3Cdefault%3E=overview чт, 21 мар. 2019 г. в 17:53, Petr Ivanov : > > According to git log, test was removed (or renamed) somewhere here [1], but > there is no hard evidence. > AFAIK, that is only possible if merging branch where this file did not exist > with branch where it was added first time — new files do not staged > automatically. > > > [1] > https://github.com/apache/ignite/commit/2ebe7b3e0d2d9671feffe44319d5b2dc743e5837 > > > On 21 Mar 2019, at 17:03, Dmitriy Pavlov wrote: > > > > Hi Igniters, > > > > TC Run config > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_PdsIndexingWalRecovery_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv > > > > > > references to non-existent suite class > > IgnitePdsWithIndexingWalAndRecoveryTestSuite > > > > Please share information if you know how it is named now. > > > > Otherwise, I suggest deleting run-config after 3 days. > > > > Sincerely, > > Dmitriy Pavlov > -- Best regards, Ivan Pavlukhin
Empty TC job PDS (Indexing / WAL & Recovery)
Hi, Does anyone know what is the purpose of job PDS (Indexing / WAL & Recovery) [1]? It refers to non-existing IgnitePdsWithIndexingWalAndRecoveryTestSuite class and runs no tests. Can we delete it safely? [1] https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PdsIndexingWalRecovery?branch=%3Cdefault%3E=overview -- Best regards, Ivan Pavlukhin
Re: Tx lock partial happens before
Anton,
Thank you for clarification.
вт, 16 июл. 2019 г. в 09:24, Anton Vinogradov :
>
> Ivan R.
>
> Thanks.
> I'll try to implement approach you proposed.
>
> Ivan P.
>
> >> what prevents primary partition relocation when
> >> Read Repair is in progress? Is there a transaction or an exlicit lock?
> Did you mean partition eviction?
> RR is almost a regular get with the same logic. It maps on some topology
> and performs regular gets.
> In case node failed or is not an owner anymore we'll just ignore this.
> See the code for details:
>
> if (invalidNodeSet.contains(affNode) || !cctx.discovery().alive(affNode)) {
> onDone(Collections.emptyMap()); // Finishing mini future with just "ok".
>
> On Tue, Jul 16, 2019 at 9:04 AM Павлухин Иван wrote:
>
> > Anton,
> >
> > You referenced to failover scenarios. I believe that everything is
> > described in IEP. But to make this discussion self-sufficient could
> > you please outline what prevents primary partition relocation when
> > Read Repair is in progress? Is there a transaction or an exlicit lock?
> >
> > пн, 15 июл. 2019 г. в 23:49, Ivan Rakov :
> > >
> > > Anton,
> > >
> > > > Step-by-step:
> > > > 1) primary locked on key mention (get/put) at
> > pessimistic/!read-committed tx
> > > > 2) backups locked on prepare
> > > > 3) primary unlocked on finish
> > > > 4) backups unlocked on finish (after the primary)
> > > > correct?
> > > Yes, this corresponds to my understanding of transactions protocol. With
> > > minor exception: steps 3 and 4 are inverted in case of one-phase commit.
> > >
> > > > Agree, but seems there is no need to acquire the lock, we have just to
> > wait
> > > > until entry becomes unlocked.
> > > > - entry locked means that previous tx's "finish" phase is in progress
> > > > - entry unlocked means reading value is up-to-date (previous "finish"
> > phase
> > > > finished)
> > > > correct?
> > > Diving deeper, entry is locked if its GridCacheMapEntry.localCandidates
> > > queue is not empty (first item in queue is actually the transaction that
> > > owns lock).
> > >
> > > > we have just to wait
> > > > until entry becomes unlocked.
> > > This may work.
> > > If consistency checking code has acquired lock on primary, backup can be
> > > in two states:
> > > - not locked - and new locks won't appear as we are holding lock on
> > primary
> > > - still locked by transaction that owned lock on primary just before our
> > > checking code - in such case checking code should just wait for lock
> > release
> > >
> > > Best Regards,
> > > Ivan Rakov
> > >
> > > On 15.07.2019 9:34, Anton Vinogradov wrote:
> > > > Ivan R.
> > > >
> > > > Thanks for joining!
> > > >
> > > > Got an idea, but not sure that got a way of a fix.
> > > >
> > > > AFAIK (can be wrong, please correct if necessary), at 2PC, locks are
> > > > acquired on backups during the "prepare" phase and released at "finish"
> > > > phase after primary fully committed.
> > > > Step-by-step:
> > > > 1) primary locked on key mention (get/put) at
> > pessimistic/!read-committed tx
> > > > 2) backups locked on prepare
> > > > 3) primary unlocked on finish
> > > > 4) backups unlocked on finish (after the primary)
> > > > correct?
> > > >
> > > > So, acquiring locks on backups, not at the "prepare" phase, may cause
> > > > unexpected behavior in case of primary fail or other errors.
> > > > That's definitely possible to update failover to solve this issue, but
> > it
> > > > seems to be an overcomplicated way.
> > > > The main question there, it there any simple way?
> > > >
> > > >>> checking read from backup will just wait for commit if it's in
> > progress.
> > > > Agree, but seems there is no need to acquire the lock, we have just to
> > wait
> > > > until entry becomes unlocked.
> > > > - entry locked means that previous tx's "finish" phase is in progress
> > > > - entry unlocked means reading value is up-to-date (previous "finish"
> > phase
> > > > finished)
> > > > correct?
> > > >
> > > > On Mon, Jul 15, 2019
Re: Tx lock partial happens before
Anton, You referenced to failover scenarios. I believe that everything is described in IEP. But to make this discussion self-sufficient could you please outline what prevents primary partition relocation when Read Repair is in progress? Is there a transaction or an exlicit lock? пн, 15 июл. 2019 г. в 23:49, Ivan Rakov : > > Anton, > > > Step-by-step: > > 1) primary locked on key mention (get/put) at pessimistic/!read-committed tx > > 2) backups locked on prepare > > 3) primary unlocked on finish > > 4) backups unlocked on finish (after the primary) > > correct? > Yes, this corresponds to my understanding of transactions protocol. With > minor exception: steps 3 and 4 are inverted in case of one-phase commit. > > > Agree, but seems there is no need to acquire the lock, we have just to wait > > until entry becomes unlocked. > > - entry locked means that previous tx's "finish" phase is in progress > > - entry unlocked means reading value is up-to-date (previous "finish" phase > > finished) > > correct? > Diving deeper, entry is locked if its GridCacheMapEntry.localCandidates > queue is not empty (first item in queue is actually the transaction that > owns lock). > > > we have just to wait > > until entry becomes unlocked. > This may work. > If consistency checking code has acquired lock on primary, backup can be > in two states: > - not locked - and new locks won't appear as we are holding lock on primary > - still locked by transaction that owned lock on primary just before our > checking code - in such case checking code should just wait for lock release > > Best Regards, > Ivan Rakov > > On 15.07.2019 9:34, Anton Vinogradov wrote: > > Ivan R. > > > > Thanks for joining! > > > > Got an idea, but not sure that got a way of a fix. > > > > AFAIK (can be wrong, please correct if necessary), at 2PC, locks are > > acquired on backups during the "prepare" phase and released at "finish" > > phase after primary fully committed. > > Step-by-step: > > 1) primary locked on key mention (get/put) at pessimistic/!read-committed tx > > 2) backups locked on prepare > > 3) primary unlocked on finish > > 4) backups unlocked on finish (after the primary) > > correct? > > > > So, acquiring locks on backups, not at the "prepare" phase, may cause > > unexpected behavior in case of primary fail or other errors. > > That's definitely possible to update failover to solve this issue, but it > > seems to be an overcomplicated way. > > The main question there, it there any simple way? > > > >>> checking read from backup will just wait for commit if it's in progress. > > Agree, but seems there is no need to acquire the lock, we have just to wait > > until entry becomes unlocked. > > - entry locked means that previous tx's "finish" phase is in progress > > - entry unlocked means reading value is up-to-date (previous "finish" phase > > finished) > > correct? > > > > On Mon, Jul 15, 2019 at 8:37 AM Павлухин Иван wrote: > > > >> Anton, > >> > >> I did not know mechanics locking entries on backups during prepare > >> phase. Thank you for pointing that out! > >> > >> пт, 12 июл. 2019 г. в 22:45, Ivan Rakov : > >>> Hi Anton, > >>> > >>>> Each get method now checks the consistency. > >>>> Check means: > >>>> 1) tx lock acquired on primary > >>>> 2) gained data from each owner (primary and backups) > >>>> 3) data compared > >>> Did you consider acquiring locks on backups as well during your check, > >>> just like 2PC prepare does? > >>> If there's HB between steps 1 (lock primary) and 2 (update primary + > >>> lock backup + update backup), you may be sure that there will be no > >>> false-positive results and no deadlocks as well. Protocol won't be > >>> complicated: checking read from backup will just wait for commit if it's > >>> in progress. > >>> > >>> Best Regards, > >>> Ivan Rakov > >>> > >>> On 12.07.2019 9:47, Anton Vinogradov wrote: > >>>> Igniters, > >>>> > >>>> Let me explain problem in detail. > >>>> Read Repair at pessimistic tx (locks acquired on primary, full sync, > >> 2pc) > >>>> able to see consistency violation because backups are not updated yet. > >>>> This seems to be not a good idea to "fix" code to unlock primary
[PROBLEM] TeamCity agents out of disk space
Hi, Currently I see that there many failed TC jobs in recent RunAll [1]. I checked logs of couple of them and it seems that it is related to no free disk space on agents, e.g. [2]. 1. Who can resolve it? 2. How can it be resolved? 3. How to avoid similar issues in future? [1] https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_RunAll_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv [2] https://ci.ignite.apache.org/viewLog.html?buildId=4330109=IgniteTests24Java8_Hibernate2=buildLog_IgniteTests24Java8=%3Cdefault%3E -- Best regards, Ivan Pavlukhin
Re: Tx lock partial happens before
Anton, I did not know mechanics locking entries on backups during prepare phase. Thank you for pointing that out! пт, 12 июл. 2019 г. в 22:45, Ivan Rakov : > > Hi Anton, > > > Each get method now checks the consistency. > > Check means: > > 1) tx lock acquired on primary > > 2) gained data from each owner (primary and backups) > > 3) data compared > Did you consider acquiring locks on backups as well during your check, > just like 2PC prepare does? > If there's HB between steps 1 (lock primary) and 2 (update primary + > lock backup + update backup), you may be sure that there will be no > false-positive results and no deadlocks as well. Protocol won't be > complicated: checking read from backup will just wait for commit if it's > in progress. > > Best Regards, > Ivan Rakov > > On 12.07.2019 9:47, Anton Vinogradov wrote: > > Igniters, > > > > Let me explain problem in detail. > > Read Repair at pessimistic tx (locks acquired on primary, full sync, 2pc) > > able to see consistency violation because backups are not updated yet. > > This seems to be not a good idea to "fix" code to unlock primary only when > > backups updated, this definitely will cause a performance drop. > > Currently, there is no explicit sync feature allows waiting for backups > > updated during the previous tx. > > Previous tx just sends GridNearTxFinishResponse to the originating node. > > > > Bad ideas how to handle this: > > - retry some times (still possible to gain false positive) > > - lock tx entry on backups (will definitely break failover logic) > > - wait for same entry version on backups during some timeout (will require > > huge changes at "get" logic and false positive still possible) > > > > Is there any simple fix for this issue? > > Thanks for tips in advance. > > > > Ivan, > > thanks for your interest > > > >>> 4. Very fast and lucky txB writes a value 2 for the key on primary and > > backup. > > AFAIK, reordering not possible since backups "prepared" before primary > > releases lock. > > So, consistency guaranteed by failover and by "prepare" feature of 2PC. > > Seems, the problem is NOT with consistency at AI, but with consistency > > detection implementation (RR) and possible "false positive" results. > > BTW, checked 1PC case (only one data node at test) and gained no issues. > > > > On Fri, Jul 12, 2019 at 9:26 AM Павлухин Иван wrote: > > > >> Anton, > >> > >> Is such behavior observed for 2PC or for 1PC optimization? Does not it > >> mean that the things can be even worse and an inconsistent write is > >> possible on a backup? E.g. in scenario: > >> 1. txA writes a value 1 for the key on primary. > >> 2. txA unlocks the key on primary. > >> 3. txA freezes before updating backup. > >> 4. Very fast and lucky txB writes a value 2 for the key on primary and > >> backup. > >> 5. txB wakes up and writes 1 for the key. > >> 6. As result there is 2 on primary and 1 on backup. > >> > >> Naively it seems that locks should be released after all replicas are > >> updated. > >> > >> ср, 10 июл. 2019 г. в 16:36, Anton Vinogradov : > >>> Folks, > >>> > >>> Investigating now unexpected repairs [1] in case of ReadRepair usage at > >>> testAccountTxNodeRestart. > >>> Updated [2] the test to check is there any repairs happen. > >>> Test's name now is "testAccountTxNodeRestartWithReadRepair". > >>> > >>> Each get method now checks the consistency. > >>> Check means: > >>> 1) tx lock acquired on primary > >>> 2) gained data from each owner (primary and backups) > >>> 3) data compared > >>> > >>> Sometime, backup may have obsolete value during such check. > >>> > >>> Seems, this happen because tx commit on primary going in the following > >> way > >>> (check code [2] for details): > >>> 1) performing localFinish (releases tx lock) > >>> 2) performing dhtFinish (commits on backups) > >>> 3) transferring control back to the caller > >>> > >>> So, seems, the problem here is that "tx lock released on primary" does > >> not > >>> mean that backups updated, but "commit() method finished at caller's > >>> thread" does. > >>> This means that, currently, there is no happens-before between > >>> 1) thread 1 committed data on primary and tx lock can be reobtained > >>> 2) thread 2 reads from backup > >>> but still strong HB between "commit() finished" and "backup updated" > >>> > >>> So, it seems to be possible, for example, to gain notification by a > >>> continuous query, then read from backup and gain obsolete value. > >>> > >>> Is this "partial happens before" behavior expected? > >>> > >>> [1] https://issues.apache.org/jira/browse/IGNITE-11973 > >>> [2] https://github.com/apache/ignite/pull/6679/files > >>> [3] > >>> > >> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal#finishTx > >> > >> > >> > >> -- > >> Best regards, > >> Ivan Pavlukhin > >> -- Best regards, Ivan Pavlukhin
Re: Tx lock partial happens before
Anton, Is such behavior observed for 2PC or for 1PC optimization? Does not it mean that the things can be even worse and an inconsistent write is possible on a backup? E.g. in scenario: 1. txA writes a value 1 for the key on primary. 2. txA unlocks the key on primary. 3. txA freezes before updating backup. 4. Very fast and lucky txB writes a value 2 for the key on primary and backup. 5. txB wakes up and writes 1 for the key. 6. As result there is 2 on primary and 1 on backup. Naively it seems that locks should be released after all replicas are updated. ср, 10 июл. 2019 г. в 16:36, Anton Vinogradov : > > Folks, > > Investigating now unexpected repairs [1] in case of ReadRepair usage at > testAccountTxNodeRestart. > Updated [2] the test to check is there any repairs happen. > Test's name now is "testAccountTxNodeRestartWithReadRepair". > > Each get method now checks the consistency. > Check means: > 1) tx lock acquired on primary > 2) gained data from each owner (primary and backups) > 3) data compared > > Sometime, backup may have obsolete value during such check. > > Seems, this happen because tx commit on primary going in the following way > (check code [2] for details): > 1) performing localFinish (releases tx lock) > 2) performing dhtFinish (commits on backups) > 3) transferring control back to the caller > > So, seems, the problem here is that "tx lock released on primary" does not > mean that backups updated, but "commit() method finished at caller's > thread" does. > This means that, currently, there is no happens-before between > 1) thread 1 committed data on primary and tx lock can be reobtained > 2) thread 2 reads from backup > but still strong HB between "commit() finished" and "backup updated" > > So, it seems to be possible, for example, to gain notification by a > continuous query, then read from backup and gain obsolete value. > > Is this "partial happens before" behavior expected? > > [1] https://issues.apache.org/jira/browse/IGNITE-11973 > [2] https://github.com/apache/ignite/pull/6679/files > [3] > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal#finishTx -- Best regards, Ivan Pavlukhin
Re: [REVIEW REQUEST] IGNITE-11951 Improvements in JdkMarshaller
Merged the patch to master [1]. Thank Alex Plekhanov for a review. [1] https://issues.apache.org/jira/browse/IGNITE-11951 ср, 10 июл. 2019 г. в 08:42, Павлухин Иван : > > Hi, > > I made some small improvements in JdkMarshaller [1]. I will be happy > if someone reviews it. Changes are quite simple. > > [1] https://issues.apache.org/jira/browse/IGNITE-11951 > > -- > Best regards, > Ivan Pavlukhin -- Best regards, Ivan Pavlukhin
Re: Revisit added java modules in ignite.sh
I merged the patch [1] by lazy consensus. [1] https://issues.apache.org/jira/browse/IGNITE-11946 пн, 8 июл. 2019 г. в 20:48, Павлухин Иван : > > Igniters, > > I present a ticket [1] removing java.transaction and java.corba > modules from sh/bat scripts for your review. I remind that it fixes > nodejs, python and php suites when running on Java 9 and 10. > > In case there is no activity on this I would like to merge it this week. > > [1] https://issues.apache.org/jira/browse/IGNITE-11946 > > ср, 3 июл. 2019 г. в 21:40, Павлухин Иван : > > > > I removed command line arguments enabling java.transaction and > > java.corba module from TC configuration. All activity in a ticket [1]. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-11946 > > > > вс, 30 июн. 2019 г. в 07:31, Павлухин Иван : > > > > > > I made 2 experiments: > > > 1. Removed --add-modules=java.transaction from TC configurations. I > > > have not observed any new test failures after that change. > > > 2. Removed that option from ignite.sh in PR. In PR problematic thin > > > client suites are green. > > > > > > See the ticket [1] for additional details. If there is no objections I > > > will cleanup that option from TC configurations and Ignite scripts. > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9687 > > > > > > пт, 28 июн. 2019 г. в 10:09, Павлухин Иван : > > > > > > > > Folks, > > > > > > > > During investigation of thin client test suite failures [1] I found > > > > that it is not clear whether --add-modules=java.transaction in > > > > ignite.sh/bat is needed (we add this module for Java 9 and 10). In > > > > thread [1] I already mentioned that launching ignite.sh with Java 9 > > > > and ignite-jta libraries on a classpath leads to an error. You can see > > > > how it can look in a log [2]. > > > > > > > > My proposal here is to get rid off --add-modules=java.transaction from > > > > stratup scripts and parameters used by TC. And I will try to explain > > > > why. > > > > > > > > The story started in a ticket [3]. Where a recipe > > > > --add-modules=java.transaction > > > > --patch-module=java.transaction=$HOME/.m2/repository/javax/transaction/javax.transaction-api/1.3/javax.transaction-api-1.3.jar > > > > allowed to execute JTA tests successfully. After that > > > > --add-modules=java.transaction was added to launch scripts. But that > > > > setup allowed to launch tests. And they failed actually due to > > > > problems with resolving needed classes for test depndencies > > > > (org.ow2.jotm:jotm-core). Needed classes were not found because the > > > > dependency required classes from java standard library which were > > > > moved to optional modules. And the thing is even trickier with a > > > > --patch-module option because javax.transaction-api-1.3.jar contains a > > > > superset of classes comparing to java.transaction module. So, I do not > > > > see a need of adding this module. Moreover that module was removed > > > > completely starting from Java 11. And even if missed something and it > > > > has some meaning for Java 9/10, I still think that having uniform > > > > setup for all Java versions simplifies matters. > > > > > > > > Have I missed something? Do we have an example how to check JTA > > > > integration? How it can be used with a standalone Ignite setup? > > > > > > > > P.S. Also I will try to do a similar reseach about java.xml.bind > > > > module if noone knows why do we need it. > > > > > > > > [1] > > > > http://apache-ignite-developers.2346864.n4.nabble.com/Thin-client-test-suites-failure-td42388.html > > > > [2] > > > > https://ci.ignite.apache.org/viewLog.html?buildId=4217786=IgniteTests24Java8_ThinClientNodeJs=buildLog_IgniteTests24Java8=pull%2F6644%2Fhead&_focus=423#_state=423 > > > > [3] https://issues.apache.org/jira/browse/IGNITE-9687 > > > > -- > > > > Best regards, > > > > Ivan Pavlukhin > > > > > > > > > > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > > > > > > -- > > Best regards, > > Ivan Pavlukhin > > > > -- > Best regards, > Ivan Pavlukhin -- Best regards, Ivan Pavlukhin
Re: [discussion] using custom build of H2 for Ignite
Dmitriy, Thank you for sharing your vision. Actually I think that an agreement within the community is the most important thing. ср, 10 июл. 2019 г. в 17:44, Dmitriy Pavlov : > > Hi Ivan, > > > > I don’t need a policy to clearly understand that the Apache project stops > to be healthy once it is controlled by one entity. This is related to > governance, not to OSI approved license (if a lib is open-source or not). > > > > Apache mission - Create software for the public good. > > Apache project is: > > - Open Source, commercial-friendly Licence, > > - Open Governance (clear, documented leader election), > > - Open Brand (Brand owner is always ASF, it is a non-profit, charity > organization). > > > > And once community adds a dependency to any vendor-controlled and released > artifact I suppose it is not anymore for the public good, it is for good of > vendor. A goal can be to be advertized using this open-source project and > the Apache brand. Vendor in some cases want just Apache brand, and rarely > shares Apache principles and Apache Way. > > > > Maybe Konstantin could explain better than me, why the community should not > accept vendor-provided dependencies. > > > Hopefully, Cos can also evaluate idea separate GitHub repository with full > control from PMC (root credentials is placed to private SVN + clear release > procedure). > > > > Sincerely, > > Dmitriy Pavlov > > Disclaimer: Opinions expressed in this email are those of the author, > and do not necessarily represent the views of any company the author > might be affiliated with at the moment of writing. > > ср, 10 июл. 2019 г. в 15:06, Nikolay Izhikov : > > > Ivan. > > > > > I suppose that I did not ever claim such thing > > > > Thanks for clarifing :) > > > > > > > GitHub project outside any commercial company accounts, there all Apache > > committers added as collaborators may work. > > > Sounds nice for me as well. > > > > +1 from me if it compatible with Apache policies. > > > > > > В Ср, 10/07/2019 в 14:59 +0300, Павлухин Иван пишет: > > > Nikolay, > > > > Why we should close H2 fork from Ignite commiters in the first place? > > > > > > I suppose that I did not ever claim such thing. I think we are > > > discussing various options. And ones might be simpler and others might > > > be harder. > > > > > > Dmitriy, > > > > > > To my shame I am not very competent in licensing and related > > > questions. I mostly think about how the problem can be solved > > > technically. If something is against Apache policies then we > > > definitely can go with it (would be great to get a reference to such > > > policies). > > > > > > > GitHub project outside any commercial company accounts, there all > > Apache committers added as collaborators may work. > > > > > > Sounds nice for me as well. > > > > > > And I would like to return to the beginning. > > > 1. What do we want? Improve SQL in Ignite. > > > 2. How do we want to do it? In the most convenient "legal" way. > > > > > > Perhaps there are other bright thoughts how to keep it simple? > > > > > > ср, 10 июл. 2019 г. в 14:18, Dmitriy Pavlov : > > > > > > > > Wearing the hat of Apache Ignite PMC: > > > > I'm against starting with dependency on GridGain code in any case. > > Gridgain > > > > can do its own forks of both. > > > > > > > > We all know how much influence the company has on the community and > > adding > > > > one more (I remind there is gridgain:shmem) > > > > means the open-governed solution is dependent on closely-governed. > > Would > > > > you use something open-source if it depends on binaries? I don't care > > about > > > > license in that case. > > > > > > > > you can count on '-1' from my side. > > > > > > > > GitHub project outside any commercial company accounts, there all > > Apache > > > > committers added as collaborators may work. > > > > > > > > > > > > ср, 10 июл. 2019 г. в 13:28, Юрий : > > > > > > > > > Agree with Ivan. > > > > > > > > > > We can start work with H2 fork owned by GG and decide change it > > later in > > > > > case it will bring some issues to Ignite community. Currently I > > don't see > > > > > any issues here. &
Re: [discussion] using custom build of H2 for Ignite
Nikolay, > Why we should close H2 fork from Ignite commiters in the first place? I suppose that I did not ever claim such thing. I think we are discussing various options. And ones might be simpler and others might be harder. Dmitriy, To my shame I am not very competent in licensing and related questions. I mostly think about how the problem can be solved technically. If something is against Apache policies then we definitely can go with it (would be great to get a reference to such policies). > GitHub project outside any commercial company accounts, there all Apache > committers added as collaborators may work. Sounds nice for me as well. And I would like to return to the beginning. 1. What do we want? Improve SQL in Ignite. 2. How do we want to do it? In the most convenient "legal" way. Perhaps there are other bright thoughts how to keep it simple? ср, 10 июл. 2019 г. в 14:18, Dmitriy Pavlov : > > Wearing the hat of Apache Ignite PMC: > I'm against starting with dependency on GridGain code in any case. Gridgain > can do its own forks of both. > > We all know how much influence the company has on the community and adding > one more (I remind there is gridgain:shmem) > means the open-governed solution is dependent on closely-governed. Would > you use something open-source if it depends on binaries? I don't care about > license in that case. > > you can count on '-1' from my side. > > GitHub project outside any commercial company accounts, there all Apache > committers added as collaborators may work. > > > ср, 10 июл. 2019 г. в 13:28, Юрий : > > > Agree with Ivan. > > > > We can start work with H2 fork owned by GG and decide change it later in > > case it will bring some issues to Ignite community. Currently I don't see > > any issues here. > > > > I'm worried about the issue, with process to synchonize changes H2 fork and > > Ignite. As possible solution it could be as follow: > > Ignite has dependency only to released version of H2 fork. > > Then we could modify the fork in any time without compatibility issues. > > When new feature is ready to use by Ignite need to modify code of Ignite > > and change dependency version for H2 fork. > > However H2 for in the case should be release more often than Ignite. > > > > WDYT? > > > > ср, 10 июл. 2019 г. в 13:08, Павлухин Иван : > > > > > Nikolay, > > > > > > Could you please elaborate why is it "closed source"? > > > > > > > What the difference for the Ignite community? > > > The difference is similar to using version X and version Y of the same > > > library. Version Y might be better. > > > > > > > I think all Ignite commiters should have write priveledges to H2 fork. > > > I agree, it is quite natural. Actually, my only point is that we can > > > do it at any point later, cannot we? > > > > > > ср, 10 июл. 2019 г. в 12:25, Nikolay Izhikov : > > > > > > > > Ivan > > > > > > > > We have closed source code dependency for now owned by H2 owners. > > > > With new fork we will have the same closed dependency owned by Grid > > Gain. > > > > > > > > What the difference for the Ignite community? > > > > > > > > > 2. Anyways some process must be established for merging changes > > > > > requiring changes in h2 library. So, I suppose it should be review of > > > > > changes in 2 repositories. > > > > > > > > The question is - *Who can apply those changes*. > > > > > > > > I think all Ignite commiters should have write priveledges to H2 fork. > > > > > > > > В Ср, 10/07/2019 в 11:30 +0300, Павлухин Иван пишет: > > > > > Folks, > > > > > > > > > > I would like to highlight a couple of points. > > > > > 1. Perhaps it is not so crucial where is this fork located if the > > code > > > > > is publicly available and can be cloned to another repository easily. > > > > > We can relocate code and use it at any point in future. > > > > > 2. Anyways some process must be established for merging changes > > > > > requiring changes in h2 library. So, I suppose it should be review of > > > > > changes in 2 repositories. > > > > > > > > > > Now (and beforehand) we use original h2. And how many of us were ever > > > > > interested what changes were made in h2? So, perhaps for the first > > > > > time we can start with GG fork? And if later on so
Re: [discussion] using custom build of H2 for Ignite
Nikolay, Could you please elaborate why is it "closed source"? > What the difference for the Ignite community? The difference is similar to using version X and version Y of the same library. Version Y might be better. > I think all Ignite commiters should have write priveledges to H2 fork. I agree, it is quite natural. Actually, my only point is that we can do it at any point later, cannot we? ср, 10 июл. 2019 г. в 12:25, Nikolay Izhikov : > > Ivan > > We have closed source code dependency for now owned by H2 owners. > With new fork we will have the same closed dependency owned by Grid Gain. > > What the difference for the Ignite community? > > > 2. Anyways some process must be established for merging changes > > requiring changes in h2 library. So, I suppose it should be review of > > changes in 2 repositories. > > The question is - *Who can apply those changes*. > > I think all Ignite commiters should have write priveledges to H2 fork. > > В Ср, 10/07/2019 в 11:30 +0300, Павлухин Иван пишет: > > Folks, > > > > I would like to highlight a couple of points. > > 1. Perhaps it is not so crucial where is this fork located if the code > > is publicly available and can be cloned to another repository easily. > > We can relocate code and use it at any point in future. > > 2. Anyways some process must be established for merging changes > > requiring changes in h2 library. So, I suppose it should be review of > > changes in 2 repositories. > > > > Now (and beforehand) we use original h2. And how many of us were ever > > interested what changes were made in h2? So, perhaps for the first > > time we can start with GG fork? And if later on some problems with > > that appear we can clone it and use that new fork without much > > trouble, can't we? > > > > ср, 10 июл. 2019 г. в 09:52, Nikolay Izhikov : > > > > > > Hello, Denis. > > > > > > > Nickolay, as for that fork which is in GG codebase - GridGain is a major > > > > contributor and maintainer but the others are welcomed to send > > > > pull-requests. > > > > > > Can we make this fork maintained by Ignite Community? > > > > > > With all respect to Grid Gain as an author of Apache Ignite I don't like > > > when some huge dependencies > > > (incompatible with community-driven analogue) belongs to the enterprise. > > > > > > This leads us to the situation when Grid Gain will decide which features > > > will be added to the SQL engine and which not. > > > > > > В Пн, 08/07/2019 в 13:51 -0700, Denis Magda пишет: > > > > Dmitry, > > > > > > > > To make this fully-vendor neutral even at the originating repository > > > > level, > > > > we can create and work with the H2 fork as a separate Github repo > > > > (separate > > > > project governed and maintained by Ignite community). That repo can't be > > > > part of Ignite due to license mismatch. Thus, during release times, we > > > > need > > > > to assemble a binary (maven artifact) from that fork. > > > > > > > > However, it's not clear to me how to use those sources during the dev > > > > time? > > > > It sounds like Ignite can use only the binary (Maven) artifact that has > > > > to > > > > be updated/regenerated if there are any changes. *SQL experts*, could > > > > you > > > > please step in? > > > > > > > > Nickolay, as for that fork which is in GG codebase - GridGain is a major > > > > contributor and maintainer but the others are welcomed to send > > > > pull-requests. > > > > > > > > - > > > > Denis > > > > > > > > > > > > On Thu, Jul 4, 2019 at 9:26 AM Dmitriy Pavlov > > > > wrote: > > > > > > > > > Hi Denis, > > > > > > > > > > As you know, some time ago I've started a discussion about removing > > > > > dependence from gridgain:shmem. Ignite community seems to be not so > > > > > much > > > > > interested in this removal, for now. So once added it could stay here > > > > > forever. Reverse dependency direction seems to be more natural. It is > > > > > like > > > > > the open-core model. > > > > > > > > > > I feel more comfortable if all Ignite dependencies are released as > > > > > part of > > >
Re: [discussion] using custom build of H2 for Ignite
Folks, I would like to highlight a couple of points. 1. Perhaps it is not so crucial where is this fork located if the code is publicly available and can be cloned to another repository easily. We can relocate code and use it at any point in future. 2. Anyways some process must be established for merging changes requiring changes in h2 library. So, I suppose it should be review of changes in 2 repositories. Now (and beforehand) we use original h2. And how many of us were ever interested what changes were made in h2? So, perhaps for the first time we can start with GG fork? And if later on some problems with that appear we can clone it and use that new fork without much trouble, can't we? ср, 10 июл. 2019 г. в 09:52, Nikolay Izhikov : > > Hello, Denis. > > > Nickolay, as for that fork which is in GG codebase - GridGain is a major > > contributor and maintainer but the others are welcomed to send > > pull-requests. > > Can we make this fork maintained by Ignite Community? > > With all respect to Grid Gain as an author of Apache Ignite I don't like when > some huge dependencies > (incompatible with community-driven analogue) belongs to the enterprise. > > This leads us to the situation when Grid Gain will decide which features will > be added to the SQL engine and which not. > > В Пн, 08/07/2019 в 13:51 -0700, Denis Magda пишет: > > Dmitry, > > > > To make this fully-vendor neutral even at the originating repository level, > > we can create and work with the H2 fork as a separate Github repo (separate > > project governed and maintained by Ignite community). That repo can't be > > part of Ignite due to license mismatch. Thus, during release times, we need > > to assemble a binary (maven artifact) from that fork. > > > > However, it's not clear to me how to use those sources during the dev time? > > It sounds like Ignite can use only the binary (Maven) artifact that has to > > be updated/regenerated if there are any changes. *SQL experts*, could you > > please step in? > > > > Nickolay, as for that fork which is in GG codebase - GridGain is a major > > contributor and maintainer but the others are welcomed to send > > pull-requests. > > > > - > > Denis > > > > > > On Thu, Jul 4, 2019 at 9:26 AM Dmitriy Pavlov wrote: > > > > > Hi Denis, > > > > > > As you know, some time ago I've started a discussion about removing > > > dependence from gridgain:shmem. Ignite community seems to be not so much > > > interested in this removal, for now. So once added it could stay here > > > forever. Reverse dependency direction seems to be more natural. It is like > > > the open-core model. > > > > > > I feel more comfortable if all Ignite dependencies are released as part of > > > the Ignite code base, or some open governed project with a license from > > > Category A https://www.apache.org/legal/resolved.html. > > > > > > It is true that H2 has Category B license, so derivative can't be > > > committed > > > into ASF repository. > > > > > > What if we consult with le...@apache.org to find additional ways to donate > > > forked version into ASF codebase? We anyway need their approval because > > > gridgain/h2 has a non-standard license, so we should approve including > > > non-standard licensed component it the product. > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > чт, 4 июл. 2019 г. в 18:57, Denis Magda : > > > > > > > Hi Igniters, > > > > > > > > As you know, Ignite SQL engine is tightly coupled with the H2 database > > > > > > that > > > > provides basic parsing and query execution capabilities. This synergy > > > > > > has > > > > worked well for a while until Ignite SQL engine got a much broader > > > > > > adoption > > > > for all sort of use cases. > > > > > > > > Presently, there is a list of impactful issues and limitations related > > > > to > > > > memory management, distributed engine optimization, and queries planning > > > > that require changes in H2. We've tried to contribute to H2 directly > > > > with > > > > no significant luck - what's needed for our distributed engine is of no > > > > interest to H2 community. At the same time, we can't leave the things as > > > > is, as long as these limitations keep Ignite SQL engine from gradual > > > > evolution. > > > > > > > > As a solution, we created an H2 fork [1] and did all of the required > > > > changes there. We would be happy to include the fork into Ignite source > > > > base, but H2's license (available under dual MPL 2.0 and EPL 1.0) is not > > > > compliant with Apache 2.0. However, if Ignite starts using our maven > > > > artifacts instead of the standard H2's ones, then the licensing issue is > > > > solved. > > > > > > > > Is the community ready to accept this solution and swap the standard H2 > > > > artifact with the one prepared by GridGain? Presently, all of those > > > > improvements are available to GridGain customers, but GridGain wants to > > > > make all of them be available for Ignite community. And that's the only > > >
[REVIEW REQUEST] IGNITE-11951 Improvements in JdkMarshaller
Hi, I made some small improvements in JdkMarshaller [1]. I will be happy if someone reviews it. Changes are quite simple. [1] https://issues.apache.org/jira/browse/IGNITE-11951 -- Best regards, Ivan Pavlukhin
Re: Revisit added java modules in ignite.sh
Igniters, I present a ticket [1] removing java.transaction and java.corba modules from sh/bat scripts for your review. I remind that it fixes nodejs, python and php suites when running on Java 9 and 10. In case there is no activity on this I would like to merge it this week. [1] https://issues.apache.org/jira/browse/IGNITE-11946 ср, 3 июл. 2019 г. в 21:40, Павлухин Иван : > > I removed command line arguments enabling java.transaction and > java.corba module from TC configuration. All activity in a ticket [1]. > > [1] https://issues.apache.org/jira/browse/IGNITE-11946 > > вс, 30 июн. 2019 г. в 07:31, Павлухин Иван : > > > > I made 2 experiments: > > 1. Removed --add-modules=java.transaction from TC configurations. I > > have not observed any new test failures after that change. > > 2. Removed that option from ignite.sh in PR. In PR problematic thin > > client suites are green. > > > > See the ticket [1] for additional details. If there is no objections I > > will cleanup that option from TC configurations and Ignite scripts. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-9687 > > > > пт, 28 июн. 2019 г. в 10:09, Павлухин Иван : > > > > > > Folks, > > > > > > During investigation of thin client test suite failures [1] I found > > > that it is not clear whether --add-modules=java.transaction in > > > ignite.sh/bat is needed (we add this module for Java 9 and 10). In > > > thread [1] I already mentioned that launching ignite.sh with Java 9 > > > and ignite-jta libraries on a classpath leads to an error. You can see > > > how it can look in a log [2]. > > > > > > My proposal here is to get rid off --add-modules=java.transaction from > > > stratup scripts and parameters used by TC. And I will try to explain > > > why. > > > > > > The story started in a ticket [3]. Where a recipe > > > --add-modules=java.transaction > > > --patch-module=java.transaction=$HOME/.m2/repository/javax/transaction/javax.transaction-api/1.3/javax.transaction-api-1.3.jar > > > allowed to execute JTA tests successfully. After that > > > --add-modules=java.transaction was added to launch scripts. But that > > > setup allowed to launch tests. And they failed actually due to > > > problems with resolving needed classes for test depndencies > > > (org.ow2.jotm:jotm-core). Needed classes were not found because the > > > dependency required classes from java standard library which were > > > moved to optional modules. And the thing is even trickier with a > > > --patch-module option because javax.transaction-api-1.3.jar contains a > > > superset of classes comparing to java.transaction module. So, I do not > > > see a need of adding this module. Moreover that module was removed > > > completely starting from Java 11. And even if missed something and it > > > has some meaning for Java 9/10, I still think that having uniform > > > setup for all Java versions simplifies matters. > > > > > > Have I missed something? Do we have an example how to check JTA > > > integration? How it can be used with a standalone Ignite setup? > > > > > > P.S. Also I will try to do a similar reseach about java.xml.bind > > > module if noone knows why do we need it. > > > > > > [1] > > > http://apache-ignite-developers.2346864.n4.nabble.com/Thin-client-test-suites-failure-td42388.html > > > [2] > > > https://ci.ignite.apache.org/viewLog.html?buildId=4217786=IgniteTests24Java8_ThinClientNodeJs=buildLog_IgniteTests24Java8=pull%2F6644%2Fhead&_focus=423#_state=423 > > > [3] https://issues.apache.org/jira/browse/IGNITE-9687 > > > -- > > > Best regards, > > > Ivan Pavlukhin > > > > > > > > -- > > Best regards, > > Ivan Pavlukhin > > > > -- > Best regards, > Ivan Pavlukhin -- Best regards, Ivan Pavlukhin
Re: Migration to JUnit 5
Ivan, I think that it is a really good that you found those not tested examples. Thank you! пт, 5 июл. 2019 г. в 11:31, Павлухин Иван : > > Ivan, > > I uncommented all tests referring to IGNITE-711 [1] in > BasicExamplesSelfTest and all they passed. > > Generally, example tests are needed to be sure that our examples > launch. And commented tests refer to existing examples. So, an ideal > way here is to uncomment them in scope of IGNITE-711 [1], removal is > not a good option. And I do not expect much problems here because we > fully support Java 8 for a long time. > > [1] https://issues.apache.org/jira/browse/IGNITE-711 > > пн, 1 июл. 2019 г. в 17:10, Ivan Fedotov : > > > > Hi, Igniters > > > > During work on IEP-30, which is about JUnit migration, I found that some > > tests in examples module were commented [1] with the remark, that they > > should be fixed in the ticket IGNITE-711 [2] which is about the > > implementation of Java 8 examples. > > > > In the context of the ticket IGNITE-10973 [3] I want to uncomment them and > > mark as @Disabled. Is it really need to disable mentioned tests or I can > > just remove them as outdated? > > > > [1] > > https://github.com/apache/ignite/pull/6606/files#diff-ed48193d25d777a2c30c187fa20a1a65L65 > > [2] https://issues.apache.org/jira/browse/IGNITE-711 > > [3] https://issues.apache.org/jira/browse/IGNITE-10973 > > > > > > вт, 26 февр. 2019 г. в 18:51, Ivan Fedotov : > > > > > Ivan, > > > I will investigate GridAbstractTest refactoring issue more precisely when > > > I finish with JUnit3Legacy classes. Anyway, I will keep in touch with you > > > and the community on the most significant moments. > > > > > > JUnit5 docs say that functionality is not full "especially with regard to > > > reporting". On the other hand, I also agree with docs that it is the > > > easiest way that does not require to touch CI infrastructure. I am going > > > to > > > try @RunWith(JUnitPlatform.class) construction with features from IEP to > > > make sure that we will have the full support of them. The alternative way > > > is dynamic tests [1], but the problem is that we add methods to suites > > > manually, not via @Test annotation. It is some kind of rollback to JUnit3 > > > syntax. > > > > > > Anton, > > > thank you for the reminder, I will update IEP according to the > > > conversation. > > > > > > [1] https://www.baeldung.com/junit5-dynamic-tests > > > > > > вт, 26 февр. 2019 г. в 17:56, Anton Vinogradov : > > > > > >> Folks, > > >> > > >> Please make sure you keep IEP updated and each issue mentioned. > > >> > > >> On Tue, Feb 26, 2019 at 4:28 PM Павлухин Иван > > >> wrote: > > >> > > >> > Ivan, > > >> > > > >> > Thank you for detailed answers! I would put a great care to > > >> > @RunWith(JUnitPlatform.class) construction. As stated in junit5 docs > > >> > [1] it does not support all features and unfortunately it is not clear > > >> > how limited it is. Also, it is some kind of transitional mechanism > > >> > which was not designed for being a long term solution. > > >> > > > >> > And I fully support an idea of refactoring GridAbstractTest. I think > > >> > it is possible to make a significant improvement here. > > >> > > > >> > [1] > > >> > > > >> https://junit.org/junit5/docs/current/user-guide/#running-tests-junit-platform-runner > > >> > > > >> > пн, 25 февр. 2019 г. в 17:41, Ivan Fedotov : > > >> > > > > >> > > Hello Nikolay. > > >> > > > > >> > > The prime benefits are more comfortable work with flaky tests, Java 8 > > >> > tests > > >> > > compatibility, user-friendly syntaxis in parametrized tests and > > >> others. > > >> > > The most significant features list you can find in IEP-30 Motivation > > >> > > section. > > >> > > > > >> > > If you have any specific questions about JUnit5 feel free to ask me. > > >> > > > > >> > > пн, 25 февр. 2019 г. в 16:55, Nikolay Izhikov : > > >> > > > > >> > > > Hello, Ivan. > > >> > > > > > >> > > > May be I miss some mail
Re: Migration to JUnit 5
Ivan, I uncommented all tests referring to IGNITE-711 [1] in BasicExamplesSelfTest and all they passed. Generally, example tests are needed to be sure that our examples launch. And commented tests refer to existing examples. So, an ideal way here is to uncomment them in scope of IGNITE-711 [1], removal is not a good option. And I do not expect much problems here because we fully support Java 8 for a long time. [1] https://issues.apache.org/jira/browse/IGNITE-711 пн, 1 июл. 2019 г. в 17:10, Ivan Fedotov : > > Hi, Igniters > > During work on IEP-30, which is about JUnit migration, I found that some > tests in examples module were commented [1] with the remark, that they > should be fixed in the ticket IGNITE-711 [2] which is about the > implementation of Java 8 examples. > > In the context of the ticket IGNITE-10973 [3] I want to uncomment them and > mark as @Disabled. Is it really need to disable mentioned tests or I can > just remove them as outdated? > > [1] > https://github.com/apache/ignite/pull/6606/files#diff-ed48193d25d777a2c30c187fa20a1a65L65 > [2] https://issues.apache.org/jira/browse/IGNITE-711 > [3] https://issues.apache.org/jira/browse/IGNITE-10973 > > > вт, 26 февр. 2019 г. в 18:51, Ivan Fedotov : > > > Ivan, > > I will investigate GridAbstractTest refactoring issue more precisely when > > I finish with JUnit3Legacy classes. Anyway, I will keep in touch with you > > and the community on the most significant moments. > > > > JUnit5 docs say that functionality is not full "especially with regard to > > reporting". On the other hand, I also agree with docs that it is the > > easiest way that does not require to touch CI infrastructure. I am going to > > try @RunWith(JUnitPlatform.class) construction with features from IEP to > > make sure that we will have the full support of them. The alternative way > > is dynamic tests [1], but the problem is that we add methods to suites > > manually, not via @Test annotation. It is some kind of rollback to JUnit3 > > syntax. > > > > Anton, > > thank you for the reminder, I will update IEP according to the > > conversation. > > > > [1] https://www.baeldung.com/junit5-dynamic-tests > > > > вт, 26 февр. 2019 г. в 17:56, Anton Vinogradov : > > > >> Folks, > >> > >> Please make sure you keep IEP updated and each issue mentioned. > >> > >> On Tue, Feb 26, 2019 at 4:28 PM Павлухин Иван > >> wrote: > >> > >> > Ivan, > >> > > >> > Thank you for detailed answers! I would put a great care to > >> > @RunWith(JUnitPlatform.class) construction. As stated in junit5 docs > >> > [1] it does not support all features and unfortunately it is not clear > >> > how limited it is. Also, it is some kind of transitional mechanism > >> > which was not designed for being a long term solution. > >> > > >> > And I fully support an idea of refactoring GridAbstractTest. I think > >> > it is possible to make a significant improvement here. > >> > > >> > [1] > >> > > >> https://junit.org/junit5/docs/current/user-guide/#running-tests-junit-platform-runner > >> > > >> > пн, 25 февр. 2019 г. в 17:41, Ivan Fedotov : > >> > > > >> > > Hello Nikolay. > >> > > > >> > > The prime benefits are more comfortable work with flaky tests, Java 8 > >> > tests > >> > > compatibility, user-friendly syntaxis in parametrized tests and > >> others. > >> > > The most significant features list you can find in IEP-30 Motivation > >> > > section. > >> > > > >> > > If you have any specific questions about JUnit5 feel free to ask me. > >> > > > >> > > пн, 25 февр. 2019 г. в 16:55, Nikolay Izhikov : > >> > > > >> > > > Hello, Ivan. > >> > > > > >> > > > May be I miss some mail - if yes, can you repeat it. > >> > > > What is advantages of migration from junit 4 to 5? > >> > > > Why we should do it? > >> > > > > >> > > > > >> > > > пн, 25 февр. 2019 г. в 16:33, Ivan Fedotov : > >> > > > > >> > > > > Ivan, > >> > > > > That is my thoughts according to your questions. > >> > > > > > >> > > > > 1. I tried to implement test suits with JUnit4 compatibility > >> layer. > >> > The > >> > > > >
Re: SpiUriDeploy TC job fails on Java 9+
Folks, I made SpiUriDeploy job successful by enforcing jdk 8 and not passing any extra jvm arguments for an extra compile step (Step 4: Pre-compile external data). Everything is running fine so far [1]. [1] https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_SpiUriDeploy=buildTypeHistoryList_IgniteTests24Java8=%3Cdefault%3E ср, 3 июл. 2019 г. в 14:46, Павлухин Иван : > > Ilya, Dmitriy, > > Are we going to drop only compilation on Java 9 and 10 but continue > running tests on all versions (8-11)? The idea sounds ok to me. But to > be honest I must say that currently we compile Ignite only using Java > 8. > > Also it does not help me to fix SpiUriDeploy job. It has an extra > compilation step into configuration and currently neither forcing jdk > 8 nor using jdk version parameter passed to a running job fixes the > issue. If we drop Java 9 and 10 compilation the only will be "careful > forcing" jdk 8 in the extra step in SpiUriDeploy. Actually sounds > unpleasant, compilation settings spreads among different jobs > configurations.. > > ср, 3 июл. 2019 г. в 13:23, Dmitriy Pavlov : > > > > It was by mistake. Let's drop 9 && 10 compilation. > > > > ср, 3 июл. 2019 г. в 12:54, Ilya Kasnacheev : > > > > > Hello! > > > > > > Dmitry, why did you switch maven.compiler.target to 11 when using Java 9+ > > > in maven.compiler.target? Indeed it breaks compilation on Java 9-10. This > > > is in IGNITE-11189 > > > > > > BTW they are unsupported now, maybe we just drop them for good? Ivan what > > > do you think? > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > ср, 3 июл. 2019 г. в 11:39, Павлухин Иван : > > > > > >> Hi Ilya, > > >> > > >> Yep there is no problem with Java 12 and 11 as well because a > > >> following option is specified in parent pom: > > >> 11 > > >> > > >> The only easy way I found is to use > > >> 9 (for java-9+ > > >> profile). Another option could be supplying maven with > > >> -Dmaven.compiler.target argument calculated for a used jdk version. > > >> > > >> What do you think? > > >> > > >> вт, 2 июл. 2019 г. в 16:58, Ilya Kasnacheev : > > >> > > > >> > Hello! > > >> > > > >> > I have just tried, looks like it is buildable with Java 12 (provided > > >> that > > >> > -Dmaven.javadoc.skip=true is specified) > > >> > > > >> > Regards, > > >> > -- > > >> > Ilya Kasnacheev > > >> > > > >> > > > >> > пн, 1 июл. 2019 г. в 22:04, Павлухин Иван : > > >> > > > >> > > Hi, > > >> > > > > >> > > SpiUriDeploy fails on TC when it is run with any Java 9+ version [1]. > > >> > > The reason is very simple, it contains one preliminary build step > > >> > > which is forced to run with Java 8. But for Java 9+ builds we pass > > >> > > module-related arguments and consequently it leads to Java 8 failure > > >> > > (unknown argument). I tried to run that step on JDK used for all > > >> > > other > > >> > > steps but faced a compilation error. It was on Java 9 and it said > > >> > > that > > >> > > target version 11 was not supported. Due to some reason we configure > > >> > > 11 in parent pom for > > >> > > java-9+ profile. > > >> > > > > >> > > In my mind simple solution here is to set target version to 9. But > > >> > > perhaps there is a clever way to use maximum version supported by a > > >> > > used JDK. > > >> > > > > >> > > Do you have an ideas? > > >> > > > > >> > > [1] > > >> > > > > >> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_SpiUriDeploy_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv > > >> > > > > >> > > -- > > >> > > Best regards, > > >> > > Ivan Pavlukhin > > >> > > > > >> > > >> > > >> > > >> -- > > >> Best regards, > > >> Ivan Pavlukhin > > >> > > > > > > > -- > Best regards, > Ivan Pavlukhin -- Best regards, Ivan Pavlukhin
Re: [MTCGA]: new failures in builds [4255938] needs to be handled
Seems to be an interference of two commits related to continuous queries [1, 2]. In one a test was added and the test was broken by another commit. Ivan, Denis, could please take a look? [1] https://github.com/apache/ignite/commit/db74e92dcd56bc5a45ed9d3a382be8bcfa2a76dc [2] https://github.com/apache/ignite/commit/2cea7e245927e8e54b93f7adea7524ce68c6d226 чт, 4 июл. 2019 г. в 03:57, : > > Hi Igniters, > > I've detected some new issue on TeamCity to be handled. You are more than > welcomed to help. > > If your changes can lead to this failure(s): We're grateful that you were a > volunteer to make the contribution to this project, but things change and you > may no longer be able to finalize your contribution. > Could you respond to this email and indicate if you wish to continue and fix > test failures or step down and some committer may revert you commit. > > *New test failure in master > ContinuousQueryDeserializationErrorOnNodeJoinTest.testDeserializationErrorOnJoiningNode > > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=2327531993912363014=%3Cdefault%3E=testDetails > Changes may lead to failure were done by > - dchu...@gridgain.com > https://ci.ignite.apache.org/viewModification.html?modId=887157 > - bessonov...@gmail.com > https://ci.ignite.apache.org/viewModification.html?modId=887155 > > - Here's a reminder of what contributors were agreed to do > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute > - Should you have any questions please contact dev@ignite.apache.org > > Best Regards, > Apache Ignite TeamCity Bot > https://github.com/apache/ignite-teamcity-bot > Notification generated at 03:57:01 04-07-2019 -- Best regards, Ivan Pavlukhin
Re: Revisit added java modules in ignite.sh
I removed command line arguments enabling java.transaction and java.corba module from TC configuration. All activity in a ticket [1]. [1] https://issues.apache.org/jira/browse/IGNITE-11946 вс, 30 июн. 2019 г. в 07:31, Павлухин Иван : > > I made 2 experiments: > 1. Removed --add-modules=java.transaction from TC configurations. I > have not observed any new test failures after that change. > 2. Removed that option from ignite.sh in PR. In PR problematic thin > client suites are green. > > See the ticket [1] for additional details. If there is no objections I > will cleanup that option from TC configurations and Ignite scripts. > > [1] https://issues.apache.org/jira/browse/IGNITE-9687 > > пт, 28 июн. 2019 г. в 10:09, Павлухин Иван : > > > > Folks, > > > > During investigation of thin client test suite failures [1] I found > > that it is not clear whether --add-modules=java.transaction in > > ignite.sh/bat is needed (we add this module for Java 9 and 10). In > > thread [1] I already mentioned that launching ignite.sh with Java 9 > > and ignite-jta libraries on a classpath leads to an error. You can see > > how it can look in a log [2]. > > > > My proposal here is to get rid off --add-modules=java.transaction from > > stratup scripts and parameters used by TC. And I will try to explain > > why. > > > > The story started in a ticket [3]. Where a recipe > > --add-modules=java.transaction > > --patch-module=java.transaction=$HOME/.m2/repository/javax/transaction/javax.transaction-api/1.3/javax.transaction-api-1.3.jar > > allowed to execute JTA tests successfully. After that > > --add-modules=java.transaction was added to launch scripts. But that > > setup allowed to launch tests. And they failed actually due to > > problems with resolving needed classes for test depndencies > > (org.ow2.jotm:jotm-core). Needed classes were not found because the > > dependency required classes from java standard library which were > > moved to optional modules. And the thing is even trickier with a > > --patch-module option because javax.transaction-api-1.3.jar contains a > > superset of classes comparing to java.transaction module. So, I do not > > see a need of adding this module. Moreover that module was removed > > completely starting from Java 11. And even if missed something and it > > has some meaning for Java 9/10, I still think that having uniform > > setup for all Java versions simplifies matters. > > > > Have I missed something? Do we have an example how to check JTA > > integration? How it can be used with a standalone Ignite setup? > > > > P.S. Also I will try to do a similar reseach about java.xml.bind > > module if noone knows why do we need it. > > > > [1] > > http://apache-ignite-developers.2346864.n4.nabble.com/Thin-client-test-suites-failure-td42388.html > > [2] > > https://ci.ignite.apache.org/viewLog.html?buildId=4217786=IgniteTests24Java8_ThinClientNodeJs=buildLog_IgniteTests24Java8=pull%2F6644%2Fhead&_focus=423#_state=423 > > [3] https://issues.apache.org/jira/browse/IGNITE-9687 > > -- > > Best regards, > > Ivan Pavlukhin > > > > -- > Best regards, > Ivan Pavlukhin -- Best regards, Ivan Pavlukhin
Re: SpiUriDeploy TC job fails on Java 9+
Ilya, Dmitriy, Are we going to drop only compilation on Java 9 and 10 but continue running tests on all versions (8-11)? The idea sounds ok to me. But to be honest I must say that currently we compile Ignite only using Java 8. Also it does not help me to fix SpiUriDeploy job. It has an extra compilation step into configuration and currently neither forcing jdk 8 nor using jdk version parameter passed to a running job fixes the issue. If we drop Java 9 and 10 compilation the only will be "careful forcing" jdk 8 in the extra step in SpiUriDeploy. Actually sounds unpleasant, compilation settings spreads among different jobs configurations.. ср, 3 июл. 2019 г. в 13:23, Dmitriy Pavlov : > > It was by mistake. Let's drop 9 && 10 compilation. > > ср, 3 июл. 2019 г. в 12:54, Ilya Kasnacheev : > > > Hello! > > > > Dmitry, why did you switch maven.compiler.target to 11 when using Java 9+ > > in maven.compiler.target? Indeed it breaks compilation on Java 9-10. This > > is in IGNITE-11189 > > > > BTW they are unsupported now, maybe we just drop them for good? Ivan what > > do you think? > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > ср, 3 июл. 2019 г. в 11:39, Павлухин Иван : > > > >> Hi Ilya, > >> > >> Yep there is no problem with Java 12 and 11 as well because a > >> following option is specified in parent pom: > >> 11 > >> > >> The only easy way I found is to use > >> 9 (for java-9+ > >> profile). Another option could be supplying maven with > >> -Dmaven.compiler.target argument calculated for a used jdk version. > >> > >> What do you think? > >> > >> вт, 2 июл. 2019 г. в 16:58, Ilya Kasnacheev : > >> > > >> > Hello! > >> > > >> > I have just tried, looks like it is buildable with Java 12 (provided > >> that > >> > -Dmaven.javadoc.skip=true is specified) > >> > > >> > Regards, > >> > -- > >> > Ilya Kasnacheev > >> > > >> > > >> > пн, 1 июл. 2019 г. в 22:04, Павлухин Иван : > >> > > >> > > Hi, > >> > > > >> > > SpiUriDeploy fails on TC when it is run with any Java 9+ version [1]. > >> > > The reason is very simple, it contains one preliminary build step > >> > > which is forced to run with Java 8. But for Java 9+ builds we pass > >> > > module-related arguments and consequently it leads to Java 8 failure > >> > > (unknown argument). I tried to run that step on JDK used for all other > >> > > steps but faced a compilation error. It was on Java 9 and it said that > >> > > target version 11 was not supported. Due to some reason we configure > >> > > 11 in parent pom for > >> > > java-9+ profile. > >> > > > >> > > In my mind simple solution here is to set target version to 9. But > >> > > perhaps there is a clever way to use maximum version supported by a > >> > > used JDK. > >> > > > >> > > Do you have an ideas? > >> > > > >> > > [1] > >> > > > >> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_SpiUriDeploy_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv > >> > > > >> > > -- > >> > > Best regards, > >> > > Ivan Pavlukhin > >> > > > >> > >> > >> > >> -- > >> Best regards, > >> Ivan Pavlukhin > >> > > -- Best regards, Ivan Pavlukhin
Re: Fwd: [DISCUSSION][IEP-35] Metrics configuration
Hi Nikolay,
Thank you for clarifications.
> What is "disabled" sensor? Why do we need it? (as long as sensort is just a
> AtomicLong)
I meant a possibility of disabling a particular metric value
calculation (skipping AtomicLong.incrementAndGet call). My current
understanding that we do NOT need it. It seems that we are on a same
page.
And how do you find named parameters approach mentioned before?
cache.my-cahe.GetLatency={
intervals: [50, 100, 250, 500]
}
пн, 1 июл. 2019 г. в 16:08, Nikolay Izhikov :
>
> Hello, Ivan.
>
> Thanks for the feedback!
>
> > 1. Specifying subset of metrics which are exported to an external system.
>
> Covered by exporter filter [1]
>
> > 2. Subset of metrics which is collected (enable/disable sensor).
>
> What is "disabled" sensor?
> Why do we need it? (as long as sensort is just a AtomicLong)
>
> > 3. A particular metric (sensor) parameters.
>
> This is the point of the discussed ticket [2]
>
>
> [1]
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/metric/PushMetricsExporterAdapter.java#L41
> [2] https://issues.apache.org/jira/browse/IGNITE-11927
>
>
> В Пн, 01/07/2019 в 15:39 +0300, Павлухин Иван пишет:
> > Nikolay, Igniters,
> >
> > In my mind there are several configuration aspects:
> > 1. Specifying subset of metrics which are exported to an external system.
> > 2. Subset of metrics which is collected (enable/disable sensor).
> > 3. A particular metric (sensor) parameters.
> >
> > Are we going to address all points in the same config (file)? In my
> > experience with metrics (actually not so rich) I have not seen metric
> > configs similar to logging configs. Are there examples of such
> > practice in industry?
> >
> > Also, looking at a line like "cache.my-cahe.GetLatency=50,100,250,500"
> > I cannot tell much from scratch. My first thought is that having named
> > parameters can be more readable, e.g. (roughly):
> > cache.my-cahe.GetLatency={
> > intervals: [50, 100, 250, 500]
> > }
> >
> > пт, 28 июн. 2019 г. в 13:02, Nikolay Izhikov :
> > >
> > > Hello, Alexey.
> > >
> > > Thanks for the feedback!
> > >
> > > > My only concert is that we should have the metrics framework
> > > > configuration
> > > > as the first-citizen of the framework itself
> > >
> > > Yes. I planned to add `void configure(String param)` method to the metric
> > > API.
> > >
> > > > but change the metrics parameters in
> > > > runtime from JMX or command-line, etc.
> > >
> > > I've add requirement of JMX method to the ticket:
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-11927
> > >
> > > > Another concern is to have an
> > > > ability to disable/enable metrics per metrics group/prefix.
> > >
> > > Yes, we discusss it.
> > > But, let's make it clear:
> > >
> > > *What is disabling metric?*
> > >
> > > Looks like exporter filter solve this task.
> > >
> > > В Чт, 27/06/2019 в 16:24 +0300, Alexey Goncharuk пишет:
> > > > Nikolay,
> > > >
> > > > My only concert is that we should have the metrics framework
> > > > configuration
> > > > as the first-citizen of the framework itself. This way, we can configure
> > > > the metrics not only from file, but change the metrics parameters in
> > > > runtime from JMX or command-line, etc. Another concern is to have an
> > > > ability to disable/enable metrics per metrics group/prefix.
> > > >
> > > > The logger-like configuration meets these suggestions given that the
> > > > configuration is generalized into the metrics framework.
> > > >
> > > > What do you think?
> > > >
> > > > чт, 27 июн. 2019 г. в 12:30, Nikolay Izhikov :
> > > >
> > > > > Hello, Igniters.
> > > > >
> > > > > As you may know, I've contributed Phase1 [1] for IEP-35 [2].
> > > > > Now we have metrics subsystem and can create and export any metrics
> > > > > from
> > > > > Ignite.
> > > > >
> > > > > I think user(administrator of Ignite) should be able to configure some
> > > > > metrics params in a common way [3]
> > > > >
> > > > > I propose to use the same way from logging frameworks.
> > &
Re: SpiUriDeploy TC job fails on Java 9+
Hi Ilya, Yep there is no problem with Java 12 and 11 as well because a following option is specified in parent pom: 11 The only easy way I found is to use 9 (for java-9+ profile). Another option could be supplying maven with -Dmaven.compiler.target argument calculated for a used jdk version. What do you think? вт, 2 июл. 2019 г. в 16:58, Ilya Kasnacheev : > > Hello! > > I have just tried, looks like it is buildable with Java 12 (provided that > -Dmaven.javadoc.skip=true is specified) > > Regards, > -- > Ilya Kasnacheev > > > пн, 1 июл. 2019 г. в 22:04, Павлухин Иван : > > > Hi, > > > > SpiUriDeploy fails on TC when it is run with any Java 9+ version [1]. > > The reason is very simple, it contains one preliminary build step > > which is forced to run with Java 8. But for Java 9+ builds we pass > > module-related arguments and consequently it leads to Java 8 failure > > (unknown argument). I tried to run that step on JDK used for all other > > steps but faced a compilation error. It was on Java 9 and it said that > > target version 11 was not supported. Due to some reason we configure > > 11 in parent pom for > > java-9+ profile. > > > > In my mind simple solution here is to set target version to 9. But > > perhaps there is a clever way to use maximum version supported by a > > used JDK. > > > > Do you have an ideas? > > > > [1] > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_SpiUriDeploy_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv > > > > -- > > Best regards, > > Ivan Pavlukhin > > -- Best regards, Ivan Pavlukhin
Re: Ignite Modularization
Denis, > I think this should be optional. Do you think we need to do it in the first > instance In my opinion we should at least have a solid understanding of the subject. I think it worth having a separate discussion regarding Java 9 modules. Today java modular subsystem seems to be not widely adopted yet. And many Java library developers are satisfied delivering their jars to be used as automatic modules. I think that today Java 8 is the most widely used version. But the situation can change soon and we should keep our eyes peeled. вт, 2 июл. 2019 г. в 00:39, Denis Magda : > > Hi Ivan, > > Thanks for looking into this. > > 1. I did not get from IEP whether Thin Clients have a separate > > repository and a release lifecycle or not. > > > Sorry for the confusion. Yes, such clients have to be in separate repos and > might have their own lifecycles. Updated the wiki. > > > > 2. Are we going to exclude tests for unsupported modules from Ignite > > TeamCity? > > > Yes, that's my thinking, an unsupported integration won't be part of Ignite > modular ecosystem and won't be tested by the community. Any objections? > > > > 3. Will we adress implementing Java 9+ modules during that process? > > > I think this should be optional. Do you think we need to do it in the > first instance? > > - > Denis > > > On Sun, Jun 30, 2019 at 10:45 PM Павлухин Иван wrote: > > > Hi Denis, > > > > I fully support the idea. Could you please clarify on following points: > > 1. I did not get from IEP whether Thin Clients have a separate > > repository and a release lifecycle or not. > > 2. Are we going to exclude tests for unsupported modules from Ignite > > TeamCity? > > 3. Will we adress implementing Java 9+ modules during that process? > > > > чт, 27 июн. 2019 г. в 18:11, Denis Magda : > > > > > > Ignite developers and users, > > > > > > I'd like us to consider Ignite modularization as part of Ignite 3.0 > > > timeframe. Presently, Ignite codebase mixes both core capabilities with > > 3rd > > > party integrations. It leads to the following: > > > > > >- Cumbersome and continuously growing codebase with many 3rd-party > > >dependencies. > > >- Some of the integrations are questionable and should no longer be > > >supported by the community at all. > > >- Integrations evolution is bound to Ignite release cycles even though > > >no changes are needed in the core. > > >- Ignite community has to support everything (test, release, fix, > > >continue development) which requires to have particular integration > > experts > > >on a permanent basis - doesn't work. > > > > > > Here is an IEP: > > > > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization > > > > > > Please review it, share feedback. Pay attention to the list of > > integrations > > > that should no longer be supported by the community. > > > > > > > > > - > > > Denis > > > > > > > > -- > > Best regards, > > Ivan Pavlukhin > > -- Best regards, Ivan Pavlukhin
Re: [PROBLEM] Is TC down?
Sorry for that. Now everything works completely fine for me. Seems to be some surprise from my ISP. вт, 2 июл. 2019 г. в 09:18, Павлухин Иван : > > Igniters, > > Currently TC web UI is not loaded in my environment and reports > ERR_CONNECTION_TIMED_OUT. Is the problem on my side or is TC down? > > -- > Best regards, > Ivan Pavlukhin -- Best regards, Ivan Pavlukhin
[PROBLEM] Is TC down?
Igniters, Currently TC web UI is not loaded in my environment and reports ERR_CONNECTION_TIMED_OUT. Is the problem on my side or is TC down? -- Best regards, Ivan Pavlukhin
SpiUriDeploy TC job fails on Java 9+
Hi, SpiUriDeploy fails on TC when it is run with any Java 9+ version [1]. The reason is very simple, it contains one preliminary build step which is forced to run with Java 8. But for Java 9+ builds we pass module-related arguments and consequently it leads to Java 8 failure (unknown argument). I tried to run that step on JDK used for all other steps but faced a compilation error. It was on Java 9 and it said that target version 11 was not supported. Due to some reason we configure 11 in parent pom for java-9+ profile. In my mind simple solution here is to set target version to 9. But perhaps there is a clever way to use maximum version supported by a used JDK. Do you have an ideas? [1] https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_SpiUriDeploy_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv -- Best regards, Ivan Pavlukhin
Fwd: [DISCUSSION][IEP-35] Metrics configuration
Nikolay, Igniters,
In my mind there are several configuration aspects:
1. Specifying subset of metrics which are exported to an external system.
2. Subset of metrics which is collected (enable/disable sensor).
3. A particular metric (sensor) parameters.
Are we going to address all points in the same config (file)? In my
experience with metrics (actually not so rich) I have not seen metric
configs similar to logging configs. Are there examples of such
practice in industry?
Also, looking at a line like "cache.my-cahe.GetLatency=50,100,250,500"
I cannot tell much from scratch. My first thought is that having named
parameters can be more readable, e.g. (roughly):
cache.my-cahe.GetLatency={
intervals: [50, 100, 250, 500]
}
пт, 28 июн. 2019 г. в 13:02, Nikolay Izhikov :
>
> Hello, Alexey.
>
> Thanks for the feedback!
>
> > My only concert is that we should have the metrics framework configuration
> > as the first-citizen of the framework itself
>
> Yes. I planned to add `void configure(String param)` method to the metric API.
>
> > but change the metrics parameters in
> > runtime from JMX or command-line, etc.
>
> I've add requirement of JMX method to the ticket:
>
> https://issues.apache.org/jira/browse/IGNITE-11927
>
> > Another concern is to have an
> > ability to disable/enable metrics per metrics group/prefix.
>
> Yes, we discusss it.
> But, let's make it clear:
>
> *What is disabling metric?*
>
> Looks like exporter filter solve this task.
>
> В Чт, 27/06/2019 в 16:24 +0300, Alexey Goncharuk пишет:
> > Nikolay,
> >
> > My only concert is that we should have the metrics framework configuration
> > as the first-citizen of the framework itself. This way, we can configure
> > the metrics not only from file, but change the metrics parameters in
> > runtime from JMX or command-line, etc. Another concern is to have an
> > ability to disable/enable metrics per metrics group/prefix.
> >
> > The logger-like configuration meets these suggestions given that the
> > configuration is generalized into the metrics framework.
> >
> > What do you think?
> >
> > чт, 27 июн. 2019 г. в 12:30, Nikolay Izhikov :
> >
> > > Hello, Igniters.
> > >
> > > As you may know, I've contributed Phase1 [1] for IEP-35 [2].
> > > Now we have metrics subsystem and can create and export any metrics from
> > > Ignite.
> > >
> > > I think user(administrator of Ignite) should be able to configure some
> > > metrics params in a common way [3]
> > >
> > > I propose to use the same way from logging frameworks.
> > > We should define some file format Ignite can understand.
> > > An administrator fills configuration file to configure one or several
> > > metrics.
> > > Ignite will analyze the file and use provided params during metrics
> > > creation.
> > >
> > > For now, we have 2 types of metrics that should be configured:
> > >
> > > * HistrogramMetric [4]
> > > This metric is a count of measurement that falls into
> > > predefined intervals.
> > > An example is "Request processing time distribution".
> > > We want to calculate a count of requests processed quicker
> > > then 50ms, 50-100, 100-250, 250-500 and slower.
> > >
> > > * HitRateMetric [5]
> > > This metric is a count of events in the last time
> > > interval.
> > > An example is the "Count of requests processed in the last
> > > 5 seconds".
> > >
> > > Example of file content:
> > >
> > >
> > > cache.my-cahe.GetLatency=50,100,250,500 #Params for the histogram metric
> > > with the name `cache.my-cahe.get`
> > > cache.my-cache.RebalancingKeysRate=6 #Param for existing HitRateMetric
> > > that hold "Estimated rebalancing speed in keys".
> > >
> > >
> > > Please, share your vision.
> > >
> > > [1]
> > > https://github.com/apache/ignite/commit/fdaa310430aefff07994eb35510d3416886b5bbe
> > > [2]
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=112820392
> > > [3] https://issues.apache.org/jira/browse/IGNITE-11927
> > > [4]
> > > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/metric/impl/HistogramMetric.java
> > > [5]
> > > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/metric/impl/HitRateMetric.java
> > >
--
Best regards,
Ivan Pavlukhin
-BEGIN PGP SIGNATURE-
iQEzBAABCgAdFiEEOiTcLcdgyP2exB5ZbiaPbjg91GUFAl0V5iYACgkQbiaPbjg9
1GWWDQf/Tfy5+lmuCTTBlrp8QetoTKrr+xdU4BuanBqLwwz8Yk+/OvA6fj1HmKDM
eIK+xGqIeYk7u8NMTURuIjTGZ+jBSd+q6riB2qaVBiR5ksRFxXCa7P+guDg8Kmm5
qQ9ocXy3tfCsQgG9Ig6d60GRuM7dW8GlUFtA/wt4zxpAA5zsdNyCJUNxjXY5zo6V
r8uSvKprxgMsXKiSiZsjoQXJtqlIoMJldeaNrq7i2zUV1w/nIeLKygLfMfI8Ty5n
2v2ocbIa07QekVBEz8JMx88grs3otmGBdJyCFwt1JZxRPsg2nEU+sJcRSs8jJl8P
tOGZbrfTByGFhwOTh8zFP55NfLm87g==
=T+jn
-END PGP SIGNATURE-
Re: Ignite Modularization
Hi Denis, I fully support the idea. Could you please clarify on following points: 1. I did not get from IEP whether Thin Clients have a separate repository and a release lifecycle or not. 2. Are we going to exclude tests for unsupported modules from Ignite TeamCity? 3. Will we adress implementing Java 9+ modules during that process? чт, 27 июн. 2019 г. в 18:11, Denis Magda : > > Ignite developers and users, > > I'd like us to consider Ignite modularization as part of Ignite 3.0 > timeframe. Presently, Ignite codebase mixes both core capabilities with 3rd > party integrations. It leads to the following: > >- Cumbersome and continuously growing codebase with many 3rd-party >dependencies. >- Some of the integrations are questionable and should no longer be >supported by the community at all. >- Integrations evolution is bound to Ignite release cycles even though >no changes are needed in the core. >- Ignite community has to support everything (test, release, fix, >continue development) which requires to have particular integration experts >on a permanent basis - doesn't work. > > Here is an IEP: > https://cwiki.apache.org/confluence/display/IGNITE/IEP-36%3A+Modularization > > Please review it, share feedback. Pay attention to the list of integrations > that should no longer be supported by the community. > > > - > Denis -- Best regards, Ivan Pavlukhin
Re: Revisit added java modules in ignite.sh
I made 2 experiments: 1. Removed --add-modules=java.transaction from TC configurations. I have not observed any new test failures after that change. 2. Removed that option from ignite.sh in PR. In PR problematic thin client suites are green. See the ticket [1] for additional details. If there is no objections I will cleanup that option from TC configurations and Ignite scripts. [1] https://issues.apache.org/jira/browse/IGNITE-9687 пт, 28 июн. 2019 г. в 10:09, Павлухин Иван : > > Folks, > > During investigation of thin client test suite failures [1] I found > that it is not clear whether --add-modules=java.transaction in > ignite.sh/bat is needed (we add this module for Java 9 and 10). In > thread [1] I already mentioned that launching ignite.sh with Java 9 > and ignite-jta libraries on a classpath leads to an error. You can see > how it can look in a log [2]. > > My proposal here is to get rid off --add-modules=java.transaction from > stratup scripts and parameters used by TC. And I will try to explain > why. > > The story started in a ticket [3]. Where a recipe > --add-modules=java.transaction > --patch-module=java.transaction=$HOME/.m2/repository/javax/transaction/javax.transaction-api/1.3/javax.transaction-api-1.3.jar > allowed to execute JTA tests successfully. After that > --add-modules=java.transaction was added to launch scripts. But that > setup allowed to launch tests. And they failed actually due to > problems with resolving needed classes for test depndencies > (org.ow2.jotm:jotm-core). Needed classes were not found because the > dependency required classes from java standard library which were > moved to optional modules. And the thing is even trickier with a > --patch-module option because javax.transaction-api-1.3.jar contains a > superset of classes comparing to java.transaction module. So, I do not > see a need of adding this module. Moreover that module was removed > completely starting from Java 11. And even if missed something and it > has some meaning for Java 9/10, I still think that having uniform > setup for all Java versions simplifies matters. > > Have I missed something? Do we have an example how to check JTA > integration? How it can be used with a standalone Ignite setup? > > P.S. Also I will try to do a similar reseach about java.xml.bind > module if noone knows why do we need it. > > [1] > http://apache-ignite-developers.2346864.n4.nabble.com/Thin-client-test-suites-failure-td42388.html > [2] > https://ci.ignite.apache.org/viewLog.html?buildId=4217786=IgniteTests24Java8_ThinClientNodeJs=buildLog_IgniteTests24Java8=pull%2F6644%2Fhead&_focus=423#_state=423 > [3] https://issues.apache.org/jira/browse/IGNITE-9687 > -- > Best regards, > Ivan Pavlukhin -- Best regards, Ivan Pavlukhin
Re: Thin client test suites failure
FYI I found out how a Java version and addinional java module parameters are passed to a build after choosing jdk version in a "Run custom build" dialog. A version is controlled by a reverse.dep.*.env.JAVA_HOME configuration parameter specified to behave like a combobox in a "~[Obsolete] Run test suites" template configuration. Addinional module parameters are present as JVM_EXTRA_ARGS in the same template configuration. A value for it is taken from "~Build Apache Ignite~" (which runs before any test suite) in a form %dep.IgniteTests24Java8_BuildApacheIgnite.JVM_EXTRA_ARGS%, simpy speaking "~Build Apache Ignite~" JVM_EXTRA_ARGS value is borrowed. "~Build Apache Ignite~" configures that parameter in a build step "Setup additional arguments" by running a special command line script, args are chosen depending on Java version. пт, 28 июн. 2019 г. в 08:46, Павлухин Иван : > > After a little bit more experimentation I tend to think that adding > java.transaction module is not needed and even brings a negative > effect. E.g. if we use an ignite binary release and enable optional > jta module (by copying libraries to lib folder) then a node simply > fails to start (failing to resolve some needed classes). I will start > a separate thread related to JTA module. > > пт, 21 июн. 2019 г. в 14:46, Павлухин Иван : > > > > My next question is why do we need adding java.transaction module > > explicitly for Java 9 and 10? As I see since 11 version there is no > > such module and classes from it in modules bundled with JDK. So, I > > suppose either we do not need java.transaction module or something > > does not work with Java 11. It is complicated... > > > > чт, 20 июн. 2019 г. в 17:13, Павлухин Иван : > > > > > > Igniters, > > > > > > I found one thing which might be reason of an observed behavior. The > > > problem does not appear for maven-based tests, while e.g. Python thin > > > client tests use ignite.sh for starting a cluster. So, I compared JVM > > > launch options and observed a following extra option in maven-based > > > runs: > > > --patch-module=java.transaction=~/.m2/repository/javax/transaction/javax.transaction-api/1.3/javax.transaction-api-1.3.jar > > > > > > I missed when this trick was employed? Can anybody help with it? Does > > > it mean that ignite.sh does not work for all supported Java versions? > > > > > > чт, 20 июн. 2019 г. в 13:30, Павлухин Иван : > > > > > > > > By the way, what about Java 9? Do we run tests using it? > > > > > > > > чт, 20 июн. 2019 г. в 13:28, Dmitriy Pavlov : > > > > > > > > > > Actually, TC Bot is not actively developed. I would like to say, > > > > > moreover, > > > > > help is needed there. > > > > > > > > > > I agree that 10095 is (intentionally) vague. Will separate > > > > > notification > > > > > based on JDK version be supported or not, depends on the time > > > > > available to > > > > > complete this task. > > > > > > > > > > чт, 20 июн. 2019 г. в 12:02, Павлухин Иван : > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > It is good to know that TC Bot is being developed actively. Just to > > > > > > double check. Will the mentioned issue [1] allow us to see alerts > > > > > > when > > > > > > a particular test fails only on a specific Java version? > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10095 > > > > > > > > > > > > чт, 20 июн. 2019 г. в 10:26, Dmitriy Pavlov : > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > About TC Bot: I was going to support filtering of builds from > > > > > > > same JVM > > > > > > for > > > > > > > TC Bot visa > > > > > > > https://issues.apache.org/jira/browse/IGNITE-10095 > > > > > > > > > > > > > > Now it is displayed only in UI as JDK8,JDK11 tags. I still do > > > > > > > investigations on how to retrieve/build run history faster than > > > > > > > it is > > > > > > now. > > > > > > > > > > > > > > Only nightly runs are random, and yes, Ignite is tested using > > > > > > > different
Revisit added java modules in ignite.sh
Folks, During investigation of thin client test suite failures [1] I found that it is not clear whether --add-modules=java.transaction in ignite.sh/bat is needed (we add this module for Java 9 and 10). In thread [1] I already mentioned that launching ignite.sh with Java 9 and ignite-jta libraries on a classpath leads to an error. You can see how it can look in a log [2]. My proposal here is to get rid off --add-modules=java.transaction from stratup scripts and parameters used by TC. And I will try to explain why. The story started in a ticket [3]. Where a recipe --add-modules=java.transaction --patch-module=java.transaction=$HOME/.m2/repository/javax/transaction/javax.transaction-api/1.3/javax.transaction-api-1.3.jar allowed to execute JTA tests successfully. After that --add-modules=java.transaction was added to launch scripts. But that setup allowed to launch tests. And they failed actually due to problems with resolving needed classes for test depndencies (org.ow2.jotm:jotm-core). Needed classes were not found because the dependency required classes from java standard library which were moved to optional modules. And the thing is even trickier with a --patch-module option because javax.transaction-api-1.3.jar contains a superset of classes comparing to java.transaction module. So, I do not see a need of adding this module. Moreover that module was removed completely starting from Java 11. And even if missed something and it has some meaning for Java 9/10, I still think that having uniform setup for all Java versions simplifies matters. Have I missed something? Do we have an example how to check JTA integration? How it can be used with a standalone Ignite setup? P.S. Also I will try to do a similar reseach about java.xml.bind module if noone knows why do we need it. [1] http://apache-ignite-developers.2346864.n4.nabble.com/Thin-client-test-suites-failure-td42388.html [2] https://ci.ignite.apache.org/viewLog.html?buildId=4217786=IgniteTests24Java8_ThinClientNodeJs=buildLog_IgniteTests24Java8=pull%2F6644%2Fhead&_focus=423#_state=423 [3] https://issues.apache.org/jira/browse/IGNITE-9687 -- Best regards, Ivan Pavlukhin
Re: Thin client test suites failure
After a little bit more experimentation I tend to think that adding java.transaction module is not needed and even brings a negative effect. E.g. if we use an ignite binary release and enable optional jta module (by copying libraries to lib folder) then a node simply fails to start (failing to resolve some needed classes). I will start a separate thread related to JTA module. пт, 21 июн. 2019 г. в 14:46, Павлухин Иван : > > My next question is why do we need adding java.transaction module > explicitly for Java 9 and 10? As I see since 11 version there is no > such module and classes from it in modules bundled with JDK. So, I > suppose either we do not need java.transaction module or something > does not work with Java 11. It is complicated... > > чт, 20 июн. 2019 г. в 17:13, Павлухин Иван : > > > > Igniters, > > > > I found one thing which might be reason of an observed behavior. The > > problem does not appear for maven-based tests, while e.g. Python thin > > client tests use ignite.sh for starting a cluster. So, I compared JVM > > launch options and observed a following extra option in maven-based > > runs: > > --patch-module=java.transaction=~/.m2/repository/javax/transaction/javax.transaction-api/1.3/javax.transaction-api-1.3.jar > > > > I missed when this trick was employed? Can anybody help with it? Does > > it mean that ignite.sh does not work for all supported Java versions? > > > > чт, 20 июн. 2019 г. в 13:30, Павлухин Иван : > > > > > > By the way, what about Java 9? Do we run tests using it? > > > > > > чт, 20 июн. 2019 г. в 13:28, Dmitriy Pavlov : > > > > > > > > Actually, TC Bot is not actively developed. I would like to say, > > > > moreover, > > > > help is needed there. > > > > > > > > I agree that 10095 is (intentionally) vague. Will separate notification > > > > based on JDK version be supported or not, depends on the time available > > > > to > > > > complete this task. > > > > > > > > чт, 20 июн. 2019 г. в 12:02, Павлухин Иван : > > > > > > > > > Dmitriy, > > > > > > > > > > It is good to know that TC Bot is being developed actively. Just to > > > > > double check. Will the mentioned issue [1] allow us to see alerts when > > > > > a particular test fails only on a specific Java version? > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10095 > > > > > > > > > > чт, 20 июн. 2019 г. в 10:26, Dmitriy Pavlov : > > > > > > > > > > > > Hi, > > > > > > > > > > > > About TC Bot: I was going to support filtering of builds from same > > > > > > JVM > > > > > for > > > > > > TC Bot visa > > > > > > https://issues.apache.org/jira/browse/IGNITE-10095 > > > > > > > > > > > > Now it is displayed only in UI as JDK8,JDK11 tags. I still do > > > > > > investigations on how to retrieve/build run history faster than it > > > > > > is > > > > > now. > > > > > > > > > > > > Only nightly runs are random, and yes, Ignite is tested using > > > > > > different > > > > > VMs > > > > > > because Ignite 2.7.5 declared support of, at least, Java 11. > > > > > > > > > > > > Sincerely, > > > > > > Dmitriy Pavlov > > > > > > > > > > > > ср, 19 июн. 2019 г. в 22:13, Павлухин Иван : > > > > > > > > > > > > > Maxim, > > > > > > > > > > > > > > Well, it is an interesting point. First of all I want to believe > > > > > > > that > > > > > > > issues occurring only on a specific Java version are rare. If so, > > > > > > > running tests on a single version for a visa should be enough. > > > > > > > > > > > > > > On the other hand, a uniform test environment sounds a good idea > > > > > > > (especially for a visa), preventing a test status change by > > > > > > > "magic". > > > > > > > > > > > > > > So, I think that it can be done as follows: > > > > > > > 1. Run tests for a visa on a baseline Java version (11?). > > > > > > > 2. Run Nightly builds
Re: [IEP-35] GridJobProcessorMetrics migration
Hi Nikolay, Alex,
A couple of my humble comments
> Aggregation should be done with the metric collect system(Prometheus,
> Graphite, etc.).
I like that statement very much!
> But, what if a user doesn't use any external monitoring system and wants to
> know the health of Ignite instance?
I think that we can add more capabilities if a real user demand
appears in future. Generally, Ignite is a cluster which almost every
time assumes an external monitoring for a production use.
And a couple of general questions regarding monitoring. If they are
answered in IEP you can simply redirect me there.
1. Are we going to preserve a compatibility with metrics present
before? Or are we going to keep only those making sense today?
2. Can we configure which supported metrics are calculated/exposed? Or
do we calculate/expose everything every time?
пн, 24 июн. 2019 г. в 12:46, Alex Plehanov :
>
> Hi Nikolay,
>
> I think "idle time" is a useful metric, but it can be calculated outside of
> Ignite using external monitoring system.
>
> About execution and waiting time, it's not the right way to calculate it
> using a jobs list. Will jobs list contain only active jobs? In this case,
> you can't calculate these metrics at all, since you don't know the time of
> finished jobs. If the list will contain all jobs (will it be unlimited?),
> iterating over this list will be resource consuming. In any way, it's much
> simpler (and sometimes only possible) for an external monitoring system to
> just get some scalar metric than iterate over a list with some condition.
>
> About aggregation, yes, in an ideal world aggregation should be done with
> the external monitoring system. But, what if a user doesn't use any
> external monitoring system and wants to know the health of Ignite instance?
> Do we have any plans to implement some simple aggregator and ship it with
> Ignite? Do we have plans to provide some presets for Ignite monitoring for
> popular monitoring systems? (These questions not related to this PR, but
> related to IEP at all)
>
> Also, some aggregation metrics ("max" for example) can't be effectively
> calculated using the external system (you should iterate over a jobs list
> again and still precision of such calculation will be no more than the time
> between probes).
--
Best regards,
Ivan Pavlukhin
Re: Thin client test suites failure
My next question is why do we need adding java.transaction module explicitly for Java 9 and 10? As I see since 11 version there is no such module and classes from it in modules bundled with JDK. So, I suppose either we do not need java.transaction module or something does not work with Java 11. It is complicated... чт, 20 июн. 2019 г. в 17:13, Павлухин Иван : > > Igniters, > > I found one thing which might be reason of an observed behavior. The > problem does not appear for maven-based tests, while e.g. Python thin > client tests use ignite.sh for starting a cluster. So, I compared JVM > launch options and observed a following extra option in maven-based > runs: > --patch-module=java.transaction=~/.m2/repository/javax/transaction/javax.transaction-api/1.3/javax.transaction-api-1.3.jar > > I missed when this trick was employed? Can anybody help with it? Does > it mean that ignite.sh does not work for all supported Java versions? > > чт, 20 июн. 2019 г. в 13:30, Павлухин Иван : > > > > By the way, what about Java 9? Do we run tests using it? > > > > чт, 20 июн. 2019 г. в 13:28, Dmitriy Pavlov : > > > > > > Actually, TC Bot is not actively developed. I would like to say, moreover, > > > help is needed there. > > > > > > I agree that 10095 is (intentionally) vague. Will separate notification > > > based on JDK version be supported or not, depends on the time available to > > > complete this task. > > > > > > чт, 20 июн. 2019 г. в 12:02, Павлухин Иван : > > > > > > > Dmitriy, > > > > > > > > It is good to know that TC Bot is being developed actively. Just to > > > > double check. Will the mentioned issue [1] allow us to see alerts when > > > > a particular test fails only on a specific Java version? > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10095 > > > > > > > > чт, 20 июн. 2019 г. в 10:26, Dmitriy Pavlov : > > > > > > > > > > Hi, > > > > > > > > > > About TC Bot: I was going to support filtering of builds from same JVM > > > > for > > > > > TC Bot visa > > > > > https://issues.apache.org/jira/browse/IGNITE-10095 > > > > > > > > > > Now it is displayed only in UI as JDK8,JDK11 tags. I still do > > > > > investigations on how to retrieve/build run history faster than it is > > > > now. > > > > > > > > > > Only nightly runs are random, and yes, Ignite is tested using > > > > > different > > > > VMs > > > > > because Ignite 2.7.5 declared support of, at least, Java 11. > > > > > > > > > > Sincerely, > > > > > Dmitriy Pavlov > > > > > > > > > > ср, 19 июн. 2019 г. в 22:13, Павлухин Иван : > > > > > > > > > > > Maxim, > > > > > > > > > > > > Well, it is an interesting point. First of all I want to believe > > > > > > that > > > > > > issues occurring only on a specific Java version are rare. If so, > > > > > > running tests on a single version for a visa should be enough. > > > > > > > > > > > > On the other hand, a uniform test environment sounds a good idea > > > > > > (especially for a visa), preventing a test status change by "magic". > > > > > > > > > > > > So, I think that it can be done as follows: > > > > > > 1. Run tests for a visa on a baseline Java version (11?). > > > > > > 2. Run Nightly builds on all supported versions, choosing version at > > > > > > random is a one of options for that. (BTW, do we need some changes > > > > > > in > > > > > > TC bot to send a proper alert when test fails on some versions and > > > > > > fails on others?). > > > > > > > > > > > > ср, 19 июн. 2019 г. в 20:33, Maxim Muzafarov : > > > > > > > > > > > > > > Folks, > > > > > > > > > > > > > > Maybe I'm missing something, but what is the reason for setting > > > > > > > JAVA_HOME randomly? For instance, some commit is working under > > > > > > > jdk8 > > > > > > > but fails under jdk9 (and we declare that 9-th version is > > > > > > > supported > > > > by > > > > > > > Igni
Re: Thin client test suites failure
Igniters, I found one thing which might be reason of an observed behavior. The problem does not appear for maven-based tests, while e.g. Python thin client tests use ignite.sh for starting a cluster. So, I compared JVM launch options and observed a following extra option in maven-based runs: --patch-module=java.transaction=~/.m2/repository/javax/transaction/javax.transaction-api/1.3/javax.transaction-api-1.3.jar I missed when this trick was employed? Can anybody help with it? Does it mean that ignite.sh does not work for all supported Java versions? чт, 20 июн. 2019 г. в 13:30, Павлухин Иван : > > By the way, what about Java 9? Do we run tests using it? > > чт, 20 июн. 2019 г. в 13:28, Dmitriy Pavlov : > > > > Actually, TC Bot is not actively developed. I would like to say, moreover, > > help is needed there. > > > > I agree that 10095 is (intentionally) vague. Will separate notification > > based on JDK version be supported or not, depends on the time available to > > complete this task. > > > > чт, 20 июн. 2019 г. в 12:02, Павлухин Иван : > > > > > Dmitriy, > > > > > > It is good to know that TC Bot is being developed actively. Just to > > > double check. Will the mentioned issue [1] allow us to see alerts when > > > a particular test fails only on a specific Java version? > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10095 > > > > > > чт, 20 июн. 2019 г. в 10:26, Dmitriy Pavlov : > > > > > > > > Hi, > > > > > > > > About TC Bot: I was going to support filtering of builds from same JVM > > > for > > > > TC Bot visa > > > > https://issues.apache.org/jira/browse/IGNITE-10095 > > > > > > > > Now it is displayed only in UI as JDK8,JDK11 tags. I still do > > > > investigations on how to retrieve/build run history faster than it is > > > now. > > > > > > > > Only nightly runs are random, and yes, Ignite is tested using different > > > VMs > > > > because Ignite 2.7.5 declared support of, at least, Java 11. > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > > > ср, 19 июн. 2019 г. в 22:13, Павлухин Иван : > > > > > > > > > Maxim, > > > > > > > > > > Well, it is an interesting point. First of all I want to believe that > > > > > issues occurring only on a specific Java version are rare. If so, > > > > > running tests on a single version for a visa should be enough. > > > > > > > > > > On the other hand, a uniform test environment sounds a good idea > > > > > (especially for a visa), preventing a test status change by "magic". > > > > > > > > > > So, I think that it can be done as follows: > > > > > 1. Run tests for a visa on a baseline Java version (11?). > > > > > 2. Run Nightly builds on all supported versions, choosing version at > > > > > random is a one of options for that. (BTW, do we need some changes in > > > > > TC bot to send a proper alert when test fails on some versions and > > > > > fails on others?). > > > > > > > > > > ср, 19 июн. 2019 г. в 20:33, Maxim Muzafarov : > > > > > > > > > > > > Folks, > > > > > > > > > > > > Maybe I'm missing something, but what is the reason for setting > > > > > > JAVA_HOME randomly? For instance, some commit is working under jdk8 > > > > > > but fails under jdk9 (and we declare that 9-th version is supported > > > by > > > > > > Ignite). Should we merge this commit to the master branch (all test > > > > > > suites can be OK under jdk8)? > > > > > > > > > > > > I know, that running all tests under all java version leads us to > > > > > > Hell, but maybe we can have separated versions nightly builds? > > > > > > > > > > > > On Wed, 19 Jun 2019 at 13:47, Павлухин Иван > > > wrote: > > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > Thank you for the hint! Java version seems to be the reason. I see > > > > > > > that tests pass with Java 8 and 11 and fail with Java 10. See the > > > > > > > exception below. Btw, do we test with Java 9?
Re: Thin client test suites failure
By the way, what about Java 9? Do we run tests using it? чт, 20 июн. 2019 г. в 13:28, Dmitriy Pavlov : > > Actually, TC Bot is not actively developed. I would like to say, moreover, > help is needed there. > > I agree that 10095 is (intentionally) vague. Will separate notification > based on JDK version be supported or not, depends on the time available to > complete this task. > > чт, 20 июн. 2019 г. в 12:02, Павлухин Иван : > > > Dmitriy, > > > > It is good to know that TC Bot is being developed actively. Just to > > double check. Will the mentioned issue [1] allow us to see alerts when > > a particular test fails only on a specific Java version? > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10095 > > > > чт, 20 июн. 2019 г. в 10:26, Dmitriy Pavlov : > > > > > > Hi, > > > > > > About TC Bot: I was going to support filtering of builds from same JVM > > for > > > TC Bot visa > > > https://issues.apache.org/jira/browse/IGNITE-10095 > > > > > > Now it is displayed only in UI as JDK8,JDK11 tags. I still do > > > investigations on how to retrieve/build run history faster than it is > > now. > > > > > > Only nightly runs are random, and yes, Ignite is tested using different > > VMs > > > because Ignite 2.7.5 declared support of, at least, Java 11. > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > ср, 19 июн. 2019 г. в 22:13, Павлухин Иван : > > > > > > > Maxim, > > > > > > > > Well, it is an interesting point. First of all I want to believe that > > > > issues occurring only on a specific Java version are rare. If so, > > > > running tests on a single version for a visa should be enough. > > > > > > > > On the other hand, a uniform test environment sounds a good idea > > > > (especially for a visa), preventing a test status change by "magic". > > > > > > > > So, I think that it can be done as follows: > > > > 1. Run tests for a visa on a baseline Java version (11?). > > > > 2. Run Nightly builds on all supported versions, choosing version at > > > > random is a one of options for that. (BTW, do we need some changes in > > > > TC bot to send a proper alert when test fails on some versions and > > > > fails on others?). > > > > > > > > ср, 19 июн. 2019 г. в 20:33, Maxim Muzafarov : > > > > > > > > > > Folks, > > > > > > > > > > Maybe I'm missing something, but what is the reason for setting > > > > > JAVA_HOME randomly? For instance, some commit is working under jdk8 > > > > > but fails under jdk9 (and we declare that 9-th version is supported > > by > > > > > Ignite). Should we merge this commit to the master branch (all test > > > > > suites can be OK under jdk8)? > > > > > > > > > > I know, that running all tests under all java version leads us to > > > > > Hell, but maybe we can have separated versions nightly builds? > > > > > > > > > > On Wed, 19 Jun 2019 at 13:47, Павлухин Иван > > wrote: > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > Thank you for the hint! Java version seems to be the reason. I see > > > > > > that tests pass with Java 8 and 11 and fail with Java 10. See the > > > > > > exception below. Btw, do we test with Java 9? Have not seen that > > > > > > version in recent runs. > > > > > > java.lang.NoClassDefFoundError: javax/transaction/SystemException > > > > > > at java.base/java.lang.Class.forName0(Native Method) > > > > > > at java.base/java.lang.Class.forName(Class.java:291) > > > > > > at > > > > > > org.apache.ignite.internal.IgniteComponentType.createOptional0(IgniteComponentType.java:249) > > > > > > at > > > > > > org.apache.ignite.internal.IgniteComponentType.createOptional(IgniteComponentType.java:235) > > > > > > at > > > > > > org.apache.ignite.internal.processors.cache.GridCacheProcessor.createSharedContext(GridCacheProcessor.java:3427) > > > > > > at > > > > > > org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:869) > > > > > > at > > > > &g
Re: Thin client test suites failure
Dmitriy, It is good to know that TC Bot is being developed actively. Just to double check. Will the mentioned issue [1] allow us to see alerts when a particular test fails only on a specific Java version? [1] https://issues.apache.org/jira/browse/IGNITE-10095 чт, 20 июн. 2019 г. в 10:26, Dmitriy Pavlov : > > Hi, > > About TC Bot: I was going to support filtering of builds from same JVM for > TC Bot visa > https://issues.apache.org/jira/browse/IGNITE-10095 > > Now it is displayed only in UI as JDK8,JDK11 tags. I still do > investigations on how to retrieve/build run history faster than it is now. > > Only nightly runs are random, and yes, Ignite is tested using different VMs > because Ignite 2.7.5 declared support of, at least, Java 11. > > Sincerely, > Dmitriy Pavlov > > ср, 19 июн. 2019 г. в 22:13, Павлухин Иван : > > > Maxim, > > > > Well, it is an interesting point. First of all I want to believe that > > issues occurring only on a specific Java version are rare. If so, > > running tests on a single version for a visa should be enough. > > > > On the other hand, a uniform test environment sounds a good idea > > (especially for a visa), preventing a test status change by "magic". > > > > So, I think that it can be done as follows: > > 1. Run tests for a visa on a baseline Java version (11?). > > 2. Run Nightly builds on all supported versions, choosing version at > > random is a one of options for that. (BTW, do we need some changes in > > TC bot to send a proper alert when test fails on some versions and > > fails on others?). > > > > ср, 19 июн. 2019 г. в 20:33, Maxim Muzafarov : > > > > > > Folks, > > > > > > Maybe I'm missing something, but what is the reason for setting > > > JAVA_HOME randomly? For instance, some commit is working under jdk8 > > > but fails under jdk9 (and we declare that 9-th version is supported by > > > Ignite). Should we merge this commit to the master branch (all test > > > suites can be OK under jdk8)? > > > > > > I know, that running all tests under all java version leads us to > > > Hell, but maybe we can have separated versions nightly builds? > > > > > > On Wed, 19 Jun 2019 at 13:47, Павлухин Иван wrote: > > > > > > > > Dmitriy, > > > > > > > > Thank you for the hint! Java version seems to be the reason. I see > > > > that tests pass with Java 8 and 11 and fail with Java 10. See the > > > > exception below. Btw, do we test with Java 9? Have not seen that > > > > version in recent runs. > > > > java.lang.NoClassDefFoundError: javax/transaction/SystemException > > > > at java.base/java.lang.Class.forName0(Native Method) > > > > at java.base/java.lang.Class.forName(Class.java:291) > > > > at > > org.apache.ignite.internal.IgniteComponentType.createOptional0(IgniteComponentType.java:249) > > > > at > > org.apache.ignite.internal.IgniteComponentType.createOptional(IgniteComponentType.java:235) > > > > at > > org.apache.ignite.internal.processors.cache.GridCacheProcessor.createSharedContext(GridCacheProcessor.java:3427) > > > > at > > org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:869) > > > > at > > org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1886) > > > > at > > org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1135) > > > > at > > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1992) > > > > at > > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1683) > > > > at > > org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1109) > > > > at > > org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1027) > > > > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:913) > > > > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:812) > > > > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:682) > > > > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:651) > > > > at org.apache.ignite.Ignition.start(Ignition.java:346) > > > > at > > org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:300) > > > > Caused by: java.lang.ClassNotFoundException: > > javax.transaction
Re: [DISCUSSION] Mirroring Teamcity settings in a GitHub repository
For curious ones I copied a RunAll DSL configuration to gist [1]. [1] https://gist.github.com/pavlukhin/e788bbebafe69ce9d9a3c9e9871ab8c4 чт, 20 июн. 2019 г. в 08:42, Павлухин Иван : > > Dmitriy, > > It is a very important topic for Ignite CI. I think it would be great > if all significant changes in a build configuration goes with a > meaningful description (a commit message). As for me, DSL sounds > promising. > > BTW, a special rights are needed to view > https://ci.ignite.apache.org/admin/editBuild.html?id=buildType:Releases_ApacheIgniteMain_ReleaseBuild > (I do not have such). > > вт, 18 июн. 2019 г. в 14:30, Nikolay Izhikov : > > > > Hello, Dmitriy. > > > > Thanks, for starting this discussion. > > I think almost all of community members don't know the difference between > > two options. > > > > I vote for the simplest solution with the human readable format. > > > > > > В Пн, 17/06/2019 в 21:07 +0300, Dmitriy Pavlov пишет: > > > Hi Igniters, > > > > > > During preparing of the release I've faced with lack of information > > > related > > > to required steps. This was one reason of too long release preparation. > > > This process was automated by several Ignite contributors (thank you, > > > BTW). > > > Now, these valued automation results are more or less stored using > > > TeamCity > > > settings at ci.ignite.apache.org. > > > > > > First of all, changes done in these steps before were not always > > > consistent. The TC's built-in audit wasn't useful, because there were tons > > > of changes. So it is not clear who and why changed something. > > > > > > Secondly, Ignite release is performed outside of ASF infrastructure. It is > > > not a problem itself (and we're grateful to GridGain to sponsoring > > > infrastructure). But I believe knowledge about release should be backed up > > > somehow inside of ASF infra. > > > > > > The last issue related not to release, but to our test suites. It is now > > > about 90 suites, which may have different settings. There is no easy and > > > clear way to grep, check consistency, check possible options we've used. > > > > > > So let's consider 2 possible options to solve these issues. > > > > > > Option A: store settings in VCS > > > https://www.jetbrains.com/help/teamcity/2019.1/storing-project-settings-in-version-control.html > > > > > > > > > Option B: use DSL > > > https://www.jetbrains.com/help/teamcity/2019.1/storing-project-settings-in-version-control.html > > > > > > > > > Please share your vision. > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > PS. to see how DSL can look like you can use > > > https://ci.ignite.apache.org/admin/editBuild.html?id=buildType:Releases_ApacheIgniteMain_ReleaseBuild > > > > > > and click view DSL. > > > > -- > Best regards, > Ivan Pavlukhin -- Best regards, Ivan Pavlukhin
Re: [DISCUSSION] Mirroring Teamcity settings in a GitHub repository
Dmitriy, It is a very important topic for Ignite CI. I think it would be great if all significant changes in a build configuration goes with a meaningful description (a commit message). As for me, DSL sounds promising. BTW, a special rights are needed to view https://ci.ignite.apache.org/admin/editBuild.html?id=buildType:Releases_ApacheIgniteMain_ReleaseBuild (I do not have such). вт, 18 июн. 2019 г. в 14:30, Nikolay Izhikov : > > Hello, Dmitriy. > > Thanks, for starting this discussion. > I think almost all of community members don't know the difference between two > options. > > I vote for the simplest solution with the human readable format. > > > В Пн, 17/06/2019 в 21:07 +0300, Dmitriy Pavlov пишет: > > Hi Igniters, > > > > During preparing of the release I've faced with lack of information related > > to required steps. This was one reason of too long release preparation. > > This process was automated by several Ignite contributors (thank you, BTW). > > Now, these valued automation results are more or less stored using TeamCity > > settings at ci.ignite.apache.org. > > > > First of all, changes done in these steps before were not always > > consistent. The TC's built-in audit wasn't useful, because there were tons > > of changes. So it is not clear who and why changed something. > > > > Secondly, Ignite release is performed outside of ASF infrastructure. It is > > not a problem itself (and we're grateful to GridGain to sponsoring > > infrastructure). But I believe knowledge about release should be backed up > > somehow inside of ASF infra. > > > > The last issue related not to release, but to our test suites. It is now > > about 90 suites, which may have different settings. There is no easy and > > clear way to grep, check consistency, check possible options we've used. > > > > So let's consider 2 possible options to solve these issues. > > > > Option A: store settings in VCS > > https://www.jetbrains.com/help/teamcity/2019.1/storing-project-settings-in-version-control.html > > > > > > Option B: use DSL > > https://www.jetbrains.com/help/teamcity/2019.1/storing-project-settings-in-version-control.html > > > > > > Please share your vision. > > > > Sincerely, > > Dmitriy Pavlov > > > > PS. to see how DSL can look like you can use > > https://ci.ignite.apache.org/admin/editBuild.html?id=buildType:Releases_ApacheIgniteMain_ReleaseBuild > > > > and click view DSL. -- Best regards, Ivan Pavlukhin
Re: [VOTE] Complete Discontinuation of IGFS and Hadoop Accelerator
+1 чт, 20 июн. 2019 г. в 04:18, Valentin Kulichenko : > > +1 > > On Wed, Jun 19, 2019 at 5:58 PM Roman Shtykh > wrote: > > > +1 > > > > > > On Thursday, June 20, 2019, 7:34:47 a.m. GMT+9, Andrey Gura < > > ag...@apache.org> wrote: > > > > +1 > > > > On Thu, Jun 20, 2019 at 12:58 AM Denis Magda wrote: > > > > > > Igniters, > > > > > > Based on our earlier discussion [1], let's formalize our decision and > > vote > > > for the following: > > > > > >- IGFS and In-Memory Hadoop Accelerator components are to be > > >discontinued and no longer supported by the community > > >- The existing source code of IGFS and In-Memory Hadoop Accelerator is > > >to be removed from Ignite master. Before that, a special branch like > > >"ignite-igfs-and-hadoop-accelerator" to be forked off the master in > > order > > >to preserve the sources in Git history for those who might need it. > > > > > > Please cast your vote: > > > > > > +1 - agree > > > 0 - indifferent > > > -1 - disagree, explain why. > > > > > > The vote lasts for the next 72 hours. > > > > > > > > > [1] > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-Complete-Discontinuation-of-IGFS-and-Hadoop-Accelerator-td42282.html#a42326 > > > > > > - > > > Denis > > -- Best regards, Ivan Pavlukhin
Re: Thin client test suites failure
Maxim, Well, it is an interesting point. First of all I want to believe that issues occurring only on a specific Java version are rare. If so, running tests on a single version for a visa should be enough. On the other hand, a uniform test environment sounds a good idea (especially for a visa), preventing a test status change by "magic". So, I think that it can be done as follows: 1. Run tests for a visa on a baseline Java version (11?). 2. Run Nightly builds on all supported versions, choosing version at random is a one of options for that. (BTW, do we need some changes in TC bot to send a proper alert when test fails on some versions and fails on others?). ср, 19 июн. 2019 г. в 20:33, Maxim Muzafarov : > > Folks, > > Maybe I'm missing something, but what is the reason for setting > JAVA_HOME randomly? For instance, some commit is working under jdk8 > but fails under jdk9 (and we declare that 9-th version is supported by > Ignite). Should we merge this commit to the master branch (all test > suites can be OK under jdk8)? > > I know, that running all tests under all java version leads us to > Hell, but maybe we can have separated versions nightly builds? > > On Wed, 19 Jun 2019 at 13:47, Павлухин Иван wrote: > > > > Dmitriy, > > > > Thank you for the hint! Java version seems to be the reason. I see > > that tests pass with Java 8 and 11 and fail with Java 10. See the > > exception below. Btw, do we test with Java 9? Have not seen that > > version in recent runs. > > java.lang.NoClassDefFoundError: javax/transaction/SystemException > > at java.base/java.lang.Class.forName0(Native Method) > > at java.base/java.lang.Class.forName(Class.java:291) > > at > > org.apache.ignite.internal.IgniteComponentType.createOptional0(IgniteComponentType.java:249) > > at > > org.apache.ignite.internal.IgniteComponentType.createOptional(IgniteComponentType.java:235) > > at > > org.apache.ignite.internal.processors.cache.GridCacheProcessor.createSharedContext(GridCacheProcessor.java:3427) > > at > > org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:869) > > at > > org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1886) > > at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1135) > > at > > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1992) > > at > > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1683) > > at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1109) > > at > > org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1027) > > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:913) > > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:812) > > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:682) > > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:651) > > at org.apache.ignite.Ignition.start(Ignition.java:346) > > at > > org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:300) > > Caused by: java.lang.ClassNotFoundException: > > javax.transaction.SystemException > > at > > java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582) > > at > > java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:190) > > at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499) > > ... 18 more > > > > вт, 18 июн. 2019 г. в 23:27, Dmitriy Pavlov : > > > > > > Hi Ivan, > > > > > > Can these failures be related to Java version? Java home is set randomly > > > by > > > TC Bot. > > > > > > Dmitriy Pavlov > > > > > > вт, 18 июн. 2019 г. в 22:11, Павлухин Иван : > > > > > > > Hi igniters, > > > > > > > > Does anyone know why python, php and nodejs [1, 2, 3] suites fail so > > > > frequently on TC? Is there any activity to deal with it? > > > > > > > > [1] > > > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ThinClientPython_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv > > > > [2] > > > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ThinClientNodeJs_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv > > > > [3] > > > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ThinClientPhp_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv > > > > > > > > -- > > > > Best regards, > > > > Ivan Pavlukhin > > > > > > > > > > > > -- > > Best regards, > > Ivan Pavlukhin -- Best regards, Ivan Pavlukhin
Re: Thin client test suites failure
Dmitriy, Thank you for the hint! Java version seems to be the reason. I see that tests pass with Java 8 and 11 and fail with Java 10. See the exception below. Btw, do we test with Java 9? Have not seen that version in recent runs. java.lang.NoClassDefFoundError: javax/transaction/SystemException at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:291) at org.apache.ignite.internal.IgniteComponentType.createOptional0(IgniteComponentType.java:249) at org.apache.ignite.internal.IgniteComponentType.createOptional(IgniteComponentType.java:235) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.createSharedContext(GridCacheProcessor.java:3427) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:869) at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1886) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1135) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1992) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1683) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1109) at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1027) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:913) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:812) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:682) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:651) at org.apache.ignite.Ignition.start(Ignition.java:346) at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:300) Caused by: java.lang.ClassNotFoundException: javax.transaction.SystemException at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:190) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:499) ... 18 more вт, 18 июн. 2019 г. в 23:27, Dmitriy Pavlov : > > Hi Ivan, > > Can these failures be related to Java version? Java home is set randomly by > TC Bot. > > Dmitriy Pavlov > > вт, 18 июн. 2019 г. в 22:11, Павлухин Иван : > > > Hi igniters, > > > > Does anyone know why python, php and nodejs [1, 2, 3] suites fail so > > frequently on TC? Is there any activity to deal with it? > > > > [1] > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ThinClientPython_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv > > [2] > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ThinClientNodeJs_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv > > [3] > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ThinClientPhp_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv > > > > -- > > Best regards, > > Ivan Pavlukhin > > -- Best regards, Ivan Pavlukhin
Thin client test suites failure
Hi igniters, Does anyone know why python, php and nodejs [1, 2, 3] suites fail so frequently on TC? Is there any activity to deal with it? [1] https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ThinClientPython_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv [2] https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ThinClientNodeJs_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv [3] https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ThinClientPhp_IgniteTests24Java8=%3Cdefault%3E=buildTypeStatusDiv -- Best regards, Ivan Pavlukhin
Re: [DISCUSSION] Ignite 3.0 and to be removed list
Do we still need onheap caches? вт, 18 июн. 2019 г. в 21:30, Denis Magda : > > +1 > > Thick (aka. standard clients) provide comprehensive compute APIs with > peer-class-loading. That's a huge differentiator for Ignite. Until thin > clients support compute and ML API at the same level as the standard client > does, I would not consider the standard clients' discontinuation. Plus, as > Alex outlined, a functional gap is even wider. > > - > Denis > > > On Mon, Jun 17, 2019 at 6:28 AM Alexey Goncharuk > wrote: > > > Nikolay, > > > > I had this thought too, but I am not too eager to implement it yet. The > > reason is transaction protocol complexity/performance issues with thin > > clients. > > > > A thick client can communicate with each primary node and coordinate > > prepare/commit phases. Thin client can only communicate with one node, so > > the change will mean an additional network hop. Of course, we can make thin > > clients implement the same protocol, but it will immediately increase the > > protocol complexity for all platforms. > > > > Plus, we do not have near cache on thin clients, we do not support p2p > > class deployment, etc. Since thin clients are positioned as > > platform-agnostic, I do not think it makes sense to expose all feature set > > of Igntie to thin clients. > > > > Instead, we can significantly simplify client node configuration - it > > currently requires the same config as a regular Ignite node, however, in > > most cases, the configuration can be reduced almost to a several host:port > > pairs. > > > > пн, 17 июн. 2019 г. в 15:58, Nikolay Izhikov : > > > > > Alexey. > > > > > > I want to share a thought (just don't drop it out in one moment :) ). > > > > > > Do we really need "client nodes"? > > > > > > We have thin client protocol that is a very convenient point to interact > > > with Ignite. > > > So, why, we need one more entity and work mode such as "client node"? > > > > > > From my point of view, client nodes were required in the time without a > > > thin client. > > > Now, we have it. > > > > > > Let's simplify Ignite codebase and drop client nodes! > > > > > > How does it sound? > > > > > > > > > В Пн, 17/06/2019 в 15:52 +0300, Alexey Goncharuk пишет: > > > > Nikolay, > > > > > > > > Local caches and scalar are already in the list :) Added the outdated > > > > metrics point. > > > > > > > > пн, 17 июн. 2019 г. в 15:32, Nikolay Izhikov : > > > > > > > > > * Scalar. > > > > > * LOCAL caches. > > > > > * Deprecated metrics. > > > > > > > > > > В Пн, 17/06/2019 в 15:18 +0300, Alexey Goncharuk пишет: > > > > > > Igniters, > > > > > > > > > > > > Even though we are still planning the Ignite 2.8 release, I would > > > like to > > > > > > kick-off a discussion related to Ignite 3.0, because the efforts > > for > > > AI > > > > > > > > > > 3.0 > > > > > > will be significantly larger than for AI 2.8, better to start > > early. > > > > > > > > > > > > As a first step, I would like to discuss the list of things to be > > > removed > > > > > > in Ignite 3.0 (partially this thread is inspired by Denis Magda's > > > IGFS > > > > > > removal thread). I've separated all to-be-removed points from > > > existing > > > > > > Ignite 3.0 wishlist [1] to a dedicated block and also added a few > > > more > > > > > > things that look right to be dropped. > > > > > > > > > > > > Please share your thoughts, probably, there are more outdated > > things > > > we > > > > > > need to add to the wishlist. > > > > > > > > > > > > As a side question: I think it makes sense to create tickets for > > such > > > > > > improvements, how do we track them. Will the 3.0 version suffice or > > > > > > > > > > should > > > > > > we add a separate label? > > > > > -- Best regards, Ivan Pavlukhin
Re: [DISCUSSION] Ignite 3.0 and to be removed list
Nikolay, Dmitriy, Should we have a separate thread devoted to client nodes? Also, my cent here is from a Hazelcast history. Once they removed their thick client (called LiteMember), but after a while they brought it back. I think we need to learn this lesson in more details. вт, 18 июн. 2019 г. в 11:42, Andrey Mashenkov : > > Also, > * Get rid of old services. > * Make system cache non-persistent or even more - drop it. Discussion on > dev list [1]. > > I have no permissions to edit wiki page and would glad if someone add all > thoughts from this thread. > > [1] > http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-System-cache-persistence-td41158.html > > On Tue, Jun 18, 2019 at 10:33 AM Alexey Goncharuk < > alexey.goncha...@gmail.com> wrote: > > > Folks, > > > > May I ask you to add the mentioned points to the wishlist to keep them in > > one place for convenience? > > > > вт, 18 июн. 2019 г. в 00:19, Alex Plehanov : > > > > > Remove "force server mode" for client nodes (already was discussed on dev > > > list earlier [1]). > > > > > > [1] : > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/Deprecate-force-server-mode-for-clients-td33614.html > > > > > > пн, 17 июн. 2019 г. в 19:22, Pavel Tupitsyn : > > > > > > > Big changes for .NET: > > > > * Remove legacy Entity Framework integration > > > > * Remove legacy ASP.NET integration > > > > * Move from .NET Framework 4.0 (released in 2010) to .NET Standard 2.0 > > > > (modern way to build libraries) > > > > > > > > Thanks, > > > > Pavel > > > > > > > > On Mon, Jun 17, 2019 at 7:14 PM Igor Sapego > > > wrote: > > > > > > > > > For the C++ I propose to drop support of VS 2010 and move to > > > > > at least 2012 (or, better yet 2013). > > > > > > > > > > Also, I'd drop x86 support for everything except for maybe ODBC. > > > > > > > > > > Best Regards, > > > > > Igor > > > > > > > > > > On Mon, Jun 17, 2019 at 7:12 PM Pavel Kovalenko > > > > > wrote: > > > > > > > > > > > I would like to add to the list following: > > > > > > > > > > > > 1. Remove ForceKeyRequests and related code. Since we have Late > > > > affinity > > > > > > assignment and primary node partitions are always up to date we > > don't > > > > > need > > > > > > to request actual data from backups. > > > > > > 2. Remove @CentralizedAffinityFunction and related code. I don't > > see > > > > any > > > > > > real usages of custom affinity functions that use this annotation. > > > > > > 3. Leave Exchanges Merge + Late Affinity assignment as the only PME > > > > > > protocol. Remove centralized affinity distribution in case of node > > > left > > > > > and > > > > > > no merge exchange legacy modes. > > > > > > 4. Remove CacheRebalanceMode.NONE and Rebalance Delay as it can > > break > > > > > data > > > > > > consistency in a cluster. Also, remove force rebalance mode as it > > can > > > > be > > > > > > used only if rebalance delay is set. > > > > > > > > > > > > пн, 17 июн. 2019 г. в 18:39, Dmitriy Pavlov : > > > > > > > > > > > > > Nikolay, > > > > > > > > > > > > > > we can (and probably should) discuss stop deploying > > caches/services > > > > to > > > > > > > client nodes. > > > > > > > > > > > > > > But I would not name it removal of client nodes at all. Any > > Apache > > > > > Ignite > > > > > > > guide I saw is starting from 2 steps: 1) start server node, 2) > > > start > > > > > > client > > > > > > > node. > > > > > > > > > > > > > > There are no reasons to write software if users are unaware of > > how > > > to > > > > > use > > > > > > > it. So I do not agree that supplementary materials are > > unimportant. > > > > > > > > > > > > > > Sincerely, > > > > > > > Dmitriy Pavlov > > > > > > > > > > > > > > пн, 17 июн. 2019 г. в 18:30, Nikolay Izhikov < > > nizhi...@apache.org > > > >: > > > > > > > > > > > > > > > Dmitriy, > > > > > > > > > > > > > > > > I think the whole concept of "client" nodes is broken. > > > > > > > > And that's why: > > > > > > > > > > > > > > > > Ignite Client node nothing to do with "client" :) > > > > > > > > We can deploy cache on it, we can execute compute tasks, > > services > > > > on > > > > > > it. > > > > > > > > "client node" is a node with special join process and some > > > > > > rebalance/PME > > > > > > > > hacks. > > > > > > > > And we put many(many!) efforts to support this concept and this > > > > > hacks. > > > > > > > > > > > > > > > > And I'm asking: What value client nodes bring to Ignite? > > > > > > > > > > > > > > > > I think, Alexey, already answered correctly: > > > > > > > > > > > > > > > > * Transactions support. > > > > > > > > * P2P deployment. > > > > > > > > > > > > > > > > I think we should, definitely, remove concept of "client nodes" > > > in > > > > > the > > > > > > > > future. > > > > > > > > It's about product design decision, in the first place, not > > about > > > > > > > > additional materials. > > > > > > > > > > > > > > > > The simpler core codebase we have, the more reliable
Re: IGNITE-7285 Add default query timeout
Hi Saikat, Thank you for driving it. I left my comments [1]. [1] https://issues.apache.org/jira/browse/IGNITE-7285 сб, 15 июн. 2019 г. в 20:48, Saikat Maitra : > > Hi Ivan, > > Thank you for your email. I have updated the PR to use default query > timeout. > > Please take a look. > > https://github.com/apache/ignite/pull/6490/files > > Regards > Saikat > > On Tue, May 7, 2019 at 12:30 AM Ivan Pavlukhina wrote: > > > Hi Saikat, > > > > It is good that we agreed how property in question should be configured. > > But I worry about the following. If the PR merged it will not contain a > > user value yet because an introduced property is not used. Consequently we > > must start using that property before a next AI release. Just one thing to > > keep in mind. > > > > Sent from my iPhone > > > > > On 4 May 2019, at 05:56, Saikat Maitra wrote: > > > > > > Hi Ivan, > > > > > > Thank you for reviewing the PR. I have updated the PR. Please review and > > > share your feedback. > > > > > > I was thinking of making a separate PR for using the defaultQueryTimeout > > > property in query execution flow. > > > > > > Regards, > > > Saikat > > > > > >> On Wed, May 1, 2019 at 1:04 AM Павлухин Иван > > wrote: > > >> > > >> Andrey K., > > >> > > >>> I think we should develop some kind of "Queries" options on Ignite > > >>> configuration. > > >> > > >> Quite a reasonable idea. We already have couple of query-related > > >> properties in IgniteConfiguration and we can move them (in a > > >> compatible way) to a query properties sub-aggregate. I think it is > > >> better to raise a separate topic for that. > > >> > > >> ср, 1 мая 2019 г. в 09:00, Павлухин Иван : > > >>> > > >>> Hi Saikat, > > >>> > > >>> I left a couple of comment on GitHub [1]. > > >>> > > >>> Perhaps I am missing it but what are the plans for making a default > > >>> query timeout workable by using an introduced property in query > > >>> execution flow? > > >>> > > >>> [1] https://github.com/apache/ignite/pull/6490 > > >>> > > >>> вт, 30 апр. 2019 г. в 02:50, Saikat Maitra : > > >>>> > > >>>> Hi Ivan, > > >>>> > > >>>> Yes, I checked this CacheQuery default value > > >>>> > > >> > > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/query/CacheQuery.java#L200 > > >>>> > > >>>> Also, Andrew recommended the same in review feedback. > > >>>> > > >>>> https://github.com/apache/ignite/pull/6490#discussion_r277730394 > > >>>> > > >>>> Regards, > > >>>> Saikat > > >>>> > > >>>> > > >>>> On Mon, Apr 29, 2019 at 3:18 AM Павлухин Иван > > >> wrote: > > >>>> > > >>>>> Hi Saikat, > > >>>>> > > >>>>> It a compatibility with previous versions the reason for an > > >> indefinite > > >>>>> timeout by default? > > >>>>> > > >>>>> сб, 27 апр. 2019 г. в 16:58, Saikat Maitra > >>> : > > >>>>>> > > >>>>>> Hi Alexey, Ivan, Andrew > > >>>>>> > > >>>>>> I think we can provide an option to configure defaultQueryOption at > > >>>>>> IgniteConfiguration and set the default value = 0 to imply if not > > >> set it > > >>>>>> will be execute indefinitely but then user can set this value > > >> based on > > >>>>> the > > >>>>>> application preferences and use it in addition to SQL query > > >> timeout. > > >>>>>> > > >>>>>> I have updated the PR accordingly. > > >>>>>> > > >>>>>> Please review and share if any changes required. > > >>>>>> > > >>>>>> Regards, > > >>>>>> Saikat > > >>>>>> > > >>>>>> On Wed, Apr 24, 2019 at 4:33 AM Alexey K
Re: [DISCUSSION] Complete Discontinuation of IGFS and Hadoop Accelerator
Denis, I must say that aforementioned solutions for a Hadoop ecosystem appear from time to time in questions on a user mailing list. So, it seems that there is a practical need for such solutions. But of course it does not mean that we should continue a support of IGFS and Hadoop Accelerator. If both are not solutions that fit well common use cases then we should discontinue it. If any of them is very good for it's purposes but we do not have a capacity to support it without sacrificing main Ignite goals then we still should discontinue it in my mind. P.S. Personally I am a fan of UNIX way. I like ideas of a single responsibility and integrations. And I suppose there are other Ignite features which could be dropped. ср, 12 июн. 2019 г. в 21:04, Denis Magda : > > Igniters, > > I'd like us to move on and finish our conversation on the IGFS [1] and > Hadoop Accelerator [2] support. > > To my knowledge, there is no single committer who maintains the > integrations; they are no longer tested and, even more, the community > stopped providing the binaries since Ignite 2.6.0 release (look for > In-Memory Hadoop Accelerator table [3]). > > Why all of that happened? Because of a little value, something succeeds > while something fails. Does it mean that Ignite cannot be used for Hadoop > acceleration, in general? No, quite the opposite, it CAN be used, but a > solution is different. Have Ignite with native persistence deployed close > to your Hadoop cluster (replace GridGain with Ignite) [4]. > > So, I propose we remove IGFS and In-Memory Hadoop Accelerator from our > master repository and rework existing public documentation showing how to > achieve the acceleration with Ignite. > > Any supporters or objections? > > > [1] https://apacheignite-fs.readme.io/docs/in-memory-file-system > [2] https://apacheignite-fs.readme.io/docs/hadoop-accelerator > [3] https://ignite.apache.org/download.cgi#binaries > [4] > https://docs.gridgain.com/docs/bdb-getting-started#section-gridgain-data-lake-accelerator > > - > Denis -- Best regards, Ivan Pavlukhin
Re: [Code Style Check] TC issues in master
Petr, Nikolay, Running some kind of smoke test might be a good idea. But I think it should be discussed separately in another thread. чт, 13 июн. 2019 г. в 11:43, Petr Ivanov : > > I do not know, some basic tests (running single instance of Ignite, > activation, simple cluster topology, etc.) > It seems that it is more critical than stylecheck, that can be fixed in > several lines of code, but will steal a day (night) of tests. > > > > On 13 Jun 2019, at 11:42, Nikolay Izhikov wrote: > > > > Hello, Petr. > > > > I think we should be specific here. > > > > For me: "code style" fail should be equal to "compilation fail". > > Can you clarify, what fundamental tests you are written about? > > > > В Чт, 13/06/2019 в 11:26 +0300, Petr Ivanov пишет: > >> How about including basic set of tests (sanity checks) into ~Build Apache > >> Ignite~? > >> Or may be we can go even further and use cascade system of tests — run > >> them group by group, stopping the whole run if more fundamental tests are > >> failing? > >> > >> > >>> On 12 Jun 2019, at 21:54, Nikolay Izhikov wrote: > >>> > >>> +1 for including checkstyl to "Build Apache Ignite". > >>> > >>> В Ср, 12/06/2019 в 21:29 +0300, Павлухин Иван пишет: > >>>> Maxim, > >>>> > >>>> Options 1 and 3 sounds fine to me. And taking into account current > >>>> state I tend to think that option 1 is even better. > >>>> > >>>> вт, 11 июн. 2019 г. в 14:19, Maxim Muzafarov : > >>>>> > >>>>> Dmitry, > >>>>> > >>>>> Thank you. > >>>>> Sure, I'll not change anything on TC without discussion with all the > >>>>> community. > >>>>> > >>>>> On Tue, 11 Jun 2019 at 13:30, Dmitriy Pavlov wrote: > >>>>>> > >>>>>> Hi Maxim, > >>>>>> > >>>>>> I've granted role Ignite Tests Admins to your account. Please check. > >>>>>> > >>>>>> Please notify community on any significant changes you do. > >>>>>> Unfortunately, > >>>>>> TC parameters audit is not so convenient as VCS-based > >>>>>> configuration/code. > >>>>>> > >>>>>> Sincerely > >>>>>> Dmitriy Pavlov > >>>>>> > >>>>>> пн, 10 июн. 2019 г. в 17:03, Maxim Muzafarov : > >>>>>> > >>>>>>> Igniters, > >>>>>>> > >>>>>>> It seems to me that building [ignite-scalar] module under JDK9+ have > >>>>>>> been successfully solved for the ~Build Apache Ignite~ suite [1] [2] > >>>>>>> [3], but it was not configured for the [Check Code Style] suite. We > >>>>>>> should configure it the same way (but it sounds to me very odd). I > >>>>>>> see, that we have several options here: > >>>>>>> > >>>>>>> 1. Enable `checkstyle` profile for the ~Build Apache Ignite~ suite as > >>>>>>> we've discussed it previously [4] and forget about any duplicate > >>>>>>> configuration once and for all. One more thing to do so is that check > >>>>>>> style has been violated for a few days and nobody mentioned it [5]. > >>>>>>> > >>>>>>> 2. Since the checkstyle plugin is not related to scala-source code (it > >>>>>>> does not check it) we can exclude scala modules from maven build > >>>>>>> procedure for the checkstyle suite by adding some command-line > >>>>>>> parameters (test them locally, but have no TC permissions to check it > >>>>>>> on TC): > >>>>>>> -pl > >>>>>>> -:ignite-scalar_2.10,-:ignite-scalar,-:ignite-visor-console,-:ignite-visor-console_2.10 > >>>>>>> > >>>>>>> 3. Configure [Check Code Style] the same way as ~Build Apache Ignite~ > >>>>>>> to support builds for JDK9+. > >>>>>>> > >>>>>>> WDYT? > >>>>>>> What options will be the best for the Apache Ignite? > >>>>>>> > >>>>>>> [1] https://github.com/scala/bug/issues/10871 &g
Re: Broken layout of Ignite javadoc on web
I went through a couple of pages and layout looks good.
A couple of aside notes:
1. There is a label "Ignite - In-Memory Database and Caching
Platform". Do we need that label at all? Can we replace it with a link
having text "Ignite" and referring to ignite.apache.org?
2. The copyright is out of date "2018 Copyright © Apache Software Foundation".
вт, 11 июн. 2019 г. в 19:22, Denis Magda :
>
> + Sergey
>
> -
> Denis
>
>
> On Tue, Jun 11, 2019 at 9:21 AM Denis Magda wrote:
>
> > Sergey,
> >
> > Could you please chime in? I do believe we have special tests to ensure
> > the javadoc is not broken. Visually, everything looks good. But I'll
> > encourage others to double check.
> >
> > -
> > Denis
> >
> >
> > On Tue, Jun 11, 2019 at 7:04 AM Dmitriy Pavlov wrote:
> >
> >> Igniters, Ivan,
> >>
> >> could you please verify new release JavaDoc doesn't have same problems:
> >> https://ignite.apache.org/releases/2.7.5/javadoc/
> >>
> >> Sincerely,
> >> Dmitriy Pavlov
> >>
> >> чт, 31 янв. 2019 г. в 21:30, Dmitriy Pavlov :
> >>
> >> > Sure, I don't see any issues with using the default style. It is
> >> > definitely better than broken CSS.
> >> >
> >> > Thank you for accurate & detailed research.
> >> >
> >> > If nobody minds I could apply a patch with a switch to defaults.
> >> >
> >> > чт, 31 янв. 2019 г. в 11:11, Павлухин Иван :
> >> >
> >> >> Hi,
> >> >>
> >> >> Continuation of the story. I build javadocs locally and checked how it
> >> >> renders with manually added images. Unfortunately there are several
> >> >> visible artifacts. I guess the reason here is that stylesheet and
> >> >> images was suitable for an older version of javadoc maven plugin but
> >> >> not for a newer one.
> >> >>
> >> >> Also I noticed that style used by Ignite is similar to Java 7 docs
> >> >> [1]. Different style is used since Java 8 [2]. And it looks like by
> >> >> default maven javadoc plugin today builds similar pages. You can see
> >> >> an example of Ignite javadoc built with default style [3].
> >> >>
> >> >> So, perhaps we can use a default style for our documentation, cannot
> >> we?
> >> >>
> >> >> [1] https://docs.oracle.com/javase/7/docs/api/java/util/BitSet.html
> >> >> [2]
> >> >>
> >> https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/util/BitSet.html
> >> >> [3] https://gist.github.com/pavlukhin/c627b31c09e4f1ae7bd04f11bafed040
> >> >>
> >> >> пн, 24 дек. 2018 г. в 17:34, Павлухин Иван :
> >> >> >
> >> >> > Hi,
> >> >> >
> >> >> > Mentioned javadoc.css references to some image resources. They
> >> present
> >> >> > for 2.3.0 release [1]. But similar url returns 404 for 2.7.0 [2]. Who
> >> >> > knows how to upload resources?
> >> >> >
> >> >> > [1] https://ignite.apache.org/releases/2.3.0/javadoc/resources
> >> >> > [2] https://ignite.apache.org/releases/2.7.0/javadoc/resources
> >> >> >
> >> >> > пн, 17 дек. 2018 г. в 16:38, Павлухин Иван :
> >> >> >
> >> >> > >
> >> >> > > Dmitriy,
> >> >> > >
> >> >> > > Yep, it looks like the problem is inside
> >> assembly/docfiles/javadoc.css
> >> >> > >
> >> >> > > I will try to dig into some time later on a spare time. Of course,
> >> if
> >> >> > > nobody fixes the problem earlier.
> >> >> > > пн, 17 дек. 2018 г. в 15:18, Dmitriy Pavlov :
> >> >> > > >
> >> >> > > > I see the same, browsers checked: Edge & Chrome. I guess it is
> >> not
> >> >> a layout
> >> >> > > > is broken, but a style missing.
> >> >> > > >
> >> >> > > > Unfortunately, I don't know how to fix. Probably we should start
> >> >> research
> >> >> > > > from ignite/pom.xml:191
> >> >> > > >
> >> >> > > >
> >> >>
> >> ${basedir}/assemb
Re: [Code Style Check] TC issues in master
Maxim, Options 1 and 3 sounds fine to me. And taking into account current state I tend to think that option 1 is even better. вт, 11 июн. 2019 г. в 14:19, Maxim Muzafarov : > > Dmitry, > > Thank you. > Sure, I'll not change anything on TC without discussion with all the > community. > > On Tue, 11 Jun 2019 at 13:30, Dmitriy Pavlov wrote: > > > > Hi Maxim, > > > > I've granted role Ignite Tests Admins to your account. Please check. > > > > Please notify community on any significant changes you do. Unfortunately, > > TC parameters audit is not so convenient as VCS-based configuration/code. > > > > Sincerely > > Dmitriy Pavlov > > > > пн, 10 июн. 2019 г. в 17:03, Maxim Muzafarov : > > > > > Igniters, > > > > > > It seems to me that building [ignite-scalar] module under JDK9+ have > > > been successfully solved for the ~Build Apache Ignite~ suite [1] [2] > > > [3], but it was not configured for the [Check Code Style] suite. We > > > should configure it the same way (but it sounds to me very odd). I > > > see, that we have several options here: > > > > > > 1. Enable `checkstyle` profile for the ~Build Apache Ignite~ suite as > > > we've discussed it previously [4] and forget about any duplicate > > > configuration once and for all. One more thing to do so is that check > > > style has been violated for a few days and nobody mentioned it [5]. > > > > > > 2. Since the checkstyle plugin is not related to scala-source code (it > > > does not check it) we can exclude scala modules from maven build > > > procedure for the checkstyle suite by adding some command-line > > > parameters (test them locally, but have no TC permissions to check it > > > on TC): > > > -pl > > > -:ignite-scalar_2.10,-:ignite-scalar,-:ignite-visor-console,-:ignite-visor-console_2.10 > > > > > > 3. Configure [Check Code Style] the same way as ~Build Apache Ignite~ > > > to support builds for JDK9+. > > > > > > WDYT? > > > What options will be the best for the Apache Ignite? > > > > > > [1] https://github.com/scala/bug/issues/10871 > > > [2] https://issues.apache.org/jira/browse/IGNITE-6730 > > > [3] https://issues.apache.org/jira/browse/IGNITE-11189 > > > [4] > > > http://apache-ignite-developers.2346864.n4.nabble.com/Code-inspection-tp27709p41297.html > > > [5] https://issues.apache.org/jira/browse/IGNITE-11899 > > > > > > On Fri, 7 Jun 2019 at 15:36, Nikolay Izhikov wrote: > > > > > > > > Hello, Petr. > > > > > > > > > at least Scala does not compile > > > > > > > > How cat I reproduce it? > > > > Do we have ticket? > > > > > > > > В Пт, 07/06/2019 в 15:28 +0300, Petr Ivanov пишет: > > > > > Suite fails because Apache Ignite compilation is not supported under > > > JDK 9+ (at least Scala does not compile). > > > > > Your build from [3] was triggered with JDK 11. > > > > > > > > > > > On 7 Jun 2019, at 14:57, Maxim Muzafarov wrote: > > > > > > > > > > > > Igniters, > > > > > > > > > > > > I've noticed a few problems with Code Style Check Suite on TC in the > > > > > > master branch. > > > > > > > > > > > > 1. Some of the rules have been violated by previous commits to the > > > > > > master branch. I've created ticket [1] and have prepared PR [2] > > > > > > which > > > > > > is fixing it. > > > > > > Dmitry, or maybe someone else, can you take a look, please? > > > > > > > > > > > > 2. The Code Style Check Stuite still fails (time to time) on TC with > > > > > > compile error on [ignite-scalar] module > > > > > > (java.lang.NoClassDefFoundError: javax/tools/ToolProvider). For > > > > > > instance, this build [3] fails and this is fully ok [4]. However, > > > > > > the > > > > > > ~Build Apache Ignite~ Suite with almost the same configuration > > > > > > passes > > > > > > normally. > > > > > > > > > > > > I'd like to create a new suite with checkstyle for debug purposes, > > > can > > > > > > anyone grant permission to copy\clone\edit suites on TC? My login: > > > > > > maxmu...@gmail.com > > > > > > > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-11899 > > > > > > [2] https://github.com/apache/ignite/pull/6597 > > > > > > [3] > > > https://ci.ignite.apache.org/viewLog.html?buildId=4020653=IgniteTests24Java8_CheckCodeStyle=buildLog_IgniteTests24Java8=%3Cdefault%3E > > > > > > [4] > > > https://ci.ignite.apache.org/viewLog.html?buildId=4021372=IgniteTests24Java8_CheckCodeStyle > > > > > > > > > > > > > -- Best regards, Ivan Pavlukhin
Re: [DISCUSSION] Release Apache Ignite 2.7.5-rc4
Vyacheslav, > Ivan, according to described issues with versions it looks like some > previously built artefacts have been cashed in Maven local repository. Versions I saw actually are consistent with a way how they had been built. But they look a little bit surprising. But actually I do not know our expectations for them.. пт, 7 июн. 2019 г. в 11:49, Vyacheslav Daradur : > > Dmitry, thank you for pointing this out, I just mixed threads. > > >> 3. Some mix of versions reported by nodes launched in different ways. > Ivan, according to described issues with versions it looks like some > previously built artefacts have been cashed in Maven local repository. > > On Fri, Jun 7, 2019 at 9:45 AM Павлухин Иван wrote: > > > > Vyacheslav, > > > > > Ivan, try to clean local repo (~/m2) and rebuild the project once again > > > according to DEVNOTES.txt > > > > Thank you for suggestion! But what problem it will solve? > > > > пт, 7 июн. 2019 г. в 00:52, Dmitriy Pavlov : > > > > > > Hi Vyacheslav, > > > > > > I'm sorry for being too formal here, although could you please cast your > > > vote in > > > https://lists.apache.org/thread.html/35cbc2d4c5b769155dc8aec15edd808a25c5cf48a5e12637528e931d@%3Cdev.ignite.apache.org%3E > > > > > > And thank you for advice on how to check examples. > > > > > > Sincerely > > > Dmitriy Pavlov > > > > > > чт, 6 июн. 2019 г. в 22:37, Vyacheslav Daradur : > > > > > > > +1 from me. Built from sources and run several examples. > > > > > > > > Ivan, try to clean local repo (~/m2) and rebuild the project once > > > > again according to DEVNOTES.txt > > > > > > > > On Thu, Jun 6, 2019 at 7:37 PM Павлухин Иван > > > > wrote: > > > > > > > > > > I spent a while running examples for RC from bin package. I did the > > > > following: > > > > > 1. "mvn clean install" for src package in order to have all > > > > > dependencies needed for examples in local maven repo on my machine. > > > > > 2. Tried to compile and launch examples from bin package. > > > > > 2.1 Failed claiming that classes from org.jetbrains.annotations are > > > > > not present on classpath. > > > > > 2.2. Fixed by adding a corresponding dependency to examples pom > > > > > manually. Run SQL examples successfully. > > > > > > > > > > So, I observed following glitches: > > > > > 1. Missing dependency containing org.jetbrains.annotations in examples > > > > > in bin package. > > > > > 2. "2018 Copyright(C) Apache Software Foundation" with year 2018 in > > > > > console output when starting ignite (using examples main classes and > > > > > ignite.sh). > > > > > 3. Some mix of versions reported by nodes launched in different ways. > > > > > 3.1 ignite.sh from bin package -- ver. 2.7.5#20190603-sha1:be4f2a15 > > > > > 3.2. examples main classes from bin package -- ver. > > > > 2.7.5#19700101-sha1:DEV > > > > > 3.3. examples main classes from src package -- ver. > > > > > 2.7.5-SNAPSHOT#19700101-sha1:DEV > > > > > > > > > > ср, 5 июн. 2019 г. в 18:59, Dmitriy Pavlov : > > > > > > > > > > > > I managed to get some build using build number from 1. Release build > > > > step > > > > > > (for that case - 20) > > > > > > > > > > > > The result can be found here > > > > > > > > > > https://ci.ignite.apache.org/viewLog.html?buildId=4049646=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages=artifacts#%2Fsrc > > > > > > > > > > > > Pavel T., Peter I., > > > > > > > > > > > > Please confirm that it is sufficient. I'll update the release > > > > > > manager > > > > notes > > > > > > with steps required to run this configuration. > > > > > > > > > > > > Sincerely > > > > > > Dmitriy Pavlov > > > > > > > > > > > > ср, 5 июн. 2019 г. в 18:07, Dmitriy Pavlov : > > > > > > > > > > > > > I've found only this step > > > > > > > > > > > > > > > > > > https://ci.ignite.apache.org/viewType.html?buildTypeId=ApacheIgni
Re: [DISCUSSION] Release Apache Ignite 2.7.5-rc4
Vyacheslav, > Ivan, try to clean local repo (~/m2) and rebuild the project once again > according to DEVNOTES.txt Thank you for suggestion! But what problem it will solve? пт, 7 июн. 2019 г. в 00:52, Dmitriy Pavlov : > > Hi Vyacheslav, > > I'm sorry for being too formal here, although could you please cast your > vote in > https://lists.apache.org/thread.html/35cbc2d4c5b769155dc8aec15edd808a25c5cf48a5e12637528e931d@%3Cdev.ignite.apache.org%3E > > And thank you for advice on how to check examples. > > Sincerely > Dmitriy Pavlov > > чт, 6 июн. 2019 г. в 22:37, Vyacheslav Daradur : > > > +1 from me. Built from sources and run several examples. > > > > Ivan, try to clean local repo (~/m2) and rebuild the project once > > again according to DEVNOTES.txt > > > > On Thu, Jun 6, 2019 at 7:37 PM Павлухин Иван wrote: > > > > > > I spent a while running examples for RC from bin package. I did the > > following: > > > 1. "mvn clean install" for src package in order to have all > > > dependencies needed for examples in local maven repo on my machine. > > > 2. Tried to compile and launch examples from bin package. > > > 2.1 Failed claiming that classes from org.jetbrains.annotations are > > > not present on classpath. > > > 2.2. Fixed by adding a corresponding dependency to examples pom > > > manually. Run SQL examples successfully. > > > > > > So, I observed following glitches: > > > 1. Missing dependency containing org.jetbrains.annotations in examples > > > in bin package. > > > 2. "2018 Copyright(C) Apache Software Foundation" with year 2018 in > > > console output when starting ignite (using examples main classes and > > > ignite.sh). > > > 3. Some mix of versions reported by nodes launched in different ways. > > > 3.1 ignite.sh from bin package -- ver. 2.7.5#20190603-sha1:be4f2a15 > > > 3.2. examples main classes from bin package -- ver. > > 2.7.5#19700101-sha1:DEV > > > 3.3. examples main classes from src package -- ver. > > > 2.7.5-SNAPSHOT#19700101-sha1:DEV > > > > > > ср, 5 июн. 2019 г. в 18:59, Dmitriy Pavlov : > > > > > > > > I managed to get some build using build number from 1. Release build > > step > > > > (for that case - 20) > > > > > > > > The result can be found here > > > > > > https://ci.ignite.apache.org/viewLog.html?buildId=4049646=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages=artifacts#%2Fsrc > > > > > > > > Pavel T., Peter I., > > > > > > > > Please confirm that it is sufficient. I'll update the release manager > > notes > > > > with steps required to run this configuration. > > > > > > > > Sincerely > > > > Dmitriy Pavlov > > > > > > > > ср, 5 июн. 2019 г. в 18:07, Dmitriy Pavlov : > > > > > > > > > I've found only this step > > > > > > > > > > > > https://ci.ignite.apache.org/viewType.html?buildTypeId=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages=buildTypeStatusDiv_Releases_ApacheIgniteMain=__all_branches__ > > > > > > > > > > > > > > > It requires some build counter. > > > > > > > > > > Peter I, could you please chime in? > > > > > > > > > > ср, 5 июн. 2019 г. в 18:02, Dmitriy Pavlov : > > > > > > > > > >> Hi Igniters, Pavel T., > > > > >> > > > > >> I'm struggling to find clear steps for building/uploading NuGet > > packages. > > > > >> How can I double check if it was built or not? What should I do? > > > > >> > > > > >> In some internal notes, I've found that NuGet upload can't be > > undone, and > > > > >> in > > https://cwiki.apache.org/confluence/display/IGNITE/Release+Process it > > > > >> is a post-release step. > > > > >> > > > > >> Please advise, both here or in private. I will update the > > instructions > > > > >> accordingly. > > > > >> > > > > >> Sincerely > > > > >> Dmitriy Pavlov > > > > >> > > > > >> ср, 5 июн. 2019 г. в 15:27, Dmitriy Pavlov : > > > > >> > > > > >>> Hi Ivan, > > > > >>> > > > > >>> In code there are 2 differences: > > > > >
