Re: [VOTE][EXTENSION] Release Apache Ignite performance-statistics-ext extension 1.0.0 RC4
+1 On Fri, Jun 25, 2021 at 6:34 AM Sergei Ryzhov wrote: > +1 > > -- > Best regards, > Sergei Ryzhov >
Re: [VOTE] Release Apache Ignite 3.0.0-alpha2 RC1
+1 (binding) On Mon, Jun 28, 2021 at 8:23 PM Вячеслав Коптилин wrote: > +1 > > Thanks, > S. > > пн, 28 июн. 2021 г. в 18:09, Igor Sapego : > > > +1 > > > > Best Regards, > > Igor > > > > > > On Sat, Jun 26, 2021 at 1:41 AM Nikita Ivanov > wrote: > > > > > +1 > > > > > > -- > > > Nikita Ivanov > > > > > > > > > > > > On Fri, Jun 25, 2021 at 3:31 PM Valentin Kulichenko < > > > valentin.kuliche...@gmail.com> wrote: > > > > > > > Dear Community, > > > > > > > > In the last several months, the development of Ignite 3 has been > moving > > > > forward significantly. On top of what we already had in the first > > Alpha, > > > we > > > > have the following features ready: > > > > > > > > - Replication infrastructure based on Raft > > > > - In-memory atomic storage with the basic insert-read functionality > > > > - New schema management engine and API > > > > > > > > In my view, this constitutes a significant milestone, and I propose > to > > > > release it as the Alpha 2. This way it will be available for > download, > > > and > > > > anyone will be able to play with the project, run examples, get a > feel > > of > > > > how Ignite will work in the future, and provide feedback. > > > > > > > > Please vote. > > > > > > > > --- > > > > > > > > The release candidate is uploaded here: > > > > https://dist.apache.org/repos/dist/dev/ignite/3.0.0-alpha2-rc1/ > > > > Maven staging: > > > > > > https://repository.apache.org/content/repositories/orgapacheignite-1522/ > > > > Git tag: https://github.com/apache/ignite-3/tree/3.0.0-alpha2-rc1 > > > > > > > > Jira tickets: > > > > > > > > > > > > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%2012315922%20AND%20fixVersion%20%3D%2012349574%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC > > > > > > > > DEVNOTES: > > > > https://github.com/apache/ignite-3/blob/3.0.0-alpha2-rc1/DEVNOTES.md > > > > > > > > The vote is formal, see voting guidelines: > > > > https://www.apache.org/foundation/voting.html > > > > > > > > +1 - accept Apache Ignite 3.0.0-alpha2 RC1 > > > > 0 - don't care either way > > > > -1 - DO NOT accept Apache Ignite 3.0.0-alpha2 RC1 (explain why) > > > > > > > > See notes on how to verify release here: > > > > https://www.apache.org/info/verification.html > > > > > > > > This vote will be open for 72 hours and will close on June 28th at > 11pm > > > > UTC: > > > > > > > > > > > > > > https://www.timeanddate.com/countdown/to?iso=20210628T16=224=Apache+Ignite+3.0.0-alpha2+RC1=cursive=1 > > > > > > > > -Val > > > > > > > > > >
Re: [VOTE] Release Apache Ignite 3.0.0-alpha2 RC1
+1 Thanks, S. пн, 28 июн. 2021 г. в 18:09, Igor Sapego : > +1 > > Best Regards, > Igor > > > On Sat, Jun 26, 2021 at 1:41 AM Nikita Ivanov wrote: > > > +1 > > > > -- > > Nikita Ivanov > > > > > > > > On Fri, Jun 25, 2021 at 3:31 PM Valentin Kulichenko < > > valentin.kuliche...@gmail.com> wrote: > > > > > Dear Community, > > > > > > In the last several months, the development of Ignite 3 has been moving > > > forward significantly. On top of what we already had in the first > Alpha, > > we > > > have the following features ready: > > > > > > - Replication infrastructure based on Raft > > > - In-memory atomic storage with the basic insert-read functionality > > > - New schema management engine and API > > > > > > In my view, this constitutes a significant milestone, and I propose to > > > release it as the Alpha 2. This way it will be available for download, > > and > > > anyone will be able to play with the project, run examples, get a feel > of > > > how Ignite will work in the future, and provide feedback. > > > > > > Please vote. > > > > > > --- > > > > > > The release candidate is uploaded here: > > > https://dist.apache.org/repos/dist/dev/ignite/3.0.0-alpha2-rc1/ > > > Maven staging: > > > > https://repository.apache.org/content/repositories/orgapacheignite-1522/ > > > Git tag: https://github.com/apache/ignite-3/tree/3.0.0-alpha2-rc1 > > > > > > Jira tickets: > > > > > > > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%2012315922%20AND%20fixVersion%20%3D%2012349574%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC > > > > > > DEVNOTES: > > > https://github.com/apache/ignite-3/blob/3.0.0-alpha2-rc1/DEVNOTES.md > > > > > > The vote is formal, see voting guidelines: > > > https://www.apache.org/foundation/voting.html > > > > > > +1 - accept Apache Ignite 3.0.0-alpha2 RC1 > > > 0 - don't care either way > > > -1 - DO NOT accept Apache Ignite 3.0.0-alpha2 RC1 (explain why) > > > > > > See notes on how to verify release here: > > > https://www.apache.org/info/verification.html > > > > > > This vote will be open for 72 hours and will close on June 28th at 11pm > > > UTC: > > > > > > > > > https://www.timeanddate.com/countdown/to?iso=20210628T16=224=Apache+Ignite+3.0.0-alpha2+RC1=cursive=1 > > > > > > -Val > > > > > >
Re: [VOTE] Release Apache Ignite 3.0.0-alpha2 RC1
+1 Best Regards, Igor On Sat, Jun 26, 2021 at 1:41 AM Nikita Ivanov wrote: > +1 > > -- > Nikita Ivanov > > > > On Fri, Jun 25, 2021 at 3:31 PM Valentin Kulichenko < > valentin.kuliche...@gmail.com> wrote: > > > Dear Community, > > > > In the last several months, the development of Ignite 3 has been moving > > forward significantly. On top of what we already had in the first Alpha, > we > > have the following features ready: > > > > - Replication infrastructure based on Raft > > - In-memory atomic storage with the basic insert-read functionality > > - New schema management engine and API > > > > In my view, this constitutes a significant milestone, and I propose to > > release it as the Alpha 2. This way it will be available for download, > and > > anyone will be able to play with the project, run examples, get a feel of > > how Ignite will work in the future, and provide feedback. > > > > Please vote. > > > > --- > > > > The release candidate is uploaded here: > > https://dist.apache.org/repos/dist/dev/ignite/3.0.0-alpha2-rc1/ > > Maven staging: > > https://repository.apache.org/content/repositories/orgapacheignite-1522/ > > Git tag: https://github.com/apache/ignite-3/tree/3.0.0-alpha2-rc1 > > > > Jira tickets: > > > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%2012315922%20AND%20fixVersion%20%3D%2012349574%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC > > > > DEVNOTES: > > https://github.com/apache/ignite-3/blob/3.0.0-alpha2-rc1/DEVNOTES.md > > > > The vote is formal, see voting guidelines: > > https://www.apache.org/foundation/voting.html > > > > +1 - accept Apache Ignite 3.0.0-alpha2 RC1 > > 0 - don't care either way > > -1 - DO NOT accept Apache Ignite 3.0.0-alpha2 RC1 (explain why) > > > > See notes on how to verify release here: > > https://www.apache.org/info/verification.html > > > > This vote will be open for 72 hours and will close on June 28th at 11pm > > UTC: > > > > > https://www.timeanddate.com/countdown/to?iso=20210628T16=224=Apache+Ignite+3.0.0-alpha2+RC1=cursive=1 > > > > -Val > > >
Re: [DISCUSSION] Brand new distributed environment testing framework ready to be reviewed/merged.
Seems, we have a silent agreement here :) We also discussed the proposal with the Ignite QA Meetup participants and found this contribution useful. Going to merge tomorrow, if there are no objections. On Mon, Jun 28, 2021 at 3:21 PM Anton Vinogradov wrote: > With regard to the request of Ivan Daschinskiy checked that we are able to > release code from the PR. > Looks like everything [1] is fine, ignite-ducketests are not a part of the > zip file. > > [1] > https://ci.ignite.apache.org/buildConfiguration/Releases_ApacheIgniteMain_ReleaseBuild/6064064?buildTab=artifacts#%2Frelease-2.11.0-rc0.zip!%2Fmaven;%2Frelease-2.11.0-rc0.zip!%2Fpackages > (you may need special permission to open this page) > > On Mon, Jun 28, 2021 at 11:01 AM Nikolay Izhikov > wrote: > >> Hello. >> >> As one of the authors of PR fully support merge of this module. >> Have plans to use it during development of other features. >> Guys, please, share your feedback. >> >> Do we need to improve something prior merge? >> >> >> > 24 июня 2021 г., в 14:41, Denis Magda написал(а): >> > >> > Anyway, it's much more community-friendly. Thanks, Anton! >> > >> > -- >> > Denis >> > >> > - >> > Denis >> > >> > On Thu, Jun 24, 2021 at 3:43 AM Anton Vinogradov wrote: >> > >> >> Denis, >> >> >> >> Unfortunately, we had some issues with this recording (low slides >> recording >> >> resolution, missed slides and random slides switch delay), which makes >> it >> >> slightly unwatchable :( >> >> Anyway, if you brave enough, you may open the slides [1] and the >> recording >> >> [2] simultaneously to solve this :) >> >> >> >> [1] >> >> >> >> >> https://go.gridgain.com/rs/491-TWR-806/images/11_IgniteSummit2021-Deep-dive-into-testing.pdf >> >> [2] https://www.youtube.com/watch?v=uRRlGrSA3NY >> >> >> >> On Wed, Jun 23, 2021 at 5:40 PM Denis Magda wrote: >> >> >> >>> Congrats, Anton, that's a valuable contribution! I attended your >> session >> >> at >> >>> the Ignite Summit and wonder if you should share that recording with >> an >> >>> English-speaking part of the community? >> >>> >> >>> - >> >>> Denis >> >>> >> >>> On Wed, Jun 23, 2021 at 7:37 AM Anton Vinogradov >> wrote: >> >>> >> Folks, >> >> Here's the video [1] that explains the proposal in detail. >> Feel free to ask questions here. >> >> [1] https://www.youtube.com/watch?v=f-i9COU5uAQ (in Russian) >> >> On Tue, Jun 8, 2021 at 2:51 PM Anton Vinogradov >> wrote: >> >> > Igniters, >> > Let me present a framework, we developed, that allows automating >> >> Apache >> > Ignite testing on a real cluster. >> > The framework was initially presented at Ignite Summit. >> > >> > In brief, >> > The framework allows automating operations with any applications on >> a >> real >> > cluster using ssh in a form of a python test. >> > >> > Features: >> > - Ignite nodes can be started/stopped on a Docker or a real cluster >> >>> with >> > any custom configuration >> > - Any AI version is supported (released or compiled from sources) >> > - Ignite forks are also supported «out of the box» >> > - Any other application execution is also possible, eg. we >> >> implemented >> > starters for Spark and Zookeeper >> > - Cluster can be managed using the control.sh, we made this a part >> of >> >>> the >> > test API >> > - Custom Java applications can be executed remotely with/without a >> > built-in Ignite node or a Thin client >> > - Any ssh command can be executed remotely, and the result will be >> > available locally (at the python test) >> > - Network can be broken by iptables change to check communication >> >>> issues >> > - Tests can be executed in parallel when cluster size bigger than >> >> tests >> > requirements >> > >> > As a result, seems, we may automate any testing scenario we can even >> > imagine. >> > >> > Framework based on Ducktape [1] library from Kafka team, that's why >> >> we >> > called it Ducktests. >> > >> > The Ducktests were developed during work on IEP-56 [2] and already >> >> were >> > used during the work on IEP-45 [3]. >> > IEP-45 measurement results examples [4.1] [4.2] were demonstrated at >> >>> the >> > HighLoad conference last month. >> > >> > Code available at PR-9117 [5] and ready to be reviewed/merged. >> > Feel free to ask questions or make proposals. >> > >> > [1] https://ducktape-docs.readthedocs.io/en/latest/index.html >> > [2] >> > >> >> >>> >> >> >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-56%3A+Distributed+environment+tests >> > [3] >> > >> >> >>> >> >> >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up >> > [4.1] https://youtu.be/UZsvCNjbkww?t=1065 (in Russian) >> > [4.2] https://youtu.be/UZsvCNjbkww?t=1609 (in Russian) >> > [5] https://github.com/apache/ignite/pull/9117 >>
Re: Defrag?
Hello! Is it WAL (wal/) that is growing or checkpoint space (db/)? If latter, any specific caches that are growing unbound? If letter, you can try creating a new cache, moving the relevant data to this new cache, switch to using it, and then drop the old cache - should reclaim the space. Regards, -- Ilya Kasnacheev пн, 28 июн. 2021 г. в 17:34, Ryan Trollip : > Is this why the native disk storage just keeps growing and does not reduce > after we delete from ignite using SQL? > We are up to 80GB on disk now on some instances. We implemented a custom > archiving feature to move older data out of ignite cache to a PostgresSQL > database but when we delete that data from ignite instance, the disk data > size ignite is using stays the same, and then keeps growing, and > growing > > On Thu, Jun 24, 2021 at 7:10 PM Denis Magda wrote: > >> Ignite fellows, >> >> I remember some of us worked on the persistence defragmentation features. >> Has it been merged? >> >> @Valentin Kulichenko probably you know >> the latest state. >> >> - >> Denis >> >> On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev < >> ilya.kasnach...@gmail.com> wrote: >> >>> Hello! >>> >>> You can probably drop the entire cache and then re-populate it via >>> loadCache(), etc. >>> >>> Regards, >>> -- >>> Ilya Kasnacheev >>> >>> >>> ср, 23 июн. 2021 г. в 21:47, Ryan Trollip : >>> Thanks, Ilya, we may have to consider moving back to non-native storage and caching more selectively as the performance degrades when there is a lot of write/delete activity or tables with large amounts of rows. This is with SQL with indexes and the use of query plans etc. Is there any easy way to rebuild the entire native database after hours? e.g. with a batch run on the weeknds? On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev < ilya.kasnach...@gmail.com> wrote: > Hello! > > I don't think there's anything ready to use, but "killing performance" > from fragmentation is also not something reported too often. > > Regards, > -- > Ilya Kasnacheev > > > ср, 16 июн. 2021 г. в 04:39, Ryan Trollip : > >> We see continual very large growth to data with ignite native. We >> have a very chatty use case that's creating and deleting stuff often. The >> data on disk just keeps growing at an explosive rate. So much so we >> ported >> this to a DB to see the difference and the DB is much smaller. I was >> searching to see if someone has the same issue. This is also killing >> performance. >> >> Founds this: >> >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation >> >> Apparently, there is no auto-rebalancing of pages? or cleanup of >> pages? >> >> Has anyone implemented a workaround to rebuild the cache and indexes >> say on a weekly basis to get it to behave reasonably? >> >> Thanks >> >
Re: Defrag?
Is this why the native disk storage just keeps growing and does not reduce after we delete from ignite using SQL? We are up to 80GB on disk now on some instances. We implemented a custom archiving feature to move older data out of ignite cache to a PostgresSQL database but when we delete that data from ignite instance, the disk data size ignite is using stays the same, and then keeps growing, and growing On Thu, Jun 24, 2021 at 7:10 PM Denis Magda wrote: > Ignite fellows, > > I remember some of us worked on the persistence defragmentation features. > Has it been merged? > > @Valentin Kulichenko probably you know > the latest state. > > - > Denis > > On Thu, Jun 24, 2021 at 11:59 AM Ilya Kasnacheev < > ilya.kasnach...@gmail.com> wrote: > >> Hello! >> >> You can probably drop the entire cache and then re-populate it via >> loadCache(), etc. >> >> Regards, >> -- >> Ilya Kasnacheev >> >> >> ср, 23 июн. 2021 г. в 21:47, Ryan Trollip : >> >>> Thanks, Ilya, we may have to consider moving back to non-native storage >>> and caching more selectively as the performance degrades when there is a >>> lot of write/delete activity or tables with large amounts of rows. This is >>> with SQL with indexes and the use of query plans etc. >>> >>> Is there any easy way to rebuild the entire native database after hours? >>> e.g. with a batch run on the weeknds? >>> >>> On Wed, Jun 23, 2021 at 7:39 AM Ilya Kasnacheev < >>> ilya.kasnach...@gmail.com> wrote: >>> Hello! I don't think there's anything ready to use, but "killing performance" from fragmentation is also not something reported too often. Regards, -- Ilya Kasnacheev ср, 16 июн. 2021 г. в 04:39, Ryan Trollip : > We see continual very large growth to data with ignite native. We have > a very chatty use case that's creating and deleting stuff often. The data > on disk just keeps growing at an explosive rate. So much so we ported this > to a DB to see the difference and the DB is much smaller. I was searching > to see if someone has the same issue. This is also killing performance. > > Founds this: > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-47%3A+Native+persistence+defragmentation > > Apparently, there is no auto-rebalancing of pages? or cleanup of > pages? > > Has anyone implemented a workaround to rebuild the cache and indexes > say on a weekly basis to get it to behave reasonably? > > Thanks >
Re: [VOTE][EXTENSION] Release Apache Ignite spring-data-all-ext extensions 1.0.0 RC4
Folks, we already have Ignite release semi-automated at TeamCity. We able to assembly, publish as RC and check published. Seems, we must do the same at extensions. This way we'll check issues like this even before starting the vote. On Mon, Jun 28, 2021 at 2:14 PM Ilya Kasnacheev wrote: > Hello! > > -1 (binding) > > The root element of the source zip file should be > ignite-spring-data-all-ext-1.0.0-src/ directory. No other directories/files > should be at the root of the source zip file. > > This is a *blocker* but should be very straightforward to fix. > > Regards, > -- > Ilya Kasnacheev > > > пт, 25 июн. 2021 г. в 17:46, Sergei Ryzhov : > > > +1 > > Building modules from source and running tests successfully on Java 8 > > > > пт, 25 июн. 2021 г. в 16:52, Alex Plehanov : > > > > > +1 > > > > > > Check sha512, sign of archives and versions of released artifacts (BTW, > > > version of parent artifact is 1.0.0-SNAPSHOT, I don't think it's a > > blocker > > > now, but it's better to be fixed in the next releases). > > > Check licenses with "mvn validate -DskipTests -P check-licenses" > > > Build modules from the source (Note: build successfully on Java 8, but > > > build on Java 11 has failed due to maven-javadoc-plugin. To build on > Java > > > 11 I have used this command: "mvn clean install -DskipTests > > > -Dmaven.javadoc.skip=true") > > > Run examples (create new maven project and copy sources from examples > > > directories): > > > from modules/spring-data-2.0-ext/examples with artifacts: Ignite > > > v2.10.0/ignite-spring-data-2.0-ext v1.0.0/spring-data v2.0 > > > from modules/spring-data-2.2-ext/examples with artifacts: Ignite > > > v2.10.0/ignite-spring-data-2.2-ext v1.0.0/spring-data v2.2 > > > > > > пт, 25 июн. 2021 г. в 16:26, Ilya Kasnacheev < > ilya.kasnach...@gmail.com > > >: > > > > > > > Hello! > > > > > > > > The contents of the source package are much more reasonable now. > > However, > > > > there's still an issue that root element of zip archive is not a > > > directory: > > > > ~/Downloads/spring% unzip ignite-spring-data-all-ext-1.0.0-src.zip > > > > Archive: ignite-spring-data-all-ext-1.0.0-src.zip > > > > creating: checkstyle/ > > > > inflating: checkstyle/checkstyle-suppressions.xml > > > > inflating: checkstyle/checkstyle.xml > > > > creating: config/ > > > > inflating: config/example-ignite.xml > > > > creating: modules/ > > > > creating: modules/spring-data-2.0-ext/ > > > > inflating: modules/spring-data-2.0-ext/README.txt > > > > > > > > > > > > whereas > > > > ~/Downloads% unzip -l apache-ignite-2.8.1-src.zip| head > > > > Archive: apache-ignite-2.8.1-src.zip > > > > 864220966caa4157c4fee8a1bc85171623963604 > > > > Length DateTimeName > > > > - -- - > > > >0 2020-05-21 16:42 apache-ignite-2.8.1-src/ > > > > 2029 2020-05-21 16:42 apache-ignite-2.8.1-src/.gitignore > > > >0 2020-05-21 16:42 apache-ignite-2.8.1-src/.idea/ > > > >0 2020-05-21 16:42 > > > > apache-ignite-2.8.1-src/.idea/inspectionProfiles/ > > > >80905 2020-05-21 16:42 > > > > > apache-ignite-2.8.1-src/.idea/inspectionProfiles/Project_Default.xml > > > > 5119 2020-05-21 16:42 apache-ignite-2.8.1-src/CONTRIBUTING.md > > > > > > > > > > > > The root element of source sip should > > > > be ignite-spring-data-all-ext-1.0.0-src/ directory. > > > > > > > > This is a small issue but a *blocker*, maybe you can do a quick > > rebuild? > > > > > > > > Another issue is that I don't understand how to run examples. If I > add > > > the > > > > entire directory as maven project, the examples/main directory is not > > > > considered source root and will not be built/run > > > > Maybe I'm missing something. > > > > > > > > Can you add is as an additional sources root, perhaps? You can even > > > build a > > > > jar with classifier 'examples' if that's OK. > > > > > > > > Regards, > > > > > > > > -- > > > > Ilya Kasnacheev > > > > > > > > > > > > пт, 25 июн. 2021 г. в 14:17, Mikhail Petrov : > > > > > > > > > +1 > > > > > > > > > > Checked on Ubuntu 20.04. Created separate Spring Applications for > > each > > > > > version of Spring Data Integrations using the Maven RC repository > and > > > > > following the documentation. Tested common use cases. > > > > > > > > > > On 25.06.2021 12:43, Nikita Amelchev wrote: > > > > > > Dear Ignite Community, > > > > > > > > > > > > I have uploaded a release candidate of the following extension > > > modules: > > > > > > > > > > > > spring-data-commons > > > > > > spring-data-ext > > > > > > spring-data-2.0-ext > > > > > > spring-data-2.2-ext > > > > > > > > > > > > The release candidate of the extensions: > > > > > > > > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-spring-data-all-ext/1.0.0-rc4/ > > > > > > > > > > > > The instruction for building from the source package is placed in > > the > > > > > > DEVNOTES file. > > > > > > > >
Re: Publishing Spring Sessions to Maven Repo
Hello! It is being voted on currently, but will need one more round probably. Regards, -- Ilya Kasnacheev пн, 28 июн. 2021 г. в 14:35, Atri Sharma : > Gentle ping > > On Thu, Jun 24, 2021 at 6:36 PM Atri Sharma wrote: > > > > Hi All, > > > > Can we please publish Spring Sessions Ext artifacts to the Maven repo? > > > > Atri > > > > -- > Regards, > > Atri > Apache Concerted >
Re: Text Queries Support
Hello! I think that number one is the most important one, then maybe it will see more use and other deficiencies become more apparent, leading to more tickets and visibility. Maybe 2. and 3. will even use a different approach when persistence is implemented. Regards, -- Ilya Kasnacheev пн, 28 июн. 2021 г. в 14:34, Atri Sharma : > Hello Again! > > I have been looking into the aforementioned and here are my follow up > thoughts: > > 1. Support persistence of Lucene indexes. > 2. https://issues.apache.org/jira/browse/IGNITE-12401 (Needs fixing of > moving partitions first) > 3. Figure out how to return scores from nodes and use them as sort > parameters on the coordinator node > (https://issues.apache.org/jira/browse/IGNITE-12291) > > Please let me know if this looks ok to make text queries functional? > > Atri > > On Mon, Jun 21, 2021 at 2:49 PM Alexei Scherbakov > wrote: > > > > Hi. > > > > One of the biggest issues with text queries is a lack of support for > lucene > > indices persistence, which makes this functionality useless if a > > persistence is enabled. > > > > I would first take care of it. > > > > пн, 21 июн. 2021 г. в 12:16, Maksim Timonin : > > > > > Hi, Atri! > > > > > > You're right, Actually there is a lack of support for TextQueries. For > the > > > last ticket I'm doing I see some obvious issues with them (no page size > > > support, for example). I'm glad that somebody wants to maintain this > > > functionality. Thanks a lot! > > > > > > For the MergeSort algorithm there is already a patch for that [1]. It's > > > currently on review. This patch introduces an abstract reducer for > > > CacheQueries with 2 implementations (unordered, merge-sort). Then > TextQuery > > > leverages on MergeSort to order results from multiple nodes by score. > This > > > patch also fixes the pageSize issue, I've mentioned before. Could you > > > please check if it fully matches your idea? Any issues or comments are > > > welcome. > > > > > > I've prepared this ticket, because I need the MergeSort algorithm for > the > > > new type of queries I'm implementing (IndexQuery, it should also > provide > > > ordered results over multiple nodes). Currently I'm not planning to go > > > further with TextQuery, so if you're going to support this it'll be a > great > > > contribution, I think. > > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14703 > > > [2] https://github.com/apache/ignite/pull/9081 > > > > > > > > > On Mon, Jun 21, 2021 at 11:11 AM Atri Sharma wrote: > > > > > > > Hi All, > > > > > > > > I have been looking into our text queries support and see that it has > > > > limited community support. > > > > > > > > Therefore, I volunteer to be the maintainer of the module and work on > > > > enhancing it further. > > > > > > > > First goal would be to move to Lucene 8.x, then work on sorted reduce > > > > - merge across nodes. Fundamentally, this is doable since Lucene > ranks > > > > documents according to their score, and documents are returned in the > > > > order of their score. Since the scoring function is homogeneous, this > > > > means that across nodes, we can compare scores and merge sort. > > > > > > > > Please let me know if I can take this up. > > > > > > > > Atri > > > > > > > > -- > > > > Regards, > > > > > > > > Atri > > > > Apache Concerted > > > > > > > > > > > > > -- > > > > Best regards, > > Alexei Scherbakov > > -- > Regards, > > Atri > Apache Concerted >
Re: [DISCUSSION] Brand new distributed environment testing framework ready to be reviewed/merged.
With regard to the request of Ivan Daschinskiy checked that we are able to release code from the PR. Looks like everything [1] is fine, ignite-ducketests are not a part of the zip file. [1] https://ci.ignite.apache.org/buildConfiguration/Releases_ApacheIgniteMain_ReleaseBuild/6064064?buildTab=artifacts#%2Frelease-2.11.0-rc0.zip!%2Fmaven;%2Frelease-2.11.0-rc0.zip!%2Fpackages (you may need special permission to open this page) On Mon, Jun 28, 2021 at 11:01 AM Nikolay Izhikov wrote: > Hello. > > As one of the authors of PR fully support merge of this module. > Have plans to use it during development of other features. > Guys, please, share your feedback. > > Do we need to improve something prior merge? > > > > 24 июня 2021 г., в 14:41, Denis Magda написал(а): > > > > Anyway, it's much more community-friendly. Thanks, Anton! > > > > -- > > Denis > > > > - > > Denis > > > > On Thu, Jun 24, 2021 at 3:43 AM Anton Vinogradov wrote: > > > >> Denis, > >> > >> Unfortunately, we had some issues with this recording (low slides > recording > >> resolution, missed slides and random slides switch delay), which makes > it > >> slightly unwatchable :( > >> Anyway, if you brave enough, you may open the slides [1] and the > recording > >> [2] simultaneously to solve this :) > >> > >> [1] > >> > >> > https://go.gridgain.com/rs/491-TWR-806/images/11_IgniteSummit2021-Deep-dive-into-testing.pdf > >> [2] https://www.youtube.com/watch?v=uRRlGrSA3NY > >> > >> On Wed, Jun 23, 2021 at 5:40 PM Denis Magda wrote: > >> > >>> Congrats, Anton, that's a valuable contribution! I attended your > session > >> at > >>> the Ignite Summit and wonder if you should share that recording with an > >>> English-speaking part of the community? > >>> > >>> - > >>> Denis > >>> > >>> On Wed, Jun 23, 2021 at 7:37 AM Anton Vinogradov > wrote: > >>> > Folks, > > Here's the video [1] that explains the proposal in detail. > Feel free to ask questions here. > > [1] https://www.youtube.com/watch?v=f-i9COU5uAQ (in Russian) > > On Tue, Jun 8, 2021 at 2:51 PM Anton Vinogradov > wrote: > > > Igniters, > > Let me present a framework, we developed, that allows automating > >> Apache > > Ignite testing on a real cluster. > > The framework was initially presented at Ignite Summit. > > > > In brief, > > The framework allows automating operations with any applications on a > real > > cluster using ssh in a form of a python test. > > > > Features: > > - Ignite nodes can be started/stopped on a Docker or a real cluster > >>> with > > any custom configuration > > - Any AI version is supported (released or compiled from sources) > > - Ignite forks are also supported «out of the box» > > - Any other application execution is also possible, eg. we > >> implemented > > starters for Spark and Zookeeper > > - Cluster can be managed using the control.sh, we made this a part of > >>> the > > test API > > - Custom Java applications can be executed remotely with/without a > > built-in Ignite node or a Thin client > > - Any ssh command can be executed remotely, and the result will be > > available locally (at the python test) > > - Network can be broken by iptables change to check communication > >>> issues > > - Tests can be executed in parallel when cluster size bigger than > >> tests > > requirements > > > > As a result, seems, we may automate any testing scenario we can even > > imagine. > > > > Framework based on Ducktape [1] library from Kafka team, that's why > >> we > > called it Ducktests. > > > > The Ducktests were developed during work on IEP-56 [2] and already > >> were > > used during the work on IEP-45 [3]. > > IEP-45 measurement results examples [4.1] [4.2] were demonstrated at > >>> the > > HighLoad conference last month. > > > > Code available at PR-9117 [5] and ready to be reviewed/merged. > > Feel free to ask questions or make proposals. > > > > [1] https://ducktape-docs.readthedocs.io/en/latest/index.html > > [2] > > > > >>> > >> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-56%3A+Distributed+environment+tests > > [3] > > > > >>> > >> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up > > [4.1] https://youtu.be/UZsvCNjbkww?t=1065 (in Russian) > > [4.2] https://youtu.be/UZsvCNjbkww?t=1609 (in Russian) > > [5] https://github.com/apache/ignite/pull/9117 > > > > >>> > >> > >
Re: Publishing Spring Sessions to Maven Repo
Gentle ping On Thu, Jun 24, 2021 at 6:36 PM Atri Sharma wrote: > > Hi All, > > Can we please publish Spring Sessions Ext artifacts to the Maven repo? > > Atri -- Regards, Atri Apache Concerted
Re: Text Queries Support
Hello Again! I have been looking into the aforementioned and here are my follow up thoughts: 1. Support persistence of Lucene indexes. 2. https://issues.apache.org/jira/browse/IGNITE-12401 (Needs fixing of moving partitions first) 3. Figure out how to return scores from nodes and use them as sort parameters on the coordinator node (https://issues.apache.org/jira/browse/IGNITE-12291) Please let me know if this looks ok to make text queries functional? Atri On Mon, Jun 21, 2021 at 2:49 PM Alexei Scherbakov wrote: > > Hi. > > One of the biggest issues with text queries is a lack of support for lucene > indices persistence, which makes this functionality useless if a > persistence is enabled. > > I would first take care of it. > > пн, 21 июн. 2021 г. в 12:16, Maksim Timonin : > > > Hi, Atri! > > > > You're right, Actually there is a lack of support for TextQueries. For the > > last ticket I'm doing I see some obvious issues with them (no page size > > support, for example). I'm glad that somebody wants to maintain this > > functionality. Thanks a lot! > > > > For the MergeSort algorithm there is already a patch for that [1]. It's > > currently on review. This patch introduces an abstract reducer for > > CacheQueries with 2 implementations (unordered, merge-sort). Then TextQuery > > leverages on MergeSort to order results from multiple nodes by score. This > > patch also fixes the pageSize issue, I've mentioned before. Could you > > please check if it fully matches your idea? Any issues or comments are > > welcome. > > > > I've prepared this ticket, because I need the MergeSort algorithm for the > > new type of queries I'm implementing (IndexQuery, it should also provide > > ordered results over multiple nodes). Currently I'm not planning to go > > further with TextQuery, so if you're going to support this it'll be a great > > contribution, I think. > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14703 > > [2] https://github.com/apache/ignite/pull/9081 > > > > > > On Mon, Jun 21, 2021 at 11:11 AM Atri Sharma wrote: > > > > > Hi All, > > > > > > I have been looking into our text queries support and see that it has > > > limited community support. > > > > > > Therefore, I volunteer to be the maintainer of the module and work on > > > enhancing it further. > > > > > > First goal would be to move to Lucene 8.x, then work on sorted reduce > > > - merge across nodes. Fundamentally, this is doable since Lucene ranks > > > documents according to their score, and documents are returned in the > > > order of their score. Since the scoring function is homogeneous, this > > > means that across nodes, we can compare scores and merge sort. > > > > > > Please let me know if I can take this up. > > > > > > Atri > > > > > > -- > > > Regards, > > > > > > Atri > > > Apache Concerted > > > > > > > > -- > > Best regards, > Alexei Scherbakov -- Regards, Atri Apache Concerted
Re: [VOTE][EXTENSION] Release Apache Ignite spring-data-all-ext extensions 1.0.0 RC4
Hello! -1 (binding) The root element of the source zip file should be ignite-spring-data-all-ext-1.0.0-src/ directory. No other directories/files should be at the root of the source zip file. This is a *blocker* but should be very straightforward to fix. Regards, -- Ilya Kasnacheev пт, 25 июн. 2021 г. в 17:46, Sergei Ryzhov : > +1 > Building modules from source and running tests successfully on Java 8 > > пт, 25 июн. 2021 г. в 16:52, Alex Plehanov : > > > +1 > > > > Check sha512, sign of archives and versions of released artifacts (BTW, > > version of parent artifact is 1.0.0-SNAPSHOT, I don't think it's a > blocker > > now, but it's better to be fixed in the next releases). > > Check licenses with "mvn validate -DskipTests -P check-licenses" > > Build modules from the source (Note: build successfully on Java 8, but > > build on Java 11 has failed due to maven-javadoc-plugin. To build on Java > > 11 I have used this command: "mvn clean install -DskipTests > > -Dmaven.javadoc.skip=true") > > Run examples (create new maven project and copy sources from examples > > directories): > > from modules/spring-data-2.0-ext/examples with artifacts: Ignite > > v2.10.0/ignite-spring-data-2.0-ext v1.0.0/spring-data v2.0 > > from modules/spring-data-2.2-ext/examples with artifacts: Ignite > > v2.10.0/ignite-spring-data-2.2-ext v1.0.0/spring-data v2.2 > > > > пт, 25 июн. 2021 г. в 16:26, Ilya Kasnacheev >: > > > > > Hello! > > > > > > The contents of the source package are much more reasonable now. > However, > > > there's still an issue that root element of zip archive is not a > > directory: > > > ~/Downloads/spring% unzip ignite-spring-data-all-ext-1.0.0-src.zip > > > Archive: ignite-spring-data-all-ext-1.0.0-src.zip > > > creating: checkstyle/ > > > inflating: checkstyle/checkstyle-suppressions.xml > > > inflating: checkstyle/checkstyle.xml > > > creating: config/ > > > inflating: config/example-ignite.xml > > > creating: modules/ > > > creating: modules/spring-data-2.0-ext/ > > > inflating: modules/spring-data-2.0-ext/README.txt > > > > > > > > > whereas > > > ~/Downloads% unzip -l apache-ignite-2.8.1-src.zip| head > > > Archive: apache-ignite-2.8.1-src.zip > > > 864220966caa4157c4fee8a1bc85171623963604 > > > Length DateTimeName > > > - -- - > > >0 2020-05-21 16:42 apache-ignite-2.8.1-src/ > > > 2029 2020-05-21 16:42 apache-ignite-2.8.1-src/.gitignore > > >0 2020-05-21 16:42 apache-ignite-2.8.1-src/.idea/ > > >0 2020-05-21 16:42 > > > apache-ignite-2.8.1-src/.idea/inspectionProfiles/ > > >80905 2020-05-21 16:42 > > > apache-ignite-2.8.1-src/.idea/inspectionProfiles/Project_Default.xml > > > 5119 2020-05-21 16:42 apache-ignite-2.8.1-src/CONTRIBUTING.md > > > > > > > > > The root element of source sip should > > > be ignite-spring-data-all-ext-1.0.0-src/ directory. > > > > > > This is a small issue but a *blocker*, maybe you can do a quick > rebuild? > > > > > > Another issue is that I don't understand how to run examples. If I add > > the > > > entire directory as maven project, the examples/main directory is not > > > considered source root and will not be built/run > > > Maybe I'm missing something. > > > > > > Can you add is as an additional sources root, perhaps? You can even > > build a > > > jar with classifier 'examples' if that's OK. > > > > > > Regards, > > > > > > -- > > > Ilya Kasnacheev > > > > > > > > > пт, 25 июн. 2021 г. в 14:17, Mikhail Petrov : > > > > > > > +1 > > > > > > > > Checked on Ubuntu 20.04. Created separate Spring Applications for > each > > > > version of Spring Data Integrations using the Maven RC repository and > > > > following the documentation. Tested common use cases. > > > > > > > > On 25.06.2021 12:43, Nikita Amelchev wrote: > > > > > Dear Ignite Community, > > > > > > > > > > I have uploaded a release candidate of the following extension > > modules: > > > > > > > > > > spring-data-commons > > > > > spring-data-ext > > > > > spring-data-2.0-ext > > > > > spring-data-2.2-ext > > > > > > > > > > The release candidate of the extensions: > > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/ignite/ignite-extensions/ignite-spring-data-all-ext/1.0.0-rc4/ > > > > > > > > > > The instruction for building from the source package is placed in > the > > > > > DEVNOTES file. > > > > > > > > > > The following staging can be used for testing: > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheignite-1520 > > > > > > > > > > Tags were created: > > > > > > > > > > ignite-spring-data-commons-1.0.0-rc4 > > > > > ignite-spring-data-ext-1.0.0-rc4 > > > > > ignite-spring-data-2.2-ext-1.0.0-rc4 > > > > > ignite-spring-data-2.0-ext-1.0.0-rc4 > > > > > ignite-spring-data-all-ext-1.0.0-rc4 > > > > > > > > > > > > > > > https://gitbox.apache.org/repos/asf?p=ignite-extensions.git;a=commit;h=56db147fc059f3a64e0c00fe9d0a9d0d70ec97b0 > > > > > > > > >
Re: [DISCUSSION] Brand new distributed environment testing framework ready to be reviewed/merged.
Hello. As one of the authors of PR fully support merge of this module. Have plans to use it during development of other features. Guys, please, share your feedback. Do we need to improve something prior merge? > 24 июня 2021 г., в 14:41, Denis Magda написал(а): > > Anyway, it's much more community-friendly. Thanks, Anton! > > -- > Denis > > - > Denis > > On Thu, Jun 24, 2021 at 3:43 AM Anton Vinogradov wrote: > >> Denis, >> >> Unfortunately, we had some issues with this recording (low slides recording >> resolution, missed slides and random slides switch delay), which makes it >> slightly unwatchable :( >> Anyway, if you brave enough, you may open the slides [1] and the recording >> [2] simultaneously to solve this :) >> >> [1] >> >> https://go.gridgain.com/rs/491-TWR-806/images/11_IgniteSummit2021-Deep-dive-into-testing.pdf >> [2] https://www.youtube.com/watch?v=uRRlGrSA3NY >> >> On Wed, Jun 23, 2021 at 5:40 PM Denis Magda wrote: >> >>> Congrats, Anton, that's a valuable contribution! I attended your session >> at >>> the Ignite Summit and wonder if you should share that recording with an >>> English-speaking part of the community? >>> >>> - >>> Denis >>> >>> On Wed, Jun 23, 2021 at 7:37 AM Anton Vinogradov wrote: >>> Folks, Here's the video [1] that explains the proposal in detail. Feel free to ask questions here. [1] https://www.youtube.com/watch?v=f-i9COU5uAQ (in Russian) On Tue, Jun 8, 2021 at 2:51 PM Anton Vinogradov wrote: > Igniters, > Let me present a framework, we developed, that allows automating >> Apache > Ignite testing on a real cluster. > The framework was initially presented at Ignite Summit. > > In brief, > The framework allows automating operations with any applications on a real > cluster using ssh in a form of a python test. > > Features: > - Ignite nodes can be started/stopped on a Docker or a real cluster >>> with > any custom configuration > - Any AI version is supported (released or compiled from sources) > - Ignite forks are also supported «out of the box» > - Any other application execution is also possible, eg. we >> implemented > starters for Spark and Zookeeper > - Cluster can be managed using the control.sh, we made this a part of >>> the > test API > - Custom Java applications can be executed remotely with/without a > built-in Ignite node or a Thin client > - Any ssh command can be executed remotely, and the result will be > available locally (at the python test) > - Network can be broken by iptables change to check communication >>> issues > - Tests can be executed in parallel when cluster size bigger than >> tests > requirements > > As a result, seems, we may automate any testing scenario we can even > imagine. > > Framework based on Ducktape [1] library from Kafka team, that's why >> we > called it Ducktests. > > The Ducktests were developed during work on IEP-56 [2] and already >> were > used during the work on IEP-45 [3]. > IEP-45 measurement results examples [4.1] [4.2] were demonstrated at >>> the > HighLoad conference last month. > > Code available at PR-9117 [5] and ready to be reviewed/merged. > Feel free to ask questions or make proposals. > > [1] https://ducktape-docs.readthedocs.io/en/latest/index.html > [2] > >>> >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-56%3A+Distributed+environment+tests > [3] > >>> >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-45%3A+Crash+Recovery+Speed-Up > [4.1] https://youtu.be/UZsvCNjbkww?t=1065 (in Russian) > [4.2] https://youtu.be/UZsvCNjbkww?t=1609 (in Russian) > [5] https://github.com/apache/ignite/pull/9117 > >>> >>
