Re: [ANNOUNCE] New committer: Mikhail Pochatkin
Great ! Good luck Mikhail ! >Congratulations, keep it going! > >Thanks, >S. > >ср, 8 мая 2024г. в 07:27, Pavel Tupitsyn < ptupit...@apache.org >: > >> Dear Igniters, >> >> The Project Management Committee (PMC) for Apache Ignite >> has invited Mikhail Pochatkin to become a committer and we are pleased >> to announce that they have accepted. >> >> Being a committer enables easier contribution to the >> project since there is no need to go via the patch >> submission process. This should enable better productivity. >> >> Mikhail, congratulations on your new role! >> >> Pavel Tupitsyn >> on behalf of Apache Ignite PMC >>
Re: [ANNOUNCE] New committer: Roman Puchkovskiy
Great ! My congrats too ! >Congrats, Roman! Well deserved! > >Thanks, >S. > >ср, 8 мая 2024г. в 07:28, Pavel Tupitsyn < ptupit...@apache.org >: > >> Dear Igniters, >> >> The Project Management Committee (PMC) for Apache Ignite >> has invited Roman Puchkovskiy to become a committer and we are pleased >> to announce that they have accepted. >> >> Being a committer enables easier contribution to the >> project since there is no need to go via the patch >> submission process. This should enable better productivity. >> >> Roman, congratulations on your new role! >> >> Pavel Tupitsyn >> on behalf of Apache Ignite PMC >>
Re: [VOTE] Release Apache Ignite 2.16.0 RC0
Finally i build it (some local maven problems). Check compilation, run calcite tests. +1 (non-binding) > >> >>>Hello, Zhenya. >>> >>>Java profiles should be activated automatically (on reload maven >>>project). Which JDK are you using? Is the project built from maven? >>> >>>I can't reproduce on HotSpot 1.8.0_201-b09, OpenJDK 21+35-2513. >>> >>>вт, 19 дек. 2023г. в 11:16, Zhenya Stanilovsky < arzamas...@mail.ru.invalid >>>>: >>>> >>>> >>>> Hi, probably it`s a kind of problem from my side, but cloning by tag and >>>> further steps: >>>> * change profile to java-8 >>>> * Reload all maven Projects >>>> * Try to run: ScriptTestSuite will raise guava dependency problem: >>>> /ignite/internal/processors/query/calcite/sql/IgniteSqlRollback.java:20:33 >>>> java: package com.google.common.collect does not exist >>>> --- >>>> * change profile to java-9 >>>> * reload >>>> * the same problem : >>>> modules/core/src/test/java/org/apache/ignite/testframework/GridTestUtils.java:79:33 >>>> java: package com.google.common.collect does not exist >>>> >>>> probably i need to clean all and try to rebuild from scratch .. for now >>>> it`s not clear for me. >>>> >>>> >+1 >>>> > >>>> >- Started the node from binaries >>>> >- Tested .NET examples >>>> >- Tested NuGet packages >>>> > >>>> >On Sun, Dec 17, 2023 at 2:48PM Nikita Amelchev < namelc...@apache.org > >>>> >wrote: >>>> > >>>> >> Dear Community, >>>> >> >>>> >> The release candidate is ready. >>>> >> >>>> >> I have uploaded release candidate to >>>> >> https://dist.apache.org/repos/dist/dev/ignite/2.16.0-rc0 >>>> >> https://dist.apache.org/repos/dist/dev/ignite/packages_2.16.0-rc0 >>>> >> >>>> >> The following staging can be used for testing: >>>> >> https://repository.apache.org/content/repositories/orgapacheignite-1559 >>>> >> >>>> >> Tag name is 2.16.0-rc0: >>>> >> https://github.com/apache/ignite/releases/tag/2.16.0-rc0 >>>> >> >>>> >> 2.16.0 most important changes: >>>> >> * Cache dumps; >>>> >> * Calcite engine: added hints, became more stable, covered with >>>> >> diagnostic tools; >>>> >> * Fixes to support JDK 14-21; >>>> >> * Cluster Management API (IEP-81); >>>> >> * Java thin client: Service Awareness feature; >>>> >> etc. >>>> >> >>>> >> RELEASE NOTES: >>>> >> >>>> >> >>>> >> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.16 >>>> >> >>>> >> Complete list of resolved issues: >>>> >> >>>> >> >>>> >> https://issues.apache.org/jira/browse/IGNITE-19650?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.16%27))%20AND%20(component%20is%20EMPTY%20OR%20component%20not%20in%20(documentation))%20and%20status%20in%20(%27CLOSED%27%2C%20%27RESOLVED%27)%20AND%20resolution%20in(Fixed%2C%20Done%2C%20Implemented%2C%20Delivered)%20ORDER%20BY%20priority >>>> >> >>>> >> DEVNOTES >>>> >> >>>> >> >>>> >> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.16 >>>> >> >>>> >> The vote is formal, see voting guidelines >>>> >> https://www.apache.org/foundation/voting.html >>>> >> >>>> >> +1 - to accept Apache Ignite 2.16.0-rc0 >>>> >> 0 - don't care either way >>>> >> -1 - DO NOT accept Apache Ignite 2.16.0-rc0 (explain why) >>>> >> >>>> >> See notes on how to verify release here >>>> >> https://www.apache.org/info/verification.html >>>> >> and >>>> >> >>>> >> >>>> >> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification >>>> >> >>>> >> This vote will be open till Sun December 24, 2023, 15:00 UTC. >>>> >> >>>> >> >>>> >> https://www.timeanddate.com/countdown/vote?iso=20231224T15=0=VOTE+on+the+Apache+Ignite+Release+2.16.0+RC0=sanserif >>>> >> >>>> >> -- >>>> >> Best wishes, >>>> >> Amelchev Nikita >>>> >> >>>> >>>> >>>> >>>> >>> >>> >>>-- >>>Best wishes, >>>Amelchev Nikita >> >> >> >>
Re: [VOTE] Release Apache Ignite 2.16.0 RC0
Thanks Nikita. remove «.idea» folder, call Invalidate and Restart from Idea but it not helps ( Idea: IntelliJ IDEA 2023.2.5 jdk 11.0.17 build from IDE not from maven >Hello, Zhenya. > >Java profiles should be activated automatically (on reload maven >project). Which JDK are you using? Is the project built from maven? > >I can't reproduce on HotSpot 1.8.0_201-b09, OpenJDK 21+35-2513. > >вт, 19 дек. 2023г. в 11:16, Zhenya Stanilovsky < arzamas...@mail.ru.invalid >: >> >> >> Hi, probably it`s a kind of problem from my side, but cloning by tag and >> further steps: >> * change profile to java-8 >> * Reload all maven Projects >> * Try to run: ScriptTestSuite will raise guava dependency problem: >> /ignite/internal/processors/query/calcite/sql/IgniteSqlRollback.java:20:33 >> java: package com.google.common.collect does not exist >> --- >> * change profile to java-9 >> * reload >> * the same problem : >> modules/core/src/test/java/org/apache/ignite/testframework/GridTestUtils.java:79:33 >> java: package com.google.common.collect does not exist >> >> probably i need to clean all and try to rebuild from scratch .. for now it`s >> not clear for me. >> >> >+1 >> > >> >- Started the node from binaries >> >- Tested .NET examples >> >- Tested NuGet packages >> > >> >On Sun, Dec 17, 2023 at 2:48PM Nikita Amelchev < namelc...@apache.org > >> >wrote: >> > >> >> Dear Community, >> >> >> >> The release candidate is ready. >> >> >> >> I have uploaded release candidate to >> >> https://dist.apache.org/repos/dist/dev/ignite/2.16.0-rc0 >> >> https://dist.apache.org/repos/dist/dev/ignite/packages_2.16.0-rc0 >> >> >> >> The following staging can be used for testing: >> >> https://repository.apache.org/content/repositories/orgapacheignite-1559 >> >> >> >> Tag name is 2.16.0-rc0: >> >> https://github.com/apache/ignite/releases/tag/2.16.0-rc0 >> >> >> >> 2.16.0 most important changes: >> >> * Cache dumps; >> >> * Calcite engine: added hints, became more stable, covered with >> >> diagnostic tools; >> >> * Fixes to support JDK 14-21; >> >> * Cluster Management API (IEP-81); >> >> * Java thin client: Service Awareness feature; >> >> etc. >> >> >> >> RELEASE NOTES: >> >> >> >> >> >> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.16 >> >> >> >> Complete list of resolved issues: >> >> >> >> >> >> https://issues.apache.org/jira/browse/IGNITE-19650?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.16%27))%20AND%20(component%20is%20EMPTY%20OR%20component%20not%20in%20(documentation))%20and%20status%20in%20(%27CLOSED%27%2C%20%27RESOLVED%27)%20AND%20resolution%20in(Fixed%2C%20Done%2C%20Implemented%2C%20Delivered)%20ORDER%20BY%20priority >> >> >> >> DEVNOTES >> >> >> >> >> >> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.16 >> >> >> >> The vote is formal, see voting guidelines >> >> https://www.apache.org/foundation/voting.html >> >> >> >> +1 - to accept Apache Ignite 2.16.0-rc0 >> >> 0 - don't care either way >> >> -1 - DO NOT accept Apache Ignite 2.16.0-rc0 (explain why) >> >> >> >> See notes on how to verify release here >> >> https://www.apache.org/info/verification.html >> >> and >> >> >> >> >> >> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification >> >> >> >> This vote will be open till Sun December 24, 2023, 15:00 UTC. >> >> >> >> >> >> https://www.timeanddate.com/countdown/vote?iso=20231224T15=0=VOTE+on+the+Apache+Ignite+Release+2.16.0+RC0=sanserif >> >> >> >> -- >> >> Best wishes, >> >> Amelchev Nikita >> >> >> >> >> >> > > >-- >Best wishes, >Amelchev Nikita
Re: [VOTE] Release Apache Ignite 2.16.0 RC0
Hi, probably it`s a kind of problem from my side, but cloning by tag and further steps: * change profile to java-8 * Reload all maven Projects * Try to run: ScriptTestSuite will raise guava dependency problem: /ignite/internal/processors/query/calcite/sql/IgniteSqlRollback.java:20:33 java: package com.google.common.collect does not exist --- * change profile to java-9 * reload * the same problem : modules/core/src/test/java/org/apache/ignite/testframework/GridTestUtils.java:79:33 java: package com.google.common.collect does not exist probably i need to clean all and try to rebuild from scratch .. for now it`s not clear for me. >+1 > >- Started the node from binaries >- Tested .NET examples >- Tested NuGet packages > >On Sun, Dec 17, 2023 at 2:48PM Nikita Amelchev < namelc...@apache.org > >wrote: > >> Dear Community, >> >> The release candidate is ready. >> >> I have uploaded release candidate to >> https://dist.apache.org/repos/dist/dev/ignite/2.16.0-rc0 >> https://dist.apache.org/repos/dist/dev/ignite/packages_2.16.0-rc0 >> >> The following staging can be used for testing: >> https://repository.apache.org/content/repositories/orgapacheignite-1559 >> >> Tag name is 2.16.0-rc0: >> https://github.com/apache/ignite/releases/tag/2.16.0-rc0 >> >> 2.16.0 most important changes: >> * Cache dumps; >> * Calcite engine: added hints, became more stable, covered with >> diagnostic tools; >> * Fixes to support JDK 14-21; >> * Cluster Management API (IEP-81); >> * Java thin client: Service Awareness feature; >> etc. >> >> RELEASE NOTES: >> >> >> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.16 >> >> Complete list of resolved issues: >> >> >> https://issues.apache.org/jira/browse/IGNITE-19650?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.16%27))%20AND%20(component%20is%20EMPTY%20OR%20component%20not%20in%20(documentation))%20and%20status%20in%20(%27CLOSED%27%2C%20%27RESOLVED%27)%20AND%20resolution%20in(Fixed%2C%20Done%2C%20Implemented%2C%20Delivered)%20ORDER%20BY%20priority >> >> DEVNOTES >> >> >> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.16 >> >> The vote is formal, see voting guidelines >> https://www.apache.org/foundation/voting.html >> >> +1 - to accept Apache Ignite 2.16.0-rc0 >> 0 - don't care either way >> -1 - DO NOT accept Apache Ignite 2.16.0-rc0 (explain why) >> >> See notes on how to verify release here >> https://www.apache.org/info/verification.html >> and >> >> >> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification >> >> This vote will be open till Sun December 24, 2023, 15:00 UTC. >> >> >> https://www.timeanddate.com/countdown/vote?iso=20231224T15=0=VOTE+on+the+Apache+Ignite+Release+2.16.0+RC0=sanserif >> >> -- >> Best wishes, >> Amelchev Nikita >>
Remove nightly builds for ignite-2.14
I found that nightly builds for 2.14 still active [1] can someone fix it ? [1] https://ci2.ignite.apache.org/buildConfiguration/IgniteTests24Java8_RunAllNightly/7245960?hideProblemsFromDependencies=false=false+Inspection=true=true
Re: Continuous integrations tests is broken on ci
Maxim, thank you for clarification ! >Hi! > >It looks like the reason is low timeout for running tests. Now it's set to >2h. See "Failure conditions" section. > >On ci2 those tests require 1h40m, pretty close for the timeout also. I >suppose we should either increase the timeout, or split the snapshots suite >into 2 parts. > >Actually we don't have a policy for this parameter, for example "Disk page >compressions 1" use 3h timeout, "Snapshots with indexes" - 1h. > >I propose to increase the timeout up to 3h as a quick solution to enable CI. > >On Fri, Jun 30, 2023 at 11:02AM Zhenya Stanilovsky >< arzamas...@mail.ru.invalid > wrote: > >> >> Igniters, seems continuous integrations tests are broken for ci.* [1] and >> are ok for ci2.* [2] seems it`s impossible to build and get visa for apache >> ignite 2 on ci.* environment. >> >> [1] >> >> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Snapshots?branch >> = >> =builds >> [2] >> >> https://ci2.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Snapshots?branch >> = >> =builds >> >> >>
Continuous integrations tests is broken on ci
Igniters, seems continuous integrations tests are broken for ci.* [1] and are ok for ci2.* [2] seems it`s impossible to build and get visa for apache ignite 2 on ci.* environment. [1] https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Snapshots?branch==builds [2] https://ci2.ignite.apache.org/buildConfiguration/IgniteTests24Java8_Snapshots?branch==builds
Re[2]: [DISCUSSION] Removal of ignitevisorcmd
+1 for remove. >+1 This module seems to be completely abandoned > >чт, 1 дек. 2022 г., 00:46 Ilya Kasnacheev < ilya.kasnach...@gmail.com >: > >> Hello! >> >> Let's go ahead and remove what we don't use. Most of that stuff is deep >> legacy, even if it contains some rare gems of functionality that nobody >> knows how to use anymore. >> >> Regards, >> -- >> Ilya Kasnacheev >> >> >> ср, 30 нояб. 2022 г. в 20:12, Nikolay Izhikov < nizhi...@apache.org >: >> >> > Hello, Igniters. >> > >> > There are legacy modules - visor-console, visor-plugins, >> visor-console-2.10 >> > Looks like that modules are not evolving by community and barely used by >> > the Ignite users. >> > >> > I propose to remove it completely. >> > There are IEP [1] for this >> > >> > It seems that removal of visor modules allow to us to remove another >> legacy >> > code from Ignite codebase: >> > >> > * support of daemon nodes. >> > * GridClient and related classes. >> > >> > What do you think? >> > >> > [1] >> > >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217392419 >> > >>
Re: [VOTE] Release pyignite 0.6.0.rc1
+1, thanks Ivan !
>Dear Igniters!
>
>Release candidate binaries for subj are uploaded and ready for vote
>You can find them here:
>https://dist.apache.org/repos/dist/dev/ignite/pyignite/0.6.0.rc1
>
>If you follow the link above, you will find source packages (*.zip)
>and binary packages (wheels) for windows (amd64), mac os x (amd64) and
>linux (x86_64)
>for pythons 37, 38, 39, 310 and 311. Also, there are sha512 and gpg
>signatures.
>Code signing keys can be found here --
>https://downloads.apache.org/ignite/KEYS
>Here you can find instructions how to verify packages
>https://www.apache.org/info/verification.html
>
>You can install binary package for specific version of python using pip
>For example do this on linux for python 3.8
>>> pip
>install
>pyignite-0.6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
>
>You can build and install package from source using this command:
>>> pip install pyignite-0.6.0.zip
>You can build wheel on your platform using this command:
>>> pip wheel --no-deps pyignite-0.6.0.zip
>
>For building C module, you should have python headers and C compiler
>installed.
>(i.e. for ubuntu sudo apt install build-essential python3-dev)
>In Mac OS X xcode-tools and python from homebrew are the best option.
>
>In order to check whether C module works, use following:
>>> from pyignite import _cutils
>>> print(_cutils.hashcode('test'))
>>> 3556498
>
>You can find documentation here:
>https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.6.0.rc1/
>
>You can find examples here (to check them, you should start ignite locally):
>https://apache-ignite-binary-protocol-client.readthedocs.io/en/0.6.0.rc0/examples.html
>Also, examples can be found in source archive in examples subfolder.
>docker-compose.yml is supplied in order to start ignite quickly. (Use
>`docker-compose up -d` to start 3 nodes cluster and `docker-compose
>down` to shut down it)
>
>Release notes:
>https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=995bda81780402116e89d76523949da88136f260
>
>Git release tag was created:
>https://gitbox.apache.org/repos/asf?p=ignite-python-thin-client.git;a=tag;h=ea3f180a0300abf25c992ed8c241defdc2b4bd4b
>
>The vote is formal, see voting guidelines
>https://www.apache.org/foundation/voting.html
>
>+1 - to accept pyignite-0.6.0.rc0
>0 - don't care either way
>-1 - DO NOT accept pyignite-0.6.0.rc0
>
>The vote finishes at 11/15/2022 15:00 UTC
Re: [VOTE] Release Apache Ignite 2.14.0 RC0
I think it`s important to mention that local caches are not supported since
this version [1].
[1] https://issues.apache.org/jira/browse/IGNITE-15759
>Dear Community,
>
>The release candidate is ready.
>
>I have uploaded release candidate to
>https://dist.apache.org/repos/dist/dev/ignite/2.14.0-rc0/
>https://dist.apache.org/repos/dist/dev/ignite/packages_2.14.0-rc0/
>
>The following staging can be used for testing:
>https://repository.apache.org/content/repositories/orgapacheignite-1552/
>
>Tag name is 2.14.0-rc0:
>https://gitbox.apache.org/repos/asf?p=ignite.git;a=tag;h=refs/tags/2.14.0-rc0
>
>2.14.0 most important changes:
>* Added new experimental, Apache Calcite based SQL engine
>* Added support of IGNITE_TO_STRING_INCLUDE_SENSITIVE option by Сonsistency
>check command
>* CPP Thin: Implemented asynchronous network events handling
>* CPP thin: Implemented continuous queries
>* Deprecated IgniteServices#service(String) and
>IgniteServices#services(String).
>* Implemented CDC for in-memory caches
>* Implemented NUMA aware allocator for Ignite durable memory
>* Implemented Read Repair strategies
>* Implemented array component type in binary object
>* Removed the legacy service grid implementation
>
>RELEASE NOTES:
>https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.14
>
>Complete list of resolved issues:
>https://issues.apache.org/jira/issues/?jql=(project%20%3D%20
>'Ignite'%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20('2.14'))%20AND%20(component%20is%20EMPTY%20OR%20component%20not%20in%20(documentation))%20and%20status%20in%20('CLOSED'%2C%20'RESOLVED')%20AND%20resolution%20in(Fixed%2C%20Done%2C%20Implemented%2C%20Delivered)%20ORDER%20BY%20priority
>
>DEVNOTES
>https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.14
>
>The vote is formal, see voting guidelines
>https://www.apache.org/foundation/voting.html
>
>+1 - to accept Apache Ignite 2.14.0-rc0
>0 - don't care either way
>-1 - DO NOT accept Apache Ignite Ignite 2.14.0-rc0 (explain why)
>
>See notes on how to verify release here
>https://www.apache.org/info/verification.html
>and
>https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification
>
>This vote will be open for at least 3 days till Mon Oct 3, 2022, 20:00 UTC.
>https://www.timeanddate.com/countdown/vote?iso=20221003T20=0=VOTE+on+the+Apache+Ignite+Release+2.14.0+RC0=sanserif
>
>--
>With best regards,
>Taras Ledkov
Re[2]: Apache calcite dependency update issue.
Hello, i plan to proceed with CALCITE-5253 soon. Great, lets move forward with new ver. >Hello! > >Apache Calcite 1.32 is already released, but [1] is still not resolved. >I propose not to wait for this ticket. "NATURAL JOIN" is a relatively rare >case, I think we can live with it for a while. Other improvements >look more valuable. >I've prepared a ticket to upgrade Apache Calcite dependency [2], if there >are no objections, I will merge it soon. > >[1]: https://issues.apache.org/jira/browse/CALCITE-5253 >[2]: https://issues.apache.org/jira/browse/IGNITE-17722 > >ср, 31 авг. 2022 г. в 08:10, Ivan Daschinsky < ivanda...@gmail.com >: > >> Oh, sorry, I see that it has been alredy reported. So let us wait for a bug >> fix. >> >> ср, 31 авг. 2022 г., 08:10 Ivan Daschinsky < ivanda...@gmail.com >: >> >> > I suppose that we are not in rush, because we have just cut off a release >> > branch for 2.14. Let us wait for a new release of Calcite. By the way, >> has >> > that bug been already reported? >> > >> > вт, 30 авг. 2022 г., 20:35 Zhenya Stanilovsky < arzamas...@mail.ru.invalid >> > >: >> > >> >> >> >> Igniters, i found that new release of apache calcite was released (1.31) >> >> [1]. This release contains great improvement [2] which makes possible to >> >> resolve [3], but also contains a bug [4] with natural join validation >> >> (possibly not one, but apache ignite sql test suite highlight only this >> >> one). I tried to override such a validation in IgniteSqlValidator but >> seems >> >> there are lot of custom private stuff, so i failed here. >> >> I don`t have my own opinion here, so we can: >> >> * update calcite version, mute test under calcite issue and wait new >> >> calcite release. >> >> * do nothing and just wait new calcite version. >> >> What do you think ? >> >> Thanks ! >> >> >> >> [1] https://calcite.apache.org/docs/history.html#v1-31-0 >> >> [2] https://issues.apache.org/jira/browse/CALCITE-4913 >> >> [3] https://issues.apache.org/jira/browse/IGNITE-16040 >> >> [4] https://issues.apache.org/jira/browse/CALCITE-5253 >> >> >> >> >> >> >> > >> > >>
Re: [ANNOUNCE] New PMC member: Ivan Daschinsky
Join the congratulations ! >Hi Igniters! > >The Project Management Committee (PMC) for Apache Ignite has invited >Ivan Daschinsky to become a member of the PMC and we are pleased to >announce that he has accepted. > >Ivan contributed the Ignite Python thin client. And he is still maintaining >it. >He is also actively supporting users. > >Please join me in congratulating Ivan on his new role. > >Best Regards, >Dmitriy Pavlov >on behalf of Apache Ignite PMC
Apache calcite dependency update issue.
Igniters, i found that new release of apache calcite was released (1.31) [1]. This release contains great improvement [2] which makes possible to resolve [3], but also contains a bug [4] with natural join validation (possibly not one, but apache ignite sql test suite highlight only this one). I tried to override such a validation in IgniteSqlValidator but seems there are lot of custom private stuff, so i failed here. I don`t have my own opinion here, so we can: * update calcite version, mute test under calcite issue and wait new calcite release. * do nothing and just wait new calcite version. What do you think ? Thanks ! [1] https://calcite.apache.org/docs/history.html#v1-31-0 [2] https://issues.apache.org/jira/browse/CALCITE-4913 [3] https://issues.apache.org/jira/browse/IGNITE-16040 [4] https://issues.apache.org/jira/browse/CALCITE-5253
Re: [VOTE] Release Apache Ignite 2.13.0 RC2
Nikita thanks for your effort ! Download sources, run examples. +1 from me. >Dear Community, > >The release candidate is ready. > >I have uploaded a release candidate to: >https://dist.apache.org/repos/dist/dev/ignite/2.13.0-rc2/ >https://dist.apache.org/repos/dist/dev/ignite/packages_2.13.0-rc2/ > >The following staging can be used for testing: >https://repository.apache.org/content/repositories/orgapacheignite-1542 > >Tag name is 2.13.0-rc1: >https://gitbox.apache.org/repos/asf?p=ignite.git;a=commit;h=refs/tags/2.13.0-rc2 > >RELEASE_NOTES: >https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.13 > >Complete list of resolved issues: >https://issues.apache.org/jira/issues/?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.13%27))%20and%20status%20in%20(%27CLOSED%27%2C%20%27RESOLVED%27)%20ORDER%20BY%20priority > >DEVNOTES: >https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.13 > >Additional checks have been performed (available for users included >into the release group on TeamCity). > >TC [Check RC: Licenses, compile, chksum] >https://ci2.ignite.apache.org/buildConfiguration/ignite2_Release_ApacheIgniteReleaseJava8_PrepareVote4CheckRcLicensesChecksum/6399934 > >TC [2] Compare w/ Previous Release >https://ci2.ignite.apache.org/buildConfiguration/ignite2_Release_ApacheIgniteReleaseJava8_IgniteRelease72CheckFileConsistency/6399932 > >The vote is formal, see voting guidelines >https://www.apache.org/foundation/voting.html > >+1 - to accept Apache Ignite 2.13.0-rc2 >0 - don't care either way >-1 - DO NOT accept Apache Ignite Ignite 2.13.0-rc2 (explain why) > >See notes on how to verify release here >https://www.apache.org/info/verification.html >and >https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification > >This vote will be open for at least 3 days till Fri Apr 23, 2022, >20:00 UTC. Please, write me down the thread if you need additional >time to check the >release. >https://www.timeanddate.com/countdown/vote?iso=20220423T20=0=VOTE+on+the+Apache+Ignite+Release+2.13.0+RC2=sanserif > >-- >Best wishes, >Amelchev Nikita
Re: [VOTE] Release Apache Ignite 2.13.0 RC1
Nikita, thanks ! I found that below link is unavailable: >TC [2] Compare w/ Previous Release >https://ci2.ignite.apache.org/buildConfiguration/ignite2_Release_ApacheIgniteReleaseJava8_IgniteRelease72CheckFileConsistency/6398741 >
Re[4]: Apache Ignite 2.13 RELEASE [Time, Scope, Manager]
Thanks Nikita, cherry-picked: IGNITE-16742 Extend calcite module documentation. >Igniters, > >I have cut the release branch: 'ignite-2.13'. Please, cherry-pick new >issues if needed. > >The 2.13 scope is freezed. Unresolved not-blocking issues will be >moved on the code freeze stage. The planning release date was updated >[1]: > >Scope Freeze: March 25, 2022 >Code Freeze: April 8, 2022 >Voting Date: April 15, 2022 >Release Date: April 22, 2022 > >[1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.13 > >чт, 10 мар. 2022 г. в 16:17, Nikita Amelchev < namelc...@apache.org >: >> >> Hello, Igniters. >> >> I have created the 2.13 release page. [1] >> >> The new SQL Calcite engine is expected to merge in the coming week. >> [2] I suggest cutting the 2.13 branch after the merge. I've updated >> the planned release dates on the release page. >> >> [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.13 >> [2] https://lists.apache.org/thread/3fkq3mn2npz326bp6gjtw3sw8xrlqyho >> >> чт, 10 февр. 2022 г. в 15:23, Nikita Amelchev < namelc...@apache.org >: >> > >> > Guys, >> > >> > +1 to await merge of the Calcite SQL engine before code-freeze phase >> > and release branch cutting. Other features and fixes can be awaited >> > too, it's discussable. >> > >> > But the 2.12 branch was cut on October 15, 2021. There are many fixes >> > and features that were merged into the master during this period. The >> > total time between branches cut is 5 months (if there is no delay >> > happens). Seems it is not so frequently. >> > >> > чт, 10 февр. 2022 г. в 15:17, Zhenya Stanilovsky < >> > arzamas...@mail.ru.invalid >: >> > > >> > > >> > > Maxim, i think that more frequent releases are useful. >> > > Ready to release branch means that it passed all known tests and also >> > > have an appropriate votes. >> > > More code changes creates more difficulties in final tests and sometimes >> > > migration. >> > > No need to switch between neighbor minor versions for user if all work >> > > properly well. >> > > I «vote» for more frequent releases. >> > > >> > > > >> > > >> >> > > >>>Hi, guys! >> > > >>> >> > > >>>Ignite 2.12 was released on 17th Jan. And here is a plan to release >> > > >>>2.13 on >> > > >>>28 Mar. It is only 2.5 months between those versions. IMHO, it's >> > > >>>better to >> > > >>>have more time between releases: >> > > >>>1. We had some bug reports after releasing 2.11, and it can be worth >> > > >>>waiting for users feedback about 2.12. Also meetup with description >> > > >>>of 2.12 >> > > >>>will be only next week (16 Feb). >> > > >>>2. In my understanding, users don't switch between versions of >> > > >>>databases >> > > >>>frequently. Actually it's hard work to upgrade such a dependency. So, >> > > >>>no >> > > >>>need for continuous delivery here. I think it's a common practice, for >> > > >>>example, MongoDB releases 1 time per year, Cassandra 2 times. >> > > >>>CockroachDB >> > > >>>releases minor versions every month, but major versions are still >> > > >>>released >> > > >>>2 times a year. But I'm not aware of all databases. >> > > >>> >> > > >>>I think we should move dates for at least 1 month. Also it depends on >> > > >>>the >> > > >>>Calcite engine readiness. WDYT? >> > > >>> >> > > >>>Anyway, from my side there are some tickets I want to include to the >> > > >>>next >> > > >>>release scope: >> > > >>>1. Partition reservation for cache queries (it will make IndexQuery >> > > >>>work on >> > > >>>unstable topology): IGNITE-16030 and IGNITE-16031 >> > > >>> >> > > >>>2. Also there is a discussion started by Andrey Mashenlov about known >> > > >>>index >> > > >>>corruption scenarios [1]. It looks like there are at least 2 issues to >> > > >>>resolve:
Re[2]: [DISCUSSION] Error handling in Ignite 3
Ivan, thanks for this effort, as for me - looks good.
>Hi everyone,
>
>I'd like to continue the discussion. I created IEP with the attempt to
>summarize
>all the information above, you can find it here [1]. What do you think?
>
>[1]
>https://cwiki.apache.org/confluence/display/IGNITE/IEP-84%3A+Error+handling
>
>ср, 21 апр. 2021 г. в 20:46, Alexei Scherbakov < alexey.scherbak...@gmail.com
>>:
>
>> Alexei,
>>
>> I think it's ok to do error conversion if it makes sense, but better to
>> preserve the root cause whenever possible.
>> Another way to solve the described scenario is to introduce something like
>> checked IgniteRetryAgainException, which forces the user to retry or ignore
>> it.
>> It's difficult to foresee exactly at this point what is the best solution
>> without knowing the exact scenario.
>>
>> Andrey,
>>
>> I've just realized your proposal to add IGN to each string code. This is ok
>> to me, for example IGN-TBL-0001
>>
>> ср, 21 апр. 2021 г. в 17:49, Alexey Goncharuk < alexey.goncha...@gmail.com
>> >:
>>
>> > Aleksei,
>> >
>> > > The method should always report root cause, in your example it will be
>> > > B-, no matter which module API is called
>> >
>> > I may be wrong, but I doubt this will be usable for an end-user. Let's
>> > imagine that the same root exception was raised in different contexts
>> > resulting in two outcomes. The first one is safe to retry (say, the root
>> > cause led to a transaction prepare failure), but the second outcome may
>> be
>> > a state in which no matter how many retries will be attempted, the
>> > operation will never succeed. Only the upper-level layer can tell the
>> > difference and return a proper message to the user, so I would say that
>> > some error conversion/wrapping will be required no matter what.
>> >
>> > --AG
>> >
>> > пт, 16 апр. 2021 г. в 16:31, Alexei Scherbakov <
>> > alexey.scherbak...@gmail.com
>> > >:
>> >
>> > > чт, 15 апр. 2021 г. в 18:21, Andrey Mashenkov <
>> > andrey.mashen...@gmail.com
>> > > >:
>> > >
>> > > > Hi Alexey,
>> > > > I like the idea.
>> > > >
>> > > > 1.
>> > > >
>> > > > > TBL-0001 is a *string representation* of the error. It is built
>> > from
>> > > 2
>> > > > > byte scope id (mapped to name TBL) and 2 byte number (0001). Both
>> > > > > internally packed in int. No any kind of parsing will be necessary
>> to
>> > > > read
>> > > > > scope/category.
>> > > >
>> > > > I think Alexey mean if it will be possible to make smth like that
>> > > >
>> > > > catch (IgniteException e) {
>> > > > if (e.getScope() == "TBL" && e.getCode() == 1234)
>> > > > continue; // E.g. retry my TX
>> > > > }
>> > > >
>> > > > It looks useful to me.
>> > > >
>> > >
>> > > I have in mind something like this:
>> > >
>> > > public class IgniteException extends RuntimeException {
>> > > private int errorCode;
>> > >
>> > > public IgniteException(ErrorScope scope, int code, String message,
>> > > Throwable cause) {
>> > > super(message, cause);
>> > > this.errorCode = make(scope, code);
>> > > }
>> > >
>> > > public boolean matches(ErrorScope scope, int code) {
>> > > return errorCode == make(scope, code);
>> > > }
>> > >
>> > > private int make(ErrorScope scope, int code) {
>> > > return ((scope.ordinal() << 16) | code);
>> > > }
>> > >
>> > > public ErrorScope scope() {
>> > > return ErrorScope.values()[errorCode >> 16];
>> > > }
>> > >
>> > > public int code() {
>> > > return 0x & errorCode;
>> > > }
>> > >
>> > > public static void main(String[] args) {
>> > > IgniteException e = new IgniteException(ErrorScope.RAFT, 1,
>> > "test",
>> > > null);
>> > >
>> > > System.out.println(e.matches(ErrorScope.RAFT, 2));
>> > > System.out.println(e.scope());
>> > > System.out.println(e.code());
>> > >
>> > > try {
>> > > throw e;
>> > > }
>> > > catch (IgniteException ee) {
>> > > System.out.println(ee.matches(ErrorScope.RAFT, 1));
>> > > }
>> > > }
>> > > }
>> > >
>> > >
>> > > >
>> > > > 2. How you see a cross-module exception throwing?
>> > > > Assume, user call -> module A, which recursively call -> module B,
>> > which
>> > > > fails.
>> > > > So, module A component calls a module B component and got an
>> Exception
>> > > with
>> > > > "B-1234" exception.
>> > > > Module A do not expect any exception here and doesn't take care of
>> > > "B-xxx"
>> > > > error codes, but only "A-.
>> > > > Should it rethrow exception with "A-unknown" (maybe "UNK-0001") code
>> > > > or reuse "B-" code with the own message, pointing original
>> > exception
>> > > as
>> > > > a cause for both cases?
>> > > >
>> > > > The first approach may looks confusing, while the second one produces
>> > too
>> > > > many "UNK-" codes.
>> > > > What code should get the user in that case?
>> > > >
>> > > >
>> > > >
>> > > The method should always report root cause, in your example it will be
>> > > B-, no matter which module API is called.
>> > >
>> > >
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Apr 15, 2021 at 5:36 PM Alexei Scherbakov <
>> > > >
Re[2]: Apache Ignite 2.13 RELEASE [Time, Scope, Manager]
Maxim, i think that more frequent releases are useful. Ready to release branch means that it passed all known tests and also have an appropriate votes. More code changes creates more difficulties in final tests and sometimes migration. No need to switch between neighbor minor versions for user if all work properly well. I «vote» for more frequent releases. > >> >>>Hi, guys! >>> >>>Ignite 2.12 was released on 17th Jan. And here is a plan to release 2.13 on >>>28 Mar. It is only 2.5 months between those versions. IMHO, it's better to >>>have more time between releases: >>>1. We had some bug reports after releasing 2.11, and it can be worth >>>waiting for users feedback about 2.12. Also meetup with description of 2.12 >>>will be only next week (16 Feb). >>>2. In my understanding, users don't switch between versions of databases >>>frequently. Actually it's hard work to upgrade such a dependency. So, no >>>need for continuous delivery here. I think it's a common practice, for >>>example, MongoDB releases 1 time per year, Cassandra 2 times. CockroachDB >>>releases minor versions every month, but major versions are still released >>>2 times a year. But I'm not aware of all databases. >>> >>>I think we should move dates for at least 1 month. Also it depends on the >>>Calcite engine readiness. WDYT? >>> >>>Anyway, from my side there are some tickets I want to include to the next >>>release scope: >>>1. Partition reservation for cache queries (it will make IndexQuery work on >>>unstable topology): IGNITE-16030 and IGNITE-16031 >>> >>>2. Also there is a discussion started by Andrey Mashenlov about known index >>>corruption scenarios [1]. It looks like there are at least 2 issues to >>>resolve: dropping of affinity index, handle of orphaned indexes. I think we >>>should fix those known issues before the next release. WDYT? >>> >>>[1] https://lists.apache.org/thread/m6dt0pn1qb01d8w6zm2fvo7lxgt0r068 >>> >>> >>>On Wed, Feb 9, 2022 at 3:13 PM Maxim Muzafarov < mmu...@apache.org > wrote: >>> Nikita, Thank you for starting this thread. +1 for these dates, but I think it's better to start the code freeze date when the Calcite engine will be actually merged to the master branch. On Tue, 8 Feb 2022 at 13:10, Nikita Amelchev < namelc...@apache.org > wrote: > > Dear Ignite Community! > > I suggest starting Apache Ignite 2.13 release activities. > > There is a plan to merge the new Calcite SQL engine. [1] I think that > 2.13 is a good candidate for it. > > Moreover, we've accumulated a hundred resolved [2] issues with new > features and bug fixes which are waiting for their release date. For > example, > > BinaryArray introduced, > Read Repair strategies implemented, > CPP Thin: asynchronous network events handling, > NUMA-aware allocator for data regions > etc. > > I want to propose myself to be the release manager of the planning release. > > I propose the following timeline: > > Scope Freeze: February 21, 2022 > Code Freeze: March 7, 2022 > Voting Date: March 21, 2022 > Release Date: March 28, 2022 > > WDYT? > > [1] https://issues.apache.org/jira/browse/IGNITE-15436 > [2] https://issues.apache.org/jira/issues/?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.13%27))%20AND%20(component%20is%20EMPTY%20OR%20component%20not%20in%20(documentation))%20and%20status%20in%20(%27CLOSED%27%2C%20%27RESOLVED%27)%20ORDER%20BY%20priority > > -- > Best wishes, > Amelchev Nikita >> >> >> >>
Re[2]: java 17 support
Petr, how can you explain the lifecycle of product ? It managed by community. I`m +1 for moving forward. > >> >>>Adding ability to compile Ignite 2 with JDK11+ will require so much >>>refactoring and, sometimes, rethinking of approaches, that it will become >>>different project in some ways. >>>And there is such different project — Ignite 3. >>> >>>I do no think that Ignite 2 will continue its lifecycle indefinitely and >>>sooner or later will be superseded by Ignite 3 which will support all latest >>>JDKs. >>> >>> >>>However — some code refactoring in order to run (not compile) Ignite 2 on >>>modern JDKs (until the end of life) seems to be worth the efforts. >>> On 2 Feb 2022, at 15:25, Nikolay Izhikov < nizhi...@apache.org > wrote: > will definitely not be moved to JDK more than 8th. I think is a matter to discuss :) Why we should stay on JDK8 forever? > 2 февр. 2022 г., в 15:23, Petr Ivanov < mr.wei...@gmail.com > написал(а): > > Hi, Davide! > > > Ignite 3 will support at least 11th JDK but that can be changed (for the > better) on initial release. > > Ignite 2 is subject to discuss, it currently runs on 11th and there > should be a patch that added JDK 17 runtime support, but codebase > (compile) will definitely not be moved to JDK more than 8th. > >> On 28 Jan 2022, at 18:28, Davide Imbriaco < davide.imbri...@gmail.com > >> wrote: >> >> hi, >> >> I've been using ignite in a project based on java 17 (which is the latest >> LTS java release). I see that ignite requires java 8 or 11... with java >> 17 >> it works fine so far (just a couple minor issues easily resolved), but I >> was wondering if there is a roadmap for official java 17 support, just to >> be sure. Maybe this is scheduled for Ignite 3? >> >> thank you, bye, >> D > >> >> >> >>
Re[2]: Adding a system view of recently completed compute tasks
Ok, thanks, now it`s clear, seems we need additional documentation here and also property renaming. >Judging by the code, this is the task session timeout.
Re[2]: Adding a system view of recently completed compute tasks
Ok, whats the purpose of END_TIME property in such a case? >Thanks Zhenya, but here we can see only the current working tasks, after they >are completed it is not possible to get the actual time of work, for example, >for statistics.
Re: Adding a system view of recently completed compute tasks
Hi, seems ignite already contain such view [1], plz explain the difference? thanks. [1] https://ignite.apache.org/docs/latest/monitoring-metrics/system-views#tasks > >> >>>Hello everyone! >>> >>>I want to add a new system view to get the last N (configurable by system >>>property, default 100) completed compute tasks, are there any objections? >> >> >> >>
Re: [PROPOSAL] Release Calcite-based SQL engine as an experimental feature
Alex, great ! If someone wants to touch codebase somehow plz use this branch [1] Test passed can be found here [2] [3] [1] https://github.com/apache/ignite/tree/sql-calcite/modules/calcite [2] https://github.com/apache/ignite/tree/sql-calcite/modules/calcite/src/test/java/org/apache/ignite/internal/processors/query/calcite [3] https://github.com/apache/ignite/tree/sql-calcite/modules/calcite/src/test/sql > >> >>>Hello, Igniters! >>> >>>As you may already know there is the new Ignite SQL engine based on Apache >>>Calcite currently under development. >>> >>>Reasons to move from H2-based engine and motivation for creating the new >>>one in details described in IEP-37 [1]. >>> >>>You can find all related to the new engine source code changes in the >>>"sql-calcite" branch [2]. >>> >>>Calcite-based SQL engine is not production-ready yet and has a lot of known >>>issues. In the future, the new engine should be fully independent of >>>"ignite-indexing" and H2, but now it relies on schema management and >>>indexes implemented in the "ignite-indexing" module and can't work without >>>the old engine. Despite all of the above mentioned, in the current state, >>>it has its own parsing, planning and execution flow and is almost as >>>functional as the H2-based SQL engine. >>> >>>Some users are already interested in the Calcite-based engine and asking >>>about the development status and release dates. Calcite-based SQL engine >>>will be the only SQL engine in Ignite 3.0. Perhaps even in 2.x we can get >>>rid of the H2-based engine at some time in the future. There is some syntax >>>difference between Calcite and H2 (Calcite is closer to SQL standards than >>>H2) and a totally new execution flow. After the release of this feature, >>>users can try their queries and determine if any adaptation for them is >>>required. With the new planning and execution flow, perhaps, some queries >>>will be executed more effectively, users can redirect such queries to the >>>new engine. >>> >>>I think we can provide an opportunity to users to try the new engine and >>>release it as an experimental feature with the next Apache Ignite version >>>(2.13). >>> >>>What do you think? >> >> >> >>
Re[2]: NUMA aware allocator, PR review request
+1 with Ivan, let`s store it in core product just because it looks like inalienable functionality and release cycle of extensions a little bit different. >Anton, I disagree. > >1. This should be released with main distro. >2. This should not be abandoned. >3. There is not any release process in ignite-extensions. >4. Everything is working now and working good. > > >So lets do not do this :) > >пн, 6 дек. 2021 г. в 14:49, Anton Vinogradov < a...@apache.org >: > >> Let's move all GCC-related parts to ignite-extensions, release, and use >> them as a maven dependency. >> >> >> On Fri, Dec 3, 2021 at 1:08 PM Ivan Daschinsky < ivanda...@gmail.com > >> wrote: >> >> > Ok, TC suite is ready [1]. >> > If there is no objections, I will merge it soon. >> > >> > Possible concerns -- now it is required to install build-essentials and >> > libnuma-dev in order to build ignite on 64 bit linux. >> > I suppose that this is not a big deal, but maybe someone will contradict? >> > >> > >> > [1] -- >> > >> > >> >> https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_NumaAllocator/?mode=builds >> > >> > чт, 2 дек. 2021 г. в 12:03, Ivan Daschinsky < ivanda...@gmail.com >: >> > >> > > >> Our runs show about 7-10 speedup, >> > > Sorry, typo 7-10% speedup >> > > >> > > чт, 2 дек. 2021 г. в 12:01, Ivan Daschinsky < ivanda...@gmail.com >: >> > > >> > >> Andrey, thanks! >> > >> >> > >> This allocator can be tested on every NUMA system. >> > >> Our runs show about 7-10 speedup, if we use allocattor with >> interleaved >> > >> strategy + -XX:+UseNUMA. >> > >> But unfortunately our yardstick benches doesn't use offheap a lot, >> > >> usually above one Gb. >> > >> We trying to do more benches with real data and share them, possibly >> in >> > >> meetup. >> > >> >> > >> AFAIK, GG lab servers are two-sockets machines, aren't they? So it is >> > >> worth to run benches with a lot data on them, using >> > >> allocator with interleaved strategy (you can skip specifying numa >> nodes, >> > >> by default it will use all available) and use -XX:+UseNUMA jvm >> > >> flag. >> > >> >> > >> >> > >> >> > >> чт, 2 дек. 2021 г. в 11:48, Andrey Mashenkov < >> > andrey.mashen...@gmail.com >> > >> >: >> > >> >> > >>> Ivan, >> > >>> >> > >>> Great job. PR looks good. >> > >>> >> > >>> This allocator in interleaved mode and passing `-XX:+UseNUMA` flag to >> > jvm >> > >>> > show promising results on yardstick benches. Technically, G1 is >> not a >> > >>> numa >> > >>> > aware collector for java versions less than 14, but allocation of >> > heap >> > >>> in >> > >>> > interleaved mode shows good results even on java 11. >> > >>> >> > >>> Can you share benchmark results? >> > >>> I'm not sure I'll have an Optane on my notebook in a reasonable time >> ;) >> > >>> >> > >>> >> > >>> On Thu, Dec 2, 2021 at 10:41 AM Ivan Daschinsky < ivanda...@gmail.com >> > >> > >>> wrote: >> > >>> >> > >>> > Semyon D. and Maks T. -- thanks a lot for review. >> > >>> > >> > >>> > BTW, Igniters, I will appreciate all opinions and feedback. >> > >>> > >> > >>> > пн, 29 нояб. 2021 г. в 10:13, Ivan Daschinsky < >> ivanda...@apache.org >> > >: >> > >>> > >> > >>> > > Hi, igniters! >> > >>> > > >> > >>> > > There is not a big secret that nowadays NUMA is quite common in >> > >>> > > multiprocessor systems. >> > >>> > > And this memory architecture should be treated in specific ways. >> > >>> > > >> > >>> > > Support for NUMA is present in many commercial and open-source >> > >>> products. >> > >>> > > >> > >>> > > I've implemented a NUMA aware allocator for Apache Ignite [1] >> > >>> > > It is a JNI wrapper around `libnuma` and supports different >> > >>> allocation >> > >>> > > options. >> > >>> > > I.e. interleaved, local, interleved_mask and so on. For more >> > >>> information, >> > >>> > > see >> > >>> > > [2], [3]. >> > >>> > > This allocator in interleaved mode and passing `-XX:+UseNUMA` >> flag >> > >>> to jvm >> > >>> > > show promising results on yardstick benches. Technically, G1 is >> > not a >> > >>> > numa >> > >>> > > aware collector for java versions less than 14, but allocation of >> > >>> heap in >> > >>> > > interleaved mode shows good results even on java 11. >> > >>> > > >> > >>> > > Currently, all needed libraries and tools for building this >> module >> > >>> are >> > >>> > > available on TC agents >> > >>> > > setup of specific test suite is in progress [4] >> > >>> > > >> > >>> > > So I am asking for a review of my patch. >> > >>> > > >> > >>> > > [1] -- https://issues.apache.org/jira/browse/IGNITE-15922 >> > >>> > > [2] -- https://man7.org/linux/man-pages/man3/numa.3.html >> > >>> > > [3] -- https://man7.org/linux/man-pages/man2/mbind.2.html >> > >>> > > [4] -- https://issues.apache.org/jira/browse/IGNITE-15994 >> > >>> > > >> > >>> > >> > >>> > >> > >>> > -- >> > >>> > Sincerely yours, Ivan Daschinskiy >> > >>> > >> > >>> >> > >>> >> > >>> -- >> > >>> Best regards, >> > >>> Andrey V. Mashenkov >> > >>> >> > >> >> > >> >> > >> --
Re: [ANNOUNCE] Welcome Semyon Danilov as a new committer
Wellcome Semyon ! Andrey what`s Ivan are you talking at the end of the message or this is some kind of phraseologism that all russians are ivans ?:) >Igniters, > >The Apache Ignite Project Management Committee (PMC) has invited >Semyon Danilov to become a new committer and are happy to announce >that he >has accepted. > >Semyon contributes actively to the project's code base (AI 2 & 3) and >helps a lot with the development environment set up (you know, all >this Maven and plugins related issues). > >Being a committer enables easier contribution to the project since there is >no need to go via the patch submission process. This should enable better >productivity. > >Please join me in welcoming Ivan, and congratulating him on the new role in >the Apache Ignite Community. > >Best Regards, >Andrey Gura
Re: [ANNOUNCE] Welcome Maxim Timonin as a new committer
Big deal, congrats Maxim ! >Igniters, > >The Project Management Committee (PMC) for Apache Ignite has invited Maxim >Timonin to become a committer and we are pleased to announce that he has >accepted. > >Maxim makes valuable contributions to the Apache Ignite code, helps >actively to applications developers on the user list, and made a good start >in Project awareness contributing by presenting at Ignite Summit: Cloud >Edition. > >Being a committer enables easier contribution to the project since there is >no need to go via the patch submission process. This should enable better >productivity. > >Please join me in welcoming Maxim, and congratulating him on the new role >in the Apache Ignite Community! > >Kseniya Romanova >on behalf of Apache Ignite PMC
Re: [2.11.0]: 'B+Tree is corrupted' exception in GridCacheTtlManager.expire() and PartitionsEvictManager$PartitionEvictionTask.run() on node start
Sergey, not so easy to recognize the problem, also with logs, plz fill the
ticket and append link to this message or duplicate all logs there.
thanks !
>From: "Sergey Korotkov" < serge.korot...@gmail.com >
>To: u...@ignite.apache.org
>Cc:
>Subject: [2.11.0]: 'B+Tree is corrupted' exception in
>GridCacheTtlManager.expire() and
>PartitionsEvictManager$PartitionEvictionTask.run() on node start
>Date: Thu, 18 Nov 2021 14:24:59 +0300
>
>Hello,
>
>We have troubles with the CorruptedTreeException: B+Tree is corrupted
>during the node start after cluster restart. Looks like the caches with
>the Expiry Policy configured are source of the problems.
>
>I have attached the log from the problem node. The exact steps with
>timestamps are as folows. Before the deactivation cluster works fine
>about 5 days
>
>2021-11-08 10:54:44 cluster deactivate request
>
>2021-11-08 10:59:33 cluster deactivated
>
>2021-11-08 11:02:30 stop all nodes
>
>2021-11-08 11:02:39 start all nodes
>
>2021-11-08 11:03:14 auto-activation start
>
>2021-11-08 11:03:16 cluster activated
>
>2021-11-08 11:03:21 'B+Tree is corrupted' exception in
>GridCacheTtlManager.expire() on one of the nodes (see the
>10.12.86.29-ignite-2021-11-08.0.log):
>
>[2021-11-08 11:03:21,820][ERROR][ttl-cleanup-worker-#215][ROOT]{}
>Critical system error detected. Will be handled accordingly to
>configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false,
>timeout=0, super=AbstractFailureHandler
>[ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED,
>SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
>[type=CRITICAL_ERROR, err=class
>o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree
>is corrupted [pages(groupId, pageId)=[], msg=Runtime failure on bounds:
>[lower=null, upper=PendingRow []
>org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
>B+Tree is corrupted [pages(groupId, pageId)=[], msg=Runtime failure on
>bounds: [lower=null, upper=PendingRow []]]
> at
>org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.corruptedTreeException(BPlusTree.java:6139)
> at
>org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1133)
> at
>org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1100)
> at
>org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1095)
> at
>org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpiredInternal(GridCacheOffheapManager.java:3076)
> at
>org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.purgeExpired(GridCacheOffheapManager.java:3023)
> at
>org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager.expire(GridCacheOffheapManager.java:1255)
> at
>org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:246)
> at
>org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.lambda$body$0(GridCacheSharedTtlCleanupManager.java:193)
> at
>java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1769)
> at
>org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:192)
> at
>org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
>Caused by:
>org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTreeRuntimeException:
>org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTreeRuntimeException:
>java.lang.IllegalStateException: Item not found: 3
> at
>org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1079)
> at
>org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1118)
> ... 11 common frames omitted
>Caused by:
>org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTreeRuntimeException:
>java.lang.IllegalStateException: Item not found: 3
> at
>org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:345)
> at
>org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:165)
> at
>org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:136)
> at
>org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:123)
> at
>org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:73)
> at
Re: [DISCUSSION] Remove VS project from C++
completely support ! >Igniters! > >Currently we have CMake build system, that works on Windows, Linux and >MacOs flawlessly > >1. CMake is supported natively in VS 2019 >2. CMake can generate VS projects for about 20 years flawlessly. >3. Sometimes even maintainers forget to add test sources to VS projects [1] >4. Currently on TC we build Ignite C++ on windows and linux flawlessly >using CMake >5. VS projects are not backward compatible. We have to add manually (or by >sed or patch) some dependencies in order to build current VS projects on >newer versions of VS. > >So I suggest simpy to remove VS projects because of reasons I've written >above. > >WDYT? > > > >[1] -- https://issues.apache.org/jira/browse/IGNITE-15511
Re[2]: Deprecating LOCAL cache
Ok, we can use node filters in such a case, i understand ) >I just dream up ) If some one have cached web session layer over >ignite with sticky cookie [1] and cross cache transaction logic through >local and global caches how this schema will transform without local ? > >[1] >https://www.imperva.com/learn/availability/sticky-session-persistence-and-cookies/ > > >> I am not about creation per se, but creation from a thin client side. >> >> This feature simply doesn't work as expected, broken and impossible to >> fix. >> It cannot broke any code, because it was already broken and it is >> impossible to use in production. >> But it still can embarrass newcomers and brings a lot of frustration. >> >> It is much more safe to ban a creation of LOCAL caches from thin clients. >> >> But it can survive for a while for ignite cluster nodes, client and >> server. >> >> вт, 14 сент. 2021 г. в 14:06, Maxim Muzafarov < mmu...@apache.org >: >> >>> Ivan, >>> >>> I don't think we should rush with this. Banning the creation of LOCAL >>> caches without a warning through the code sounds not good. Will it be >>> better to do everything in three steps (releases)? 2.12 deprecate, >>> 2.13 forbid new cache creation, 2.14 remove. >>> >>> On Tue, 14 Sept 2021 at 12:09, Ivan Daschinsky < ivanda...@gmail.com > >>> wrote: >>> > >>> > Few thoughts about LOCAL caches on thin client: >>> > >>> > 1. If partition awareness is disabled: >>> > a. Inconsistent behaviour if node to which client is connected goes >>> down. >>> > 2. If partition awareness is enabled: >>> > a. For Java and .NET -- same as 1a >>> > b. For C++ and python -- use random routing for caches that are not >>> > PARTITIONED, so inconsistent behaviour from the beginning. >>> > >>> > So I suppose we should ban creation of LOCAL caches from thin client >>> in >>> > 2.12 (fail attempt to create such caches in ClientRequestHandler >>> > >>> > вт, 14 сент. 2021 г. в 11:31, Ivan Daschinsky < ivanda...@gmail.com >: >>> > >>> > > >> Unsupported operation exception. >>> > > Binary protocol doesn't have a concept of exception, only error >>> status >>> and >>> > > message, but it is just a remark >>> > > I suppose that response with error status and message is ok, but >>> may be >>> > > others have different opinion? >>> > > >>> > > >> Removal should happen at 2.13. >>> > > A few thin clients are released separately. I suppose that it is >>> better to >>> > > remove this feature from pyignite a little bit earlier. >>> > > >>> > > вт, 14 сент. 2021 г. в 11:21, Anton Vinogradov < a...@apache.org >: >>> > > >>> > >> > 1. What is expected behaviour if an old thin client requests >>> creation of >>> > >> > LOCAL cache on the newest ignite cluster? >>> > >> Unsupported operation exception. >>> > >> >>> > >> > 2. Should we completely remove LOCAL caches support in thin >>> clients >>> > >> (i.e. >>> > >> pyignite) before 2.13 release? >>> > >> Removal should happen at 2.13. >>> > >> >>> > >> On Tue, Sep 14, 2021 at 10:30 AM Ivan Daschinsky < >>> ivanda...@gmail.com >>> > >>> > >> wrote: >>> > >> >>> > >> > >> 2. 2.13 - complete removal LOCAL caches from codebase. >>> > >> > Let's discuss this step with more details. >>> > >> > 1. What is expected behaviour if an old thin client requests >>> creation of >>> > >> > LOCAL cache on the newest ignite cluster? >>> > >> > 2. Should we completely remove LOCAL caches support in thin >>> clients >>> > >> (i.e. >>> > >> > pyignite) before 2.13 release? >>> > >> > >>> > >> > вт, 14 сент. 2021 г. в 10:11, Nikolay Izhikov < >>> nizhi...@apache.org >>> >: >>> > >> > >>> > >> > > I proposed the following plan: >>> > >> > > >>> > >> > > 1. 2.12 - deprecation of LOCAL caches. >>> > >> > > 2. 2.13 - complete removal LOCAL caches from codebase. >>> > >> > > >>> > >> > > > 13 сент. 2021 г., в 13:30, Ivan Daschinsky < >>> ivanda...@gmail.com >>> > >>> > >> > > написал(а): >>> > >> > > > >>> > >> > > > I personally support deprecation, but we should at least >>> have a >>> > >> plan. >>> > >> > > > I suppose that putting annotations and removing documentation >>> are >>> > >> not >>> > >> > > > enough. >>> > >> > > > >>> > >> > > > >>> > >> > > > пн, 13 сент. 2021 г. в 13:22, Maxim Muzafarov < >>> mmu...@apache.org >: >>> > >> > > > >>> > >> > > >> Ivan, >>> > >> > > >> >>> > >> > > >> I don't think we can remove LOCAL caches at the nearest >>> time, >>> so >>> > >> there >>> > >> > > >> is no plan for that. I can only imagine a single release >>> that >>> will >>> > >> > > >> contain all the breaking changes we want to apply in 2.x >>> version. >>> > >> > > >> >>> > >> > > >> My point here is only about deprecation: >>> > >> > > >> - there are a lot of motivation points to remove written in >>> this >>> > >> > thread; >>> > >> > > >> - I always hear from the support team that they do not >>> recommend >>> > >> using >>> > >> > > >> local caches; >>> > >> > > >> - I haven't seen any bugs fixed for a long time for local >>> caches >>> > >> > > >>
Re[2]: Deprecating LOCAL cache
I just dream up ) If some one have cached web session layer over ignite with sticky cookie [1] and cross cache transaction logic through local and global caches how this schema will transform without local ? [1] https://www.imperva.com/learn/availability/sticky-session-persistence-and-cookies/ >I am not about creation per se, but creation from a thin client side. > >This feature simply doesn't work as expected, broken and impossible to fix. >It cannot broke any code, because it was already broken and it is >impossible to use in production. >But it still can embarrass newcomers and brings a lot of frustration. > >It is much more safe to ban a creation of LOCAL caches from thin clients. > >But it can survive for a while for ignite cluster nodes, client and server. > >вт, 14 сент. 2021 г. в 14:06, Maxim Muzafarov < mmu...@apache.org >: > >> Ivan, >> >> I don't think we should rush with this. Banning the creation of LOCAL >> caches without a warning through the code sounds not good. Will it be >> better to do everything in three steps (releases)? 2.12 deprecate, >> 2.13 forbid new cache creation, 2.14 remove. >> >> On Tue, 14 Sept 2021 at 12:09, Ivan Daschinsky < ivanda...@gmail.com > >> wrote: >> > >> > Few thoughts about LOCAL caches on thin client: >> > >> > 1. If partition awareness is disabled: >> > a. Inconsistent behaviour if node to which client is connected goes down. >> > 2. If partition awareness is enabled: >> > a. For Java and .NET -- same as 1a >> > b. For C++ and python -- use random routing for caches that are not >> > PARTITIONED, so inconsistent behaviour from the beginning. >> > >> > So I suppose we should ban creation of LOCAL caches from thin client in >> > 2.12 (fail attempt to create such caches in ClientRequestHandler >> > >> > вт, 14 сент. 2021 г. в 11:31, Ivan Daschinsky < ivanda...@gmail.com >: >> > >> > > >> Unsupported operation exception. >> > > Binary protocol doesn't have a concept of exception, only error status >> and >> > > message, but it is just a remark >> > > I suppose that response with error status and message is ok, but may be >> > > others have different opinion? >> > > >> > > >> Removal should happen at 2.13. >> > > A few thin clients are released separately. I suppose that it is >> better to >> > > remove this feature from pyignite a little bit earlier. >> > > >> > > вт, 14 сент. 2021 г. в 11:21, Anton Vinogradov < a...@apache.org >: >> > > >> > >> > 1. What is expected behaviour if an old thin client requests >> creation of >> > >> > LOCAL cache on the newest ignite cluster? >> > >> Unsupported operation exception. >> > >> >> > >> > 2. Should we completely remove LOCAL caches support in thin clients >> > >> (i.e. >> > >> pyignite) before 2.13 release? >> > >> Removal should happen at 2.13. >> > >> >> > >> On Tue, Sep 14, 2021 at 10:30 AM Ivan Daschinsky < ivanda...@gmail.com >> > >> > >> wrote: >> > >> >> > >> > >> 2. 2.13 - complete removal LOCAL caches from codebase. >> > >> > Let's discuss this step with more details. >> > >> > 1. What is expected behaviour if an old thin client requests >> creation of >> > >> > LOCAL cache on the newest ignite cluster? >> > >> > 2. Should we completely remove LOCAL caches support in thin clients >> > >> (i.e. >> > >> > pyignite) before 2.13 release? >> > >> > >> > >> > вт, 14 сент. 2021 г. в 10:11, Nikolay Izhikov < nizhi...@apache.org >> >: >> > >> > >> > >> > > I proposed the following plan: >> > >> > > >> > >> > > 1. 2.12 - deprecation of LOCAL caches. >> > >> > > 2. 2.13 - complete removal LOCAL caches from codebase. >> > >> > > >> > >> > > > 13 сент. 2021 г., в 13:30, Ivan Daschinsky < ivanda...@gmail.com >> > >> > >> > > написал(а): >> > >> > > > >> > >> > > > I personally support deprecation, but we should at least have a >> > >> plan. >> > >> > > > I suppose that putting annotations and removing documentation >> are >> > >> not >> > >> > > > enough. >> > >> > > > >> > >> > > > >> > >> > > > пн, 13 сент. 2021 г. в 13:22, Maxim Muzafarov < >> mmu...@apache.org >: >> > >> > > > >> > >> > > >> Ivan, >> > >> > > >> >> > >> > > >> I don't think we can remove LOCAL caches at the nearest time, >> so >> > >> there >> > >> > > >> is no plan for that. I can only imagine a single release that >> will >> > >> > > >> contain all the breaking changes we want to apply in 2.x >> version. >> > >> > > >> >> > >> > > >> My point here is only about deprecation: >> > >> > > >> - there are a lot of motivation points to remove written in >> this >> > >> > thread; >> > >> > > >> - I always hear from the support team that they do not >> recommend >> > >> using >> > >> > > >> local caches; >> > >> > > >> - I haven't seen any bugs fixed for a long time for local >> caches >> > >> > > >> (suppose that we are not maintaining them); >> > >> > > >> >> > >> > > >> I just want to make sure that all these points are reflected >> in the >> > >> > > >> code base, so propose to mark them as deprecated. >> > >> > > >> >> > >> > > >> On Mon, 13
Re: [VOTE] Release Apache Ignite 2.11.0 RC2
Thanks Maxim ! I tries to compare this ver with 2.10 (some performance tests with persistence and transactional\atomic payload) and seems all ok there. +1 from me. > > >Dear Community, > > >The release candidate for the 2.11 version is ready. > > >I have uploaded a release candidate to: >https://dist.apache.org/repos/dist/dev/ignite/2.11.0-rc2/ >https://dist.apache.org/repos/dist/dev/ignite/packages_2.11.0-rc2/ > >The following staging can be used for testing: >https://repository.apache.org/content/repositories/orgapacheignite-1526/ >https://www.myget.org/feed/apache-ignite-staging/package/nuget/Apache.Ignite > >Tag name is 2.11.0-rc2: >https://gitbox.apache.org/repos/asf?p=ignite.git;a=tag;h=refs/tags/2.11.0-rc2 > >RELEASE_NOTES: >https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.11 > >Complete list of resolved issues: >https://issues.apache.org/jira/issues/?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.11%27 > )) > >DEVNOTES: >https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.11 > > >Additional checks have been performed (available for users included into >the release group on TeamCity). > >TC [Check RC: Licenses, compile, chksum] >https://ci.ignite.apache.org/viewLog.html?buildId=6176219=ApacheIgniteReleaseJava8_PrepareVote4CheckRcLicensesChecksum=buildResultsDiv > > >The vote is formal, see voting guidelines >https://www.apache.org/foundation/voting.html > >+1 - to accept Apache Ignite 2.11.0-rc2 >0 - don't care either way >-1 - DO NOT accept Apache Ignite Ignite 2.11.0-rc2 (explain why) > >See notes on how to verify release here >https://www.apache.org/info/verification.html >and >https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification > > >This vote will be open for 4 days until Thu Sep 16, 12:00 2021 (Moscow >time). >Please, write me down the thread if you need additional time to check >the release. >https://www.timeanddate.com/countdown/vote?iso=20210916T12=166=%5BVOTE%5D+Release+Apache+Ignite+2.10.0+RC2=serif=1 >
Re: Ban Java Streams usage in Ignite 3 code
Pavel, thanks ! And what about stream usage not in a hot path ? I.e. create\drop table operation, may be such a cases will leave for committer\contributor consideration ? >Igniters, > >Java streams are known to be slower and cause more GC pressure than an >equivalent loop. >Below is a simple filter/map/reduce scenario (code [1]): > > * Benchmark Mode Cnt >Score Error Units > > * StreamVsLoopBenchmark.loopSum thrpt 3 > 7987.016 ± 293.013 ops/ms > * StreamVsLoopBenchmark.loopSum:·gc.alloc.rate thrpt 3 > ≈ 10⁻⁴ MB/sec > * StreamVsLoopBenchmark.loopSum:·gc.count thrpt 3 > ≈ 0 counts > > * StreamVsLoopBenchmark.streamSum thrpt 3 > 1060.244 ± 36.485 ops/ms > * StreamVsLoopBenchmark.streamSum:·gc.alloc.rate thrpt 3 > 315.819 ± 10.844 MB/sec > * StreamVsLoopBenchmark.streamSum:·gc.count thrpt 3 > 55.000 counts > >Loop is several times faster and does not allocate at all. > >1. Performance is one of the most important features of our product. >2. Most of our APIs will be on the hot path. > >One can argue about performance differences in real-world scenarios, >but increasing GC pressure just to make the code a little bit nicer is >unacceptable. > >I propose to ban streams usage in the codebase (except for the tests). > >Thoughts, objections? > >[1] https://gist.github.com/ptupitsyn/5934bbbf8f92ac4937e534af9386da97
Re: [VOTE] Allow or prohibit usages of the Guava library methods
Aleksandr, thanks for this activity. -1 from my side, all my decisions are in linked discussion. >Dear Igniters, > >In this thread >< >https://lists.apache.org/thread.html/r4120a03a2bf32098e54e21ae02e509b0d68f413bc7cc1f8f6d85c93d%40%3Cdev.ignite.apache.org%3E > > >we've been discussing the problems and opportunities of using Guava >< https://github.com/google/guava > in Ignite 3. We have agreed that it >should be added as a shaded dependency, but we haven't decided whether to >allow using Guava methods in the Ignite codebase or not. Therefore I would >like to propose a vote: > >[+1 Allow]: allow using Guava methods, if justified. >[-1 Prohibit]: prohibit using all Guava methods. > >The voting will commence on Monday, September 13th at 9:00 UTC. Also feel >free to express your opinion in the original discussion thread. > >-- >With regards, >Aleksandr Polovtcev
Re[4]: Google Guava in Ignite 3
Alex, what about : «compatibility when the transitive Guava dependency will be updated to a new version.» ? I don`t know is it an issue for us, but we need to answer it before move forward. >Andrey, >I think that the benefits of using Guava methods instead of copy-pasting >them are quite obvious: you don't have to copy-paste code and support it. I >also find this situation quite strange: we have a dependency and copy-paste >code from it instead of using it directly. > > >On Thu, Sep 2, 2021 at 1:58 PM Andrey Mashenkov < andrey.mashen...@gmail.com > >wrote: > >> Alex, >> >> As I understand we agree to shade Guava transitive dependency and >> you said a 'maven-shade-plugin' can drop unused Guava methods to reduce the >> footprint of Ignite jar. >> >> At this point, there is no difference between copy-pasting Guava method to >> Ignite and use Guava one. >> The resulted jar will have one copy of such method, but in the second case, >> we have to care about compatibility when the transitive Guava dependency >> will be updated to a new version. >> >> So, I see no benefits from using Guava directly. >> >> >> On Thu, Sep 2, 2021 at 11:56 AM Alexander Polovtcev < >> alexpolovt...@gmail.com > >> wrote: >> >> > Hi, Andrey! >> > I mostly agree with your proposal, but, since we already have some >> > copy-paste in our code, can we at least use Guava to remove it? So I >> would >> > propose to allow at least *some* methods that we consider useful, while >> > disallowing everything else. I understand that it may be difficult to >> > formalize, so maybe we can create some kind of a whitelist of Guava >> > methods? What do you think? >> > >> > On Mon, Aug 30, 2021 at 2:54 PM Andrey Gura < ag...@apache.org > wrote: >> > >> > > Follow up >> > > >> > > On Mon, Aug 23, 2021 at 1:22 PM Andrey Gura < ag...@apache.org > wrote: >> > > > >> > > > Igniters, >> > > > >> > > > What I actually didn't understand from this thread: Is Guava allowed >> > > > in production code of Ignite 3 modules (not dependencies like >> > > > Calcite)? >> > > > >> > > > While we agree with using shading I don't see any arguments about >> > > > using Guava library in our code base except for the fact that we have >> > > > some copy-paste of Guava code in the project. >> > > > >> > > > Guava is rich enough library and it has both advantages and >> > > > disadvantages. Ignite code base always strived to be not overloaded >> > > > and minimalistic. For example we usually use immutability very rare, >> > > > we try to avoid complex and overloaded APIs on hot path, we try >> reduce >> > > > GC pressure, etc. In turn, Guava is some sense forces using of >> > > > immutable collections, has a lot of utility methods which produces >> > > > temp object (yes, I don't trust to escap analysys), etc. Rich set of >> > > > collections and utility functions will also lead to different >> > > > programming styles during Ignite development. I can't predict it, but >> > > > it is usual enough when some developers force immutablity while >> others >> > > > remove static type declaration and replace it by var :) >> > > > >> > > > This is a common pratice to reduce language possibilies subset and >> > > > limiting using of some libraries in order to provide more or less >> > > > unified approach to coding. So I propose to disallow using Guava in >> > > > our code base. >> > > > >> > > > WDYT? >> > > > >> > > > >> > > > On Fri, Aug 20, 2021 at 7:37 PM Alexander Polovtcev >> > > > < alexpolovt...@gmail.com > wrote: >> > > > > >> > > > > Guys, >> > > > > >> > > > > Thanks again for your responses. I've created a ticket about using >> > the >> > > > > Shade plugin in order to understand if it is possible to shade and >> > > minimize >> > > > > Guava, I will start working on it shortly: >> > > > > https://issues.apache.org/jira/browse/IGNITE-15354 >> > > > > >> > > > > On Tue, Aug 10, 2021 at 1:51 PM Courtney Robinson < >> > > courtney.robin...@hypi.io >
Re[2]: Google Guava in Ignite 3
Alexander, first of all looks like Ivan Daschinsky approach about thin client use only and shadow plugin are cover all Andrey Mashenkov listing problems. But there is no restrictions from running ignite server nodes from some other code with it`s own guava version seems we obtain fast path to jar hell here? >Zhenya, > >My intentions are the following: > >1. Remove some copy-pasted code (like the "bytecode" module or some utility >methods). Please see my original message for the links to the code. >2. Explicitly pin the Guava version to avoid conflicts in the runtime. > >About allowing to use Guava in the codebase, my thoughts are the following: > >1. We *already* use some code from Guava either directly (like in the >"calcite" module) or by copy-pasting it into a utility class. >2. I understand that some Guava methods are obsolete as of Java 11, but >some of them still don't have any standard library counterparts, in which >case I think using Guava is justified (which is supported by point 1). > >Can you please explain why you would disapprove of my proposal? > >On Thu, Aug 5, 2021 at 7:56 PM Zhenya Stanilovsky >< arzamas...@mail.ru.invalid > wrote: > >> >> alexpolovtcev please clarify what do you mean under : «possibility of >> using Guava in Ignite 3», using how necessary dependency of calcite or >> using like «using in our code» ? If using in code, i -1 here. >> thanks. >> >> >> >Hello, dear Igniters! >> > >> >I would like to discuss the possibility of using Guava >> >< https://github.com/google/guava > in Ignite 3. I know about the >> restrictive >> >policy of using it in Ignite 2, but I have the following reasons: >> > >> >1. We are de-facto using it already as an implicit dependency, since the >> >Calcite module depends on it, and Calcite is going to stay for a while =) >> >2. AFAIK, the "bytecode" module is copied into the codebase only to strip >> >Guava away from it. We can remove this module, which will improve the >> >maintainability of the project. >> >3. We have some copy-paste of Guava code in the project. For example, see >> >this >> >< >> >> https://github.com/apache/ignite-3/blob/main/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L136 >> > >> >and this >> >< >> >> https://github.com/apache/ignite-3/blob/main/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L428 >> > >> >. >> >4. Regarding security concerns, this report >> >< >> https://www.cvedetails.com/product/52274/Google-Guava.html?vendor_id=1224 >> > >> >shows no major vulnerability issues for the last three years. >> > >> >Taking these points into account, I propose to allow using Guava both in >> >production and test code and to add it as an explicit dependency. >> > >> >What do you think? >> > >> >-- >> >With regards, >> >Aleksandr Polovtcev >> >> >> >> > > >-- >With regards, >Aleksandr Polovtcev
Re: Google Guava in Ignite 3
alexpolovtcev please clarify what do you mean under : «possibility of using Guava in Ignite 3», using how necessary dependency of calcite or using like «using in our code» ? If using in code, i -1 here. thanks. >Hello, dear Igniters! > >I would like to discuss the possibility of using Guava >< https://github.com/google/guava > in Ignite 3. I know about the restrictive >policy of using it in Ignite 2, but I have the following reasons: > >1. We are de-facto using it already as an implicit dependency, since the >Calcite module depends on it, and Calcite is going to stay for a while =) >2. AFAIK, the "bytecode" module is copied into the codebase only to strip >Guava away from it. We can remove this module, which will improve the >maintainability of the project. >3. We have some copy-paste of Guava code in the project. For example, see >this >< >https://github.com/apache/ignite-3/blob/main/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L136 > > >and this >< >https://github.com/apache/ignite-3/blob/main/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L428 > > >. >4. Regarding security concerns, this report >< https://www.cvedetails.com/product/52274/Google-Guava.html?vendor_id=1224 > >shows no major vulnerability issues for the last three years. > >Taking these points into account, I propose to allow using Guava both in >production and test code and to add it as an explicit dependency. > >What do you think? > >-- >With regards, >Aleksandr Polovtcev
Re[2]: Google Guava in Ignite 3
Andrey, seems we can use [1] it help us with point 1 in your comment, isn`t it ? [1] https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html >-1 >It is sad to say -1, as Guava has very useful stuff and it looks easier to >add it as a dependency rather than copy-paste a code. My concerns are: 1. >Original Bytecode module depends on 26.0-jre Calcite depends on 29.0-jre We >maybe will use some other version. A user might want to use one more >version. So, I'd disagree legalizing Guava will help with maintainability >anyhow. 2. Guava supports JDK-8. Is it possible to handle different >versions of Guava in dependencies with JigSaw? What impact will have >potential future CVEs (and the current one) with the JigSaw? 3. Guava has >an unresolved CVE [1]. They just mark a vulnerable method as Deprecated and >didn't actually fix it [2]. [1] https://github.com/google/guava/issues/4011 >[2] https://github.com/google/guava/issues/4011 > >On Thu, Aug 5, 2021 at 4:54 PM Konstantin Orlov < kor...@gridgain.com > wrote: > >> +1, I considered it a necessary evil >> >> -- >> Regards, >> Konstantin Orlov >> >> >> >> > On 5 Aug 2021, at 16:37, Alexei Scherbakov < alexey.scherbak...@gmail.com > >> wrote: >> > >> > +1 >> > >> > чт, 5 авг. 2021 г. в 16:12, Alexander Polovtcev < alexpolovt...@gmail.com >> >: >> > >> >> Hello, dear Igniters! >> >> >> >> I would like to discuss the possibility of using Guava >> >> < https://github.com/google/guava > in Ignite 3. I know about the >> >> restrictive >> >> policy of using it in Ignite 2, but I have the following reasons: >> >> >> >> 1. We are de-facto using it already as an implicit dependency, since the >> >> Calcite module depends on it, and Calcite is going to stay for a while >> =) >> >> 2. AFAIK, the "bytecode" module is copied into the codebase only to >> strip >> >> Guava away from it. We can remove this module, which will improve the >> >> maintainability of the project. >> >> 3. We have some copy-paste of Guava code in the project. For example, >> see >> >> this >> >> < >> >> >> >> https://github.com/apache/ignite-3/blob/main/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L136 >> >>> >> >> and this >> >> < >> >> >> >> https://github.com/apache/ignite-3/blob/main/modules/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java#L428 >> >>> >> >> . >> >> 4. Regarding security concerns, this report >> >> < >> https://www.cvedetails.com/product/52274/Google-Guava.html?vendor_id=1224 >> >>> >> >> shows no major vulnerability issues for the last three years. >> >> >> >> Taking these points into account, I propose to allow using Guava both in >> >> production and test code and to add it as an explicit dependency. >> >> >> >> What do you think? >> >> >> >> -- >> >> With regards, >> >> Aleksandr Polovtcev >> >> >> > >> > >> > -- >> > >> > Best regards, >> > Alexei Scherbakov >> >> >-- >Best regards, >Andrey V. Mashenkov
Re[2]: [ANNOUNCE] Welcome Zhenya Stanilovsky as a new committer
Guys, thank you very much !! >Zhenya, > >Congrats! > >-- >Regards, >Konstantin Orlov > > > > >> On 30 Jul 2021, at 12:20, Вячеслав Коптилин < slava.kopti...@gmail.com > >> wrote: >> >> Hooray! >> >> Congrats! May the Force be with you! >> >> Thanks, >> S. >> >> пт, 30 июл. 2021 г. в 11:17, Anton Vinogradov < a...@apache.org >: >> >>> Congrats! >>> >>> On Fri, Jul 30, 2021 at 10:19 AM ткаленко кирилл < tkalkir...@yandex.ru > >>> wrote: >>> >>>> Zhenya, congratulations! >>>> >>>> Пересылаемое сообщение >>>> 30.07.2021, 09:50, "Ivan Daschinsky" < ivanda...@apache.org >: >>>> >>>> >>>> Zhenya, congrats, well deserved! >>>> >>>> пт, 30 июл. 2021 г. в 00:44, Andrey Mashenkov < >>> andrey.mashen...@gmail.com >>>>> : >>>> >>>>> Congratulations Zhenya! >>>>> >>>>> On Fri, Jul 30, 2021 at 12:06 AM Maxim Muzafarov < mmu...@apache.org > >>>>> wrote: >>>>> >>>>>> The Project Management Committee (PMC) for Apache Ignite has invited >>>>>> Zhenya Stanilovsky to become a committer and we are pleased to >>>> announce >>>>>> that >>>>>> he has accepted. >>>>>> >>>>>> For the last few years, Zhenya made a lot of performance fixes >>>>>> especially for the core modules and important contributions to the >>>>>> Apache Ignite codebase. He is actively involved in integrating the >>>>>> Calcite framework with Apache Ignite. Moreover, he has been a great >>>>>> help in the preparation of several Apache Ignite major releases by >>>>>> carrying out stress-load tests. Besides the code contributions, >>> Zhenya >>>>>> is also an active community member and help users on dev and users >>>>>> lists. >>>>>> >>>>>> Being a committer enables easier contribution to the project since >>>> there >>>>> is >>>>>> no need to go via the patch submission process. This should enable >>>> better >>>>>> productivity. >>>>>> >>>>>> Please join me in welcoming Ivan, and congratulating him on the new >>>> role >>>>> in >>>>>> the Apache Ignite Community. >>>>>> >>>>>> Best Regards, >>>>>> Maxim Muzafarov >>>>>> >>>>> >>>>> -- >>>>> Best regards, >>>>> Andrey V. Mashenkov >>>> >>>> Конец пересылаемого сообщения >>>> >>>
[DISCUSS] Confuse default inspections.
Igniters, i understand that this is very long and fundamental story. but …
still want to rise up this discussion, we have 2 very strange inspections:
* «public» modifier in interface methods.
* Illegal ‘{}’ for one line statement. — i found it harmful.
I don`t want to link additional discussion about pos. 2 i think that harm is
obvious.
I suggest to get rid of them.
what do you think ?
Re[2]: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output
-1 for extra arg, +1 for Ivan`s upper proposal : @IgniteSystemProperty annotation. Look, someone will set some of IGNITE_* option and after possibly cluster problems will give this logs into analysis and engineer can`t reproduce such a case, cause no param is applied. >An extra argument for IgniteSystemProperty sounds reasonable. > >-Val > >On Thu, Jul 1, 2021 at 10:04 AM Ivan Daschinsky < ivanda...@gmail.com > wrote: > >> Ok, this can be excluded using blocklist-jvm-params.properties or just by >> providing and extra arg to annotation, as I have just suggested >> >> чт, 1 июл. 2021 г., 19:51 Valentin Kulichenko < >> valentin.kuliche...@gmail.com >> >: >> >> > Ivan, >> > >> > IP addresses (e.g. IGNITE_TCP_DISCOVERY_ADDRESSES) and file paths >> > (e.g. IGNITE_CONFIG_URL) are often considered sensitive information. Data >> > related to authentication (e.g. IGNITE_SSH_USER_NAME) is very likely to >> be >> > sensitive. >> > >> > Once again - I would exclude any property that can contain user-specific >> > information. Only our internal settings (stuff >> > like IGNITE_SQL_MERGE_TABLE_MAX_SIZE) are OK to appear in the logs. >> > >> > -Val >> > >> > On Thu, Jul 1, 2021 at 9:47 AM Ivan Daschinsky < ivanda...@gmail.com > >> > wrote: >> > >> > > We can add add an extra param in annotation, that blocks param to be >> > > printed, just set it to false by default and block it wheb set to true >> > > >> > > чт, 1 июл. 2021 г., 19:45 Atri Sharma < a...@apache.org >: >> > > >> > > > What if we allowed a blocklist of parameters that are never printed? >> > > > >> > > > On Thu, 1 Jul 2021, 22:06 Valentin Kulichenko, < >> > > > valentin.kuliche...@gmail.com > wrote: >> > > > >> > > > > Not all of them are OK to be printed out. At the very least, we >> > should >> > > > have >> > > > > a mechanism to exclude some of them. I would still go with opt-in >> > > rather >> > > > > than opt-out though, but I guess that is up to a discussion. >> > > > > >> > > > > -Val >> > > > > >> > > > > On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky < >> ivanda...@gmail.com > >> > > > > wrote: >> > > > > >> > > > > > This is security through obscurity, an obvious and a well-known >> > anti >> > > > > > pattern. I suppose that printing jvm options, that is registered >> by >> > > > > > @IgniteSystemProperty annotation is an ideal variant >> > > > > > >> > > > > > чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko < >> > > > > > valentin.kuliche...@gmail.com >> > > > > > >: >> > > > > > >> > > > > > > Folks, >> > > > > > > >> > > > > > > *Anything* that a user provides to the system can potentially >> be >> > > > > > considered >> > > > > > > sensitive information. This includes the VM arguments. We can't >> > > > predict >> > > > > > > what exactly one can put there, so let's not make assumptions. >> > > > > > > >> > > > > > > When dealing with security, we should be as conservative as >> > > possible. >> > > > > > That >> > > > > > > said, I do not even agree with the pattern approach - there >> might >> > > be >> > > > a >> > > > > > > user's system property named IGNITE_xxx. It is also possible >> for >> > > our >> > > > > > > internal properties to contain sensitive information (not all >> of >> > > them >> > > > > are >> > > > > > > boolean flags). >> > > > > > > >> > > > > > > The only option I see is to print out specific properties for >> > which >> > > > we >> > > > > > > agree that they are safe. For example, we can introduce an >> > > annotation >> > > > > > that >> > > > > > > would mark "safe" properties in IgniteSystemProperties; we will >> > > then >> > > > > > print >> > > > > > > out only those that are marked with the annotation. >> > > > > > > >> > > > > > > -Val >> > > > > > > >> > > > > > > >> > > > > > > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин < >> > > > > > slava.kopti...@gmail.com >> > > > > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > > > Hello Ivan, >> > > > > > > > >> > > > > > > > > At least, we could just hide params that match a specific >> > > pattern >> > > > > > > > Yes, we can filter out all vm options that do not relate to >> > > Ignite, >> > > > > for >> > > > > > > > instance. >> > > > > > > > >> > > > > > > > > Ilya, go ahead, file ticket and prepare a PR. >> > > > > > > > Please do not rush. Let's listen to other community members. >> > This >> > > > > > > question >> > > > > > > > is about security and it should not be discussed in a hurry >> > (even >> > > > > > though >> > > > > > > it >> > > > > > > > looks like an obvious thing). >> > > > > > > > >> > > > > > > > Thanks, >> > > > > > > > S. >> > > > > > > > >> > > > > > > > чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky < >> > ivanda...@gmail.com >> > > >: >> > > > > > > > >> > > > > > > > > I suppose, that all normal users should not suffer from >> this >> > > > > > > > restrictions. >> > > > > > > > > Nobody will pass password using jvm options. It is >> absolutely >> > > > > insane, >> > > > > > > > > normal users pass
Re[2]: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output
+1 for reverting, can anybody (possibly ticket starter?) explain how jvm settings will rise some security problems ? >I suppose, that we should revert this particular line. I don't understand >who ever considers vm options as sensitive info. > >ср, 30 июн. 2021 г., 22:52 Shishkov Ilya < shishkovi...@gmail.com >: > >> Hi Igniters, >> >> This feature [1, 2] prevents logging of the VM arguments when >> IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false. Till now, method >> IgniteKernal#ackVmArguments remains mostly the same [3]. >> >> Is this behaviour actual now? Often, we should be able to get from logs the >> actual VM options used at startup even if output of sensitive data is >> restricted. >> >> 1. https://issues.apache.org/jira/browse/IGNITE-4991 >> 2. >> >> >> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93 >> 3. >> >> >> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002 >>
Re[2]: [Discussion] Apache Ignite 2.11 Scope Freeze
Hi ! I found that very important issue [1] (already in master) is not planned to be in 2.11, may be it still possible to take it into scope ? thanks ! [1] https://issues.apache.org/jira/browse/IGNITE-14739 >Hi Folks, > >Branch divergence has been completed. Sorry for the delay, it was my fault. >Now branch for 2.11 is now available >https://github.com/apache/ignite/tree/ignite-2.11 >https://gitbox.apache.org/repos/asf?p=ignite.git;a=shortlog;h=refs/heads/ignite-2.11 > >Who could help with updating TC bot configuration (json)? >I suggest changing tracked branch ignite-2.10-nightly to ignite-2.11-nightly >and run 'ignite-2.11' instread. See also >https://mtcga.gridgain.com/current.html?branch=ignite-2.10-nightly > >Sincerely, >Dmitriy Pavlov > > >On 2021/06/04 13:51:28, Pavel Pereslegin < xxt...@gmail.com > wrote: >> Hopefully we'll be done by the end of next week (2021.06.11). >> >> пт, 4 июн. 2021 г. в 16:06, Alexey Gidaspov < olive.c...@gmail.com >: >> > >> > Hello, Pavel. >> > >> > Can you specify the date more precisly? >> > >> > On 2021/06/04 12:05:53, Pavel Pereslegin < xxt...@gmail.com > wrote: >> > > Hello Alexey. >> > > >> > > We have implemented a feature to automatically restore the cache group >> > > from a snapshot [1]. >> > > But this feature requires CLI management [2], which is currently on >> > > review and should also be included in 2.11. We plan to finish with it >> > > next week. >> > > >> > > [1] https://issues.apache.org/jira/browse/IGNITE-13805 >> > > [2] https://issues.apache.org/jira/browse/IGNITE-14723 >> > > >> > > пт, 4 июн. 2021 г. в 13:01, Dmitry Pavlov < dpav...@apache.org >: >> > > > >> > > > I've updated the page >> > > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.11 >> > > > >> > > > since final freeze is technically possible only after branch >> > > > divergence (see also >> > > > https://cwiki.apache.org/confluence/display/IGNITE/Release+Process ) >> > > > >> > > > Sincerely >> > > > >> > > > On 2021/06/04 09:56:15, Alexey Gidaspov < olive.c...@gmail.com > wrote: >> > > > > I'm planning to diverge branch at 16:00 MSK 7 June, 2021 >> > > > > >> > > > > And here is 2.11 release wiki page [1] >> > > > > >> > > > > [1] >> > > > > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.11 >> > > > > >> > > > > On 2021/06/04 09:11:49, Alexey Gidaspov < olive.c...@gmail.com > >> > > > > wrote: >> > > > > > Hi, Igniters! >> > > > > > >> > > > > > I'm going to start scope freeze and leave only Resolved tickets in >> > > > > > it [1]. You are welcome to discuss. >> > > > > > >> > > > > > [1] >> > > > > > https://lists.apache.org/thread.html/a4be115d2bd4318ae958ac91aba40294dfd62f293b3cb264c8fcf0ae%40%3Cdev.ignite.apache.org%3E >> > > > > > >> > > > > >> > > >>
[DISCUSSION] Transactional cache could be in inconsistent state after node recovery.
Igniters, as previously was found [1] in some cases transactional cache can contain unexpected data after node crash and further recovery. Short explanation: it`s all due to ignite does not save transactional records into the WAL. The simplest example: 1 node cluster and transactional cache, if crash has occurred during transaction processing data records will be partially stored into the wal and further recovery procedure will apply them, as is, thus if transaction contain keys [k1, k2, k3] after recovery we can obtain only [k1, k2], of course this is unexpected and erroneous behavior. There are numerous variants with multi nodes on how they can obtain inconsistent data after recovery [2]. PR [1] (already merged into master) has changed this situation and i suggest to turn on by default transactional records storing and remove flag IGNITE_WAL_LOG_TX_RECORDS at all. I benchmarked (attached in issue) and found no potential slowdowns here. Any comments ? Review is appreciated too. Thanks! [1] https://issues.apache.org/jira/browse/IGNITE-6324 [2] https://github.com/apache/ignite/pull/8987/files#diff-96ba20321a66ec20d0df55abcdeb0808ff6dd1d4b87b782c892496369321a593R75 [3] https://issues.apache.org/jira/browse/IGNITE-14739
Re: [DISCUSS] Python thin client development approach.
Ivan, if Petr can`t help here, how can we change it out own ? May be we need some help from pmc chair or someone else ? > >> >>Igniters. Almost half of a year passed since this thread begun. We >>released 0.4.0, we adopted travis-ci and use it as primary source for >>test results but nothing changed in TC >> >>1. We use deprecated pytest-runner (even here >>( https://pypi.org/project/pytest-runner ) you can see deprecation >>notice and advice to use tox instead) in TC. >> This dependency is absolutely unnecessary and cause a bunch of >>problems when pyignite is installed as source from pypi.org local >>mirror >> We just need at least tox in linux agents on TC. >>2. We build release 0.4.0 manually on my and Igor Sapego's laptops. >>There are no release build on TC, despite the fact that all needed >>scripts are included in git repo and comprehensive instruction is >>available. >>For running these scripts, we just need docker available on linux >>agents and some additional packages on windows agents. >> >>I, Igor and Petr Ivanov had personal talk about this few months ago >>but nothing changed. >> >>пт, 22 янв. 2021 г. в 16:12, Ivan Daschinsky < ivanda...@gmail.com >: >>> >>> Igor, I've never talked about complete removal of TC builds. >>> I just suggested to add TC jobs for different python versions and use >>> travis heavily. >>> >>> Currently, we have done most of the tasks, except of run TC tests on >>> different python's versions. >>> >>> >>> >>> пт, 22 янв. 2021 г. в 15:16, Igor Sapego < isap...@apache.org >: Ivan, Though I generally agree with the approach you've suggested, I can see a problem here. Since we now have a separate repos for thin clients, for some features we may need to introduce changes to Ignite and python-thin repos in a single ticket and we should have an ability to run tests on with changes on both python client and server nodes. Current TC suites provide such ability, Travis does not. So I believe, it's too soon to abandon TC for thin clients, at least until we could solve the issue I've described. Best Regards, Igor On Fri, Dec 25, 2020 at 1:49 PM Nikolay Izhikov < nizhi...@apache.org > wrote: > Hello, Ivan. > > I’m +1 for your proposal. > > > 25 дек. 2020 г., в 13:14, Ivan Daschinsky < ivanda...@gmail.com > > написал(а): > > > > Hi folks! > > > > Since we already have a separate repo for thin-clients [1], [2] > > I'd like to propose some improvements in development process/ > > > > 1. We should simplify and automate unit tests run for different versions > of > > python > > 2. We should add travis integration per commit and pr. Tests could be run > > against latest dockered image of ignite > > 3. There should be ability to run tests against multiple pythons on TC > > 4. For thin client development process, travis should be the first > option. > > TC suite should be used only to check that compatibility is not broken > > and when new functionality is developed (rare case). > > > > I've prepared fix [3], you can see successful builds for travis. It uses > > tox [5], the most common tool to run tests in multiple environments. > > There are few environments set up in tox.ini -- with and without docker, > > with or without ssl, etc. This helped a lot > > to setup travis CI build (you can see in commits list of PR) and simplify > > run tests for developers. Also docker-compose was introduced > > to help python thin client developers. > > > > But I need some assistance for TC: > > 1. There is outdated python setuptools on TC agents, so tests cannot be > run > > with updated pytest etc. > > 2. Multiple pythons should be installed on TC agents (via > > https://github.com/pyenv/pyenv ), latest minor versions > > for 3.6, 3.7 and 3.8 > > 3. After that, TC job should be changed to utilize tox > > > > WDYT about this initiative? > > > > > > [1] -- https://issues.apache.org/jira/browse/IGNITE-13767 > > [2] -- https://github.com/apache/ignite-python-thin-client > > [3] -- https://issues.apache.org/jira/browse/IGNITE-13903 > > [4] -- > https://github.com/apache/ignite-python-thin-client/pull/1/commits > > [5] -- https://tox.readthedocs.io/en/latest/ > > > > -- > > Sincerely yours, Ivan Daschinskiy > > >>> >>> >>> >>> -- >>> Sincerely yours, Ivan Daschinskiy > > > >
IgniteCompute can lead to OOM in some cases.
Igniters, i found some problems with running p2p tasks concurrently. Description and patch available here, can someone review plz? [1] https://issues.apache.org/jira/browse/IGNITE-14131
Re: Ignite 2.10. Performance tests in Azure
hello, can you recheck with OPTIMISTIC SERILIZABLE tx`s ? >Hello all, > >For our project we need a distributed database with transactional support, >and Ignite is one of the options we are testing. > >Scalability is one of our must have, so we created an Ignite Kubernetes >cluster in Azure to test it, but we found that the results were not what we >expected. > >To discard the problem was in our code or in using transactional caches, we >created a small test program for writing/reading 1.8M keys of 528 bytes >each >(it represents one of our data types). > >As you can see in this graph, reading doesn't seem to scale. Especially >for >the transactional cache, where having 4, 8 or 16 nodes in the cluster >performs worse than having only 2: >< http://apache-ignite-users.70518.x6.nabble.com/file/t3059/reading.png > > >While writing in atomic caches does... until 8 nodes, then it gets steady >(No transactional times because of this >< https://issues.apache.org/jira/browse/IGNITE-14076 > ): >< http://apache-ignite-users.70518.x6.nabble.com/file/t3059/writing.png > > >Another strange thing is that, for atomic caches, reading seems to be >slower >than writing: >< http://apache-ignite-users.70518.x6.nabble.com/file/t3059/atomic.png > > >So, my questions are: >- Could I been doing something wrong that could lead to this results? >- How could it be possible to get worse reading timings in a 4/8/16 >nodes >cluster than in a 2 nodes cluster for a transactional cache? >- How could reading be slower than writing in atomic caches? > >These are the source code and configuration files we're using: >Test.cpp >< http://apache-ignite-users.70518.x6.nabble.com/file/t3059/Test.cpp > >Order.h < http://apache-ignite-users.70518.x6.nabble.com/file/t3059/Order.h > >node-configuration.xml >< >http://apache-ignite-users.70518.x6.nabble.com/file/t3059/node-configuration.xml > > > >Best regards and thanks in advance! > > > > >-- >Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite 2.10. Performance tests in Azure
hello, can you try OPTIMISTIC SERILIZABLE tx`s ? >Hello all, > >For our project we need a distributed database with transactional support, >and Ignite is one of the options we are testing. > >Scalability is one of our must have, so we created an Ignite Kubernetes >cluster in Azure to test it, but we found that the results were not what we >expected. > >To discard the problem was in our code or in using transactional caches, we >created a small test program for writing/reading 1.8M keys of 528 bytes >each >(it represents one of our data types). > >As you can see in this graph, reading doesn't seem to scale. Especially >for >the transactional cache, where having 4, 8 or 16 nodes in the cluster >performs worse than having only 2: >< http://apache-ignite-users.70518.x6.nabble.com/file/t3059/reading.png > > >While writing in atomic caches does... until 8 nodes, then it gets steady >(No transactional times because of this >< https://issues.apache.org/jira/browse/IGNITE-14076 > ): >< http://apache-ignite-users.70518.x6.nabble.com/file/t3059/writing.png > > >Another strange thing is that, for atomic caches, reading seems to be >slower >than writing: >< http://apache-ignite-users.70518.x6.nabble.com/file/t3059/atomic.png > > >So, my questions are: >- Could I been doing something wrong that could lead to this results? >- How could it be possible to get worse reading timings in a 4/8/16 >nodes >cluster than in a 2 nodes cluster for a transactional cache? >- How could reading be slower than writing in atomic caches? > >These are the source code and configuration files we're using: >Test.cpp >< http://apache-ignite-users.70518.x6.nabble.com/file/t3059/Test.cpp > >Order.h < http://apache-ignite-users.70518.x6.nabble.com/file/t3059/Order.h > >node-configuration.xml >< >http://apache-ignite-users.70518.x6.nabble.com/file/t3059/node-configuration.xml > > > >Best regards and thanks in advance! > > > > >-- >Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re[2]: [DISCUSSION] MaxLineLength checkstyle rule
Python is not so verbose as java ) +1 for 140 >Hi! >Personally, I suppose that 120 chars per line is OK. Moreover, many >codestyles suggests less chars per line. >For example PEP8 recommends 80 (but we use 120 in pyignite and flake8 >codestyle checks it). Google java codestyle insists on 100. > >More than 120 chars is too long as for me and is not convenient for 3-way >merges. > >чт, 15 апр. 2021 г. в 12:28, Nikolay Izhikov < nizhi...@apache.org >: > >> Hello, Ilya. >> >> Thanks for the feedback. >> >> 140 characters is fine for me. >> >> > 15 апр. 2021 г., в 12:25, Ilya Kasnacheev < ilya.kasnach...@gmail.com > >> написал(а): >> > >> > Hello! >> > >> > Please find attached the distribution of line lengths in the project, in >> the form of (count, line length). >> > >> > I think that we can enforce a hard limit of 140 chars per line. I think >> that having longer lines is excessive and does not benefit readability. >> > >> > Having a limit of 150 or 180 does not give us much since there's still >> a long tail which has to be fixed. >> > >> > Regards, >> > -- >> > Ilya Kasnacheev >> > >> > >> > чт, 15 апр. 2021 г. в 11:30, Nikolay Izhikov < nizhi...@apache.org >: >> > Hello, Igniters. >> > >> > Right now, we have a code style rule [1] - the line should fit in 120 >> characters. >> > But, this rule violated in many and many places through code. >> > I have a plan to add a check style rule to force maximum line length. >> > >> > For me, personally, 120 characters a bit old-fashioned restriction. >> > Should we increase the maximum line length to 150 or even 180 characters? >> > >> > [1] https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines >> > >> >> >-- >Sincerely yours, Ivan Daschinskiy
Re: [ANNOUNCE] Welcome Ivan Daschinsky as a new committer
Big deal ! Ivan, ignite it !) >The Project Management Committee (PMC) for Apache Ignite has invited >Ivan Daschinsky to become a committer and we are pleased to announce that >he has accepted. > >Ivan made a lot of contributions to Apache Ignite. >He helped a lot to improve our Python Thin Client fixing a lot of different >bugs and contributing major feature such as asyncio support and C-extension >which improved performance significantly for many cases. Thanks to Ivan >Python >Thin client has become much more stable and production-ready. He also >introduced the CMake building system for Ignite C++, and has made a number >of >other important improvements. Besides the code contributions, Ivan is also >an >active community member. > >Being a committer enables easier contribution to the project since there is >no need to go via the patch submission process. This should enable better >productivity. > >Please join me in welcoming Ivan, and congratulating him on the new role in >the Apache Ignite Community. > >Best Regards, >Igor
Re: Long transaction suspended
hi jjimeno, plz check your throughput once more by using OPTIMISTIC, SERIALIZABLE options, hope it would be more faster than default. > > >--- Forwarded message --- >From: jjimeno < jjim...@omp.com > >To: u...@ignite.apache.org >Cc: >Subject: Long transaction suspended >Date: Thu, 04 Feb 2021 12:32:20 +0300 > >Hi all, > >I'm trying to commit a very large transaction (8M keys and ~4GB of data). > >After a while, I can see this diagnostics message in node log: >[08:56:31,721][WARNING][sys-#989][diagnostic] >>> Transaction >[startTime=08:55:22.095, curTime=08:56:31.712, ... *state=SUSPENDED* ... > >Does anyone know why it is suspended, and how to avoid it? > >Thanks in advance >José > > > > > >-- >Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re[2]: [VOTE] Release Apache Ignite 2.10.0 RC2
Build from sources, run .net tests, looks good. +1
>+1 (binding)
>
>Downloaded binary packages, started nodes, .NET examples, .NET nodes.
>Downloaded source package, built Java and .NET parts.
>
>On Thu, Mar 11, 2021 at 4:24 AM Maxim Muzafarov < mmu...@apache.org > wrote:
>
>> Dear Community,
>>
>> I have uploaded a release candidate to:
>>
>> https://dist.apache.org/repos/dist/dev/ignite/2.10.0-rc2/
>> https://dist.apache.org/repos/dist/dev/ignite/packages_2.10.0-rc2/
>>
>> The following staging can be used for testing:
>> https://repository.apache.org/content/repositories/orgapacheignite-1507/
>>
>> https://www.myget.org/feed/apache-ignite-staging/package/nuget/Apache.Ignite
>>
>> Tag name is 2.10.0-rc2:
>>
>>
>> https://gitbox.apache.org/repos/asf?p=ignite.git;a=tag;h=refs/tags/2.10.0-rc2
>>
>> RELEASE_NOTES:
>>
>>
>> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.10
>>
>> Complete list of resolved issues:
>>
>> https://issues.apache.org/jira/issues/?jql=(project%20%3D%20
>> 'Ignite'%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20('2.10'))
>>
>> DEVNOTES:
>>
>>
>> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.10
>>
>>
>> Additional checks have been performed (available for users included into
>> the release group on TeamCity).
>>
>> TC [Check RC: Licenses, compile, chksum]
>>
>>
>> https://ci.ignite.apache.org/viewLog.html?buildId=5909007=ApacheIgniteReleaseJava8_PrepareVote4CheckRcLicensesChecksum
>>
>> TC [3] Build & Upload Nuget Staging Packages
>>
>>
>> https://ci.ignite.apache.org/viewLog.html?buildId=5909005=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages=artifacts
>>
>> TC [2] Compare w/ Previous Release
>>
>>
>> https://ci.ignite.apache.org/viewLog.html?buildId=5909003=ApacheIgniteReleaseJava8_IgniteRelease72CheckFileConsistency=artifacts#
>>
>>
>> The vote is formal, see voting guidelines
>> https://www.apache.org/foundation/voting.html
>>
>> +1 - to accept Apache Ignite 2.10.0-rc2
>> 0 - don't care either way
>> -1 - DO NOT accept Apache Ignite Ignite 2.10.0-rc2 (explain why)
>>
>> See notes on how to verify release here
>> https://www.apache.org/info/verification.html
>> and
>>
>>
>> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification
>>
>> This vote will be open until Sun Mar 14, 09:00 UTC.
>> Please, write down the thread if you need additional time to check
>> the release.
>>
>>
>> https://www.timeanddate.com/countdown/vote?iso=20210314T12=166=Vote+for+the+Apache+Ignite+2.10.0+RC2=slab
>>
Re[2]: [DISCUSSION] Apache Ignite Release 2.10 (time, scope, manager)
I fill the ticket with drop problem, plz take a look [1] [1] https://issues.apache.org/jira/browse/IGNITE-14205 >Ilya, > >Thanks! >I've added this step to the Release Process wiki page also [1]. > >[1] >https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P1.2ImplementationandScopeFinalization > >On Wed, 17 Feb 2021 at 18:05, Ilya Kasnacheev < ilya.kasnach...@gmail.com > >wrote: >> >> Hello! >> >> I have added ignite-2.10 to Nightly build triggering. I hope we will have a >> 2.10 nightly tomorrow. Shame that I didn't do it earlier. >> >> https://ci.ignite.apache.org/buildConfiguration/Releases_NightlyRelease_RunApacheIgniteNightlyRelease?branch=ignite-2.10=overview=builds >> >> Regards, >> -- >> Ilya Kasnacheev >> >> >> ср, 17 февр. 2021 г. в 17:37, Maxim Muzafarov < mmu...@apache.org >: >> >> > Folks, >> > >> > There is a possible issue with the performance for 2.9.1 vs 2.10. I'm >> > trying to reproduce on a stress-testing environment. >> > [1] >> > >> > https://cwiki.apache.org/confluence/download/attachments/165224767/atomicPutRandomValueFullSync.jpg?api=v2 >> > >> > On Fri, 12 Feb 2021 at 20:46, Maxim Muzafarov < mmu...@apache.org > wrote: >> > > >> > > Folks, >> > > >> > > I'm going to cherry-pick these issues to the release branch, any >> > objections? >> > > >> > > >> > > Checkpointer thread holds write lock too long >> > > https://issues.apache.org/jira/browse/IGNITE-14140 >> > > >> > > Incorrect initialize checkpoint-runner-cpu thread pool >> > > https://issues.apache.org/jira/browse/IGNITE-14139 >> > > >> > > On Wed, 10 Feb 2021 at 21:59, Maxim Muzafarov < mmu...@apache.org > >> > > wrote: >> > > > >> > > > Folks, >> > > > >> > > > Do we need any other critical issues from the master branch that need >> > > > to be cherry-picked picked from the master branch? I've marked the >> > > > latest select issues with patching version 2.10. >> > > > >> > > > - benchmarks completed (I'll do another one prior to preparing rc) >> > > > - the release notes merged >> > > > - cherry-picked issue (IGNITE-14073 Fixed transactions failover) >> > > > - most of the documentation pages also merged >> > > > >> > > > Hopefully, by Friday the 12th everything will be ready for the >> > > > preparation of a release candidate. >> > > > >> > > > On Tue, 9 Feb 2021 at 05:09, Никита Сафонов < >> > > > vlasovpavel2...@gmail.com > >> > wrote: >> > > > > >> > > > > Hi everyone, >> > > > > >> > > > > Below are two lists of items representing all the remaining (and >> > completed) >> > > > > documentation tasks for the Ignite 2.10 release. >> > > > > >> > > > > The "*Improvements*" part includes PRs on reworked documentation. >> > > > > The "*Finished*" part includes PRs on newly added documentation. >> > > > > >> > > > > *Improvements:* >> > > > > >> > > > > Documentation for .NET thin client service invocation >> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14129 >> > > > > [2] https://github.com/apache/ignite/pull/8756 >> > > > > >> > > > > Documentation for cache warm-up strategy >> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-13385 >> > > > > [2] https://github.com/apache/ignite/pull/8703 >> > > > > >> > > > > *Finished:* >> > > > > >> > > > > Document control.(sh|bin) command to get an arbitrary SystemView >> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14142 >> > > > > [2] https://github.com/apache/ignite/pull/8775 >> > > > > >> > > > > Document metric for processed keys when rebuilding indexes >> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14143 >> > > > > [2] https://github.com/apache/ignite/pull/8776 >> > > > > >> > > > > Document C++ thin client transactions >> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14144 >> > > > > [2] https://github.com/apache/ignite/pull/8777 >> > > > > >> > > > > Please don't hesitate to ask me if you have any questions or >> > concerns. >> > > > > >> > > > > With best regards, >> > > > > Nikita >> > > > > >> > > > > сб, 6 февр. 2021 г. в 02:14, Никита Сафонов < >> > vlasovpavel2...@gmail.com >: >> > > > > >> > > > > > Maxim, >> > > > > > >> > > > > > Thank you for being ready to help! >> > > > > > >> > > > > > As I mentioned before, I'm sharing the completed doc items today. >> > > > > > Below is the list of tickets with the prepared PR's: >> > > > > > >> > > > > > *- Documentation: SQL tracing.* >> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-13981 >> > > > > > [2] https://github.com/apache/ignite/pull/8762 >> > > > > > >> > > > > > *- Documentation for async API (Thin client Java API)* >> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14128 >> > > > > > [2] https://github.com/apache/ignite/pull/8753 >> > > > > > >> > > > > > *- Documentation for .NET: Thin Client: Service invocation* >> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14129 >> > > > > > [2] https://github.com/apache/ignite/pull/8756 >> > > > > > >> > > > > > *-
Re[2]: Adding metrics of using WAL archive
Kirill, is it good practice to have a metrics for internal use? Don`t think so. +1 witk Nikolay size is more readable than abstract segments count. >Hi, Nikolay! > >For internal use, leave the metric that I propose and also add the metric: >Count of bytes logged in WAL. Why not "written" because for the mmap we cannot >track when the physical writting will occur. > >16.02.2021, 15:42, "Nikolay Izhikov" < nizhi...@apache.org >: >> Kirill. >> >> «Count of segments» is a very internal thing for a regular user. >> Regular user don’t want to know about such things. >> >> You suggest to calculate the number (space required to store WAL) with some >> kind of rough calculation, and with the «Count of bytes written in WAL» we >> can have exact number without any suggestions or calculations. >> >> Moreover, «Count of bytes written in WAL» is independent on internal WAL >> implementation. >> >> So, I think exact number is always better to have then some approximation. >> >> What do you think? >> >>> 15 февр. 2021 г., в 20:45, ткаленко кирилл < tkalkir...@yandex.ru > >>> написал(а): >>> >>> Hi, Nikolay! >>> >>> We set the number of segments in the working directory, we also delete by >>> segment, it seems that this is a matter of usability. I prefer to dwell on >>> my own version, this is a simple metric that does not hurt and you can add >>> more as needed. >>> >>> 15.02.2021, 17:10, "Nikolay Izhikov" < nizhi...@apache.org >: My suggestion that «count of files» is meaningless number. And «count of bytes written to the files» is useful number to know and use for capacity planning.. > 15 февр. 2021 г., в 15:59, ткаленко кирилл < tkalkir...@yandex.ru > > написал(а): > > Hi, Nikolay! > > There may be a number (count of segments * segment size) or there may > be a count of segments, whichever is more convenient for the user. > > 15.02.2021, 13:14, "Nikolay Izhikov" < nizhi...@apache.org >: >> Hello, Kirill. >> >> Thanks for an answers. >> Now, I understand your intentions. >> >>> t also seems that it will be more natural to operate not just bytes >>> but multiples of a segment. >> >> Can’t agree here. >> From my point of view - it’s better to know exact number, not just >> «count of segments». >> >>> 15 февр. 2021 г., в 13:00, ткаленко кирилл < tkalkir...@yandex.ru > >>> написал(а): >>> >>> Hello, Nikolay! >>> >>> The period of one day (24h) seems more natural, you can take more or >>> less, I think that one day may not be enough, and it is worth getting >>> the metric for several days (collect statistics) for example a week. >>> Yes, the total size of the segments may not be >>> DataStorageConfiguration#getMaxWalArchiveSize, but for capacity >>> planning, accuracy is not so important to us, since the load can always >>> change, it will hurt users more if we overflow the archive and it will >>> not be able to start the node. So to say that more is better than less, >>> it also seems that it will be more natural to operate not just bytes >>> but multiples of a segment. >>> >>> In separate threads, you can discuss the metric that you propose >>> about page memory and indexes estimates. >>> >>> 14.02.2021, 11:54, "Nikolay Izhikov" < nizhi...@apache.org >: Hello, Kirill Your conclusions still not clear for me. > It is not possible for us to estimate how much space a user will > need in the archive so as not to overflow it under its load > We take the maximum 44 and multiply it by a > DataStorageConfiguration#getWalSegmentSize Why you take a single day (24h) for a standard period? Is there any rationale behind this? 1. We have `walAutoArchiveAfterInactivity` property. So WAL segment can have a size less than the maximum. 2. For CDC feature I want to introduce «WAL force rollover timeout» to make data available for a consumer in a guaranteed period [1]. Why does the user want to estimate those numbers in the first place? Are we talking about some kind of capacity planning? If yes, then maybe it will be better to have a metric for a count of bytes written in the WAL? With it, we will have an exact number of space we need for WAL. How user should estimate capacity for a page memory and indexes? [1] https://issues.apache.org/jira/browse/IGNITE-13582 > 14 февр. 2021 г., в 09:48, ткаленко кирилл < tkalkir...@yandex.ru > > написал(а): > > Hi, Nikolay! > > The user will be able to take the getLastArchivedSegmentIndex >
Re[4]: [DISCUSSION] Request for thread unsafe Compute functionality deprecation.
Hello Ilya ! >Hello! > >In the test GridDifferentLocalDeploymentSelfTest, I can see that some count >of internal data structure yanked from a class field is not equal expected >(without the fix). GridDifferentLocalDeploymentSelfTest checks that inner deployment machinery are not «leak» the only test without fix will generate huge number of SharedDeployment instances which will lead to jvm metaspace oom. Previous algorithm just replace old execution implementation with new one what led to new ClassLoaderId generation each time classloader was changed in client node. The second IgniteExplicitImplicitDeploymentSelfTest demonstrates correct concurrent job calling. >But I don't see why it is a problem. Do you have some reproducer where bad >things (such as task failure, deadlock, critical error) happen? > >Regards, >-- >Ilya Kasnacheev > > >пт, 5 февр. 2021 г. в 16:00, Zhenya Stanilovsky < arzamas...@mail.ru >: > >> >> Ilya, as previously agreed, ticket [1], examples of concurrent tests you >> can find here GridDifferentLocalDeploymentSelfTest and here >> IgniteExplicitImplicitDeploymentSelfTest >> < >> https://github.com/apache/ignite/pull/8759/files#diff-802af604a112f46b1282431673f3b143da5f2bfc4da70dbc8f0cbc93a3ed0f80 >> >, >> TC in progress. >> >> thanks ! >> >> [1] https://issues.apache.org/jira/browse/IGNITE-14131 >> >> >> >> Hello! >> >> Please publish it. I don't see why not. >> >> Regards, >> >> >> >> >> >> >> -- >> >> >> >>
Re[4]: [DISCUSSION] Request for thread unsafe Compute functionality deprecation.
As previously agreed, ticket [1], examples of concurrent tests you can find here GridDifferentLocalDeploymentSelfTest and here IgniteExplicitImplicitDeploymentSelfTest , TC in progress. Fabric — possibly i mention incorrect word here, more clear it need to be — factory ) I suppose additional tests would be helpful. thanks ! [1] https://issues.apache.org/jira/browse/IGNITE-14131 >Zhenya, > >Could you clarify what you mean by "one instance is shared between numerous >of fabric"? What is the exact scenario and what are the implications of >running multiple compute tasks in parallel? A code sample demonstrating the >scenario might be helpful as well. > >-Val > >On Fri, Feb 5, 2021 at 12:58 PM Vladislav Pyatkov < vldpyat...@gmail.com > >wrote: > >> Hi Zhenya, >> >> I don't understand your proposal without a patch. >> I see that you want to mark as @Depricate three methods in IgniteCompute >> interface, but what will you give instead of? >> >> It seems to me, this interface can invoke some job without contains a class >> locally. >> How will this be supported in your mind? >> >> On Thu, Jan 28, 2021 at 6:58 PM Ilya Kasnacheev < ilya.kasnach...@gmail.com >> > >> wrote: >> >> > Hello! >> > >> > Please publish it. I don't see why not. >> > >> > Regards, >> > -- >> > Ilya Kasnacheev >> > >> > >> > чт, 28 янв. 2021 г. в 14:28, Zhenya Stanilovsky >> < arzamas...@mail.ru.invalid >> > >: >> > >> > > >> > > >> > > Hi Ilya , of course it contains in my PR (i don`t know if it shout be >> > > published before this discussion will be finished). >> > > Little changes from single thread into multiple, for example here [1] >> > will >> > > highlight a problem, or i can just publish my PR. >> > > >> > > [1] >> > > >> > >> >> https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/IgniteExplicitImplicitDeploymentSelfTest.java#L221 >> > > >> > > > >> > > >> >> > > >>>Hello! >> > > >>> >> > > >>>Do you have some kind of reproducer which demonstrates the issue? >> > > >>> >> > > >>>Regards, >> > > >>>-- >> > > >>>Ilya Kasnacheev >> > > >>> >> > > >>> >> > > >>>чт, 28 янв. 2021 г. в 10:32, Zhenya Stanilovsky < >> > > arzamas...@mail.ru.invalid >> > > >>>>: >> > > >>> >> > > >>>> >> > > >>>> Hello Igniters ! >> > > >>>> In the process of Ignite usage i found that some part of Compute >> > > >>>> functionality are thread unsafe and seems was designed with such >> > > >>>> limitations initially. >> > > >>>> Example : one (client, but it doesn`t matter at all) instance is >> > > >>>> shared between numerous of fabric, all of them calls something >> like >> > : >> > > >>>> IgniteCompute#execute(ComputeTask, T) >> > > >>>> or >> > > >>>> IgniteCompute#execute(java.lang.Class>, >> > T) >> > > >>>> and appropriate «async» methods — what kind of instance will be >> > > called is >> > > >>>> nondeterministic for now and as a confirmation of my words — i >> found >> > > no >> > > >>>> tests covered multi thread usage of Computing i also found nothing >> > on >> > > >>>> documentation page [1]. >> > > >>>> We have all necessary info for correct processing of such cases: >> > > >>>> from initiator (ignite.compute(...) starter) side we have Class or >> > it >> > > >>>> instance and appropriate class loader which will be wired by class >> > > loader >> > > >>>> id from execution side. >> > > >>>> I create a fix and seems all work perfectly well besides one >> place, >> > > this >> > > >>>> functionality : >> > > >>>> /** >> > > >>>> * Executes given task within the cluster group. For step-by-step >> > > >>>> explanation of task execution
Re[2]: [DISCUSSION] Request for thread unsafe Compute functionality deprecation.
Ilya, as previously agreed, ticket [1], examples of concurrent tests you can find here GridDifferentLocalDeploymentSelfTest and here IgniteExplicitImplicitDeploymentSelfTest , TC in progress. thanks ! [1] https://issues.apache.org/jira/browse/IGNITE-14131 >Hello! > >Please publish it. I don't see why not. > >Regards, --
Re[4]: [DISCUSSION] Fail on non-colocated join
I think it`s « absolutely needed » case ) Many people will find erroneous places. >Hello! > >Many people do read logs, especially developers. You would be amazed how >many people come to discuss even the most benign of warnings. > >I think it violates the Apache Ignite compatibility policy, where we do not >break existing code during a major release. > >We do not always hold on to that principle, for example Baseline Topology >introduced some changes which needed to be explicitly handled by the user. >But in this case, it looks like a violation. > >https://cwiki.apache.org/confluence/display/IGNITE/Review+Checklist > >It's checklist, 1b. > >Regards, >-- >Ilya Kasnacheev > > >ср, 3 февр. 2021 г. в 14:48, Zhenya Stanilovsky < arzamas...@mail.ru.invalid >>: > >> >> >> >> >> >If it breaks existing working code it may not be done that way. >> >> Who reads the logs ? >> Is it violates apache way approach or some existing rules ? >> >> thanks ! >> >> >> >Regards, >> >-- >> >Ilya Kasnacheev >> > >> > >> >ср, 3 февр. 2021 г. в 09:05, Zhenya Stanilovsky < >> arzamas...@mail.ru.invalid >> >>: >> > >> >> >> >> >> >> Maxim it`s cool that it`s moved :) >> >> +1 for exception, but take into account such use case : >> >> T1 (country, city) affinity_key=country and T2 (country,city) >> >> affinity_key=country join with «city» usage — will be correct here (i >> hope, >> >> need to recheck of course) thus seems you must give some flag\hint what >> >> ever to run such reqs. >> >> >> >> thanks ! >> >> >> >> >Hi, Igniters! >> >> > >> >> >Last week I investigated a bug [1]. It's about an incorrect result for >> >> >non-colocated joins. For such joins it's required to set up the >> >> >"distributedJoin" flag, or try to make joined tables colocated. It is >> >> >covered in docs [2]. But it's not obvious and some users don't read >> that >> >> or >> >> >forget about that. In result there are wrong results for some queries >> that >> >> >are pretty hard to debug. >> >> > >> >> >There is a ticket [3] with a comment, where it's suggested to add a >> check >> >> >for such joins. I tried to implement it and found a place where it's >> >> >possible to put this check. But there is an open question what this >> check >> >> >should do. Currently I see 2 ways for that: >> >> >1. Forbid non-colocated joins that aren't marked with the >> distributedJoin >> >> >flag, and throw an exception. >> >> >2. Check every query for such joins and implicitly setup a >> distributedJoin >> >> >flag for them. >> >> > >> >> >Both solutions may break compatibility, but is this compatibility OK? >> >> > >> >> >Igniters, what do you think? >> >> > >> >> >[1] https://issues.apache.org/jira/browse/IGNITE-12847 >> >> >[2] >> >> > >> >> >> >> https://ignite.apache.org/docs/latest/SQL/distributed-joins#distributed-joins >> >> >[3] https://issues.apache.org/jira/browse/IGNITE-13019 >> >> >> >> >> >> >> >> >> >> >> >>
Re[2]: [DISCUSSION] Fail on non-colocated join
>If it breaks existing working code it may not be done that way. Who reads the logs ? Is it violates apache way approach or some existing rules ? thanks ! >Regards, >-- >Ilya Kasnacheev > > >ср, 3 февр. 2021 г. в 09:05, Zhenya Stanilovsky < arzamas...@mail.ru.invalid >>: > >> >> >> Maxim it`s cool that it`s moved :) >> +1 for exception, but take into account such use case : >> T1 (country, city) affinity_key=country and T2 (country,city) >> affinity_key=country join with «city» usage — will be correct here (i hope, >> need to recheck of course) thus seems you must give some flag\hint what >> ever to run such reqs. >> >> thanks ! >> >> >Hi, Igniters! >> > >> >Last week I investigated a bug [1]. It's about an incorrect result for >> >non-colocated joins. For such joins it's required to set up the >> >"distributedJoin" flag, or try to make joined tables colocated. It is >> >covered in docs [2]. But it's not obvious and some users don't read that >> or >> >forget about that. In result there are wrong results for some queries that >> >are pretty hard to debug. >> > >> >There is a ticket [3] with a comment, where it's suggested to add a check >> >for such joins. I tried to implement it and found a place where it's >> >possible to put this check. But there is an open question what this check >> >should do. Currently I see 2 ways for that: >> >1. Forbid non-colocated joins that aren't marked with the distributedJoin >> >flag, and throw an exception. >> >2. Check every query for such joins and implicitly setup a distributedJoin >> >flag for them. >> > >> >Both solutions may break compatibility, but is this compatibility OK? >> > >> >Igniters, what do you think? >> > >> >[1] https://issues.apache.org/jira/browse/IGNITE-12847 >> >[2] >> > >> >> https://ignite.apache.org/docs/latest/SQL/distributed-joins#distributed-joins >> >[3] https://issues.apache.org/jira/browse/IGNITE-13019 >> >> >> >>
Re: [DISCUSSION] Fail on non-colocated join
Maxim it`s cool that it`s moved :) +1 for exception, but take into account such use case : T1 (country, city) affinity_key=country and T2 (country,city) affinity_key=country join with «city» usage — will be correct here (i hope, need to recheck of course) thus seems you must give some flag\hint what ever to run such reqs. thanks ! >Hi, Igniters! > >Last week I investigated a bug [1]. It's about an incorrect result for >non-colocated joins. For such joins it's required to set up the >"distributedJoin" flag, or try to make joined tables colocated. It is >covered in docs [2]. But it's not obvious and some users don't read that or >forget about that. In result there are wrong results for some queries that >are pretty hard to debug. > >There is a ticket [3] with a comment, where it's suggested to add a check >for such joins. I tried to implement it and found a place where it's >possible to put this check. But there is an open question what this check >should do. Currently I see 2 ways for that: >1. Forbid non-colocated joins that aren't marked with the distributedJoin >flag, and throw an exception. >2. Check every query for such joins and implicitly setup a distributedJoin >flag for them. > >Both solutions may break compatibility, but is this compatibility OK? > >Igniters, what do you think? > >[1] https://issues.apache.org/jira/browse/IGNITE-12847 >[2] >https://ignite.apache.org/docs/latest/SQL/distributed-joins#distributed-joins >[3] https://issues.apache.org/jira/browse/IGNITE-13019
Re[2]: [DISCUSSION] Request for thread unsafe Compute functionality deprecation.
Hi Ilya , of course it contains in my PR (i don`t know if it shout be published
before this discussion will be finished).
Little changes from single thread into multiple, for example here [1] will
highlight a problem, or i can just publish my PR.
[1]
https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/IgniteExplicitImplicitDeploymentSelfTest.java#L221
>
>>
>>>Hello!
>>>
>>>Do you have some kind of reproducer which demonstrates the issue?
>>>
>>>Regards,
>>>--
>>>Ilya Kasnacheev
>>>
>>>
>>>чт, 28 янв. 2021 г. в 10:32, Zhenya Stanilovsky < arzamas...@mail.ru.invalid
>>>>:
>>>
>>>>
>>>> Hello Igniters !
>>>> In the process of Ignite usage i found that some part of Compute
>>>> functionality are thread unsafe and seems was designed with such
>>>> limitations initially.
>>>> Example : one (client, but it doesn`t matter at all) instance is
>>>> shared between numerous of fabric, all of them calls something like :
>>>> IgniteCompute#execute(ComputeTask, T)
>>>> or
>>>> IgniteCompute#execute(java.lang.Class>, T)
>>>> and appropriate «async» methods — what kind of instance will be called is
>>>> nondeterministic for now and as a confirmation of my words — i found no
>>>> tests covered multi thread usage of Computing i also found nothing on
>>>> documentation page [1].
>>>> We have all necessary info for correct processing of such cases:
>>>> from initiator (ignite.compute(...) starter) side we have Class or it
>>>> instance and appropriate class loader which will be wired by class loader
>>>> id from execution side.
>>>> I create a fix and seems all work perfectly well besides one place, this
>>>> functionality :
>>>> /**
>>>> * Executes given task within the cluster group. For step-by-step
>>>> explanation of task execution process
>>>> * refer to {@link ComputeTask} documentation.
>>>> *
>>>> * If task for given name has not been deployed yet, then {@code taskName}
>>>> will be
>>>> * used as task class name to auto-deploy the task (see {@link
>>>> #localDeployTask(Class, ClassLoader)} method).
>>>> */
>>>> public R execute(String taskName, T arg) throws IgniteException;
>>>> and attendant
>>>> /**
>>>> * Finds class loader for the given class.
>>>> *
>>>> * @param rsrcName Class name or class alias to find class loader for.
>>>> * @return Deployed class loader, or {@code null} if not deployed.
>>>> */
>>>> public DeploymentResource findResource(String rsrcName);
>>>> is thread unsafe by default, no guarantee that concurrent call of
>>>> localDeployTask and execute will bring expected result.
>>>> My proposal is to deprecate (or probably annotate [2], as a minimal
>>>> — additionally document it) this methods and to append additional :
>>>> public DeploymentResource findResource(String rsrcName, ClassLoader
>>>> clsLdr);
>>>> Only one problem i can observe here, if someone creates new class loaders
>>>> and appropriate class instances in loop (i don`t know the purpose) and
>>>> doesn`t undeploy them then he will get possibly OOM here.
>>>>
>>>> Such approach will give a possibility to use compute in concurrent
>>>> scenario. If there is no objections here i will mark this methods and
>>>> publish my PR, of course with additional tests.
>>>>
>>>> What do you think ?
>>>>
>>>>
>>>> [1]
>>>> https://ignite.apache.org/docs/latest/code-deployment/peer-class-loading
>>>> [2]
>>>> https://jcip.net/annotations/doc/net/jcip/annotations/NotThreadSafe.html
>>>>
>>>>
>>
>>
>>
>>
[DISCUSSION] Request for thread unsafe Compute functionality deprecation.
Hello Igniters !
In the process of Ignite usage i found that some part of Compute functionality
are thread unsafe and seems was designed with such limitations initially.
Example : one (client, but it doesn`t matter at all) instance is shared between
numerous of fabric, all of them calls something like :
IgniteCompute#execute(ComputeTask, T)
or
IgniteCompute#execute(java.lang.Class>, T)
and appropriate «async» methods — what kind of instance will be called is
nondeterministic for now and as a confirmation of my words — i found no tests
covered multi thread usage of Computing i also found nothing on documentation
page [1].
We have all necessary info for correct processing of such cases:
from initiator (ignite.compute(...) starter) side we have Class or it instance
and appropriate class loader which will be wired by class loader id from
execution side.
I create a fix and seems all work perfectly well besides one place, this
functionality :
/**
* Executes given task within the cluster group. For step-by-step explanation
of task execution process
* refer to {@link ComputeTask} documentation.
*
* If task for given name has not been deployed yet, then {@code taskName} will
be
* used as task class name to auto-deploy the task (see {@link
#localDeployTask(Class, ClassLoader)} method).
*/
public R execute(String taskName, T arg) throws IgniteException;
and attendant
/**
* Finds class loader for the given class.
*
* @param rsrcName Class name or class alias to find class loader for.
* @return Deployed class loader, or {@code null} if not deployed.
*/
public DeploymentResource findResource(String rsrcName);
is thread unsafe by default, no guarantee that concurrent call of
localDeployTask and execute will bring expected result.
My proposal is to deprecate (or probably annotate [2], as a minimal —
additionally document it) this methods and to append additional :
public DeploymentResource findResource(String rsrcName, ClassLoader clsLdr);
Only one problem i can observe here, if someone creates new class loaders and
appropriate class instances in loop (i don`t know the purpose) and doesn`t
undeploy them then he will get possibly OOM here.
Such approach will give a possibility to use compute in concurrent scenario. If
there is no objections here i will mark this methods and publish my PR, of
course with additional tests.
What do you think ?
[1] https://ignite.apache.org/docs/latest/code-deployment/peer-class-loading
[2] https://jcip.net/annotations/doc/net/jcip/annotations/NotThreadSafe.html
Re: Hard limit WAL archive size
Hello ! this is unclear for me, all you described near brings no info why node work improperly and why FH can possibly fail this node. Can you explain ? >Hello, everyone! > >Currently, property DataStorageConfiguration#maxWalArchiveSize is not working >as expected by users. We can easily go beyond this limit and overflow the >disk, which will lead to errors and a crash of the node. I propose to fix this >behavior and not let WAL archive overflow. > >It is suggested not to add segments to the archive if we can exceed the >DataStorageConfiguration#maxWalArchiveSize and wait until space becomes >available for this. > >Thus, we may have a deadlock: >Get checkpontReadLock -> write to WAL -> need to rollover WAL segment -> need >to clean WAL archive -> need to complete checkpoint (impossible because of >checkpontReadLock taken). > >To avoid such situations, I suggest adding a custom heuristic - do not give a >IgniteCacheDatabaseSharedManager#checkpointReadLock if there are few (default >1) segments left. >But this will not allow us to completely avoid archive overflow situations. >Therefore, I suggest fail node by FH when a deadlock is detected, since it >could be the same if there was no disk space left.
Re[2]: [DISCUSSION] Modules organization in Ignite 3
Hello Nikolay, if i find out introduced features structure in some project, i would prefer to choose different one ) > >> >>>Hello, Alexey. >>> >>>Think we can extend our @IgniteExperimental annotation. >>> >>>`@IgniteExperimental` - mark features that are truly experimental and can be >>>completely removed in future releases. >>>`@NotRecommended` - mark features that widely adopted by the users but >>>implemented wrong or have known issues that can’t be fixed. >>>`@NotStable` - mark features supported by community but API not stable and >>>can be reworked in the next release. >>>`@Stable` - mark features that are completely OK and here to stay. >>> >>>We should output notes about these annotations in the JavaDoc, also. >>>What do you think? >>> >>> 8 дек. 2020 г., в 12:49, Alexey Goncharuk < alexey.goncha...@gmail.com > написал(а): Igniters, I want to tackle the topic of modules structure in Ignite 3. So far, the modules in Ignite are mostly defined intuitively which leads to some complications: - Ignite public API is separated from the rest of the code only by package name. This leads to private classes leaking to public API which is very hard to catch even during the review process (we missed a bunch of such leaks for new metrics API [1] and I remember this happening for almost every SPI) - Classes from 'internal' packages are considered to be 'free for grabs' in every place of the code. This leads to tight coupling and abstraction leakage in the code. An example of such a case - an often cast of WALPointer to FileWALPointer, so that the community decided to get rid of the WALPointer interface altogether [2] - Overall code complexity. Because of the lack of inter-module interaction rules, we are free to add new methods and callbacks to any class, which leads to duplicating entities and verbose interfaces. A good example of this is the clear duplication of methods in IgniteCacheOffheapManager and IgniteCacheOffheapManager.DataStore [3] I think we need to work out some rules that will help us define and control both Ignite public API and module internal API which still defines a clear contract for other modules. Some ideas: - Perhaps we can move all user public classed and interfaces to an Ignite-API module which will have no dependencies on implementation modules. This will prevent private classes from leaking to the API module. - We need somehow define which classes from a module are exposed to other modules, and which classes are left for module-private usage. Maybe Java's jigsaw will help us here, but maybe we will be ok with just more strict java access modifiers usage :) The idea here is that a module should never touch a dependent module's private classes, ever. The exported classes and interfaces are still free to be modified between releases, as long as it is not a user public API. - A module should be logically complete, thus it may be beneficial if module name matches with the code package it provides (e.g. configuration -> org.apache.ignite.configuration, replication -> org.apache.ignite.replication, raft->org.apache.ignite.raft, etc) Any other principles/rules we can apply to make the code structure more concise? Thoughts? --AG [1] https://issues.apache.org/jira/browse/IGNITE-12552 [2] https://issues.apache.org/jira/browse/IGNITE-13513 [3] https://issues.apache.org/jira/browse/IGNITE-13220 >> >> >> >>
Re[2]: 2.9.1 release scope and dates
hello ! seems it [1] need to be included too, if it`s not too late, of course. May be you can bump reviewer somehow?) [1] https://issues.apache.org/jira/browse/IGNITE-13765 >Ivan, it's added, thanks for pointing that out > >On Mon, Nov 30, 2020 at 5:44 PM Ivan Daschinsky < ivanda...@gmail.com > wrote: > >> Yaroslav, could you explain why you decided to remove from scope >> https://issues.apache.org/jira/browse/IGNITE-13699 ? >> This ticket also incorporate few fixes for metrics. Currently metrics is a >> little bit broken (JoinedCount shows invalid results for example) >> >> >> пн, 30 нояб. 2020 г. в 17:26, Yaroslav Molochkov < molochko...@gmail.com >: >> >> > Igniters, hello! >> > >> > First of all, sorry that the release process is taking so long. >> > >> > Secondly, the release build seems stable and no blockers were introduced >> > within that list >> > < >> > >> >> https://issues.apache.org/jira/browse/IGNITE-11312?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.9.1%20and%20status%20%3D%20Resolved >> > > >> > (at >> > least on RunAll compared to 2.9) >> > >> > I've also prepared release notes: >> > >> > Ignite Core: >> > * Added support to graceful shutdown for ZookeeperDiscoverySpi >> > * Added System view for binary metadata >> > * Added System view for metastorage items >> > * Added RebalancingPartitionsTotal metrics >> > * Improved checkpoint concurrent behaviour >> > * Fixed critical system error when unregister a JMX bean >> > * Fixed IgniteCache#isClosed() returns false on server node even if >> > the cache had been closed before >> > * Fixed inability to eagerly remove expired cache entries >> > * Fixed partial index rebuild issue in case indexed cache contains >> > different datatypes >> > * Fixed local metastorage system view error if unmarshallable values >> > present >> > * Fixed deadlock between grid-timeout-worker and a thread opening a >> > communication connection >> > * Fixed deadlock in IgniteServiceProcessor >> > * Fixed issue when rebalance future might hang in no final state >> > though all partitions are owned >> > * Fixed issue when scan query fails with an assertion error: Unexpected >> > row key >> > * Fixed issue with archiving and enabled wal compaction setting on >> > server restart >> > * Fixed NPE during Cassandra Store initialization with PRIMITIVE strategy >> > * Fixed synchronization problems when different classloaders are used >> > for deployment of same class >> > * Fixed exception on SQL caches when client reconnect >> > * Fixed deadlock on multiple cache delete >> > * Fixed NPE in IgniteServiceProcessor when destroying a cache >> > * Fixed issue when DurableBackgroundTask can abandon incomplete task >> > * Fixed issue related to cache interceptor deserialization on client >> nodes >> > * Fixed issue when control.sh doesn't start if JMX port was set >> > * Fixed issue when ZookeeperDiscoverySpiMBeanImpl#getCoordinator can >> > return invalid node as coordinator >> > * Fixed issue when valid blocking section in GridNioWorker and >> > GridNioClientWorker leads to false positive blocking thread detection >> > * Fixed several logging issues >> > * Fixed issue when exchange worker, waiting for new task from queue, >> > considered as blocked >> > * Fixed incorrect topology snapshot logger output about coordinator >> change >> > * Fixed slowdown during node initialization >> > * Fixed incorrect usage of Class.isAssignableFrom in SystemViewLocal >> > and SystemViewMBean classes >> > * Removed unnecessary dependency to curator-client from and improved >> > ZookeeperDiscoverySpi >> > * Removed unnecessary failure trace in IgnitionEx >> > >> > Java thin-client: >> > * Fixed issue when thin client connect/disconnect during topology >> > update may lead to partition divergence in ignite-sys-cache >> > * Fixed issue when thin client silently closes channel after inactivity >> > * Fixed unsupported protocol version exception when getting cache >> > configuration from Java thin client >> > >> > JDBC: >> > * Fixed thin driver reports incorrect property names >> > * Updated JDBC metadata to match actual capabilities >> > >> > Ignite .NET: >> > * Improved slow Enum serialization >> > * Fixed issue when deserializing IBinaryObject containing an >> > IBinaryObject field fails >> > * Fixed issue TransactionImpl finalizer can crash the process >> > * Fixed child processes become zombies when persistence is used with >> > direct-io on Linux >> > >> > Ignite C++: >> > * Fixed issue when odbc-example loses values if run with 1 additional >> node >> > * Added Windows support to CMake build system >> > >> > >> > Thoughts? >> > >> > >> > >> > >> > On Thu, Nov 26, 2020 at 11:55 PM Yaroslav Molochkov < >> molochko...@gmail.com >> > > >> > wrote: >> > >> > > Maxim, >> > > >> > > Yeah, I guess I did all I could with my access rights, all there’s to >> do >> > > is to merge my PR in the branch and publish necessary artifacts >> > > >> > > And, of course, sorry that I’m a bit
Re[2]: [DISCUSS] Page replacement improvement
Nikolay, i hope such case ignite users already observed) I suggest to: put data bigger then available, full scan, get data only for available inmem data in loop, check PageReplacement metric, how match iterations will bring to zero this metric. >Hello, Alex. > >> Perhaps we can implement a special benchmark for this case with manually >> triggered "batch page replacement" using yardstick (but I'm not sure if >> ducktape can help here). > >I think we should carefully describe the issues with the current approach and >then we can choose right tool to benchmark improvements. >You said: > >> we use Random-LRU algorithm … it has many disadvantages and affects >> performance very much when replacement is started > >Can you list disadvantages of the Random-LRU? > >AFAIU the first benchmark should be: > >1. Start cluster with persistence and put data bigger then available RAM to it. >2. Measure performance of the queries that selects one entry from the cache. >3. Make some queries that will iterate throw all data - `SELECT SUM(x) FROM t` >or something similar. >4. Repeat step 2. in this time performance of random queries should be lower >due to the page replacement. > >Is this scenario correct? > >> 23 нояб. 2020 г., в 09:12, Alex Plehanov < plehanov.a...@gmail.com > >> написал(а): >> >> Nikolay, Zhenya, >> >> Benchmark from IGNITE-13034 is fully synthetic, it makes random puts >> uniformly. It can be used to compare different page replacement algorithms >> (random-LRU vs segmented-LRU), but it is totally inapplicable to benchmark >> batch page replacement. >> Perhaps we can implement a special benchmark for this case with manually >> triggered "batch page replacement" using yardstick (but I'm not sure >> if ducktape can help here). >> >>> I understand case you described, but who will pull the switch ? Human, >> artificial intelligence ? >> As I described before, we can implement both manual and automatic "batch >> page replacement" triggering. For automatic triggering, there is no >> artificial intelligence needed, just several conditions with reasonable >> thresholds. Automatic triggering also can be disabled by default. >> >> пт, 20 нояб. 2020 г. в 13:32, Zhenya Stanilovsky < arzamas...@mail.ru.invalid >>> : >> >>> >>> >>> >>> >>> >>> >>>> Zhenya, >>>> >>>>> Alexey, we already have changes that partially fixes this issue [1] >>>> IGNITE-13086 it's a minor improvement. We still have major problems with >>>> our page replacement algorithm (slow page selection and non-optimal >>>> page-fault rate). I think changing from random 5 pages to 7 will make >>>> things even worse (it's better for page-fault rate, but page selection >>> will >>>> be slower). >>> All this words above need to be proven, i hope. + 1 with Nikolay, we need >>> correct reproduces or some graphs from 2.9 ver. >>> >>>> >>>>> This approach still not applicable for real life >>>> Why do you think batch replacement is not applicable for real-life? It can >>>> be applied for workloads, where some big amount of data periodically used, >>>> but not very often. For example, when OLAP request over historical data >>>> raised pages to page-memory, and after such request this data is not >>> needed >>>> for a long time. Or when OLTP transactions mostly add new data and process >>>> recent data but rarely touch historical data. In these cases with the >>>> current approach, we will enter "page replacement mode" after some period >>>> of time and never leave it. With batch page replacement there is a chance >>>> to prevent random-LRU page replacement or postpone it. >>> I understand case you described, but who will pull the switch ? Human, >>> artificial intelligence ? >>> You approach assume some triggering from inner, i don`t like this. >>> >>>> >>>>> But request once more, do you really observe such problems with 2.9 ver >>> ? >>>> Any graphs maybe ? >>>> I don't have production usage feedback after IGNITE-13086, but I doubt >>>> something changed significantly. >>> >>> Lets wait ?:) In any case (Nikolay, Alex) IGNITE-13086 includes yardstik >>> bench for PR proven, we can use it once more. >>> >>> Thanks ! >>>> >>>> >>>> чт, 19 нояб. 2020
Re[2]: [DISCUSS] Page replacement improvement
>Zhenya, > >> Alexey, we already have changes that partially fixes this issue [1] >IGNITE-13086 it's a minor improvement. We still have major problems with >our page replacement algorithm (slow page selection and non-optimal >page-fault rate). I think changing from random 5 pages to 7 will make >things even worse (it's better for page-fault rate, but page selection will >be slower). All this words above need to be proven, i hope. + 1 with Nikolay, we need correct reproduces or some graphs from 2.9 ver. > >> This approach still not applicable for real life >Why do you think batch replacement is not applicable for real-life? It can >be applied for workloads, where some big amount of data periodically used, >but not very often. For example, when OLAP request over historical data >raised pages to page-memory, and after such request this data is not needed >for a long time. Or when OLTP transactions mostly add new data and process >recent data but rarely touch historical data. In these cases with the >current approach, we will enter "page replacement mode" after some period >of time and never leave it. With batch page replacement there is a chance >to prevent random-LRU page replacement or postpone it. I understand case you described, but who will pull the switch ? Human, artificial intelligence ? You approach assume some triggering from inner, i don`t like this. > >> But request once more, do you really observe such problems with 2.9 ver ? >Any graphs maybe ? >I don't have production usage feedback after IGNITE-13086, but I doubt >something changed significantly. Lets wait ?:) In any case (Nikolay, Alex) IGNITE-13086 includes yardstik bench for PR proven, we can use it once more. Thanks ! > > >чт, 19 нояб. 2020 г. в 09:18, Zhenya Stanilovsky < arzamas...@mail.ru.invalid >>: > >> >> Alexey, we already have changes that partially fixes this issue [1] >> Easy way: >> Looks like we already have converge in page replacement. >> If we change 5 times touch iterator from random lru algo into, for >> example — 7 we will obtain fast improvement from scratch. >> >> » Batch page replacement >> This approach still not applicable for real life if you wan`t to observe >> ugly people for threshold (i.e. 12 h) interval. And, of course, you >> understand that dramatically reduce of such interval gives nothing? >> >> » Change the page replacement algorithm. >> That`s way i vote for ) But request once more, do you really observe such >> problems with 2.9 ver ? Any graphs maybe ? >> >> thanks ! >> >> [1] https://issues.apache.org/jira/browse/IGNITE-13086 >> >Hello, Igniters! >> > >> >Currently, for page replacement (page rotation between page-memory and >> >disk) we use Random-LRU algorithm. It has a low maintenance cost and >> >relatively simple implementation, but it has many disadvantages and >> affects >> >performance very much when replacement is started. We even have warnings >> in >> >the log when page replacement started and a special event for this. I know >> >Ignite deployments where administrators force to restart cluster nodes >> >periodically to avoid page replacement. >> > >> >I have a couple of proposals to improve page replacement in Ignite: >> > >> >*Batch page replacement.* >> > >> >Main idea: in some cases start background task to evict cold pages from >> >page-memory (for example, pages, last touched more than 12 hours ago). >> > >> >The task can be started: >> >- Automatically, triggered by some events, for example, when we expect a >> >start of Random-LRU page replacing soon (allocated more than 90% of >> >page-memory) + we have enough amount of cold pages (we need some metric to >> >calculate the number of cold pages) + some time passed since last batch >> >page replacement (to avoid too much resource consumption by background >> >batch replacement). >> >- Manually (JMX or control.sh), if an administrator wants to control the >> >time of batch replacement more precisely (for example, to avoid the start >> >of this task during peak time). >> > >> >Batch page replacement will be helpful in some workloads (when some data >> >much colder than another), it can prevent the starting of Random-LRU page >> >replacement, or if Random-LRU already started it can provide conditions to >> >stop it. >> > >> >*Change the page replacement algorithm.* >> > >> >Good page replacement algorithm should satisfy the requirements: >> >- low page-fa
[DISCUSS] Page replacement improvement
Alexey, we already have changes that partially fixes this issue [1] Easy way: Looks like we already have converge in page replacement. If we change 5 times touch iterator from random lru algo into, for example — 7 we will obtain fast improvement from scratch. » Batch page replacement This approach still not applicable for real life if you wan`t to observe ugly people for threshold (i.e. 12 h) interval. And, of course, you understand that dramatically reduce of such interval gives nothing? » Change the page replacement algorithm. That`s way i vote for ) But request once more, do you really observe such problems with 2.9 ver ? Any graphs maybe ? thanks ! [1] https://issues.apache.org/jira/browse/IGNITE-13086 >Hello, Igniters! > >Currently, for page replacement (page rotation between page-memory and >disk) we use Random-LRU algorithm. It has a low maintenance cost and >relatively simple implementation, but it has many disadvantages and affects >performance very much when replacement is started. We even have warnings in >the log when page replacement started and a special event for this. I know >Ignite deployments where administrators force to restart cluster nodes >periodically to avoid page replacement. > >I have a couple of proposals to improve page replacement in Ignite: > >*Batch page replacement.* > >Main idea: in some cases start background task to evict cold pages from >page-memory (for example, pages, last touched more than 12 hours ago). > >The task can be started: >- Automatically, triggered by some events, for example, when we expect a >start of Random-LRU page replacing soon (allocated more than 90% of >page-memory) + we have enough amount of cold pages (we need some metric to >calculate the number of cold pages) + some time passed since last batch >page replacement (to avoid too much resource consumption by background >batch replacement). >- Manually (JMX or control.sh), if an administrator wants to control the >time of batch replacement more precisely (for example, to avoid the start >of this task during peak time). > >Batch page replacement will be helpful in some workloads (when some data >much colder than another), it can prevent the starting of Random-LRU page >replacement, or if Random-LRU already started it can provide conditions to >stop it. > >*Change the page replacement algorithm.* > >Good page replacement algorithm should satisfy the requirements: >- low page-fault rates for typical workload >- low maintenance cost (low resource consumption to maintain additional >structures required for page replacement) >- fast searching of next page for replacement >- sequential scans resistance (one sequential scan should not evict all >relatively hot pages from page-memory) > >Our Random-LRU has low maintenance cost and sequential scan resistant, but >to find the next page for replacement in the best case we scan 5 pages, in >the worst case we can scan all data region segment. Also, due to random >nature, it's not very effective in predicting the right page for >replacement to minimize the page-fault rate. And it's much time required to >totally evict old cold data. > >Usually, database management systems and operating systems use >modifications of LRU algorithms. These algorithms have higher maintenance >costs (pages list should be modified on each page access), but often they >are effective from a "page-fault rate" point of view and have O(1) >complexity for a searching page to replace. Simple LRU is not sequential >scan resistant, but modifications that utilize page access frequency are >resistant to sequential scan. > >We can try one of the modifications of LRU as well (for example, "segmented >LRU" seems suitable for Ignite). > >Ignite is a memory-centric product, so "low maintenance cost" is very >critical. And there is a risk that page replacement algorithm can affect >workloads, where page replacement is not used (enough RAM to store all >data). Of course, any page replacement solution should be carefully >benchmarked. > > >Igniters, WDYT? If any of these proposals look reasonable to you, I will >create IEP and start implementation. > >Also, I have a draft implementation of system view to determine how hot are >pages in page-memory [1]. I think it will be useful for any of these >approaches (and even if we decide to left page replacement as is). > >[1]: https://issues.apache.org/jira/browse/IGNITE-13726 >
Re[2]: IEP-58: Statistics
Andrey, thanks for firing this ! Sasha it`s unclear for me « These part consists of two processes: statistics collection process itself and acquiring statistics by the client. »: * I agree that in both cases local statistics are useless. May be we need more informative use cases for such statistics usage ? Can someone append additional columns (possible not presented in index) statistics? * Client — can you unfold this term ? If this means — ignite client node ? Does sql best plan is chosen in request starter node ? If so — what about this client with limited cpu here? * « If there are no statistics in all of them - client will choose random » — not random but affinity concerted isn`t it ? * « After getting statistics client will cache it and server node it to renew statistics from same node. » I don`t understand this approach, can you clarify it plz ? * Whats the storage mechanism for client node statistics? * Can we use thin client without discs in such cases? thanks ! >: > >Follow up > >Igniters, > >is there any comment to this IEP? > >JFYI, IEP is renamed and placed here [1] > >[1] >https://cwiki.apache.org/confluence/display/IGNITE/IEP-58%3A+Statistics+for+SQL+query+optimization > >On Thu, Sep 24, 2020 at 2:30 PM Sasha Belyak < rtsfo...@gmail.com > wrote: >> >> Igniters, >> I'e prepared an IEP [1], please review and let me know what you think. >> >> In particular, I'd like to discuss the new subsystem to collect statistics >> to optimize sql queries execution. >> [1] https://cwiki.apache.org/confluence/display/IGNITE/IEP-58+Statistics
Re[2]: [VOTE] Release Apache Ignite 2.9.0 RC4
ok, thus +1 from me, rebuild from sources. >Zhenya, > >It's not ok, but I think it's not a release blocker. >Sources can be compiled using instructions given in DEVNOTES.txt (there are no >requirements to enable checkstyle in our documentation). >But we can fix build scripts to check compilation with enabled checkstyle for >future releases. >Guys, WDYT? >чт, 15 окт. 2020 г. в 16:21, Zhenya Stanilovsky < arzamas...@mail.ru.invalid >: > >>build with checkstyle from sources is failed: >>Starting audit... >>[ERROR] >>/home/zstan/Downloads/apache-ignite-2.9.0-src/modules/indexing/src/test/java/org/apache/ignite/internal/processors/query/WrongQueryEntityFieldTypeTest.java:35:8: >> Unused import - org.apache.ignite.client.ClientException. [UnusedImports] >>Audit done. >> >>is it ok? >> >>>Dear Community, >>> >>> >>>I have uploaded a release candidate to: >>> https://dist.apache.org/repos/dist/dev/ignite/2.9.0-rc4/ >>> https://dist.apache.org/repos/dist/dev/ignite/packages_2.9.0-rc4/ >>> >>>The following staging can be used for testing: >>> https://repository.apache.org/content/repositories/orgapacheignite-1487 >>> >>>Tag name is 2.9.0-rc4: >>> https://gitbox.apache.org/repos/asf?p=ignite.git;a=commit;h=refs/tags/2.9.0-rc4 >>> >>>RELEASE_NOTES: >>> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.9 >>> >>>Complete list of resolved issues: >>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.9%20and%20status%20%3D%20Resolved%20and%20resolution%20%3D%20Fixed%20order%20by%20updated >>> >>>DEVNOTES: >>> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.9 >>> >>> >>>Additional checks have been performed (available for users included into >>>the release group on TeamCity). >>> >>>TC [Check RC: Licenses, compile, chksum] >>> https://ci.ignite.apache.org/viewLog.html?buildId=5666942=ApacheIgniteReleaseJava8_PrepareVote4CheckRcLicensesChecksum >>> >>>TC [3] Build & Upload Nuget Staging Packages >>> https://ci.ignite.apache.org/viewLog.html?buildId=5666940=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages >>> >>>TC [2] Compare w/ Previous Release >>> https://ci.ignite.apache.org/viewLog.html?buildId=5666938=ApacheIgniteReleaseJava8_IgniteRelease72CheckFileConsistency >>> >>> >>>The vote is formal, see voting guidelines >>> https://www.apache.org/foundation/voting.html >>> >>>+1 - to accept Apache Ignite 2.9.0-rc4 >>>0 - don't care either way >>>-1 - DO NOT accept Apache Ignite Ignite 2.9.0-rc4 (explain why) >>> >>>See notes on how to verify release here >>> https://www.apache.org/info/verification.html >>>and >>> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification >>> >>>This vote will be open at least until Tue Oct 20, 12:00 UTC >>> https://www.timeanddate.com/countdown/generic?iso=20201020T12=%3A=Apache+Ignite+2.9.0-rc4+Vote=1 >>> >>
Re: [VOTE] Release Apache Ignite 2.9.0 RC4
build with checkstyle from sources is failed: Starting audit... [ERROR] /home/zstan/Downloads/apache-ignite-2.9.0-src/modules/indexing/src/test/java/org/apache/ignite/internal/processors/query/WrongQueryEntityFieldTypeTest.java:35:8: Unused import - org.apache.ignite.client.ClientException. [UnusedImports] Audit done. is it ok? >Dear Community, > > >I have uploaded a release candidate to: >https://dist.apache.org/repos/dist/dev/ignite/2.9.0-rc4/ >https://dist.apache.org/repos/dist/dev/ignite/packages_2.9.0-rc4/ > >The following staging can be used for testing: >https://repository.apache.org/content/repositories/orgapacheignite-1487 > >Tag name is 2.9.0-rc4: >https://gitbox.apache.org/repos/asf?p=ignite.git;a=commit;h=refs/tags/2.9.0-rc4 > >RELEASE_NOTES: >https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.9 > >Complete list of resolved issues: >https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.9%20and%20status%20%3D%20Resolved%20and%20resolution%20%3D%20Fixed%20order%20by%20updated > >DEVNOTES: >https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.9 > > >Additional checks have been performed (available for users included into >the release group on TeamCity). > >TC [Check RC: Licenses, compile, chksum] >https://ci.ignite.apache.org/viewLog.html?buildId=5666942=ApacheIgniteReleaseJava8_PrepareVote4CheckRcLicensesChecksum > >TC [3] Build & Upload Nuget Staging Packages >https://ci.ignite.apache.org/viewLog.html?buildId=5666940=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages > >TC [2] Compare w/ Previous Release >https://ci.ignite.apache.org/viewLog.html?buildId=5666938=ApacheIgniteReleaseJava8_IgniteRelease72CheckFileConsistency > > >The vote is formal, see voting guidelines >https://www.apache.org/foundation/voting.html > >+1 - to accept Apache Ignite 2.9.0-rc4 >0 - don't care either way >-1 - DO NOT accept Apache Ignite Ignite 2.9.0-rc4 (explain why) > >See notes on how to verify release here >https://www.apache.org/info/verification.html >and >https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification > >This vote will be open at least until Tue Oct 20, 12:00 UTC >https://www.timeanddate.com/countdown/generic?iso=20201020T12=%3A=Apache+Ignite+2.9.0-rc4+Vote=1 >
Re[2]: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
without [2] and [3] we obtain unexpected fields in index creation and as a consequence buggy sql execution and pla of course. >Guys, > >I've found 3 more tickets which were targeted to 2.9 but merged only to the >master branch ([1], [2], [3]). >Do we need these tickets in 2.9? Are they critical? > >[1]: https://issues.apache.org/jira/browse/IGNITE-12350 >[2]: https://issues.apache.org/jira/browse/IGNITE-13376 >[3]: https://issues.apache.org/jira/browse/IGNITE-13280 > >вс, 11 окт. 2020 г. в 16:11, Nikolay Izhikov < nizhi...@apache.org >: > >> Igniters, >> >> IGNITE-13553 fixed and cherry-picked to 2.9. >> We can continue with the release. >> >> > 10 окт. 2020 г., в 00:00, Denis Magda < dma...@apache.org > написал(а): >> > >> > Nikolay, >> > >> > re: the minor improvements. I'm not against of including those if we can >> > prepare the docs before the vote starts. Presently, the docs are "frozen" >> > for the 2.9 release, but I can scratch some time and take part in the >> docs >> > review the next week. >> > >> > - >> > Denis >> > >> > >> > On Fri, Oct 9, 2020 at 12:40 AM Nikolay Izhikov < nizhi...@apache.org > >> wrote: >> > >> >> Hello, Igniters. >> >> >> >> I prepared a patch [1] for blocker ticket [2] - «Server node fail and >> >> stops in case wrong datatype put in indexed field» >> >> Can someone, please, help me with the review? >> >> >> >> [1] https://github.com/apache/ignite/pull/8330 >> >> [2] https://issues.apache.org/jira/browse/IGNITE-13553 >> >> >> >> == >> >> >> >> I propose to include several minor tickets to the scope of the 2.9 that >> >> increase Ignite User Experience >> >> >> >> CMD tools improvements: >> >> >> >> IGNITE-13488 - Command to print metric value >> >> IGNITE-13426 - Command to print system view content >> >> IGNITE-13422 - Parameter to explicitly enable experimental commands >> >> >> >> IGNITE-13380 - Output IgniteSystemProperties via ignite.sh >> >> >> >> New system views: >> >> >> >> IGNITE-13409 Metastorage and DistributedMetastorage viewы. >> >> IGNITE-13408 BinaryMetadata view. >> >> >> >>> 9 окт. 2020 г., в 04:04, Denis Magda < dma...@apache.org > написал(а): >> >>> >> >>> @Alex Plehanov < plehanov.a...@gmail.com >, >> >>> >> >>> If it still makes sense and not too late, you can cherry-pick the >> commit >> >>> with the new docs into the 2.9 branch: >> >>> >> >> >> >> https://github.com/apache/ignite/commit/073488ac97517bbaad9f6b94b781fc404646f191 >> >>> >> >>> Anyway, it's not crucial as long as we agreed to make an exception for >> >> this >> >>> release. The docs were already published to the website and assembled >> >> from >> >>> the master. Once the vote passes, I'll make them visible and traceable >> >> from >> >>> the website menus. >> >>> >> >>> - >> >>> Denis >> >>> >> >>> >> >>> On Tue, Oct 6, 2020 at 7:20 AM Alexey Goncharuk < >> >> alexey.goncha...@gmail.com > >> >>> wrote: >> >>> >> Saikat, >> >> The plan sounds good to me! Do you have an idea for the timeline of >> the >> module releases? Let me know if I can help in any way. >> >> Also, I was looking into the modularization IEP and noticed that OSGI >> module is discussed to be moved to the extensions, but there is no >> corresponding ticket. Should I create one? I will be happy to help >> with >> moving this module to extensions. >> >> вт, 29 сент. 2020 г. в 03:03, Saikat Maitra < saikat.mai...@gmail.com >> >: >> >> > Hi Alexey, >> > >> > All the modules as planned in IEP-36 >> > >> >> >> >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-36:+Modularization >> > are migrated to ignite-extensions repository. >> > >> > We can definitely plan to release ignite-extensions modules. >> > >> > I have a pending PR related to the kafka module and the PR is in >> review >> > https://github.com/apache/ignite/pull/8222 but its corresponding PR >> >> has >> > been merged in ignite-extensions repository. >> > >> > My thoughts / action items for the migration efforts and releases >> were >> > as follows: >> > >> > 1. Merge the changes for https://github.com/apache/ignite/pull/8222 >> > 2. Fix the upload nightly packages task for ignite-core to help fix >> the >> > travis build failure in ignite-extensions repository. >> > 3. Review and update the README.md files for the ignite-extensions >> modules >> > as required. >> > 4. Update the docs for ignite-extensions modules in ignite-website. >> > 5. Release each module separately and share updates. >> > >> > Please let me know if you have feedback. >> > >> > Regards, >> > Saikat >> > >> > >> > >> > >> > >> > >> > >> > >> > On Mon, Sep 28, 2020 at 7:36 AM Petr Ivanov < mr.wei...@gmail.com > >> >> wrote: >> > >> >> It seems that gitbox.apache.org is not available on CI/CD. >> >> >> >> I will try to update
Re[2]: Broken test in master: BasicIndexTest
Of course, i still have no ticket filled. As was discussed in private i will fill other ticket that can lead to potential problems. >Hi everyone, > >I believe I have a fix for this bug - >https://issues.apache.org/jira/browse/IGNITE-13500, So Zhenya you can leave >this problem to me. > >-- >Best regards, >Anton Kalashnikov > > > >09.10.2020, 10:19, "Sergey Chugunov" < sergey.chugu...@gmail.com >: >> Max, >> >> Thanks for spotting this, great catch! >> >> Zhenya, could you please file a ticket of at least Critical priority? >> >> On Fri, Oct 9, 2020 at 9:24 AM Zhenya Stanilovsky >> < arzamas...@mail.ru.invalid > wrote: >> >>> Thanks Maxim, the test is correct no need for removal. >>> I checked 2.9 too, but looks it all ok there. I will take a look. >>> >Hi, Igniters! >>> > >>> >I was discovering how indexes work and found a failed test. >>> >BasicIndexTest#testInlineSizeChange is broken in master and it's not a >>> >flaky case [1]. But it has been failing since 25/09 only. >>> > >>> >I discovered that it happened after the IGNITE-13207 ticket merged >>> >(Checkpointer code refactoring) [2]. I'm not sure about the expected >>> >behaviour of the inline index and how checkpointer affects it. But let's >>> >fix it if it is a bug or completely remove this test. >>> > >>> >[1] >>> > >>> >>> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6131871779633595667=%3Cdefault%3E=testDetails >>> > >>> >[2] https://issues.apache.org/jira/browse/IGNITE-13207 >>> >
Re: Broken test in master: BasicIndexTest
Thanks Maxim, the test is correct no need for removal. I checked 2.9 too, but looks it all ok there. I will take a look. >Hi, Igniters! > >I was discovering how indexes work and found a failed test. >BasicIndexTest#testInlineSizeChange is broken in master and it's not a >flaky case [1]. But it has been failing since 25/09 only. > >I discovered that it happened after the IGNITE-13207 ticket merged >(Checkpointer code refactoring) [2]. I'm not sure about the expected >behaviour of the inline index and how checkpointer affects it. But let's >fix it if it is a bug or completely remove this test. > >[1] >https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6131871779633595667=%3Cdefault%3E=testDetails > >[2] https://issues.apache.org/jira/browse/IGNITE-13207 >
Re[6]: Thin Client ping operation?
I understand now, thanks Pavel, initial discussion didn`t touch kuber theme ... >Вторник, 15 сентября 2020, 18:22 +03:00 от Pavel Tupitsyn >: > >Zhenya, sure, let me explain. > >Health checks are a common practice in modern deployments, quote [1]: >"Health probes can be used by container orchestrators and load balancers to >check an app's status. >For example, a container orchestrator may respond to a failing health check by >halting a rolling deployment or restarting a container. >A load balancer might react to an unhealthy app by routing traffic away from >the failing instance to a healthy instance." > >Kubernetes has various probes [2] to determine the pod status. > >So Ignite users need a proper mechanism to determine connectivity status of >the thin client >to integrate with frameworks and orchestrators. > >[1] https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks >[2] >https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ > >On Tue, Sep 15, 2020 at 4:36 PM Zhenya Stanilovsky < >arzamas...@mail.ru.invalid > wrote: > >>Pavel, i read whole thread, show me the reason why this functionality need to >>be external ? >> >>> >>> >>>Health checks are the primary use case. See linked user list thread. >>> >>>On Tue, Sep 15, 2020 at 12:26 PM Zhenya Stanilovsky >>>< arzamas...@mail.ru.invalid > wrote: >>> >>>> >>>> Whats the usage of such API ? Igor can you clarify please ? >>>> >>>> >Personally I believe that public API still can be helpful, as it gives >>>> user >>>> >an ability to check connection in the specific point in time, even if >>>> >automatic >>>> >ping is implemented (which is more complex and hard-to-maintain feature >>>> >by the way). >>>> > >>>> >Not sure there should be "ping" in API though, maybe something more like >>>> >client.checkConnection(); >>>> > >>>> >Best Regards, >>>> >Igor >>>> > >>>> > >>>> >On Mon, Sep 14, 2020 at 11:37 AM Alex Plehanov < plehanov.a...@gmail.com >>>> > >>>> >wrote: >>>> > >>>> >> Hello guys, >>>> >> >>>> >> We've already raised the question about ping requests here [1]. >>>> >> >>>> >> I'm not sure about public API, but at least we can have auto-ping as an >>>> >> internal mechanism. This will be helpful if the client doesn't send any >>>> new >>>> >> requests but only waits for server-side notifications (for example, if >>>> the >>>> >> client subscribed to CQ events). The client can't detect a connection >>>> lost >>>> >> until sending something to the server. Using periodic ping requests this >>>> >> problem can be solved. >>>> >> >>>> >> So, +1 to add ping to the protocol, +0 to expose it to public API. >>>> >> >>>> >> [1] >>>> >> >>>> >> >>>> >>>>http://apache-ignite-developers.2346864.n4.nabble.com/IEP-44-Thin-Client-Discovery-tp47129p47318.html >>>> >> >>>> >> пн, 14 сент. 2020 г. в 10:32, Pavel Tupitsyn < ptupit...@apache.org >: >>>> >> >>>> >> > Nikolay, >>>> >> > >>>> >> > See the discussion on the user list: >>>> >> > >>>> >> > 1. It is not immediately obvious which APIs perform server calls and >>>> >> which >>>> >> > don't. >>>> >> > 2. It is not clear which APIs can cause heavy resource usage on the >>>> >> server >>>> >> > side. >>>> >> > We don't want to stress servers by pinging them. >>>> >> > cache.size() is an example - it is tempting to use and seems to be >>>> >> > simple, but actually queries every server node in the cluster. >>>> >> > >>>> >> > > dedicated `ping` operation makes our API heavier >>>> >> > The operation is so trivial that I would not worry about increased >>>> >> > complexity or future maintenance. >>>> >> > >>>> >> > On Mon, Sep 14, 2020 at 10
Re[4]: Thin Client ping operation?
Pavel, i read whole thread, show me the reason why this functionality need to be external ? > > >Health checks are the primary use case. See linked user list thread. > >On Tue, Sep 15, 2020 at 12:26 PM Zhenya Stanilovsky >< arzamas...@mail.ru.invalid > wrote: > >> >> Whats the usage of such API ? Igor can you clarify please ? >> >> >Personally I believe that public API still can be helpful, as it gives >> user >> >an ability to check connection in the specific point in time, even if >> >automatic >> >ping is implemented (which is more complex and hard-to-maintain feature >> >by the way). >> > >> >Not sure there should be "ping" in API though, maybe something more like >> >client.checkConnection(); >> > >> >Best Regards, >> >Igor >> > >> > >> >On Mon, Sep 14, 2020 at 11:37 AM Alex Plehanov < plehanov.a...@gmail.com >> > >> >wrote: >> > >> >> Hello guys, >> >> >> >> We've already raised the question about ping requests here [1]. >> >> >> >> I'm not sure about public API, but at least we can have auto-ping as an >> >> internal mechanism. This will be helpful if the client doesn't send any >> new >> >> requests but only waits for server-side notifications (for example, if >> the >> >> client subscribed to CQ events). The client can't detect a connection >> lost >> >> until sending something to the server. Using periodic ping requests this >> >> problem can be solved. >> >> >> >> So, +1 to add ping to the protocol, +0 to expose it to public API. >> >> >> >> [1] >> >> >> >> >> >> http://apache-ignite-developers.2346864.n4.nabble.com/IEP-44-Thin-Client-Discovery-tp47129p47318.html >> >> >> >> пн, 14 сент. 2020 г. в 10:32, Pavel Tupitsyn < ptupit...@apache.org >: >> >> >> >> > Nikolay, >> >> > >> >> > See the discussion on the user list: >> >> > >> >> > 1. It is not immediately obvious which APIs perform server calls and >> >> which >> >> > don't. >> >> > 2. It is not clear which APIs can cause heavy resource usage on the >> >> server >> >> > side. >> >> > We don't want to stress servers by pinging them. >> >> > cache.size() is an example - it is tempting to use and seems to be >> >> > simple, but actually queries every server node in the cluster. >> >> > >> >> > > dedicated `ping` operation makes our API heavier >> >> > The operation is so trivial that I would not worry about increased >> >> > complexity or future maintenance. >> >> > >> >> > On Mon, Sep 14, 2020 at 10:17 AM Nikolay Izhikov < >> nizhi...@apache.org > >> >> > wrote: >> >> > >> >> > > Hello, Igor. >> >> > > >> >> > > On the other hand, dedicated `ping` operation makes our API heavier >> >> > > without adding new feature - >> >> > > We can do the same with the other part of the API. >> >> > > >> >> > > Moreover, response to the ping doesn’t mean that SQL or cache query >> can >> >> > be >> >> > > served. >> >> > > >> >> > > > 14 сент. 2020 г., в 10:08, Igor Sapego < isap...@apache.org > >> >> > написал(а): >> >> > > > >> >> > > > Николай, >> >> > > > >> >> > > > It looks a little bit hacky to me. I believe SQL drivers usually >> use >> >> > that >> >> > > > approach >> >> > > > as a workaround because there is no other common way to do that. >> >> > > > >> >> > > > Sure we can recommend users to use cache.size() or anything other >> >> > > > similar way >> >> > > > to ensure the connection is alive, but it still looks like a >> >> workaround >> >> > > to >> >> > > > me. >> >> > > > >> >> > > > Best Regards, >> >> > > > Igor >> >> > > > >> >> > > > >> >> > > > On Sun, Sep 13, 2020 at 10:16 PM Николай Ижиков < >> nizhi...@apache.org >> >> > >> >> > > wrote: >> >> > > > >> >> > > >> Hello, Pavel. >> >> > > >> >> >> > > >> SQL drivers usually use “SELECT 1” query to ensure connection is >> >> > alive. >> >> > > >> >> >> > > >> Can we use similar approach? >> >> > > >> >> >> > > >> Отправлено с iPhone >> >> > > >> >> >> > > >>> 13 сент. 2020 г., в 13:26, Pavel Tupitsyn < >> ptupit...@apache.org > >> >> > > >> написал(а): >> >> > > >>> >> >> > > >>> Igniters, >> >> > > >>> >> >> > > >>> There is a feature request for a thin client Ping operation on >> the >> >> > user >> >> > > >>> list [1]. >> >> > > >>> I think that is a good idea - IgniteClient.ping() will be a >> >> valuable >> >> > > >>> addition. >> >> > > >>> >> >> > > >>> Any objections? >> >> > > >>> >> >> > > >>> [1] >> >> > > >>> >> >> > > >> >> >> > > >> >> > >> >> >> >> http://apache-ignite-users.70518.x6.nabble.com/Feature-request-method-to-test-active-connection-in-Ignite-thin-client-td33985.html >> >> > > >> >> >> > > >> >> > > >> >> > >> >> >> >> >> >>
Re[2]: Thin Client ping operation?
Whats the usage of such API ? Igor can you clarify please ? >Personally I believe that public API still can be helpful, as it gives user >an ability to check connection in the specific point in time, even if >automatic >ping is implemented (which is more complex and hard-to-maintain feature >by the way). > >Not sure there should be "ping" in API though, maybe something more like >client.checkConnection(); > >Best Regards, >Igor > > >On Mon, Sep 14, 2020 at 11:37 AM Alex Plehanov < plehanov.a...@gmail.com > >wrote: > >> Hello guys, >> >> We've already raised the question about ping requests here [1]. >> >> I'm not sure about public API, but at least we can have auto-ping as an >> internal mechanism. This will be helpful if the client doesn't send any new >> requests but only waits for server-side notifications (for example, if the >> client subscribed to CQ events). The client can't detect a connection lost >> until sending something to the server. Using periodic ping requests this >> problem can be solved. >> >> So, +1 to add ping to the protocol, +0 to expose it to public API. >> >> [1] >> >> >> http://apache-ignite-developers.2346864.n4.nabble.com/IEP-44-Thin-Client-Discovery-tp47129p47318.html >> >> пн, 14 сент. 2020 г. в 10:32, Pavel Tupitsyn < ptupit...@apache.org >: >> >> > Nikolay, >> > >> > See the discussion on the user list: >> > >> > 1. It is not immediately obvious which APIs perform server calls and >> which >> > don't. >> > 2. It is not clear which APIs can cause heavy resource usage on the >> server >> > side. >> > We don't want to stress servers by pinging them. >> > cache.size() is an example - it is tempting to use and seems to be >> > simple, but actually queries every server node in the cluster. >> > >> > > dedicated `ping` operation makes our API heavier >> > The operation is so trivial that I would not worry about increased >> > complexity or future maintenance. >> > >> > On Mon, Sep 14, 2020 at 10:17 AM Nikolay Izhikov < nizhi...@apache.org > >> > wrote: >> > >> > > Hello, Igor. >> > > >> > > On the other hand, dedicated `ping` operation makes our API heavier >> > > without adding new feature - >> > > We can do the same with the other part of the API. >> > > >> > > Moreover, response to the ping doesn’t mean that SQL or cache query can >> > be >> > > served. >> > > >> > > > 14 сент. 2020 г., в 10:08, Igor Sapego < isap...@apache.org > >> > написал(а): >> > > > >> > > > Николай, >> > > > >> > > > It looks a little bit hacky to me. I believe SQL drivers usually use >> > that >> > > > approach >> > > > as a workaround because there is no other common way to do that. >> > > > >> > > > Sure we can recommend users to use cache.size() or anything other >> > > > similar way >> > > > to ensure the connection is alive, but it still looks like a >> workaround >> > > to >> > > > me. >> > > > >> > > > Best Regards, >> > > > Igor >> > > > >> > > > >> > > > On Sun, Sep 13, 2020 at 10:16 PM Николай Ижиков < nizhi...@apache.org >> > >> > > wrote: >> > > > >> > > >> Hello, Pavel. >> > > >> >> > > >> SQL drivers usually use “SELECT 1” query to ensure connection is >> > alive. >> > > >> >> > > >> Can we use similar approach? >> > > >> >> > > >> Отправлено с iPhone >> > > >> >> > > >>> 13 сент. 2020 г., в 13:26, Pavel Tupitsyn < ptupit...@apache.org > >> > > >> написал(а): >> > > >>> >> > > >>> Igniters, >> > > >>> >> > > >>> There is a feature request for a thin client Ping operation on the >> > user >> > > >>> list [1]. >> > > >>> I think that is a good idea - IgniteClient.ping() will be a >> valuable >> > > >>> addition. >> > > >>> >> > > >>> Any objections? >> > > >>> >> > > >>> [1] >> > > >>> >> > > >> >> > > >> > >> >> http://apache-ignite-users.70518.x6.nabble.com/Feature-request-method-to-test-active-connection-in-Ignite-thin-client-td33985.html >> > > >> >> > > >> > > >> > >>
Cpp thin client transactions support ready for review.
Thanks for all who participated in review. Cpp transactions functionality already in master. Appropriate wiki page [1] was updated. [1] https://cwiki.apache.org/confluence/display/IGNITE/Thin+clients+features > > >Great, I'll take a look. > >Best Regards, >Igor > > >On Mon, Aug 24, 2020 at 8:33 AM Zhenya Stanilovsky >< arzamas...@mail.ru.invalid > wrote: > >> >> >> Thanks Ivan Daschinsky for review, does anyone more who could check it ? >> >> thanks ! >> >Igniters, seems i complete with transactions in thin cpp client >> >implementation [1], part of iep-34 [2]. >> >Can anyone review my decision ? >> >Failed test seems not mine, looks like after fresh master rebase it will >> >gone. >> > >> >[1] https://issues.apache.org/jira/browse/IGNITE-13308 >> >[2] >> > >> >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-34+Thin+client%3A+transactions+support >> > >> > >> > >> >> >>
[SQL] Correct index usage with primary key constraint.
Folks, want to pay your attention that after [1] was merged into master, primary key index usage became predictable and correct and as a consequence old behavior was broken. for example table creation: CREATE TABLE PUBLIC.T1 (F1 VARCHAR, F2 VARCHAR, F3 VARCHAR, CONSTRAINT PK PRIMARY KEY (F2, F1)) in the old ver columns (F2, F1) were reordered, thus request : select … from PUBLIC.T1 WHERE F2 > … will not use primary key and index at all. after fix was merged primary key index used correctly. [1] https://issues.apache.org/jira/browse/IGNITE-13376
Re[2]: Exception handling in thin client: should we pass stack traces to the client?
Thanks ! now its clear for me, ones more want to repeat my position — sending exceptions to the client side looks like bad design, because exceptions its about unhealthy system recognition from administrator side, not from client of course, but looks like for now it`s more informative than existing decision. >> >> Alexey. >> >> This option should be on the server side, so server administrator can >> disable stack trace for all clients. >> Correct? >> >Yes, correct. >
Cpp thin client transactions support ready for review.
Thanks Ivan Daschinsky for review, does anyone more who could check it ? thanks ! >Igniters, seems i complete with transactions in thin cpp client >implementation [1], part of iep-34 [2]. >Can anyone review my decision ? >Failed test seems not mine, looks like after fresh master rebase it will >gone. > >[1] https://issues.apache.org/jira/browse/IGNITE-13308 >[2] >https://cwiki.apache.org/confluence/display/IGNITE/IEP-34+Thin+client%3A+transactions+support > > >
Re: [DISCUSSION] Output IgniteSystemProperties via ignite.sh
Good catch, as for me, do you plan some autogeneration here?
>
>>
>>>Hello, Igniters.
>>>
>>>For now, we have dozens of the `IgniteSystemProperties` [1] that can tweak
>>>Ignite behaviour in the very wide limits.
>>>But, the issue, for the administrator is the following
>>>
>>>- Documentation about existing properties can be outdated.
>>>- The only place of the truth is the source code.
>>>- It’s hard to understand what flag is supported in what version.
>>>
>>>I propose to implement output of all available properties with it’s
>>>descriptions in the `ignite.sh -X` command.
>>>
>>>Example of the JVM output:
>>>
>>>```
>>>[16:25:49]~/src/ignite:[master]$ java -X
>>>
>>>-Xbatch disable background compilation
>>>-Xbootclasspath/a:
>>> append to end of bootstrap class path
>>>-Xcheck:jni perform additional checks for JNI functions
>>>-Xcomp forces compilation of methods on first invocation
>>>-Xdebug provided for backward compatibility
>>>-Xdiag show additional diagnostic messages
>>>-Xfuture enable strictest checks, anticipating future default
>>>-Xint interpreted mode execution only
>>>-Xinternalversion
>>> displays more detailed JVM version information than the
>>>
>>>[16:28:45]~/src/ignite:[master]$ java -XX:+UnlockDiagnosticVMOptions
>>>-XX:+PrintFlagsFinal -version
>>>
>>>[Global flags]
>>>ccstrlist AOTLibrary = {product} {default}
>>> bool AbortVMOnCompilationFailure = false {diagnostic} {default}
>>>ccstr AbortVMOnException = {diagnostic} {default}
>>>ccstr AbortVMOnExceptionMessage = {diagnostic} {default}
>>> bool AbortVMOnSafepointTimeout = false {diagnostic} {default}
>>> bool AbortVMOnVMOperationTimeout = false {diagnostic} {default}
>>> intx AbortVMOnVMOperationTimeoutDelay = 1000 {diagnostic} {default}
>>> int ActiveProcessorCount = -1 {product} {default}
>>>
>>>```
>>>
>>>Example of the Ignite output:
>>>
>>>
ignite.sh -X IGNITE_CONFIG_URL - System property to hold optional
configuration URL.
>>>IGNITE_SSH_HOST - System property to hold SSH host for visor-started nodes.
>>>IGNITE_MIN_BUFFERED_COMMUNICATION_MSG_CNT - [DEPRECATED] System property to
>>>disable buffered communication if node sends less messages count than
>>>specified by this property. Default value is {@code 512}.
>>>
>>>…
>>>
>>>```
>>>
>>>WDYT?
>>>
>>>[1]
>>>https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/IgniteSystemProperties.java#L56
>>>
>>
>>
>>
>>
Exception handling in thin client: should we pass stack traces to the client?
Guys, does anyone else bothers about it ?) Ones more — i really not sure that all exceptions have been thrown from server side have informative message visible from client side, if client would receive full stack trace from server side it would reduce additional efforts from cluster administrator side, in one case i agree here — this is not pretty a bit. If no quorum would be here — ok, i fill a ticket for optionally enable such behavior, as was discussed earlier, and leave the current one as it is. thanks ! >Hi, > >I agree with Zhenya, that a stack from server side will be able to help in >investigation of issues, but it really confused in production environment. >I see all participants tell the same. > >Pavel, do you mean this behavior should be switching by configuration? > >On Thu, Aug 20, 2020 at 5:00 PM Pavel Tupitsyn < ptupit...@apache.org > wrote: > >> Link to the original discussion: >> >> >> >> http://apache-ignite-developers.2346864.n4.nabble.com/Exception-handling-in-thin-client-should-we-pass-stack-traces-to-the-client-td22392.html >> >> On Thu, Aug 20, 2020 at 4:46 PM Zhenya Stanilovsky >> < arzamas...@mail.ru.invalid > wrote: >> >> > >> > I want to resurrect this discussion, i don`t understand what sensitive >> > information you are talking about ? >> > Can you show some examples or something else ? I never listen that thread >> > dumps belong to sensitive info. >> > I believe that one linear error can`t help user to recognize problem and >> > logs from server side can be simple unreachable or logging disabled at >> all. >> > So i suggest to request full thread dump in case of server side error >> > occurred. >> > >> > what do you think ? >> > >> > >> > >Igniters, >> > > >> > >We had a discussion about how to propagate error information from >> cluster >> > >nodes to the client. My opinion is that we should pass a kind of vendor >> > >code plus optional error message, if vendor code is not very specific. >> > > >> > >Alternative idea is to pass the whole stack trace as well. I agree that >> > >this is very useful for debugging purposes, but on the other hand IMO it >> > >imposes security risk. By sending invalid requests to the server user >> > might >> > >get sensitive information about server configuration, such as it's >> > version, >> > >version of the underlying database, frameworks etc.. This information >> may >> > >help attacker to apply some version-specific attacks. This is precise >> > >reason why default error pages of web servers with stack traces are >> always >> > >replaces with some stubs. >> > > >> > >This is why I think we should not include stack traces. >> > > >> > >What do you think? >> > > >> > >Vladimir. >> > >> > >> > >> > >> > >-- >Vladislav Pyatkov >
Exception handling in thin client: should we pass stack traces to the client?
I want to resurrect this discussion, i don`t understand what sensitive information you are talking about ? Can you show some examples or something else ? I never listen that thread dumps belong to sensitive info. I believe that one linear error can`t help user to recognize problem and logs from server side can be simple unreachable or logging disabled at all. So i suggest to request full thread dump in case of server side error occurred. what do you think ? >Igniters, > >We had a discussion about how to propagate error information from cluster >nodes to the client. My opinion is that we should pass a kind of vendor >code plus optional error message, if vendor code is not very specific. > >Alternative idea is to pass the whole stack trace as well. I agree that >this is very useful for debugging purposes, but on the other hand IMO it >imposes security risk. By sending invalid requests to the server user might >get sensitive information about server configuration, such as it's version, >version of the underlying database, frameworks etc.. This information may >help attacker to apply some version-specific attacks. This is precise >reason why default error pages of web servers with stack traces are always >replaces with some stubs. > >This is why I think we should not include stack traces. > >What do you think? > >Vladimir.
Re[2]: Update of the default inline size for variable types
Huge +1 with Ilya I check your pr, this looks like stub : Pattern . compile( " \\ w+ \\ (( \\ d+) \\ ) " ); * Do we have some normalization before it ? varchar(whitespace + N) looks like not matching. * Can we obtain this info not from regexp ? >Hello! > >I can see where you are getting at but, as far as my experience tells me, >64 is already too large for the average use case. It will also start to >drag on the performance since you don't have too many entries in one page >anymore, and your tree starts to grow up, not to mention more i/o. > >I think we should benchmark it, see at which value we see a sharp decline. >Maybe 64 is OK after all, if it's a maximum for a complex index. Just make >sure that a single VARCHAR without length is still 10 and not 64. > >Regards, >-- >Ilya Kasnacheev > > >чт, 20 авг. 2020 г. в 11:15, Evgeniy Rudenko < e.a.rude...@gmail.com >: > >> Hi guys, >> >> Thank you for your feedback. >> >> Current calculation of the default size is not completely correct. If it >> meets a field of the variable length (such as byte array or string) it just >> stops any attempt to make index size more reasonable and uses >> IGNITE_MAX_INDEX_PAYLOAD_SIZE_DEFAULT as its size. Such approach doesn't >> seem correct to me in any case. First part of the update changes this logic >> and starts to calculate size based on all indexed columns. This update can >> even save some space for the users with varchars and high >> IGNITE_MAX_INDEX_PAYLOAD_SIZE_DEFAULT value. >> >> Second part of the update increases IGNITE_MAX_INDEX_PAYLOAD_SIZE_DEFAULT. >> Please note that we are changing only upper bound of the default size. >> Obviously this can lead to some increase of the used space, but we are >> trading size for the speed here. Current default value is too small for the >> average usage case. Users which care about size of the data still can set >> exact size of each index or limit all sizes by >> IGNITE_MAX_INDEX_PAYLOAD_SIZE_DEFAULT. So after the update users which >> would want to keep previous data size will just need to set >> IGNITE_MAX_INDEX_PAYLOAD_SIZE_DEFAULT=10. >> >> >> >> On Wed, Aug 19, 2020 at 5:20 PM Vladislav Pyatkov < vldpyat...@gmail.com > >> wrote: >> >> > Hi, >> > >> > In my mind, the inline size 64 will be able to significant grow of >> storage >> > size. >> > It can be difficult to understand by users. >> > >> > Earlier I remember we panned to replace inline value to hash code in the >> > case where size of value more than inline size. >> > It will help to comparison of "==", "!=", but will not grow size of >> > storage. >> > >> > I think optimization with hash code looks more preferable and in last way >> > anyone can to grow size of baseline though API. >> > >> > >> > On Wed, Aug 19, 2020 at 9:22 AM Zhenya Stanilovsky >> > < arzamas...@mail.ru.invalid > wrote: >> > >> > > >> > > >> > > >Hi guys, >> > > >> > > Evgeniy, hola! >> > > > >> > > >Currently if a varlength type (such as String or byte[]) is >> encountered >> > in >> > > >the composite index inline size just defaults to 10, which is almost >> > > always >> > > >not enough. I am going to change this and implement following changes: >> > > > >> > > >1) For a column of the variable length keep using 10 as the default >> size >> > > in >> > > >case of the one-column index. But if the index is composite the >> default >> > > >index size will be calculated as the sum of sizes of all indexed >> > columns. >> > > >For example, for the index like (INT, VARCHAR, VARCHAR, INT) default >> > > inline >> > > >size will be 5 + 10 + 10 + 5 = 30 (5 for each int, 10 for each >> string). >> > > >> > > Why exactly this approach ? Why not 5 + 10 and its all here ? Do you >> have >> > > some logical base, statistical distribution or something near it, for >> now >> > > this look as your own decision and nothing more, i`m wrong ? >> > > > >> > > >2) For sql varchar and binary columns with defined length (for example >> > > >VARCHAR(XX)) use XX + 3 as default inline size for the column (need 3 >> > > extra >> > > >bytes for the inner representation of the type). >> > > >> > > The same question here, why you want o cover all varchar len ? do you >> > > compare with other vendors approach ? >> > > > >> > > >3) Maximum default index size still will be limited by >> > > >IGNITE_MAX_INDEX_PAYLOAD_SIZE, but its default value will be increased >> > to >> > > >64. For example for the index (VARCHAR, VARCHAR, VARCHAR, VARCHAR, >> > > VARCHAR, >> > > >VARCHAR, VARCHAR) default index size will be only 64. Same for the >> > columns >> > > >with defined length: by default VARCHAR(100) column will create index >> > only >> > > >with size equal to 64. >> > > > >> > > >Please tell if you have any concerns. Update can be found at >> > > > https://github.com/apache/ignite/pull/8161 >> > > > >> > > >Best regards, >> > > >Evgeniy >> > > > >> > > >> > > >> > > >> > > >> > >> > >> > >> > -- >> > Vladislav Pyatkov >> > >> >> >> -- >> Best regards, >> Evgeniy >>
Re: Update of the default inline size for variable types
>Hi guys, Evgeniy, hola! > >Currently if a varlength type (such as String or byte[]) is encountered in >the composite index inline size just defaults to 10, which is almost always >not enough. I am going to change this and implement following changes: > >1) For a column of the variable length keep using 10 as the default size in >case of the one-column index. But if the index is composite the default >index size will be calculated as the sum of sizes of all indexed columns. >For example, for the index like (INT, VARCHAR, VARCHAR, INT) default inline >size will be 5 + 10 + 10 + 5 = 30 (5 for each int, 10 for each string). Why exactly this approach ? Why not 5 + 10 and its all here ? Do you have some logical base, statistical distribution or something near it, for now this look as your own decision and nothing more, i`m wrong ? > >2) For sql varchar and binary columns with defined length (for example >VARCHAR(XX)) use XX + 3 as default inline size for the column (need 3 extra >bytes for the inner representation of the type). The same question here, why you want o cover all varchar len ? do you compare with other vendors approach ? > >3) Maximum default index size still will be limited by >IGNITE_MAX_INDEX_PAYLOAD_SIZE, but its default value will be increased to >64. For example for the index (VARCHAR, VARCHAR, VARCHAR, VARCHAR, VARCHAR, >VARCHAR, VARCHAR) default index size will be only 64. Same for the columns >with defined length: by default VARCHAR(100) column will create index only >with size equal to 64. > >Please tell if you have any concerns. Update can be found at >https://github.com/apache/ignite/pull/8161 > >Best regards, >Evgeniy >
Cpp thin client transactions support ready for review.
Igniters, seems i complete with transactions in thin cpp client implementation [1], part of iep-34 [2]. Can anyone review my decision ? Failed test seems not mine, looks like after fresh master rebase it will gone. [1] https://issues.apache.org/jira/browse/IGNITE-13308 [2] https://cwiki.apache.org/confluence/display/IGNITE/IEP-34+Thin+client%3A+transactions+support
Re[2]: new connection event
may be jmx would be enough here ? >Hello! > >Why not the Failure Handler then? > >(I'm only half-joking). > >Regards, > >-- >Ilya Kasnacheev > > >ср, 12 авг. 2020 г. в 09:54, Oleg Ostanin < oleg.alex.osta...@gmail.com >: > >> Thank you for the response. Yes, we have a simple warning in log, but in >> case SSL error needs an immediate human attention it would be better to set >> a listener for such events. >> >> вт, 11 авг. 2020 г. в 17:59, Ilya Kasnacheev < ilya.kasnach...@gmail.com >: >> >> > Hello! >> > >> > I'm not sure this is a good event. Events are computer-processable, while >> > SSL error either needs attention from a human or none at all. >> > >> > Then again, sometimes it's not an SSL error (come to think or it, >> > unauthorized people tend to not use SSL since success is not an option >> > here) but any unexpected input. Such input will mean that something (or >> > somebody) wrong is trying to connect, possibly using a wrong port or >> hoping >> > that SSL is disabled. >> > >> > Maybe we need a simple warning. I guess we already have it, actually. >> Don't >> > we? >> > >> > Regards, >> > -- >> > Ilya Kasnacheev >> > >> > >> > вт, 11 авг. 2020 г. в 14:27, Oleg Ostanin < oleg.alex.osta...@gmail.com >: >> > >> > > Hello igniters! I wonder if we can add a new event for a failed ssl >> > > connection? Considering security hazards it can be quite helpful to get >> > > notifications in case someone unauthorized is trying to connect to the >> > > cluster. I've created a small pull-request to illustrate a possible >> > > solution. >> > > https://github.com/apache/ignite/pull/8139 >> > > Please tell me what do you think. >> > > >> > >>
Re[2]: Please grant me privileges to edit ignite wiki pages.
Ilya, thanks ! full name : evgeny stanilovsky , short : zstan > >> >>>Hello! >>> >>>Do you have a Wiki account? What's its username? >>> >>>Thanks, >>>-- >>>Ilya Kasnacheev >>> >>> >>>чт, 6 авг. 2020 г. в 11:01, Zhenya Stanilovsky < arzamas...@mail.ru.invalid >>>>: >>> >>>> >>>> I`m currently working on cpp thin client transactions support [1] and need >>>> to edit, for example this page [2]. >>>> Can someone grant me this privileges ? >>>> thanks ! >>>> >>>> [1] https://issues.apache.org/jira/browse/IGNITE-13308 >>>> [2] >>>> https://cwiki.apache.org/confluence/display/IGNITE/Thin+clients+features >>>> >>>> >> >> >> >>
Please grant me privileges to edit ignite wiki pages.
I`m currently working on cpp thin client transactions support [1] and need to edit, for example this page [2]. Can someone grant me this privileges ? thanks ! [1] https://issues.apache.org/jira/browse/IGNITE-13308 [2] https://cwiki.apache.org/confluence/display/IGNITE/Thin+clients+features
Re: [DISCUSSION] Cache warmup
Looks like we need additional func for static caches, for example:
warmup(List cconf) it would be helpful for spring too.
>
>--- Forwarded message ---
>From: "Вячеслав Коптилин" < slava.kopti...@gmail.com >
>To: dev@ignite.apache.org
>Cc:
>Subject: Re: [DISCUSSION] Cache warmup
>Date: Mon, 27 Jul 2020 16:47:48 +0300
>
>Hello Kirill,
>
>Thanks a lot for driving this activity. If I am not mistaken, this
>discussion relates to IEP-40.
>
>> I suggest adding a warmup phase after recovery here [1] after [2], before
>discovery.
>This means that the user's thread, which starts Ignite via
>Ignition.start(), will wait for ana additional step - cache warm-up.
>I think this fact has to be clearly mentioned in our documentation (at
>Javadocat least) because this step can be time-consuming.
>
>> I suggest adding a new interface:
>I would change it a bit. First of all, it would be nice to place this
>interface to a public package and get rid of using GridCacheContext,
>which is an internal class and it should not leak to the public API in any
>case.
>Perhaps, this parameter is not needed at all or we should add some public
>abstraction instead of internal class.
>
>package org.apache.ignite.configuration;
>
>import org.apache.ignite.IgniteCheckedException;
>import org.apache.ignite.lang.IgniteFuture;
>
>public interface CacheWarmupper {
> /**
> * Warmup cache.
> *
> * @param cachename Cache name.
> * @return Future cache warmup.
> * @throws IgniteCheckedException If failed.
> */
> IgniteFuture warmup(String cachename) throws
>IgniteCheckedException;
>}
>
>Thanks,
>S.
>
>пн, 27 июл. 2020 г. в 15:03, ткаленко кирилл < tkalkir...@yandex.ru >:
>
>> Now, after restarting node, we have only cold caches, which at first
>> requests to them will gradually load data from disks, which can slow down
>> first calls to them.
>> If node has more RAM than data on disk, then they can be loaded at start
>> "warmup", thereby solving the issue of slowdowns during first calls to
>> caches.
>>
>> I suggest adding a warmup phase after recovery here [1] after [2], before
>> descovery.
>>
>> I suggest adding a new interface:
>>
>> package org.apache.ignite.internal.processors.cache;
>>
>> import org.apache.ignite.IgniteCheckedException;
>> import org.apache.ignite.internal.IgniteInternalFuture;
>> import org.jetbrains.annotations.Nullable;
>>
>> /**
>> * Interface for warming up cache.
>> */
>> public interface CacheWarmup {
>> /**
>> * Warmup cache.
>> *
>> * @param cacheCtx Cache context.
>> * @return Future cache warmup.
>> * @throws IgniteCheckedException if failed.
>> */
>> @Nullable IgniteInternalFuture process(GridCacheContext cacheCtx)
>> throws IgniteCheckedException;
>> }
>>
>> Which will allow to warm up caches in parallel and asynchronously. Warmup
>> phase will end after all IgniteInternalFuture for all caches isDone.
>>
>> Also adding the ability to customize via methods:
>> org.apache.ignite.configuration.IgniteConfiguration#setDefaultCacheWarmup
>> org.apache.ignite.configuration.CacheConfiguration#setCacheWarmup
>>
>> Which will allow for each cache to set implementation of cache warming
>> up,
>> both for a specific cache, and for all if necessary.
>>
>> I suggest adding an implementation of SequentialWarmup that will use [3].
>>
>> Questions, suggestions, comments?
>>
>> [1] -
>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>> [2] -
>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>> [3] -
>> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManager.CacheDataStore#preload
>>
Re[2]: Apache Ignite 2.9.0 RELEASE [Time, Scope, Manager]
Alex, i also suggest to merge this https://issues.apache.org/jira/browse/IGNITE-13229 too, GridClient leakage and further TC OOM preventing. >Ivan, > >It was already in release scope as discussed in this thread. > >вт, 14 июл. 2020 г. в 14:31, Ivan Rakov < ivan.glu...@gmail.com >: > >> Hi, >> >> We are still waiting for a final review of Tracing functionality [1] until >> the end of tomorrow (July 15). >> We anticipate that it will be merged to Ignite master no later than July >> 16. >> >> Sorry for being a bit late here. Alex P., can you include [1] to the >> release scope? >> >> [1]: https://issues.apache.org/jira/browse/IGNITE-13060 >> >> -- >> Best Regards, >> Ivan Rakov >> >> On Tue, Jul 14, 2020 at 6:16 AM Alexey Kuznetsov < akuznet...@gridgain.com > >> wrote: >> >>> Alex, >>> >>> Can you cherry-pick to Ignite 2.9 this issue: >>> https://issues.apache.org/jira/browse/IGNITE-13246 ? >>> >>> This issue is about BASELINE events and it is very useful for notification >>> external tools about changes in baseline. >>> >>> Thank you! >>> >>> --- >>> Alexey Kuznetsov >>> >>
Re[2]: Request for contributors permissions
now its all ok, i rerun your pr here : https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_RunAll_IgniteTests24Java8=pull%2F7906%2Fhead=buildTypeStatusDiv >Hi Ivan, > >let me first apologize for the question, sure they are stupid and I am >missing something obvious but I do not get what. > >I forked ignite there https://github.com/hostettler/ignite >Then I comitted everything in my fork and then I create a pull request at >https://github.com/apache/ignite/pull/7906 > >and updated the ticket at >https://issues.apache.org/jira/browse/IGNITE-6499?src=confmacro > >Can you point me to what I do wrong? > > > >-- >Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re[2]: IGNITE-6499 Compact NULL fields
request it, check for example [1] also you need to run [2] tests. [1] http://apache-ignite-developers.2346864.n4.nabble.com/Phani-Introduction-td47788.html [2] https://mtcga.gridgain.com >Hello, > >Look at the ticket and the only comment I can see is creating a branch on >git in the main repo and not in my fork. I do not have the right to create a >branch in the main repository. Am i missing something? > >Sorry I probably misread the document but I though that I should fork the >repo and then pull request as I do not have the rights to create a branch. > >Thanks for your help > > > >-- >Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: IGNITE-6499 Compact NULL fields
Steve i place some comments in ticket, still have no response. > > >--- Forwarded message --- >From: " steve.hostett...@gmail.com " < steve.hostett...@gmail.com > >To: dev@ignite.apache.org >Cc: >Subject: Re: Re[4]: IGNITE-6499 Compact NULL fields >Date: Fri, 12 Jun 2020 16:15:37 +0300 > >Hello, > >Stanilovsky Evgeny : would you mind having a look at the pull request. > >Thanks in advance > > > >-- >Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re[2]: [DISCUSS] Extra test coverage for ACTIVE_READ_ONLY cluster state
Sergey, changes looks good to me. >Четверг, 4 июня 2020, 12:39 +03:00 от Sergey Antonov >: > >Igniters, I faced several problems during write tests for the read-only >mode: > > 1. You can create/destroy cache on the read-only cluster. Fixed in [1]. > 2. The read-only mode doesn't affect LOCAL caches. Ticket created [2]. > 3. IgniteCache#isClosed() doesn't work on server nodes. Ticket created > [3]. > >Also, I wrote tests for: > > 1. IgniteCache API (including create(), destroy(), invoke()) > 2. Near caches. > 3. 3rd party CacheStore. > 4. Services (deploy, execution, cancel) > 5. Datastructures (IgniteAtomicLong, IgniteAtomicReference, > IgniteAtomicSequence, IgniteAtomicStamped, IgniteCountDownLatch, > IgniteQueue, IgniteSet) > 6. Updates to meta storage. > >The patch [4] ready for review. > >[1] https://issues.apache.org/jira/browse/IGNITE-13071 >[2] https://issues.apache.org/jira/browse/IGNITE-13076 >[3] https://issues.apache.org/jira/browse/IGNITE-13102 >[4] https://github.com/apache/ignite/pull/7853/files > > >вт, 26 мая 2020 г. в 20:28, Sergey Antonov < antonovserge...@gmail.com >: > >> Maxim, I've created a ticket [1] for this change. >> >> [1] https://issues.apache.org/jira/browse/IGNITE-13079 >> >> вт, 26 мая 2020 г. в 20:09, Sergey Antonov < antonovserge...@gmail.com >: >> >>> Maxim, I'd prefer to do this with a separate ticket. >>> >>> вт, 26 мая 2020 г. в 19:59, Maxim Muzafarov < mmu...@apache.org >: >>> Sergey, Sounds good! Should we consider removing the deprecated methods `active()`, `active(boolean active)` from tests also? On Tue, 26 May 2020 at 12:18, Sergey Antonov < antonovserge...@gmail.com > wrote: > > Hello, Igniters. > > I introduced cluster read-only mode [1] and a new API for cluster state > change [2]. At the moment we don't have good test coverage for this > feature. I'm going to fix it and write tests and check that operations > are *denied > *in read-only mode: > > - data structures usage > - cache create/clear/destroy > - DDL requests > - cache updates by task's execution / deployed service > > And the following operations are *allowed *in read-only mode: > > - update of metastorage / distributed metastorage > - updates to ignite-sys-cache > - task's execution, but updates must be rejected > - service deploy/undeploy, but updates must be rejected > - data recovery on node join > > I'll work under these tests in ticket [3]. > Any objections? > > [1] https://issues.apache.org/jira/browse/IGNITE-11256 > [2] https://issues.apache.org/jira/browse/IGNITE-12225 > [3] https://issues.apache.org/jira/browse/IGNITE-13071 > -- > BR, Sergey Antonov >>> >>> >>> -- >>> BR, Sergey Antonov >>> >> >> >> -- >> BR, Sergey Antonov >> > >-- >BR, Sergey Antonov >
Re: [DISCUSSION] Add autocompletion for commands in control.sh
good catch ! it`s a little bit pain for now to working with it. >Hi, Igniters! > >At the moment to work with the control.sh we need to know exactly what the >name of the command and its options are and so the user can often make >mistakes when using it. So I think it would be useful to do control.sh more >user-friendly by adding autocomplete as in modern command-line utilities. > >For this purpose, I suggest using framework [1] and to do this, take out >control.sh together with its associated classes in a separate module such as >"modules/control-utility". > >Comments, suggestions? > >[1] - https://picocli.info/
Re[2]: [DISCUSSION] Ignite.C++ and CMake
Ivan besides documentation [1] -s no — no works -- catch_system_errors =no — works properly well, tests are passed. boost 1.65 [1] https://www.boost.org/doc/libs/1_65_0/libs/test/doc/html/boost_test/utf_reference/rt_param_reference/catch_system.html >Hello! > >I didn't check tests since I don't develop AI C++, merely use it as user. >That's where we should wait for Igor Sapego to check. > >Regards, >-- >Ilya Kasnacheev > > >пт, 29 мая 2020 г. в 12:20, Ivan Daschinsky < ivanda...@gmail.com >: > >> Ilya, thanks a lot! What about tests? I found one flag that must be >> supplied to boost.tests. >> This flag should fix JVM crash of C++ suites on Linux. >> >> Zhenya Stanilovsky and me have checked, that without this flag tests failed >> with SIGSEGV early (boost catch this signal from jvm, but it is ok for >> jvm). >> Flag is -catch_system_errors=no. I added it to CTest runner. You can >> invoke it manually and using make test ARGS="-V" >> >> >> >> пт, 29 мая 2020 г. в 11:54, Ilya Kasnacheev < ilya.kasnach...@gmail.com >: >> >> > Hello! >> > >> > Looks good to me! But we probably also ask Igor to take a look. >> > >> > Compiled debug and release, without and with odbc, checked running thick >> > node and ODBC connection on Linux. >> > >> > Regards, >> > -- >> > Ilya Kasnacheev >> > >> > >> > чт, 28 мая 2020 г. в 17:31, Ivan Daschinsky < ivanda...@gmail.com >: >> > >> > > Ok, PR is ready >> > > https://issues.apache.org/jira/browse/IGNITE-13078 >> > > >> > > Build tested on Mac OS X 10.15 and Ubuntu 20.04 with CMake 3.17.2 and >> > 3.6.1 >> > > Unfortunately, I was not able to test on Windows, but principally it >> > should >> > > works, but minor issues are probable. >> > > >> > > Instruction is attached in PR. >> > > Any use reports are welcomed! >> > > >> > > вт, 26 мая 2020 г. в 18:51, Ivan Daschinsky < ivanda...@gmail.com >: >> > > >> > > > Stephen, looks great! I do mostly the same things in C++ code. Thank >> > you! >> > > > >> > > > вт, 26 мая 2020 г. в 18:33, Stephen Darlington < >> > > > stephen.darling...@gridgain.com >: >> > > > >> > > >> Not sure if it’ll help, but I made some changes to get it working >> on a >> > > >> Mac with the current built system. There may be some code worth >> > taking. >> > > >> >> > > >> https://github.com/apache/ignite/pull/4872 < >> > > >> https://github.com/apache/ignite/pull/4872 > >> > > >> >> > > >> Regards, >> > > >> Stephen >> > > >> >> > > >> > On 26 May 2020, at 16:02, Ivan Daschinsky < ivanda...@gmail.com > >> > > wrote: >> > > >> > >> > > >> > I appreciate any help, thank you, Ilya. >> > > >> > >> > > >> > Currently I have a small PR without ticket (link in first >> post),but >> > I >> > > >> > decided not to file a jira issue before discussion. >> > > >> > Now I see, that this feature are of great interest to community. >> So >> > I >> > > >> file >> > > >> > a ticket, test myself on my home laptop (ubuntu 20.04) >> > > >> > and add detailed instructions to DEVNOTES.txt in a few days. >> > > >> > >> > > >> > I would be happy if my someone can follow the instruction and >> write >> > > >> > possible issues. >> > > >> > >> > > >> > I will notify about status update in this thread in next few days. >> > > >> > >> > > >> > Thank you all very much for support! >> > > >> > >> > > >> > >> > > >> > вт, 26 мая 2020 г. в 17:50, Ilya Kasnacheev < >> > > ilya.kasnach...@gmail.com >> > > >> >: >> > > >> > >> > > >> >> Hello! >> > > >> >> >> > > >> >> I will assist with checking on Linux if you would contribute a >> > patch. >> > > >> >> Please start with a ticket (or even an IEP maybe?) >> > > >> >> >>
Re[4]: Proposal: set default transaction timeout to 5 minutes
Of course, i just thinking about huge persistent installation and guys who not carefully reads Release Notes ) In case of long tx timeouts by design, they can easily fix default timeout with just one jmx call. >Zhenya, > >Can you please elaborate? >Why we need to change default TX timeout via JMX? It looks feasible and >perhaps may work as a hotfix for live deployments experiencing issues with >long transactions, but it's definitely a separate issue. > >On Fri, May 22, 2020 at 6:20 PM Zhenya Stanilovsky >< arzamas...@mail.ru.invalid > wrote: > >> >> Ivan, does global timeout change through jmx in scope of this ticket ? If >> so, can you add it ? Opposite we need additional ticket, i hope ? We >> still have no somehow store for jmx changed params, every one need to >> remember that cluster restart will reset this setting to default, in this >> case system param need to be appended. >> >> >> >> > https://issues.apache.org/jira/browse/IGNITE-13064 is raised with label >> >"newbie". >> > >> >On Tue, May 19, 2020 at 4:10 PM Ivan Rakov < ivan.glu...@gmail.com > >> wrote: >> > >> >> Support this idea in general but why 5 minutes and not less? >> >> >> >> This value looks to me greater than any value that can possibly affect >> >> existing deployments (existing long transactions may suddenly start to >> >> rollback), but less than reaction time of users that are only starting >> to >> >> get along with Ignite and suddenly experience TX deadlock. >> >> >> >> -- >> >> Best Regards, >> >> Ivan Rakov >> >> >> >> On Tue, May 19, 2020 at 10:31 AM Anton Vinogradov < a...@apache.org > >> wrote: >> >> >> >>> +1 >> >>> >> >>> On Mon, May 18, 2020 at 9:45 PM Sergey Antonov < >> antonovserge...@gmail.com >> >>> > >> >>> wrote: >> >>> >> >>> > +1 >> >>> > >> >>> > пн, 18 мая 2020 г. в 21:26, Andrey Mashenkov < >> >>> andrey.mashen...@gmail.com >: >> >>> > >> >>> > > +1 >> >>> > > >> >>> > > On Mon, May 18, 2020 at 9:19 PM Ivan Rakov < ivan.glu...@gmail.com >> > >> >>> > wrote: >> >>> > > >> >>> > > > Hi Igniters, >> >>> > > > >> >>> > > > I have a very simple proposal. Let's set default TX timeout to 5 >> >>> > minutes >> >>> > > > (right now it's 0 = no timeout). >> >>> > > > Pros: >> >>> > > > 1. Deadlock detection procedure is triggered on timeout. In case >> >>> user >> >>> > > will >> >>> > > > get into key-level deadlock, he'll be able to discover root cause >> >>> from >> >>> > > the >> >>> > > > logs (even though load will hang for a while) and skip step with >> >>> > googling >> >>> > > > and debugging. >> >>> > > > 2. Almost every system with transactions has timeout enabled by >> >>> > default. >> >>> > > > >> >>> > > > WDYT? >> >>> > > > >> >>> > > > -- >> >>> > > > Best Regards, >> >>> > > > Ivan Rakov >> >>> > > > >> >>> > > >> >>> > > >> >>> > > -- >> >>> > > Best regards, >> >>> > > Andrey V. Mashenkov >> >>> > > >> >>> > >> >>> > >> >>> > -- >> >>> > BR, Sergey Antonov >> >>> > >> >>> >> >> >> >> >> >>
Re: [DISCUSSION] Ignite.C++ and CMake
Ivan huge +1 with your proposal. I had some problems with odbc tests building too, looks like cmake will make it more easy. >Hello Igniters. > >I’d like to discuss porting build process of Ignite.C++. I think that there is >time to change it. > >*Motivation* >Currently, it is hard to build Ignite.C++. Different build process for windows >and linux, lack of building support on Mac OS X (quite popular OS among >developers), absolutely not IDE support, except windows and only Visual Studio >is supported. > >*Suggestion* >I’d suggest to migrate to CMake build system. It is very popular among open >source projects, and in Apache Software Foundation too. Notable user: Apache >Mesos, Apache Zookeeper (C client offers CMake as an alternative to autoconf >and only option on windows), Apache Kafka (librdkafka - C/C++ client), Apache >Thrift. Popular column-oriented database ClickHouse also uses CMake. > >CMake is widely supported in many IDE’s on various platforms, notably Visual >Studio, CLion, Xcode, QtCreator, KDevelop. > >*Current status* > >Currently, the most of work is done (see [1]) and tested on Mac OS X 10.15 >(some C++ porting). All tests are run without any flaws, including odbc >(unixodbc), ssl, thin and thick client, installation, IDE integration (CLion). >Next steps is to test linux and windows. > >But full migration isn’t possible without agreement and help of community. >Even if most of all you agree, migration requires additional efforts in TC >agents tuning and so on (event though test running fully automated by CMake >CTest). > >Lets discuss my proposition and idea. > >[1] - https://github.com/apache/ignite/pull/7845 > > >
