Re: Re[2]: IGNITE-6499 Compact NULL fields
Hello! If we do improve it, I think we should go for a full re-think as opposed to a single breaking change that doesn't actually improve that much. Nevertheless, I think we can commit some improvements with opt-in BinaryConfiguration. Regards, -- Ilya Kasnacheev сб, 11 июл. 2020 г. в 01:27, steve.hostett...@gmail.com < steve.hostett...@gmail.com>: > Ok gotcha, so it is not going to make it. > > Just to note that we are dragging this since before v2.0 and just a > reminder > that someone else tried a similar thing before v2 and it got blocked > because > it was too much of a change for v2. > > Typically the type of things that we can never change because it is too > much > of an impact. > > BinaryObject format is not optimal far for it and if we cannot change it > incrementally nor in a big bang... > > > > -- > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ >
Re: Re[2]: IGNITE-6499 Compact NULL fields
Ok gotcha, so it is not going to make it. Just to note that we are dragging this since before v2.0 and just a reminder that someone else tried a similar thing before v2 and it got blocked because it was too much of a change for v2. Typically the type of things that we can never change because it is too much of an impact. BinaryObject format is not optimal far for it and if we cannot change it incrementally nor in a big bang... -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Re[2]: IGNITE-6499 Compact NULL fields
Hello! Yes, I think this is a sensible approach. Regards, -- Ilya Kasnacheev ср, 8 июл. 2020 г. в 14:46, Ivan Daschinsky : > I think that this feature can be handled as compactFooter. For example, C++ > doesn't support compactFooter and it is not an issue. > Of course, this feature should be disabled by default, and should be > enabled explicitly in BinaryConfiguration. > Also, subsequent issues in jira about this feature support in platforms > should be created. > > ср, 8 июл. 2020 г. в 14:31, Ilya Kasnacheev : > > > Hello! > > > > I think this is a blocker for this change. We already have binary format > > published: > > > > > https://apacheignite.readme.io/docs/binary-client-protocol-data-format#complex-object > > > > Arguably, we cannot change it in a minor version of Apache Ignite, so > this > > change has to target AI 3.0. > > > > Extending this binary format with e.g. new operations could probably be > OK. > > But we have clients released on a different schedule in their own repos > > (and there are some 3rd party clients too), we can't release a minor > > version which will change this format unilaterally without any change of > > operation (same data, same calls, different result after upgrade, broken > > clients). > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > ср, 8 июл. 2020 г. в 13:43, Ivan Daschinsky : > > > > > Hi! > > > Ilya, unfortunatelly yes, subsequent changes should be made in C++, > .NET > > > and other platform code. > > > > > > ср, 8 июл. 2020 г. в 12:22, Ilya Kasnacheev >: > > > > > > > Hello fellow devs, > > > > > > > > I just wanted to ask, how would this Binary Object format change > affect > > > > thin clients? C++/.Net nodes? Etc? > > > > > > > > Is it fully backwards compatible or not? > > > > > > > > I think that realistically, we can only add binary-incompatible > changes > > > to > > > > Binary Object format in 3.0. > > > > > > > > Regards, > > > > -- > > > > Ilya Kasnacheev > > > > > > > > > > > > ср, 8 июл. 2020 г. в 09:05, Ivan Pavlukhin : > > > > > > > > > A side note. Now we have a neat URL for TC bot > > > > > https://mtcga.ignite.apache.org/ (along with one in a gridgain > > > > > domain). > > > > > > > > > > 2020-07-07 18:43 GMT+03:00, Zhenya Stanilovsky > > > > > > > > >: > > > > > > > > > > > > request it, check for example [1] > > > > > > > > > > > > also you need to run [2] tests. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/Phani-Introduction-td47788.html > > > > > > [2] https://mtcga.gridgain.com > > > > > >>Hello, > > > > > >> > > > > > >>Look at the ticket and the only comment I can see is creating a > > > branch > > > > on > > > > > >>git in the main repo and not in my fork. I do not have the right > to > > > > > create > > > > > >> a > > > > > >>branch in the main repository. Am i missing something? > > > > > >> > > > > > >>Sorry I probably misread the document but I though that I should > > fork > > > > the > > > > > >>repo and then pull request as I do not have the rights to create > a > > > > > branch. > > > > > >> > > > > > >>Thanks for your help > > > > > >> > > > > > >> > > > > > >> > > > > > >>-- > > > > > >>Sent from: > http://apache-ignite-developers.2346864.n4.nabble.com/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Best regards, > > > > > Ivan Pavlukhin > > > > > > > > > > > > > > > > > > -- > > > Sincerely yours, Ivan Daschinskiy > > > > > > > > -- > Sincerely yours, Ivan Daschinskiy >
Re: Re[2]: IGNITE-6499 Compact NULL fields
I think that this feature can be handled as compactFooter. For example, C++ doesn't support compactFooter and it is not an issue. Of course, this feature should be disabled by default, and should be enabled explicitly in BinaryConfiguration. Also, subsequent issues in jira about this feature support in platforms should be created. ср, 8 июл. 2020 г. в 14:31, Ilya Kasnacheev : > Hello! > > I think this is a blocker for this change. We already have binary format > published: > > https://apacheignite.readme.io/docs/binary-client-protocol-data-format#complex-object > > Arguably, we cannot change it in a minor version of Apache Ignite, so this > change has to target AI 3.0. > > Extending this binary format with e.g. new operations could probably be OK. > But we have clients released on a different schedule in their own repos > (and there are some 3rd party clients too), we can't release a minor > version which will change this format unilaterally without any change of > operation (same data, same calls, different result after upgrade, broken > clients). > > Regards, > -- > Ilya Kasnacheev > > > ср, 8 июл. 2020 г. в 13:43, Ivan Daschinsky : > > > Hi! > > Ilya, unfortunatelly yes, subsequent changes should be made in C++, .NET > > and other platform code. > > > > ср, 8 июл. 2020 г. в 12:22, Ilya Kasnacheev : > > > > > Hello fellow devs, > > > > > > I just wanted to ask, how would this Binary Object format change affect > > > thin clients? C++/.Net nodes? Etc? > > > > > > Is it fully backwards compatible or not? > > > > > > I think that realistically, we can only add binary-incompatible changes > > to > > > Binary Object format in 3.0. > > > > > > Regards, > > > -- > > > Ilya Kasnacheev > > > > > > > > > ср, 8 июл. 2020 г. в 09:05, Ivan Pavlukhin : > > > > > > > A side note. Now we have a neat URL for TC bot > > > > https://mtcga.ignite.apache.org/ (along with one in a gridgain > > > > domain). > > > > > > > > 2020-07-07 18:43 GMT+03:00, Zhenya Stanilovsky > > > > > > >: > > > > > > > > > > request it, check for example [1] > > > > > > > > > > also you need to run [2] tests. > > > > > > > > > > [1] > > > > > > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/Phani-Introduction-td47788.html > > > > > [2] https://mtcga.gridgain.com > > > > >>Hello, > > > > >> > > > > >>Look at the ticket and the only comment I can see is creating a > > branch > > > on > > > > >>git in the main repo and not in my fork. I do not have the right to > > > > create > > > > >> a > > > > >>branch in the main repository. Am i missing something? > > > > >> > > > > >>Sorry I probably misread the document but I though that I should > fork > > > the > > > > >>repo and then pull request as I do not have the rights to create a > > > > branch. > > > > >> > > > > >>Thanks for your help > > > > >> > > > > >> > > > > >> > > > > >>-- > > > > >>Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Best regards, > > > > Ivan Pavlukhin > > > > > > > > > > > > > -- > > Sincerely yours, Ivan Daschinskiy > > > -- Sincerely yours, Ivan Daschinskiy
Re: Re[2]: IGNITE-6499 Compact NULL fields
Hello! I think this is a blocker for this change. We already have binary format published: https://apacheignite.readme.io/docs/binary-client-protocol-data-format#complex-object Arguably, we cannot change it in a minor version of Apache Ignite, so this change has to target AI 3.0. Extending this binary format with e.g. new operations could probably be OK. But we have clients released on a different schedule in their own repos (and there are some 3rd party clients too), we can't release a minor version which will change this format unilaterally without any change of operation (same data, same calls, different result after upgrade, broken clients). Regards, -- Ilya Kasnacheev ср, 8 июл. 2020 г. в 13:43, Ivan Daschinsky : > Hi! > Ilya, unfortunatelly yes, subsequent changes should be made in C++, .NET > and other platform code. > > ср, 8 июл. 2020 г. в 12:22, Ilya Kasnacheev : > > > Hello fellow devs, > > > > I just wanted to ask, how would this Binary Object format change affect > > thin clients? C++/.Net nodes? Etc? > > > > Is it fully backwards compatible or not? > > > > I think that realistically, we can only add binary-incompatible changes > to > > Binary Object format in 3.0. > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > ср, 8 июл. 2020 г. в 09:05, Ivan Pavlukhin : > > > > > A side note. Now we have a neat URL for TC bot > > > https://mtcga.ignite.apache.org/ (along with one in a gridgain > > > domain). > > > > > > 2020-07-07 18:43 GMT+03:00, Zhenya Stanilovsky > > > > >: > > > > > > > > request it, check for example [1] > > > > > > > > also you need to run [2] tests. > > > > > > > > [1] > > > > > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/Phani-Introduction-td47788.html > > > > [2] https://mtcga.gridgain.com > > > >>Hello, > > > >> > > > >>Look at the ticket and the only comment I can see is creating a > branch > > on > > > >>git in the main repo and not in my fork. I do not have the right to > > > create > > > >> a > > > >>branch in the main repository. Am i missing something? > > > >> > > > >>Sorry I probably misread the document but I though that I should fork > > the > > > >>repo and then pull request as I do not have the rights to create a > > > branch. > > > >> > > > >>Thanks for your help > > > >> > > > >> > > > >> > > > >>-- > > > >>Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best regards, > > > Ivan Pavlukhin > > > > > > > > -- > Sincerely yours, Ivan Daschinskiy >
Re: Re[2]: IGNITE-6499 Compact NULL fields
Hi! Ilya, unfortunatelly yes, subsequent changes should be made in C++, .NET and other platform code. ср, 8 июл. 2020 г. в 12:22, Ilya Kasnacheev : > Hello fellow devs, > > I just wanted to ask, how would this Binary Object format change affect > thin clients? C++/.Net nodes? Etc? > > Is it fully backwards compatible or not? > > I think that realistically, we can only add binary-incompatible changes to > Binary Object format in 3.0. > > Regards, > -- > Ilya Kasnacheev > > > ср, 8 июл. 2020 г. в 09:05, Ivan Pavlukhin : > > > A side note. Now we have a neat URL for TC bot > > https://mtcga.ignite.apache.org/ (along with one in a gridgain > > domain). > > > > 2020-07-07 18:43 GMT+03:00, Zhenya Stanilovsky > > >: > > > > > > request it, check for example [1] > > > > > > also you need to run [2] tests. > > > > > > [1] > > > > > > http://apache-ignite-developers.2346864.n4.nabble.com/Phani-Introduction-td47788.html > > > [2] https://mtcga.gridgain.com > > >>Hello, > > >> > > >>Look at the ticket and the only comment I can see is creating a branch > on > > >>git in the main repo and not in my fork. I do not have the right to > > create > > >> a > > >>branch in the main repository. Am i missing something? > > >> > > >>Sorry I probably misread the document but I though that I should fork > the > > >>repo and then pull request as I do not have the rights to create a > > branch. > > >> > > >>Thanks for your help > > >> > > >> > > >> > > >>-- > > >>Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > > > > > > > > > > > > > > > > > -- > > > > Best regards, > > Ivan Pavlukhin > > > -- Sincerely yours, Ivan Daschinskiy
Re: Re[2]: IGNITE-6499 Compact NULL fields
Hello fellow devs, I just wanted to ask, how would this Binary Object format change affect thin clients? C++/.Net nodes? Etc? Is it fully backwards compatible or not? I think that realistically, we can only add binary-incompatible changes to Binary Object format in 3.0. Regards, -- Ilya Kasnacheev ср, 8 июл. 2020 г. в 09:05, Ivan Pavlukhin : > A side note. Now we have a neat URL for TC bot > https://mtcga.ignite.apache.org/ (along with one in a gridgain > domain). > > 2020-07-07 18:43 GMT+03:00, Zhenya Stanilovsky >: > > > > request it, check for example [1] > > > > also you need to run [2] tests. > > > > [1] > > > http://apache-ignite-developers.2346864.n4.nabble.com/Phani-Introduction-td47788.html > > [2] https://mtcga.gridgain.com > >>Hello, > >> > >>Look at the ticket and the only comment I can see is creating a branch on > >>git in the main repo and not in my fork. I do not have the right to > create > >> a > >>branch in the main repository. Am i missing something? > >> > >>Sorry I probably misread the document but I though that I should fork the > >>repo and then pull request as I do not have the rights to create a > branch. > >> > >>Thanks for your help > >> > >> > >> > >>-- > >>Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > > > > > > > > > > -- > > Best regards, > Ivan Pavlukhin >
Re: Re[2]: IGNITE-6499 Compact NULL fields
A side note. Now we have a neat URL for TC bot https://mtcga.ignite.apache.org/ (along with one in a gridgain domain). 2020-07-07 18:43 GMT+03:00, Zhenya Stanilovsky : > > request it, check for example [1] > > also you need to run [2] tests. > > [1] > http://apache-ignite-developers.2346864.n4.nabble.com/Phani-Introduction-td47788.html > [2] https://mtcga.gridgain.com >>Hello, >> >>Look at the ticket and the only comment I can see is creating a branch on >>git in the main repo and not in my fork. I do not have the right to create >> a >>branch in the main repository. Am i missing something? >> >>Sorry I probably misread the document but I though that I should fork the >>repo and then pull request as I do not have the rights to create a branch. >> >>Thanks for your help >> >> >> >>-- >>Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > > > -- Best regards, Ivan Pavlukhin
Re[2]: IGNITE-6499 Compact NULL fields
request it, check for example [1] also you need to run [2] tests. [1] http://apache-ignite-developers.2346864.n4.nabble.com/Phani-Introduction-td47788.html [2] https://mtcga.gridgain.com >Hello, > >Look at the ticket and the only comment I can see is creating a branch on >git in the main repo and not in my fork. I do not have the right to create a >branch in the main repository. Am i missing something? > >Sorry I probably misread the document but I though that I should fork the >repo and then pull request as I do not have the rights to create a branch. > >Thanks for your help > > > >-- >Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re: Re[2]: IGNITE-6499 Compact NULL fields
Sorry I do not actual get what are you opposing? the compress of the binary or the null compaction or both? And can you ellaborate on why you are opposing it? -- Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
Re[2]: IGNITE-6499 Compact NULL fields
I`m currently against this approach, everyone can previously compress Binary Object for further using, no additional code need here. This discussion only about currently not optimal null storing and looks like we can improve it without performance pay. >Понедельник, 25 мая 2020, 13:42 +03:00 от Ilya Kasnacheev >: > >Hello! > >That would be nice! My preferred compression method is zstd (it also has >dictionary generation built in). > >Regards, >-- >Ilya Kasnacheev > > >пн, 25 мая 2020 г. в 13:25, Hostettler, Steve < >steve.hostett...@wolterskluwer.com >: > >> I like the idea, especially because it also would apply across the board. >> So you propose to build the binary object and to apply dictionary based >> compression on top. >> >> I could quickly generate a bunch of binary objects from the tests and >> apply java compress/deflate with a dictionary based on the BinaryUtils >> elements. >> To compare with the null compaction and the varint. >> >> >> -Original Message- >> From: Ilya Kasnacheev < ilya.kasnach...@gmail.com > >> Sent: Monday, May 25, 2020 12:05 PM >> To: dev < dev@ignite.apache.org > >> Subject: Re: IGNITE-6499 Compact NULL fields >> >> Caution, this email may be from a sender outside Wolters Kluwer. Verify >> the sender and know the content is safe. >> >> Hello! >> >> My take is the following: if conserving memory is needed at all, then we >> better invest in compression (such as dictionary-based row compression) >> rather than implementing varint, compact nulls, etc. >> >> Dictionary-based compression can easily tackle varints, null patterns >> while also compressing strings and repeated values and even things we would >> never think out on our own. >> >> It also has low complexity of our own code, no compatibility issues >> (people store binary objects in 3rd party storage, they do indeed) and low >> incidence of bugs. >> >> Regards, >> -- >> Ilya Kasnacheev >> >> >> пн, 25 мая 2020 г. в 12:51, Hostettler, Steve < >> steve.hostett...@wolterskluwer.com >: >> >> > I went for a simpler approach (only with null mask( and yes the gain >> > is high for smaller object but low otherwise. I gain between 5-20% on >> > my objects. But to me it is the step stone to easily implement other >> > optimisations like varint and schemaless without using raw. Trying to >> > solve the latest unit tests to give you a better idea. If not worth >> > then let's not do it but it is worth a try I think. >> > >> > >> > -Original Message- >> > From: Ilya Kasnacheev < ilya.kasnach...@gmail.com > >> > Sent: Monday, May 25, 2020 11:48 AM >> > To: dev < dev@ignite.apache.org > >> > Subject: Re: IGNITE-6499 Compact NULL fields >> > >> > Caution, this email may be from a sender outside Wolters Kluwer. >> > Verify the sender and know the content is safe. >> > >> > Hello! >> > >> > I can't help myself but wonder how large of a benefit will it give. I >> > have checked the ticket description, it looks the proposed scheme is >> > elaborate and benefit for non-extreme binary objects rather tiny. >> > >> > WDYT? >> > >> > Regards, >> > -- >> > Ilya Kasnacheev >> > >> > >> > пн, 18 мая 2020 г. в 22:54, steve.hostett...@gmail.com < >> > steve.hostett...@gmail.com >: >> > >> > > Hello igniters, >> > > >> > > while I would like to help on the calcite because H2 optimiser (or >> > > the lack >> > > thereof) is really killing us, I think that it would be wiser to >> > > start by contributing on something easier. >> > > >> > > Therefore I will tackle another problem that we have which is the >> > > memory consumption. I stumbled upon this IEP >> > > >> > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcw >> > > ik >> > > i.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FIEP-2%253A%2BBinary%2 >> > > Bo >> > > bject%2Bformat%2Bimprovementsdata=02%7C01%7CSteve.Hostettler%40 >> > > wo >> > > lterskluwer.com%7C7568148487434617407b08d80090b1f2%7C8ac76c91e7f141f >> > > fa >> > > 89c3553b2da2c17%7C0%7C0%7C637259968758509764sdata=ZNFJ5gqEXRv5K >> > > R3 >> > > HJUfYZ4rmnGwCiFVGg4IrWTORT2k%3Dreserved=0 >> > > < >> > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcw >> > > ik >> > > i.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FIEP-2%253A%2BBinary%2 >> > > Bo >> > > bject%2Bformat%2Bimprovementsdata=02%7C01%7CSteve.Hostettler%40 >> > > wo >> > > lterskluwer.com%7C7568148487434617407b08d80090b1f2%7C8ac76c91e7f141f >> > > fa >> > > 89c3553b2da2c17%7C0%7C0%7C637259968758509764sdata=ZNFJ5gqEXRv5K >> > > R3 HJUfYZ4rmnGwCiFVGg4IrWTORT2k%3Dreserved=0> >> > > >> > > that is about optimising the binary marshaller. >> > > >> > > The low hanging fruit seemed to be the null compaction so I decided >> > > to start with it. Though I am sure I do see some hidden complexity. >> > > >> > > Here a couple of questions: >> > > - Can I assign myself IGNITE-6499 and attach a patch? >> > > - Who can I contact to help with the review. In the following page >> > >
