Re: IGNITE-13364: Improve index inline defaults
Hi! JFYI, it was merged to master today, then this patch will be part of 2.12 only. On Thu, Jun 10, 2021 at 5:00 PM Maksim Timonin wrote: > Hi, > > I've prepared a patch for this feature. Can anybody have a look at it? > > https://github.com/apache/ignite/pull/9167 > https://issues.apache.org/jira/browse/IGNITE-13364 > > On Mon, Jun 7, 2021 at 12:14 PM Maksim Timonin > wrote: > >> Hi, I've left a comment in your PR, could you please have a look? >> >> On Sat, Jun 5, 2021 at 6:09 AM 18624049226 <18624049...@163.com> wrote: >> >>> Hello team, >>> >>> I think this is a very valuable optimization. This patch has been >>> developed for a long time, I don't think there is any reason to continue >>> to delay the release of this patch to version 2.12. >>> Is anyone willing to push this forward and merge this patch into the >>> master? >>> >>> 2021/1/27 下午4:52, Maxim Muzafarov 写道: >>> > Stanislav, >>> > >>> >> I think the most compatibility impact will be on the in-memory caches >>> with SQL and without explicitly specified inline sizes. >>> > I don't think that this is `true` compatibility issue. But I think we >>> > should at least mentioned it on our documentation pages and in the >>> > release notes. So, I'm +1 to proceed with the merge. >>> > >>> > Should we include the issue into 2.10 release? >>> > >>> > On Fri, 8 Jan 2021 at 16:50, Stanislav Lukyanov < >>> stanlukya...@gmail.com> wrote: >>> >> Hi Igniters, >>> >> >>> >> I'd like to discuss the change implemented by Evgeniy Rudenko in the >>> ticket https://issues.apache.org/jira/browse/IGNITE-13364. >>> >> I see that the fix is ready for review and merging, and I'm >>> interested in it so I'd like to pick it up on the last mile. >>> >> I also wanted to bring community's attention to it before the merge >>> as it changes the default behavior. >>> >> >>> >> The patch changes how SQL index inline size is calculated. >>> >> >>> >> Specifically: >>> >> >>> >> 1. When inline size is calculated for a variable-sized field, a >>> constant 10 (configurable via IGNITE_VARIABLE_TYPE_DEFAULT_INDEX_SIZE) is >>> added to the calculated size instead of setting the entire calculation >>> result to 10. >>> >> For example, consider the following cases >>> >> >>> >> Index (int, int, string) >>> >> Before the change: inline size = 10 >>> >> After the change: inline size = 5 + 5 + 10 = 20 >>> >> >>> >> Index (long, long, long, long, string) >>> >> Before the change: inline size = 10 >>> >> After the change: inline size = 9 + 9 + 9 + 9 + 10 = 46 >>> >> >>> >> 2. If there is a VARCHAR_FIXED, e.g. VARCHAR(5), then instead of the >>> default IGNITE_VARIABLE_TYPE_DEFAULT_INDEX_SIZE Ignite will use the value >>> provided in the calculation >>> >> >>> >> 3. If the calculated size exceeds >>> IGNITE_MAX_INDEX_PAYLOAD_SIZE_DEFAULT=64, it will be truncated to 64. >>> >> >>> >> All of this only affects calculated inline sizes of new indexes. >>> >> Existing indexes should not be affected. >>> >> Indexes with explicitly specified inline size should not be affected. >>> >> >>> >> I think the most compatibility impact will be on the in-memory caches >>> with SQL and without explicitly specified inline sizes. >>> >> After the upgrade these caches may slightly increase in size (because >>> the inline is likely to become bigger) while SQL is also likely to become >>> faster. >>> >> >>> >> Please comment. >>> >> If there are no concerns, I'll proceed with finding a committer to >>> review and merge the fix at the end of the next week. >>> >> >>> >> Thanks, >>> >> Stan >>> >>
Re: IGNITE-13364: Improve index inline defaults
Hi, I've prepared a patch for this feature. Can anybody have a look at it? https://github.com/apache/ignite/pull/9167 https://issues.apache.org/jira/browse/IGNITE-13364 On Mon, Jun 7, 2021 at 12:14 PM Maksim Timonin wrote: > Hi, I've left a comment in your PR, could you please have a look? > > On Sat, Jun 5, 2021 at 6:09 AM 18624049226 <18624049...@163.com> wrote: > >> Hello team, >> >> I think this is a very valuable optimization. This patch has been >> developed for a long time, I don't think there is any reason to continue >> to delay the release of this patch to version 2.12. >> Is anyone willing to push this forward and merge this patch into the >> master? >> >> 2021/1/27 下午4:52, Maxim Muzafarov 写道: >> > Stanislav, >> > >> >> I think the most compatibility impact will be on the in-memory caches >> with SQL and without explicitly specified inline sizes. >> > I don't think that this is `true` compatibility issue. But I think we >> > should at least mentioned it on our documentation pages and in the >> > release notes. So, I'm +1 to proceed with the merge. >> > >> > Should we include the issue into 2.10 release? >> > >> > On Fri, 8 Jan 2021 at 16:50, Stanislav Lukyanov >> wrote: >> >> Hi Igniters, >> >> >> >> I'd like to discuss the change implemented by Evgeniy Rudenko in the >> ticket https://issues.apache.org/jira/browse/IGNITE-13364. >> >> I see that the fix is ready for review and merging, and I'm interested >> in it so I'd like to pick it up on the last mile. >> >> I also wanted to bring community's attention to it before the merge as >> it changes the default behavior. >> >> >> >> The patch changes how SQL index inline size is calculated. >> >> >> >> Specifically: >> >> >> >> 1. When inline size is calculated for a variable-sized field, a >> constant 10 (configurable via IGNITE_VARIABLE_TYPE_DEFAULT_INDEX_SIZE) is >> added to the calculated size instead of setting the entire calculation >> result to 10. >> >> For example, consider the following cases >> >> >> >> Index (int, int, string) >> >> Before the change: inline size = 10 >> >> After the change: inline size = 5 + 5 + 10 = 20 >> >> >> >> Index (long, long, long, long, string) >> >> Before the change: inline size = 10 >> >> After the change: inline size = 9 + 9 + 9 + 9 + 10 = 46 >> >> >> >> 2. If there is a VARCHAR_FIXED, e.g. VARCHAR(5), then instead of the >> default IGNITE_VARIABLE_TYPE_DEFAULT_INDEX_SIZE Ignite will use the value >> provided in the calculation >> >> >> >> 3. If the calculated size exceeds >> IGNITE_MAX_INDEX_PAYLOAD_SIZE_DEFAULT=64, it will be truncated to 64. >> >> >> >> All of this only affects calculated inline sizes of new indexes. >> >> Existing indexes should not be affected. >> >> Indexes with explicitly specified inline size should not be affected. >> >> >> >> I think the most compatibility impact will be on the in-memory caches >> with SQL and without explicitly specified inline sizes. >> >> After the upgrade these caches may slightly increase in size (because >> the inline is likely to become bigger) while SQL is also likely to become >> faster. >> >> >> >> Please comment. >> >> If there are no concerns, I'll proceed with finding a committer to >> review and merge the fix at the end of the next week. >> >> >> >> Thanks, >> >> Stan >> >
Re: IGNITE-13364: Improve index inline defaults
Hi, I've left a comment in your PR, could you please have a look? On Sat, Jun 5, 2021 at 6:09 AM 18624049226 <18624049...@163.com> wrote: > Hello team, > > I think this is a very valuable optimization. This patch has been > developed for a long time, I don't think there is any reason to continue > to delay the release of this patch to version 2.12. > Is anyone willing to push this forward and merge this patch into the > master? > > 2021/1/27 下午4:52, Maxim Muzafarov 写道: > > Stanislav, > > > >> I think the most compatibility impact will be on the in-memory caches > with SQL and without explicitly specified inline sizes. > > I don't think that this is `true` compatibility issue. But I think we > > should at least mentioned it on our documentation pages and in the > > release notes. So, I'm +1 to proceed with the merge. > > > > Should we include the issue into 2.10 release? > > > > On Fri, 8 Jan 2021 at 16:50, Stanislav Lukyanov > wrote: > >> Hi Igniters, > >> > >> I'd like to discuss the change implemented by Evgeniy Rudenko in the > ticket https://issues.apache.org/jira/browse/IGNITE-13364. > >> I see that the fix is ready for review and merging, and I'm interested > in it so I'd like to pick it up on the last mile. > >> I also wanted to bring community's attention to it before the merge as > it changes the default behavior. > >> > >> The patch changes how SQL index inline size is calculated. > >> > >> Specifically: > >> > >> 1. When inline size is calculated for a variable-sized field, a > constant 10 (configurable via IGNITE_VARIABLE_TYPE_DEFAULT_INDEX_SIZE) is > added to the calculated size instead of setting the entire calculation > result to 10. > >> For example, consider the following cases > >> > >> Index (int, int, string) > >> Before the change: inline size = 10 > >> After the change: inline size = 5 + 5 + 10 = 20 > >> > >> Index (long, long, long, long, string) > >> Before the change: inline size = 10 > >> After the change: inline size = 9 + 9 + 9 + 9 + 10 = 46 > >> > >> 2. If there is a VARCHAR_FIXED, e.g. VARCHAR(5), then instead of the > default IGNITE_VARIABLE_TYPE_DEFAULT_INDEX_SIZE Ignite will use the value > provided in the calculation > >> > >> 3. If the calculated size exceeds > IGNITE_MAX_INDEX_PAYLOAD_SIZE_DEFAULT=64, it will be truncated to 64. > >> > >> All of this only affects calculated inline sizes of new indexes. > >> Existing indexes should not be affected. > >> Indexes with explicitly specified inline size should not be affected. > >> > >> I think the most compatibility impact will be on the in-memory caches > with SQL and without explicitly specified inline sizes. > >> After the upgrade these caches may slightly increase in size (because > the inline is likely to become bigger) while SQL is also likely to become > faster. > >> > >> Please comment. > >> If there are no concerns, I'll proceed with finding a committer to > review and merge the fix at the end of the next week. > >> > >> Thanks, > >> Stan >
Re: IGNITE-13364: Improve index inline defaults
Hello team, I think this is a very valuable optimization. This patch has been developed for a long time, I don't think there is any reason to continue to delay the release of this patch to version 2.12. Is anyone willing to push this forward and merge this patch into the master? 2021/1/27 下午4:52, Maxim Muzafarov 写道: Stanislav, I think the most compatibility impact will be on the in-memory caches with SQL and without explicitly specified inline sizes. I don't think that this is `true` compatibility issue. But I think we should at least mentioned it on our documentation pages and in the release notes. So, I'm +1 to proceed with the merge. Should we include the issue into 2.10 release? On Fri, 8 Jan 2021 at 16:50, Stanislav Lukyanov wrote: Hi Igniters, I'd like to discuss the change implemented by Evgeniy Rudenko in the ticket https://issues.apache.org/jira/browse/IGNITE-13364. I see that the fix is ready for review and merging, and I'm interested in it so I'd like to pick it up on the last mile. I also wanted to bring community's attention to it before the merge as it changes the default behavior. The patch changes how SQL index inline size is calculated. Specifically: 1. When inline size is calculated for a variable-sized field, a constant 10 (configurable via IGNITE_VARIABLE_TYPE_DEFAULT_INDEX_SIZE) is added to the calculated size instead of setting the entire calculation result to 10. For example, consider the following cases Index (int, int, string) Before the change: inline size = 10 After the change: inline size = 5 + 5 + 10 = 20 Index (long, long, long, long, string) Before the change: inline size = 10 After the change: inline size = 9 + 9 + 9 + 9 + 10 = 46 2. If there is a VARCHAR_FIXED, e.g. VARCHAR(5), then instead of the default IGNITE_VARIABLE_TYPE_DEFAULT_INDEX_SIZE Ignite will use the value provided in the calculation 3. If the calculated size exceeds IGNITE_MAX_INDEX_PAYLOAD_SIZE_DEFAULT=64, it will be truncated to 64. All of this only affects calculated inline sizes of new indexes. Existing indexes should not be affected. Indexes with explicitly specified inline size should not be affected. I think the most compatibility impact will be on the in-memory caches with SQL and without explicitly specified inline sizes. After the upgrade these caches may slightly increase in size (because the inline is likely to become bigger) while SQL is also likely to become faster. Please comment. If there are no concerns, I'll proceed with finding a committer to review and merge the fix at the end of the next week. Thanks, Stan
Re: IGNITE-13364: Improve index inline defaults
Stanislav, > I think the most compatibility impact will be on the in-memory caches with > SQL and without explicitly specified inline sizes. I don't think that this is `true` compatibility issue. But I think we should at least mentioned it on our documentation pages and in the release notes. So, I'm +1 to proceed with the merge. Should we include the issue into 2.10 release? On Fri, 8 Jan 2021 at 16:50, Stanislav Lukyanov wrote: > > Hi Igniters, > > I'd like to discuss the change implemented by Evgeniy Rudenko in the ticket > https://issues.apache.org/jira/browse/IGNITE-13364. > I see that the fix is ready for review and merging, and I'm interested in it > so I'd like to pick it up on the last mile. > I also wanted to bring community's attention to it before the merge as it > changes the default behavior. > > The patch changes how SQL index inline size is calculated. > > Specifically: > > 1. When inline size is calculated for a variable-sized field, a constant 10 > (configurable via IGNITE_VARIABLE_TYPE_DEFAULT_INDEX_SIZE) is added to the > calculated size instead of setting the entire calculation result to 10. > For example, consider the following cases > > Index (int, int, string) > Before the change: inline size = 10 > After the change: inline size = 5 + 5 + 10 = 20 > > Index (long, long, long, long, string) > Before the change: inline size = 10 > After the change: inline size = 9 + 9 + 9 + 9 + 10 = 46 > > 2. If there is a VARCHAR_FIXED, e.g. VARCHAR(5), then instead of the default > IGNITE_VARIABLE_TYPE_DEFAULT_INDEX_SIZE Ignite will use the value provided in > the calculation > > 3. If the calculated size exceeds IGNITE_MAX_INDEX_PAYLOAD_SIZE_DEFAULT=64, > it will be truncated to 64. > > All of this only affects calculated inline sizes of new indexes. > Existing indexes should not be affected. > Indexes with explicitly specified inline size should not be affected. > > I think the most compatibility impact will be on the in-memory caches with > SQL and without explicitly specified inline sizes. > After the upgrade these caches may slightly increase in size (because the > inline is likely to become bigger) while SQL is also likely to become faster. > > Please comment. > If there are no concerns, I'll proceed with finding a committer to review and > merge the fix at the end of the next week. > > Thanks, > Stan
IGNITE-13364: Improve index inline defaults
Hi Igniters, I'd like to discuss the change implemented by Evgeniy Rudenko in the ticket https://issues.apache.org/jira/browse/IGNITE-13364. I see that the fix is ready for review and merging, and I'm interested in it so I'd like to pick it up on the last mile. I also wanted to bring community's attention to it before the merge as it changes the default behavior. The patch changes how SQL index inline size is calculated. Specifically: 1. When inline size is calculated for a variable-sized field, a constant 10 (configurable via IGNITE_VARIABLE_TYPE_DEFAULT_INDEX_SIZE) is added to the calculated size instead of setting the entire calculation result to 10. For example, consider the following cases Index (int, int, string) Before the change: inline size = 10 After the change: inline size = 5 + 5 + 10 = 20 Index (long, long, long, long, string) Before the change: inline size = 10 After the change: inline size = 9 + 9 + 9 + 9 + 10 = 46 2. If there is a VARCHAR_FIXED, e.g. VARCHAR(5), then instead of the default IGNITE_VARIABLE_TYPE_DEFAULT_INDEX_SIZE Ignite will use the value provided in the calculation 3. If the calculated size exceeds IGNITE_MAX_INDEX_PAYLOAD_SIZE_DEFAULT=64, it will be truncated to 64. All of this only affects calculated inline sizes of new indexes. Existing indexes should not be affected. Indexes with explicitly specified inline size should not be affected. I think the most compatibility impact will be on the in-memory caches with SQL and without explicitly specified inline sizes. After the upgrade these caches may slightly increase in size (because the inline is likely to become bigger) while SQL is also likely to become faster. Please comment. If there are no concerns, I'll proceed with finding a committer to review and merge the fix at the end of the next week. Thanks, Stan
[jira] [Created] (IGNITE-13364) Improve index inline defaults
Evgeniy Rudenko created IGNITE-13364: Summary: Improve index inline defaults Key: IGNITE-13364 URL: https://issues.apache.org/jira/browse/IGNITE-13364 Project: Ignite Issue Type: Improvement Reporter: Evgeniy Rudenko Assignee: Evgeniy Rudenko We need to improve how inline size is calculated by default for variable-length types. Currently if a varlength type is encountered inline size just defaults to 10, which is almost always not enough. A more sensible behavior would be the following: 1. Add a fixed default to the inline size calculation for every variable-length type. For example, if the default inlined size for a string is 10 then an index like (INT, VARCHAR, VARCHAR, INT) should have inline size default as 5 + 10 + 10 + 5 = 30 (5 for each int, 10 for each string). 2. Add special support for VARCHAR_FIXED - if a VARCHAR has known length then that length is used for inline size calculation -- This message was sent by Atlassian Jira (v8.3.4#803005)
