[
https://issues.apache.org/jira/browse/IGNITE-19530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17725856#comment-17725856
]
Roman Puchkovskiy commented on IGNITE-19530:
--------------------------------------------
It is not that easy to implement what was requested, so I conducted an
experiment and reduced key sizes in a dirty way. For the longest table-related
keys, the reduction is almost twofold, and (with the test in IGNITE-19275)
there are 40 such keys of 55 per table (the remaining keys are much shorter),
so the total reduction of keys is around 30%.
After enabling this optimization, I did not see any difference: with it and
without it, the test was failing after creating approximately 45 tables.
I also discovered that most of the commands that are sent to the Metastorage
RAFT group are writes under placementdriver.lease (around 16k writes, versus 93
writes for table metadata). When I disabled writing of leases, the situation
improved drasticatlly, even though a big dispersion emerged on my machine: the
test started failing after 341 or 676 tables, and once it even created all 1000
tables.
But even with disabled lease writing, enabling or disabling the keys
optimization I mentioned in the beginning has no influence on the results.
So I suggest the following:
1. Create an issue about making table config keys lighter, but with low priority
2. Close this ticket as the suggested optimization does not change anything in
the 'problem of 1000 tables'
[~Denis Chudov] , [~sanpwc] what do you think?
> Reduce size of configuration keys
> ---------------------------------
>
> Key: IGNITE-19530
> URL: https://issues.apache.org/jira/browse/IGNITE-19530
> Project: Ignite
> Issue Type: Improvement
> Reporter: Denis Chudov
> Assignee: Roman Puchkovskiy
> Priority: Major
> Labels: ignite-3
>
> *Motivation*
> The distributed configuration keys are byte arrays formed from strings that
> contain some constant prefixes, postfixes, delimiters and identificators,
> mostly UUIDs. Example of the configuration key for default value provider
> type of table column:
> {{dst-cfg.table.tables.d7b99c6a-de10-454d-9370-38d18b65e9c0.columns.d8482dae-cfb8-42b8-a759-9727dd3763a6.defaultValueProvider.type}}
> It contains 2 UUIDs in string representation. Unfortunately, there are
> several configuration entries for each table column (having similar keys) and
> besides that about a dozen of keys for table itself.
> As a result, configuration keys take 68% of a meta storage message related to
> table creation (for one node cluster, for a table of 2 columns and 25
> partitions) which creates excessive load on meta storage raft group in case
> of mass table creation (see IGNITE-19275 )
> *Definition of done*
> We should get rid of string representation of UUIDs in configuration keys,
> UUIDs should be written as 16 bytes each into byte array directly. Also,
> string constants should be reduced (or even replaced to constants consisting
> of few bytes) because there is no need to keep them human readable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
