[jira] [Commented] (IGNITE-12684) Optimize count of threads for rebuild indexes
[
https://issues.apache.org/jira/browse/IGNITE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051895#comment-17051895
]
Kirill Tkalenko commented on IGNITE-12684:
--
[~alex_pl] Through the setting, you can set how much you need.
[~mmuzaf] here is IGNITE-12750 ticket.
> Optimize count of threads for rebuild indexes
> -
>
> Key: IGNITE-12684
> URL: https://issues.apache.org/jira/browse/IGNITE-12684
> Project: Ignite
> Issue Type: Improvement
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
> Fix For: 2.9
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We were recently implemented the new property
> {{INDEX_REBUILDING_PARALLELISM}} (by issue IGNITE-11979).
> That fix do possible to rebuild indexes, after restart without index.bin or
> restore from snapshot to another topology, in parallel threads.
> But now the process of rebuild can to consume a lot of threads, which limited
> on INDEX_REBUILDING_PARALLELISM * AVAILABLE_PROC_CNT (count of thread in
> public pool).
> For example in case of INDEX_REBUILDING_PARALLELISM = 28 and 64 CPU it will
> be more then 1.5К threads.
> Look at the method, it will create dedicate thread on each invocation
> `org.apache.ignite.internal.processors.query.schema.SchemaIndexCacheVisitorImpl#processPartitionsAsync`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-12684) Optimize count of threads for rebuild indexes
[
https://issues.apache.org/jira/browse/IGNITE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051150#comment-17051150
]
Maxim Muzafarov commented on IGNITE-12684:
--
[~ktkale...@gridgain.com]
Will you file a JIRA ticket to fix the issues mentioned above?
> Optimize count of threads for rebuild indexes
> -
>
> Key: IGNITE-12684
> URL: https://issues.apache.org/jira/browse/IGNITE-12684
> Project: Ignite
> Issue Type: Improvement
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
> Fix For: 2.9
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We were recently implemented the new property
> {{INDEX_REBUILDING_PARALLELISM}} (by issue IGNITE-11979).
> That fix do possible to rebuild indexes, after restart without index.bin or
> restore from snapshot to another topology, in parallel threads.
> But now the process of rebuild can to consume a lot of threads, which limited
> on INDEX_REBUILDING_PARALLELISM * AVAILABLE_PROC_CNT (count of thread in
> public pool).
> For example in case of INDEX_REBUILDING_PARALLELISM = 28 and 64 CPU it will
> be more then 1.5К threads.
> Look at the method, it will create dedicate thread on each invocation
> `org.apache.ignite.internal.processors.query.schema.SchemaIndexCacheVisitorImpl#processPartitionsAsync`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-12684) Optimize count of threads for rebuild indexes
[
https://issues.apache.org/jira/browse/IGNITE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17047479#comment-17047479
]
Aleksey Plekhanov commented on IGNITE-12684:
[~ktkale...@gridgain.com] also, maximum 4 threads as default value for index
rebuilding seems too low. There were 4 threads for each cache before, but now
by default it will be 4 threads for entire node.
> Optimize count of threads for rebuild indexes
> -
>
> Key: IGNITE-12684
> URL: https://issues.apache.org/jira/browse/IGNITE-12684
> Project: Ignite
> Issue Type: Improvement
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
> Fix For: 2.9
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We were recently implemented the new property
> {{INDEX_REBUILDING_PARALLELISM}} (by issue IGNITE-11979).
> That fix do possible to rebuild indexes, after restart without index.bin or
> restore from snapshot to another topology, in parallel threads.
> But now the process of rebuild can to consume a lot of threads, which limited
> on INDEX_REBUILDING_PARALLELISM * AVAILABLE_PROC_CNT (count of thread in
> public pool).
> For example in case of INDEX_REBUILDING_PARALLELISM = 28 and 64 CPU it will
> be more then 1.5К threads.
> Look at the method, it will create dedicate thread on each invocation
> `org.apache.ignite.internal.processors.query.schema.SchemaIndexCacheVisitorImpl#processPartitionsAsync`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-12684) Optimize count of threads for rebuild indexes
[
https://issues.apache.org/jira/browse/IGNITE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17047267#comment-17047267
]
Kirill Tkalenko commented on IGNITE-12684:
--
[~mmuzaf] thank you, you are right, I tried locally and it turned out that only
1 thread works.
> Optimize count of threads for rebuild indexes
> -
>
> Key: IGNITE-12684
> URL: https://issues.apache.org/jira/browse/IGNITE-12684
> Project: Ignite
> Issue Type: Improvement
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
> Fix For: 2.9
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We were recently implemented the new property
> {{INDEX_REBUILDING_PARALLELISM}} (by issue IGNITE-11979).
> That fix do possible to rebuild indexes, after restart without index.bin or
> restore from snapshot to another topology, in parallel threads.
> But now the process of rebuild can to consume a lot of threads, which limited
> on INDEX_REBUILDING_PARALLELISM * AVAILABLE_PROC_CNT (count of thread in
> public pool).
> For example in case of INDEX_REBUILDING_PARALLELISM = 28 and 64 CPU it will
> be more then 1.5К threads.
> Look at the method, it will create dedicate thread on each invocation
> `org.apache.ignite.internal.processors.query.schema.SchemaIndexCacheVisitorImpl#processPartitionsAsync`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-12684) Optimize count of threads for rebuild indexes
[
https://issues.apache.org/jira/browse/IGNITE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046835#comment-17046835
]
Maxim Muzafarov commented on IGNITE-12684:
--
[~ktkale...@gridgain.com], [~ascherbakov], [~zstan]
Folks,
I've just suddenly found that your fix would not work as you expected. The root
cause is related to this line of code:
https://github.com/apache/ignite/pull/7432/files#diff-e7d7d29811367d4d13a05c912b0ffbdaR1963
With this configuration, all index rebuild procedures will be processed
single-threaded. This is my explanation of why. Look here [1]: If there are
more than corePoolSize but less than maximumPoolSize threads running, a new
thread will be created only if the queue is full. This means that if you define
a thread pool with the following code (corePoolSize==0), it will never start
the 2nd thread because the LinkedBlockingQueue is unbounded.
[1]
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/java/util/concurrent/ThreadPoolExecutor.java#l83
> Optimize count of threads for rebuild indexes
> -
>
> Key: IGNITE-12684
> URL: https://issues.apache.org/jira/browse/IGNITE-12684
> Project: Ignite
> Issue Type: Improvement
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
> Fix For: 2.9
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We were recently implemented the new property
> {{INDEX_REBUILDING_PARALLELISM}} (by issue IGNITE-11979).
> That fix do possible to rebuild indexes, after restart without index.bin or
> restore from snapshot to another topology, in parallel threads.
> But now the process of rebuild can to consume a lot of threads, which limited
> on INDEX_REBUILDING_PARALLELISM * AVAILABLE_PROC_CNT (count of thread in
> public pool).
> For example in case of INDEX_REBUILDING_PARALLELISM = 28 and 64 CPU it will
> be more then 1.5К threads.
> Look at the method, it will create dedicate thread on each invocation
> `org.apache.ignite.internal.processors.query.schema.SchemaIndexCacheVisitorImpl#processPartitionsAsync`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-12684) Optimize count of threads for rebuild indexes
[
https://issues.apache.org/jira/browse/IGNITE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17041679#comment-17041679
]
Alexey Scherbakov commented on IGNITE-12684:
[~ktkale...@gridgain.com]
Looks good.
Merged to master f52350c552f957e4c867bab9431b45ea1f16dc62.
Thanks for the contribution.
> Optimize count of threads for rebuild indexes
> -
>
> Key: IGNITE-12684
> URL: https://issues.apache.org/jira/browse/IGNITE-12684
> Project: Ignite
> Issue Type: Improvement
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
> Fix For: 2.9
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We were recently implemented the new property
> {{INDEX_REBUILDING_PARALLELISM}} (by issue IGNITE-11979).
> That fix do possible to rebuild indexes, after restart without index.bin or
> restore from snapshot to another topology, in parallel threads.
> But now the process of rebuild can to consume a lot of threads, which limited
> on INDEX_REBUILDING_PARALLELISM * AVAILABLE_PROC_CNT (count of thread in
> public pool).
> For example in case of INDEX_REBUILDING_PARALLELISM = 28 and 64 CPU it will
> be more then 1.5К threads.
> Look at the method, it will create dedicate thread on each invocation
> `org.apache.ignite.internal.processors.query.schema.SchemaIndexCacheVisitorImpl#processPartitionsAsync`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-12684) Optimize count of threads for rebuild indexes
[
https://issues.apache.org/jira/browse/IGNITE-12684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039485#comment-17039485
]
Ignite TC Bot commented on IGNITE-12684:
{panel:title=Branch: [pull/7432/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=5060658buildTypeId=IgniteTests24Java8_RunAll]
> Optimize count of threads for rebuild indexes
> -
>
> Key: IGNITE-12684
> URL: https://issues.apache.org/jira/browse/IGNITE-12684
> Project: Ignite
> Issue Type: Improvement
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
> Fix For: 2.9
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We were recently implemented the new property
> {{INDEX_REBUILDING_PARALLELISM}} (by issue IGNITE-11979).
> That fix do possible to rebuild indexes, after restart without index.bin or
> restore from snapshot to another topology, in parallel threads.
> But now the process of rebuild can to consume a lot of threads, which limited
> on INDEX_REBUILDING_PARALLELISM * AVAILABLE_PROC_CNT (count of thread in
> public pool).
> For example in case of INDEX_REBUILDING_PARALLELISM = 28 and 64 CPU it will
> be more then 1.5К threads.
> Look at the method, it will create dedicate thread on each invocation
> `org.apache.ignite.internal.processors.query.schema.SchemaIndexCacheVisitorImpl#processPartitionsAsync`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
