[jira] [Comment Edited] (IGNITE-12374) Too low performance ~200TPS for single ODBC client
[ https://issues.apache.org/jira/browse/IGNITE-12374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982051#comment-16982051 ] swy edited comment on IGNITE-12374 at 11/26/19 2:19 AM: [~isapego] any clue to share? Or can I conclude this as the limit of apache ignite in ODBC case? Or any other channel we can bring this forward? was (Author: yow): [~isapego] any clue to share? Or can I conclude this is the limit of apache ignite? > Too low performance ~200TPS for single ODBC client > -- > > Key: IGNITE-12374 > URL: https://issues.apache.org/jira/browse/IGNITE-12374 > Project: Ignite > Issue Type: Improvement > Components: clients, odbc >Affects Versions: 2.7.5 > Environment: Ignite server run on top of Kubernetes, with 2 server > nodes, persistence enabled. Both CPU and RAM at server/client server is > sufficient according to system reports. >Reporter: swy >Priority: Major > Attachments: odbcsample.allchar.rebind.cc, odbcsample.cc, values.yaml > > > Hi, in our test ignite performance with ODBC connection is too bad to proceed > with product integration. It is about ~200 TPS, each transaction with > select+update operation. > Please refer to attach sample program. It is just a simple test case. > Based on the profiling most of the time consumed by sql execution. Please > advice if the application did not do the right thing. > Thank you. > local ignite server, but the result same to remote container Ignite > deployment too. > cat /etc/apache-ignite/my-config.xml > > http://www.springframework.org/schema/beans; >xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; >xsi:schemaLocation="http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans.xsd;> > class="org.apache.ignite.configuration.IgniteConfiguration"> > > class="org.apache.ignite.configuration.ClientConnectorConfiguration"/> > > > > > > class="org.apache.ignite.binary.BinaryBasicIdMapper"> > > > > > > > > g++ -I/usr/include -I./ignite/binary/include -I./ignite/common/include > -I./ignite/common/os/linux/include -I./ignite/common/os/win/include > -I./ignite/core/include -I./ignite/jni/include odbcsample.cc -o odbcsample > -lodbc -L./ignite-libs/libignite.so -L./ignite-libs/libignite-odbc.so -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-12374) Too low performance ~200TPS for single ODBC client
[ https://issues.apache.org/jira/browse/IGNITE-12374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16982051#comment-16982051 ] swy commented on IGNITE-12374: -- [~isapego] any clue to share? Or can I conclude this is the limit of apache ignite? > Too low performance ~200TPS for single ODBC client > -- > > Key: IGNITE-12374 > URL: https://issues.apache.org/jira/browse/IGNITE-12374 > Project: Ignite > Issue Type: Improvement > Components: clients, odbc >Affects Versions: 2.7.5 > Environment: Ignite server run on top of Kubernetes, with 2 server > nodes, persistence enabled. Both CPU and RAM at server/client server is > sufficient according to system reports. >Reporter: swy >Priority: Major > Attachments: odbcsample.allchar.rebind.cc, odbcsample.cc, values.yaml > > > Hi, in our test ignite performance with ODBC connection is too bad to proceed > with product integration. It is about ~200 TPS, each transaction with > select+update operation. > Please refer to attach sample program. It is just a simple test case. > Based on the profiling most of the time consumed by sql execution. Please > advice if the application did not do the right thing. > Thank you. > local ignite server, but the result same to remote container Ignite > deployment too. > cat /etc/apache-ignite/my-config.xml > > http://www.springframework.org/schema/beans; >xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; >xsi:schemaLocation="http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans.xsd;> > class="org.apache.ignite.configuration.IgniteConfiguration"> > > class="org.apache.ignite.configuration.ClientConnectorConfiguration"/> > > > > > > class="org.apache.ignite.binary.BinaryBasicIdMapper"> > > > > > > > > g++ -I/usr/include -I./ignite/binary/include -I./ignite/common/include > -I./ignite/common/os/linux/include -I./ignite/common/os/win/include > -I./ignite/core/include -I./ignite/jni/include odbcsample.cc -o odbcsample > -lodbc -L./ignite-libs/libignite.so -L./ignite-libs/libignite-odbc.so -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-12394) Confusing message and thread dumps about ignored failures
[
https://issues.apache.org/jira/browse/IGNITE-12394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981966#comment-16981966
]
Ignite TC Bot commented on IGNITE-12394:
{panel:title=Branch: [pull/7075/head] Base: [master] : Possible Blockers
(114)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}
{color:#d04437}Platform C++ (Linux)*{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792319]]
{color:#d04437}Cache 6{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792348]]
{color:#d04437}ZooKeeper (Discovery) 3{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792372]]
{color:#d04437}Scala (Examples){color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792315]]
{color:#d04437}PDS (Compatibility){color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792357]]
{color:#d04437}SPI{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792310]]
{color:#d04437}Platform .NET (Long Running){color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792368]]
{color:#d04437}PDS (Indexing){color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792358]]
{color:#d04437}ZooKeeper (Discovery) 1{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792317]]
{color:#d04437}Cache 8{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792350]]
{color:#d04437}ZooKeeper (Discovery) 2{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792318]]
{color:#d04437}Scala (Visor Console){color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792316]]
{color:#d04437}Examples{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792291]]
{color:#d04437}Cache 5{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792347]]
{color:#d04437}Cache (Expiry Policy){color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792333]]
{color:#d04437}Platform .NET{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792364]]
{color:#d04437}Streamers{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792308]]
{color:#d04437}Start Nodes{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792309]]
{color:#d04437}Cache 2{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792344]]
{color:#d04437}ZooKeeper (Discovery) 4{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792373]]
{color:#d04437}MVCC Queries{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792328]]
{color:#d04437}Thin Client: Java{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792301]]
{color:#d04437}Platform .NET (Core Linux){color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792365]]
{color:#d04437}Queries 1{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792370]]
{color:#d04437}Continuous Query 4{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792354]]
{color:#d04437}Continuous Query 3{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792353]]
{color:#d04437}Cache 1{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792343]]
{color:#d04437}PDS 1{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792360]]
{color:#d04437}Compute (Grid){color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792288]]
{color:#d04437}Cache 7{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792349]]
{color:#d04437}MVCC Cache 1{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792378]]
{color:#d04437}Basic 1{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792322]]
{color:#d04437}Queries 2{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792305]]
{color:#d04437}MVCC PDS 2{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792388]]
{color:#d04437}Cache (Failover) 3{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792337]]
{color:#d04437}MVCC Cache 5{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792382]]
{color:#d04437}MVCC PDS 1{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792387]]
{color:#d04437}MVCC PDS 4{color} [[tests 0
CANCELLED|https://ci.ignite.apache.org/viewLog.html?buildId=4792390]]
{color:#d04437}Cache (Failover SSL){color} [[tests 0
[jira] [Updated] (IGNITE-12390) .NET: Add automatic NuGet verification as part of the release process
[ https://issues.apache.org/jira/browse/IGNITE-12390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Tupitsyn updated IGNITE-12390: Issue Type: Improvement (was: Bug) > .NET: Add automatic NuGet verification as part of the release process > - > > Key: IGNITE-12390 > URL: https://issues.apache.org/jira/browse/IGNITE-12390 > Project: Ignite > Issue Type: Improvement > Components: platforms >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: .NET > Fix For: 2.8 > > Time Spent: 10m > Remaining Estimate: 0h > > Use dotnet CLI to verify that resulting nupkg file can be installed and > Ignite node starts (dotnet new console, dotnet add package, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-12390) .NET: Add automatic NuGet verification as part of the release process
[ https://issues.apache.org/jira/browse/IGNITE-12390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981804#comment-16981804 ] Pavel Tupitsyn commented on IGNITE-12390: - Merged to maaster: 08cca283078128cdc9fefdb986bb5a497881c23d > .NET: Add automatic NuGet verification as part of the release process > - > > Key: IGNITE-12390 > URL: https://issues.apache.org/jira/browse/IGNITE-12390 > Project: Ignite > Issue Type: Improvement > Components: platforms >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: .NET > Fix For: 2.8 > > Time Spent: 20m > Remaining Estimate: 0h > > Use dotnet CLI to verify that resulting nupkg file can be installed and > Ignite node starts (dotnet new console, dotnet add package, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-12390) .NET: Add automatic NuGet verification as part of the release process
[ https://issues.apache.org/jira/browse/IGNITE-12390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981800#comment-16981800 ] Igor Sapego commented on IGNITE-12390: -- [~ptupitsyn] Looks good to me > .NET: Add automatic NuGet verification as part of the release process > - > > Key: IGNITE-12390 > URL: https://issues.apache.org/jira/browse/IGNITE-12390 > Project: Ignite > Issue Type: Improvement > Components: platforms >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: .NET > Fix For: 2.8 > > Time Spent: 10m > Remaining Estimate: 0h > > Use dotnet CLI to verify that resulting nupkg file can be installed and > Ignite node starts (dotnet new console, dotnet add package, etc). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12196) [Phase-4] Deprecate old rebalancing cache metrics
[ https://issues.apache.org/jira/browse/IGNITE-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Muzafarov updated IGNITE-12196: - Fix Version/s: (was: 2.8) > [Phase-4] Deprecate old rebalancing cache metrics > - > > Key: IGNITE-12196 > URL: https://issues.apache.org/jira/browse/IGNITE-12196 > Project: Ignite > Issue Type: Sub-task >Reporter: Maxim Muzafarov >Priority: Major > Labels: IEP-35 > > We need to mark rebalancing CacheMetrics deprecated and remove them from > metrics a newly introduced metrics framework IGNITE-11961. Such cache metrics > should be implemented in an old-fashion way (like they were before the > metrics framework added) to keep backwards compatibility. > Removed it Apache Ignite 3.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12195) [Phase-3] Rebalance HitRate metrics
[ https://issues.apache.org/jira/browse/IGNITE-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Muzafarov updated IGNITE-12195: - Fix Version/s: (was: 2.8) > [Phase-3] Rebalance HitRate metrics > --- > > Key: IGNITE-12195 > URL: https://issues.apache.org/jira/browse/IGNITE-12195 > Project: Ignite > Issue Type: Sub-task >Reporter: Maxim Muzafarov >Priority: Major > Labels: IEP-35 > > Currently, HitRate metric can consume a lot of CPU resources. We need to > investigate such performance issues and disable these metrics by default. > * rebalancingKeysRate HitRate metric > * rebalancingBytesRate HitRate metric -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (IGNITE-7609) .NET: FieldsQueryCursor should expose data types too
[
https://issues.apache.org/jira/browse/IGNITE-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981716#comment-16981716
]
Pavel Tupitsyn edited comment on IGNITE-7609 at 11/25/19 5:22 PM:
--
Despite Java-based ticket being closed as Won't Fix, I still think this ticket
is important.
See GetFieldType and GetDataTypeName methods in DbDataReader:
https://docs.microsoft.com/en-us/dotnet/api/system.data.common.dbdatareader?view=netcore-2.0
For example, if we try to build some kind of a generic tool that runs SQL
against Ignite, we want to know the type of every field in the query results to
show a nice table with formatting and so on.
The information is already available in QueryCursorEx.fieldsMeta(), so this
ticket is a matter of simple change to PlatformFieldsQueryCursor on Java side.
was (Author: ptupitsyn):
Despite Java-based ticket being closed as Won't Fix, I still think this ticket
is important.
For example, if we try to build some kind of a generic tool that runs SQL
against Ignite, we want to know the type of every field in the query results to
show a nice table with formatting and so on.
The information is already available in QueryCursorEx.fieldsMeta(), so this
ticket is a matter of simple change to PlatformFieldsQueryCursor on Java side.
> .NET: FieldsQueryCursor should expose data types too
>
>
> Key: IGNITE-7609
> URL: https://issues.apache.org/jira/browse/IGNITE-7609
> Project: Ignite
> Issue Type: Improvement
> Components: platforms
>Affects Versions: 2.4
>Reporter: Pavel Tupitsyn
>Priority: Major
> Labels: .NET
>
> See IGNITE-7607 for Java, add same thing to .NET:
> {code}
> public interface IFieldsQueryCursor
> {
> ...
> IList FieldTypes
> }
> {code}
> T could be string or Type, needs investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-7609) .NET: FieldsQueryCursor should expose data types too
[
https://issues.apache.org/jira/browse/IGNITE-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981716#comment-16981716
]
Pavel Tupitsyn commented on IGNITE-7609:
Despite Java-based ticket being closed as Won't Fix, I still think this ticket
is important.
For example, if we try to build some kind of a generic tool that runs SQL
against Ignite, we want to know the type of every field in the query results to
show a nice table with formatting and so on.
The information is already available in QueryCursorEx.fieldsMeta(), so this
ticket is a matter of simple change to PlatformFieldsQueryCursor on Java side.
> .NET: FieldsQueryCursor should expose data types too
>
>
> Key: IGNITE-7609
> URL: https://issues.apache.org/jira/browse/IGNITE-7609
> Project: Ignite
> Issue Type: Improvement
> Components: platforms
>Affects Versions: 2.4
>Reporter: Pavel Tupitsyn
>Priority: Major
> Labels: .NET
>
> See IGNITE-7607 for Java, add same thing to .NET:
> {code}
> public interface IFieldsQueryCursor
> {
> ...
> IList FieldTypes
> }
> {code}
> T could be string or Type, needs investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-11410) Sandbox for user-defined code
[
https://issues.apache.org/jira/browse/IGNITE-11410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981710#comment-16981710
]
Ignite TC Bot commented on IGNITE-11410:
{panel:title=Branch: [pull/6707/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=4790950buildTypeId=IgniteTests24Java8_RunAll]
> Sandbox for user-defined code
> -
>
> Key: IGNITE-11410
> URL: https://issues.apache.org/jira/browse/IGNITE-11410
> Project: Ignite
> Issue Type: Task
>Reporter: Anton Vinogradov
>Assignee: Denis Garus
>Priority: Major
> Labels: iep-38
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> We should provide a restricted environment (sandbox) in which to run
> user-defined code securely. To get it done, we would use the java sandbox
> model.
> The java sandbox model allows restricting access from user-defined code to
> the system resources or security-sensitive feature of java, for example,
> reflection.
> The user-defined code contains:
> - StreamReceiver for DataStreamer:
> - EntryProcessor;
> - ComputeJob;
> - filter and transformer for ScanQuery.
> The user-defined code will get permissions from GridSecuerityProcessor
> (security plugin).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (IGNITE-7552) .NET: AtomicConfiguration.DefaultBackups should be 1
[
https://issues.apache.org/jira/browse/IGNITE-7552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavel Tupitsyn updated IGNITE-7552:
---
Labels: .NET newbie (was: .NET)
> .NET: AtomicConfiguration.DefaultBackups should be 1
>
>
> Key: IGNITE-7552
> URL: https://issues.apache.org/jira/browse/IGNITE-7552
> Project: Ignite
> Issue Type: Bug
> Components: platforms
>Affects Versions: 2.4
>Reporter: Pavel Tupitsyn
>Priority: Major
> Labels: .NET, newbie
> Fix For: 2.9
>
>
> Defaults have changed in Java (see {{AtomicConfiguration.DFLT_BACKUPS}}),
> update .NET part, add test (we usually check that .NET and Java defaults
> match in {{IgniteConfigurationTest.TestSpringXml}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (IGNITE-9017) .NET: Clear cache statistics
[ https://issues.apache.org/jira/browse/IGNITE-9017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Tupitsyn updated IGNITE-9017: --- Labels: .NET newbie (was: .NET) > .NET: Clear cache statistics > > > Key: IGNITE-9017 > URL: https://issues.apache.org/jira/browse/IGNITE-9017 > Project: Ignite > Issue Type: Improvement > Components: platforms >Affects Versions: 2.7 >Reporter: Denis Garus >Priority: Major > Labels: .NET, newbie > > ICache.ClearStatistics, ICluster.ClearStatistics. > See IGNITE-8705 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IGNITE-12219) Cache operations performance metrics
[ https://issues.apache.org/jira/browse/IGNITE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amelchev Nikita reassigned IGNITE-12219: Assignee: Amelchev Nikita > Cache operations performance metrics > > > Key: IGNITE-12219 > URL: https://issues.apache.org/jira/browse/IGNITE-12219 > Project: Ignite > Issue Type: Improvement >Reporter: Nikolay Izhikov >Assignee: Amelchev Nikita >Priority: Major > Labels: IEP-35, await > Fix For: 2.8 > > > We need to provide cache operations histogram metrics > Next API and its variants should be covered: > * get > * put > * remove -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12219) Cache operations performance metrics
[ https://issues.apache.org/jira/browse/IGNITE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Izhikov updated IGNITE-12219: - Description: We need to provide cache operations histogram metrics Next API and its variants should be covered: * get * put * remove was: We need to provide cache operations histogram metrics Next API and its variants should be covered: * get * getEntry * getAll * put * putAll * remove * replace * lock * invoke * containsKey > Cache operations performance metrics > > > Key: IGNITE-12219 > URL: https://issues.apache.org/jira/browse/IGNITE-12219 > Project: Ignite > Issue Type: Improvement >Reporter: Nikolay Izhikov >Priority: Major > Labels: IEP-35, await > Fix For: 2.8 > > > We need to provide cache operations histogram metrics > Next API and its variants should be covered: > * get > * put > * remove -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12219) Cache operations performance metrics
[ https://issues.apache.org/jira/browse/IGNITE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikolay Izhikov updated IGNITE-12219: - Description: We need to provide cache operations histogram metrics Next API and its variants should be covered: * get * getEntry * getAll * put * putAll * remove * replace * lock * invoke * containsKey was: We need to provide cache operations histogram metrics Next API and its variants should be covered: * get * getEntry * getAll * put * remove * replace * lock * invoke * containsKey > Cache operations performance metrics > > > Key: IGNITE-12219 > URL: https://issues.apache.org/jira/browse/IGNITE-12219 > Project: Ignite > Issue Type: Improvement >Reporter: Nikolay Izhikov >Priority: Major > Labels: IEP-35, await > Fix For: 2.8 > > > We need to provide cache operations histogram metrics > Next API and its variants should be covered: > * get > * getEntry > * getAll > * put > * putAll > * remove > * replace > * lock > * invoke > * containsKey -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (IGNITE-12249) Concurrency guarantees for TransactionView
[
https://issues.apache.org/jira/browse/IGNITE-12249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolay Izhikov resolved IGNITE-12249.
--
Resolution: Invalid
All required guarantees provided by {{ConcurrentHashMap#values()#iterator}}
> Concurrency guarantees for TransactionView
> --
>
> Key: IGNITE-12249
> URL: https://issues.apache.org/jira/browse/IGNITE-12249
> Project: Ignite
> Issue Type: Bug
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Labels: IEP-35, await
> Fix For: 2.8
>
>
> Currently, {{TransactionView#keysCount}} and {{TransactionView#cacheIds}}
> works with Collections that not provides concurrent guarantees.
> We should research the possibility to provide consistent transaction view for
> these data structures.
> Performance of the transaction engine can be limitation here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (IGNITE-12249) Concurrency guarantees for TransactionView
[
https://issues.apache.org/jira/browse/IGNITE-12249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolay Izhikov reassigned IGNITE-12249:
Assignee: Nikolay Izhikov
> Concurrency guarantees for TransactionView
> --
>
> Key: IGNITE-12249
> URL: https://issues.apache.org/jira/browse/IGNITE-12249
> Project: Ignite
> Issue Type: Bug
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Labels: IEP-35, await
> Fix For: 2.8
>
>
> Currently, {{TransactionView#keysCount}} and {{TransactionView#cacheIds}}
> works with Collections that not provides concurrent guarantees.
> We should research the possibility to provide consistent transaction view for
> these data structures.
> Performance of the transaction engine can be limitation here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-12388) The testReconnectServersRestart_3 test is flacky
[
https://issues.apache.org/jira/browse/IGNITE-12388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981660#comment-16981660
]
Amelchev Nikita commented on IGNITE-12388:
--
I have fixed the test and checked it on TC more than [400 times.
|https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ZooKeeperDiscovery1_IgniteTests24Java8=pull%2F7064%2Fhead=true=buildTypeStatusDiv]
Tests look good.
[~alex_pl], could you help with a review, please?
> The testReconnectServersRestart_3 test is flacky
>
>
> Key: IGNITE-12388
> URL: https://issues.apache.org/jira/browse/IGNITE-12388
> Project: Ignite
> Issue Type: Test
>Reporter: Amelchev Nikita
>Assignee: Amelchev Nikita
>Priority: Minor
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The testReconnectServersRestart_3 test is flacky. [TC
> history.|https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=534893108987486091=testDetails]
> Sometimes it fails by timeout. The reason is the incorrect client mode
> condition during nodes starts:
> {noformat}
> helper.clientModeThreadLocal(threadIdx == srvIdx ||
> ThreadLocalRandom.current().nextBoolean())
> {noformat}
> There is no server node to connect when all started nodes are clients:
> {noformat}
> [2019-11-22 15:42:24,998][WARN ][start-node-5][ZookeeperDiscoveryImpl]
> Waiting for local join event [nodeId=74a33ea3-6d7c-4ab4-b86c-3d9305c4,
> name=internal.ZookeeperDiscoveryClientReconnectTest4]
> [2019-11-22 15:42:25,010][WARN ][start-node-1][ZookeeperDiscoveryImpl]
> Waiting for local join event [nodeId=b113bb09-5d6a-48b8-a656-d22cb920,
> name=internal.ZookeeperDiscoveryClientReconnectTest0]
> [2019-11-22 15:42:25,015][WARN ][start-node-4][ZookeeperDiscoveryImpl]
> Waiting for local join event [nodeId=121ca03d-31de-4de4-a81c-20bd1373,
> name=internal.ZookeeperDiscoveryClientReconnectTest3]
> [2019-11-22 15:42:25,017][WARN ][start-node-2][ZookeeperDiscoveryImpl]
> Waiting for local join event [nodeId=4a6b5c7f-fc06-4251-b78f-c49277f1,
> name=internal.ZookeeperDiscoveryClientReconnectTest1]
> [2019-11-22 15:42:25,020][WARN ][start-node-3][ZookeeperDiscoveryImpl]
> Waiting for local join event [nodeId=b0468b08-7d57-4b29-b2fb-6d142592,
> name=internal.ZookeeperDiscoveryClientReconnectTest2]
> {noformat}
> Then the test fails by timeout (5min).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-12388) The testReconnectServersRestart_3 test is flacky
[
https://issues.apache.org/jira/browse/IGNITE-12388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981656#comment-16981656
]
Ignite TC Bot commented on IGNITE-12388:
{panel:title=Branch: [pull/7064/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=4785191buildTypeId=IgniteTests24Java8_RunAll]
> The testReconnectServersRestart_3 test is flacky
>
>
> Key: IGNITE-12388
> URL: https://issues.apache.org/jira/browse/IGNITE-12388
> Project: Ignite
> Issue Type: Test
>Reporter: Amelchev Nikita
>Assignee: Amelchev Nikita
>Priority: Minor
> Fix For: 2.8
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The testReconnectServersRestart_3 test is flacky. [TC
> history.|https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=534893108987486091=testDetails]
> Sometimes it fails by timeout. The reason is the incorrect client mode
> condition during nodes starts:
> {noformat}
> helper.clientModeThreadLocal(threadIdx == srvIdx ||
> ThreadLocalRandom.current().nextBoolean())
> {noformat}
> There is no server node to connect when all started nodes are clients:
> {noformat}
> [2019-11-22 15:42:24,998][WARN ][start-node-5][ZookeeperDiscoveryImpl]
> Waiting for local join event [nodeId=74a33ea3-6d7c-4ab4-b86c-3d9305c4,
> name=internal.ZookeeperDiscoveryClientReconnectTest4]
> [2019-11-22 15:42:25,010][WARN ][start-node-1][ZookeeperDiscoveryImpl]
> Waiting for local join event [nodeId=b113bb09-5d6a-48b8-a656-d22cb920,
> name=internal.ZookeeperDiscoveryClientReconnectTest0]
> [2019-11-22 15:42:25,015][WARN ][start-node-4][ZookeeperDiscoveryImpl]
> Waiting for local join event [nodeId=121ca03d-31de-4de4-a81c-20bd1373,
> name=internal.ZookeeperDiscoveryClientReconnectTest3]
> [2019-11-22 15:42:25,017][WARN ][start-node-2][ZookeeperDiscoveryImpl]
> Waiting for local join event [nodeId=4a6b5c7f-fc06-4251-b78f-c49277f1,
> name=internal.ZookeeperDiscoveryClientReconnectTest1]
> [2019-11-22 15:42:25,020][WARN ][start-node-3][ZookeeperDiscoveryImpl]
> Waiting for local join event [nodeId=b0468b08-7d57-4b29-b2fb-6d142592,
> name=internal.ZookeeperDiscoveryClientReconnectTest2]
> {noformat}
> Then the test fails by timeout (5min).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (IGNITE-11923) [IEP-35] Migrate IgniteMXBean
[ https://issues.apache.org/jira/browse/IGNITE-11923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Kalashnikov reassigned IGNITE-11923: --- Assignee: Sergey Kalashnikov > [IEP-35] Migrate IgniteMXBean > - > > Key: IGNITE-11923 > URL: https://issues.apache.org/jira/browse/IGNITE-11923 > Project: Ignite > Issue Type: Improvement >Reporter: Nikolay Izhikov >Assignee: Sergey Kalashnikov >Priority: Major > Labels: IEP-35, await > Fix For: 2.8 > > Time Spent: 10m > Remaining Estimate: 0h > > After merging of IGNITE-11848 we should migrate `IgniteMXBean` to the new > metric framework. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IGNITE-12183) Rebalancing process metrics for cache groups
[
https://issues.apache.org/jira/browse/IGNITE-12183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolay Izhikov reassigned IGNITE-12183:
Assignee: (was: Nikolay Izhikov)
> Rebalancing process metrics for cache groups
>
>
> Key: IGNITE-12183
> URL: https://issues.apache.org/jira/browse/IGNITE-12183
> Project: Ignite
> Issue Type: Improvement
>Reporter: Aleksandr Brazhnikov
>Priority: Major
> Labels: IEP-35, await
> Fix For: 2.8
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> h3. Intro
> Currently, some of the Apache Ignite rebalance process metrics are not
> working well enough. For instance, `EstimatedRebalancingKeys` keys time to
> time returns `-1` value due to a bug, or `rebalanceKeysReceived` metric
> treated as CacheMetric in fact calculated for the whole cache group (e.g.
> historical rebalance, see IGNITE-11330 and code block comment below).
> All the rebalance process metrics must be re-worked.
> {code:java}
> /**
> * Update rebalancing metrics.
> */
> private void updateGroupMetrics() {
> // TODO: IGNITE-11330: Update metrics for touched cache only.
> // Due to historical rebalancing "EstimatedRebalancingKeys" metric is
> currently calculated for the whole cache
> // group (by partition counters), so "RebalancedKeys" and
> "RebalancingKeysRate" is calculated in the same way.
> for (GridCacheContext cctx0 : grp.caches()) {
> if (cctx0.statisticsEnabled())
> cctx0.cache().metrics0().onRebalanceKeyReceived();
> }
> }
> {code}
> h3. What we have
> _CacheMetrics_ - statistics must be enabled to see these metrics.
> * getRebalancedKeys
> * getKeysToRebalanceLeft
> * getEstimatedRebalancingKeys
> * getEstimatedRebalancingFinishTime
> * getRebalancingStartTime
> * getRebalanceClearingPartitionsLeft
> * getRebalancingKeysRate
> * getRebalancingBytesRate
> h3. What to do
> All such metrics (or their analogue) must be available for the
> _CacheGroupMetrics_. I'd suggest to do the following:
> # Phase-1
> #* rebalancingPartitionsLeft long metric
> #* rebalancingReceivedKeys long metric
> #* rebalancingReceivedBytes long metric
> #* rebalancingStartTime long metric
> #* rebalancingFinishTime long metric
> # Phase-2
> #* rebalancingExpectedKeys long metric
> #* rebalancingExpectedBytes long metric
> #* rebalancingEvictedPartitionsLeft long metric
> # Phase-3 (statistics must be enabled)
> #* rebalancingKeysRate HitRate metric
> #* rebalancingBytesRate HitRate metric
> # Phase-4
> #* Mark rebalancing _CacheMetrics_ deprecated and remove from metrics
> framework IGNITE-11961.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (IGNITE-12196) [Phase-4] Deprecate old rebalancing cache metrics
[ https://issues.apache.org/jira/browse/IGNITE-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Muzafarov updated IGNITE-12196: - Labels: IEP-35 (was: ) > [Phase-4] Deprecate old rebalancing cache metrics > - > > Key: IGNITE-12196 > URL: https://issues.apache.org/jira/browse/IGNITE-12196 > Project: Ignite > Issue Type: Sub-task >Reporter: Maxim Muzafarov >Priority: Major > Labels: IEP-35 > Fix For: 2.8 > > > We need to mark rebalancing CacheMetrics deprecated and remove them from > metrics a newly introduced metrics framework IGNITE-11961. Such cache metrics > should be implemented in an old-fashion way (like they were before the > metrics framework added) to keep backwards compatibility. > Removed it Apache Ignite 3.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12194) [Phase-2] Calculate expected rebalancing cache group keys
[ https://issues.apache.org/jira/browse/IGNITE-12194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Muzafarov updated IGNITE-12194: - Fix Version/s: 2.8 > [Phase-2] Calculate expected rebalancing cache group keys > - > > Key: IGNITE-12194 > URL: https://issues.apache.org/jira/browse/IGNITE-12194 > Project: Ignite > Issue Type: Sub-task >Reporter: Maxim Muzafarov >Priority: Major > Labels: IEP-35 > Fix For: 2.8 > > > We need to implement expected to be rebalanced cache group keys and total > bytes. Currently, 'estimatedKeysCount' cache metric returns '-1' for some of > the cases (see comments IGNITE-11330). > * rebalancingExpectedKeys long metric > * rebalancingExpectedBytes long metric > * rebalancingEvictedPartitionsLeft long metric -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12196) [Phase-4] Deprecate old rebalancing cache metrics
[ https://issues.apache.org/jira/browse/IGNITE-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Muzafarov updated IGNITE-12196: - Fix Version/s: 2.8 > [Phase-4] Deprecate old rebalancing cache metrics > - > > Key: IGNITE-12196 > URL: https://issues.apache.org/jira/browse/IGNITE-12196 > Project: Ignite > Issue Type: Sub-task >Reporter: Maxim Muzafarov >Priority: Major > Fix For: 2.8 > > > We need to mark rebalancing CacheMetrics deprecated and remove them from > metrics a newly introduced metrics framework IGNITE-11961. Such cache metrics > should be implemented in an old-fashion way (like they were before the > metrics framework added) to keep backwards compatibility. > Removed it Apache Ignite 3.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12085) ThreadPool metrics register after all components start
[
https://issues.apache.org/jira/browse/IGNITE-12085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxim Muzafarov updated IGNITE-12085:
-
Fix Version/s: (was: 2.8)
> ThreadPool metrics register after all components start
> --
>
> Key: IGNITE-12085
> URL: https://issues.apache.org/jira/browse/IGNITE-12085
> Project: Ignite
> Issue Type: Improvement
>Reporter: Nikolay Izhikov
>Priority: Major
> Labels: IEP-35, await
>
> For now, thread pool metrics register after all {{GridComponent}} starts.
> But there are specific scenarios when some component blocks {{onKernalStart}}
> execution for a long time. {{GridCacheProcessor}} can be taken as an example.
> This leads to the situation when some metric info is lost.
> Seems, we can register thread pool metrics right after only **required**
> components are started and don't wait for all components.
> {code:java}
> // Callbacks.
> for (GridComponent comp : ctx) {
> comp.onKernalStart(active);
> }
> // Start plugins.
> for (PluginProvider provider : ctx.plugins().allProviders())
> provider.onIgniteStart();
> ctx.metric().registerThreadPools(utilityCachePool, execSvc,
> svcExecSvc, sysExecSvc, stripedExecSvc,
> p2pExecSvc, mgmtExecSvc, igfsExecSvc, dataStreamExecSvc,
> restExecSvc, affExecSvc, idxExecSvc,
> callbackExecSvc, qryExecSvc, schemaExecSvc, rebalanceExecSvc,
> rebalanceStripedExecSvc, customExecSvcs);
> // Register MBeans.
> mBeansMgr.registerAllMBeans(utilityCachePool, execSvc,
> svcExecSvc, sysExecSvc, stripedExecSvc, p2pExecSvc,
> mgmtExecSvc, igfsExecSvc, dataStreamExecSvc, restExecSvc,
> affExecSvc, idxExecSvc, callbackExecSvc,
> qryExecSvc, schemaExecSvc, rebalanceExecSvc,
> rebalanceStripedExecSvc, customExecSvcs, ctx.workersRegistry());
> {code}
> {code:java}
> public class GridCacheProcessor {
> @Override public void onKernalStart(boolean active) throws
> IgniteCheckedException {
> //.
> final List syncFuts = new
> ArrayList<>(caches.size());
> sharedCtx.forAllCaches(new CIX1() {
> @Override public void applyx(GridCacheContext cctx) {
> CacheConfiguration cfg = cctx.config();
> if (cctx.affinityNode() &&
> cfg.getRebalanceMode() == SYNC &&
> startTopVer.equals(cctx.startTopologyVersion())) {
> CacheMode cacheMode = cfg.getCacheMode();
> if (cacheMode == REPLICATED || (cacheMode == PARTITIONED
> && cfg.getRebalanceDelay() >= 0))
> // Need to wait outside to avoid a deadlock
> syncFuts.add(cctx.preloader().syncFuture());
> }
> }
> });
> for (int i = 0, size = syncFuts.size(); i < size; i++)
> syncFuts.get(i).get();
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (IGNITE-12194) [Phase-2] Calculate expected rebalancing cache group keys
[ https://issues.apache.org/jira/browse/IGNITE-12194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Muzafarov updated IGNITE-12194: - Labels: IEP-35 (was: ) > [Phase-2] Calculate expected rebalancing cache group keys > - > > Key: IGNITE-12194 > URL: https://issues.apache.org/jira/browse/IGNITE-12194 > Project: Ignite > Issue Type: Sub-task >Reporter: Maxim Muzafarov >Priority: Major > Labels: IEP-35 > > We need to implement expected to be rebalanced cache group keys and total > bytes. Currently, 'estimatedKeysCount' cache metric returns '-1' for some of > the cases (see comments IGNITE-11330). > * rebalancingExpectedKeys long metric > * rebalancingExpectedBytes long metric > * rebalancingEvictedPartitionsLeft long metric -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12193) [Phase-1] Rebalancing cache group keys metric counters
[ https://issues.apache.org/jira/browse/IGNITE-12193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Muzafarov updated IGNITE-12193: - Fix Version/s: 2.8 > [Phase-1] Rebalancing cache group keys metric counters > -- > > Key: IGNITE-12193 > URL: https://issues.apache.org/jira/browse/IGNITE-12193 > Project: Ignite > Issue Type: Sub-task >Reporter: Maxim Muzafarov >Assignee: Nikolai Kulagin >Priority: Major > Labels: IEP-35 > Fix For: 2.8 > > Time Spent: 20m > Remaining Estimate: 0h > > Implement metrics counters related to the cache group: > * rebalancingPartitionsLeft long metric > * rebalancingReceivedKeys long metric > * rebalancingReceivedBytes long metric > * rebalancingStartTime long metric > * rebalancingFinishTime long metric -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12195) [Phase-3] Rebalance HitRate metrics
[ https://issues.apache.org/jira/browse/IGNITE-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Muzafarov updated IGNITE-12195: - Fix Version/s: 2.8 > [Phase-3] Rebalance HitRate metrics > --- > > Key: IGNITE-12195 > URL: https://issues.apache.org/jira/browse/IGNITE-12195 > Project: Ignite > Issue Type: Sub-task >Reporter: Maxim Muzafarov >Priority: Major > Labels: IEP-35 > Fix For: 2.8 > > > Currently, HitRate metric can consume a lot of CPU resources. We need to > investigate such performance issues and disable these metrics by default. > * rebalancingKeysRate HitRate metric > * rebalancingBytesRate HitRate metric -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12193) [Phase-1] Rebalancing cache group keys metric counters
[ https://issues.apache.org/jira/browse/IGNITE-12193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Muzafarov updated IGNITE-12193: - Labels: IEP-35 (was: ) > [Phase-1] Rebalancing cache group keys metric counters > -- > > Key: IGNITE-12193 > URL: https://issues.apache.org/jira/browse/IGNITE-12193 > Project: Ignite > Issue Type: Sub-task >Reporter: Maxim Muzafarov >Assignee: Nikolai Kulagin >Priority: Major > Labels: IEP-35 > Time Spent: 20m > Remaining Estimate: 0h > > Implement metrics counters related to the cache group: > * rebalancingPartitionsLeft long metric > * rebalancingReceivedKeys long metric > * rebalancingReceivedBytes long metric > * rebalancingStartTime long metric > * rebalancingFinishTime long metric -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12195) [Phase-3] Rebalance HitRate metrics
[ https://issues.apache.org/jira/browse/IGNITE-12195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Muzafarov updated IGNITE-12195: - Labels: IEP-35 (was: ) > [Phase-3] Rebalance HitRate metrics > --- > > Key: IGNITE-12195 > URL: https://issues.apache.org/jira/browse/IGNITE-12195 > Project: Ignite > Issue Type: Sub-task >Reporter: Maxim Muzafarov >Priority: Major > Labels: IEP-35 > > Currently, HitRate metric can consume a lot of CPU resources. We need to > investigate such performance issues and disable these metrics by default. > * rebalancingKeysRate HitRate metric > * rebalancingBytesRate HitRate metric -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12249) Concurrency guarantees for TransactionView
[
https://issues.apache.org/jira/browse/IGNITE-12249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxim Muzafarov updated IGNITE-12249:
-
Fix Version/s: 2.8
> Concurrency guarantees for TransactionView
> --
>
> Key: IGNITE-12249
> URL: https://issues.apache.org/jira/browse/IGNITE-12249
> Project: Ignite
> Issue Type: Bug
>Reporter: Nikolay Izhikov
>Priority: Major
> Labels: IEP-35, await
> Fix For: 2.8
>
>
> Currently, {{TransactionView#keysCount}} and {{TransactionView#cacheIds}}
> works with Collections that not provides concurrent guarantees.
> We should research the possibility to provide consistent transaction view for
> these data structures.
> Performance of the transaction engine can be limitation here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (IGNITE-12393) Thread pool queue system view
[
https://issues.apache.org/jira/browse/IGNITE-12393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxim Muzafarov updated IGNITE-12393:
-
Fix Version/s: 2.8
> Thread pool queue system view
> -
>
> Key: IGNITE-12393
> URL: https://issues.apache.org/jira/browse/IGNITE-12393
> Project: Ignite
> Issue Type: Sub-task
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Labels: IEP-35
> Fix For: 2.8
>
>
> When in the production environment exist some cluster performance issues
> usually it leads to the large striped executor queue size.
> The number of tasks in the queue can observe by
> {StripedExecutorMXBean#getTotalQueueSize} metric. In the case queue size
> becomes large it's useful to have the ability to know what tasks are waiting
> for execution in the thread pool.
> Especially, for dealing with failover scenarios.
> We should create a system views to expose information about striped executor
> services queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (IGNITE-12278) Add metric showing how many nodes may safely leave the cluster wihout partition loss
[ https://issues.apache.org/jira/browse/IGNITE-12278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Muzafarov updated IGNITE-12278: - Fix Version/s: (was: 2.8) > Add metric showing how many nodes may safely leave the cluster wihout > partition loss > > > Key: IGNITE-12278 > URL: https://issues.apache.org/jira/browse/IGNITE-12278 > Project: Ignite > Issue Type: Improvement >Reporter: Ivan Rakov >Assignee: Albert Iskhakov >Priority: Major > Labels: IEP-35, newbie > > We already have CacheGroupMetricsMXBean#getMinimumNumberOfPartitionCopies > metrics that shows partitions redundancy number for a specific cache group. > It would be handy if user has single aggregated metric for all cache groups > showing how many nodes may leave the cluster without partition loss in any > cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (IGNITE-12393) Thread pool queue system view
[
https://issues.apache.org/jira/browse/IGNITE-12393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolay Izhikov reassigned IGNITE-12393:
Assignee: Nikolay Izhikov
> Thread pool queue system view
> -
>
> Key: IGNITE-12393
> URL: https://issues.apache.org/jira/browse/IGNITE-12393
> Project: Ignite
> Issue Type: Sub-task
>Reporter: Nikolay Izhikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Labels: IEP-35
>
> When in the production environment exist some cluster performance issues
> usually it leads to the large striped executor queue size.
> The number of tasks in the queue can observe by
> {StripedExecutorMXBean#getTotalQueueSize} metric. In the case queue size
> becomes large it's useful to have the ability to know what tasks are waiting
> for execution in the thread pool.
> Especially, for dealing with failover scenarios.
> We should create a system views to expose information about striped executor
> services queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (IGNITE-12183) Rebalancing process metrics for cache groups
[
https://issues.apache.org/jira/browse/IGNITE-12183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolay Izhikov reassigned IGNITE-12183:
Assignee: Nikolay Izhikov (was: Surkov Aleksandr)
> Rebalancing process metrics for cache groups
>
>
> Key: IGNITE-12183
> URL: https://issues.apache.org/jira/browse/IGNITE-12183
> Project: Ignite
> Issue Type: Improvement
>Reporter: Aleksandr Brazhnikov
>Assignee: Nikolay Izhikov
>Priority: Major
> Labels: IEP-35, await
> Fix For: 2.8
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> h3. Intro
> Currently, some of the Apache Ignite rebalance process metrics are not
> working well enough. For instance, `EstimatedRebalancingKeys` keys time to
> time returns `-1` value due to a bug, or `rebalanceKeysReceived` metric
> treated as CacheMetric in fact calculated for the whole cache group (e.g.
> historical rebalance, see IGNITE-11330 and code block comment below).
> All the rebalance process metrics must be re-worked.
> {code:java}
> /**
> * Update rebalancing metrics.
> */
> private void updateGroupMetrics() {
> // TODO: IGNITE-11330: Update metrics for touched cache only.
> // Due to historical rebalancing "EstimatedRebalancingKeys" metric is
> currently calculated for the whole cache
> // group (by partition counters), so "RebalancedKeys" and
> "RebalancingKeysRate" is calculated in the same way.
> for (GridCacheContext cctx0 : grp.caches()) {
> if (cctx0.statisticsEnabled())
> cctx0.cache().metrics0().onRebalanceKeyReceived();
> }
> }
> {code}
> h3. What we have
> _CacheMetrics_ - statistics must be enabled to see these metrics.
> * getRebalancedKeys
> * getKeysToRebalanceLeft
> * getEstimatedRebalancingKeys
> * getEstimatedRebalancingFinishTime
> * getRebalancingStartTime
> * getRebalanceClearingPartitionsLeft
> * getRebalancingKeysRate
> * getRebalancingBytesRate
> h3. What to do
> All such metrics (or their analogue) must be available for the
> _CacheGroupMetrics_. I'd suggest to do the following:
> # Phase-1
> #* rebalancingPartitionsLeft long metric
> #* rebalancingReceivedKeys long metric
> #* rebalancingReceivedBytes long metric
> #* rebalancingStartTime long metric
> #* rebalancingFinishTime long metric
> # Phase-2
> #* rebalancingExpectedKeys long metric
> #* rebalancingExpectedBytes long metric
> #* rebalancingEvictedPartitionsLeft long metric
> # Phase-3 (statistics must be enabled)
> #* rebalancingKeysRate HitRate metric
> #* rebalancingBytesRate HitRate metric
> # Phase-4
> #* Mark rebalancing _CacheMetrics_ deprecated and remove from metrics
> framework IGNITE-11961.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (IGNITE-12393) Thread pool queue system view
[
https://issues.apache.org/jira/browse/IGNITE-12393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolay Izhikov updated IGNITE-12393:
-
Labels: IEP-35 (was: )
> Thread pool queue system view
> -
>
> Key: IGNITE-12393
> URL: https://issues.apache.org/jira/browse/IGNITE-12393
> Project: Ignite
> Issue Type: Sub-task
>Reporter: Nikolay Izhikov
>Priority: Major
> Labels: IEP-35
>
> When in the production environment exist some cluster performance issues
> usually it leads to the large striped executor queue size.
> The number of tasks in the queue can observe by
> {StripedExecutorMXBean#getTotalQueueSize} metric. In the case queue size
> becomes large it's useful to have the ability to know what tasks are waiting
> for execution in the thread pool.
> Especially, for dealing with failover scenarios.
> We should create a system views to expose information about striped executor
> services queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-12054) [Umbrella][Spark] Upgrade Spark module to 2.4
[ https://issues.apache.org/jira/browse/IGNITE-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981551#comment-16981551 ] Shenson Joseph commented on IGNITE-12054: - [~zaleslaw] [~nizhikov] Just want to let you know that I am using Ignite-2.7.0 & spark-2.4.4 > [Umbrella][Spark] Upgrade Spark module to 2.4 > - > > Key: IGNITE-12054 > URL: https://issues.apache.org/jira/browse/IGNITE-12054 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.8 >Reporter: Denis A. Magda >Assignee: Alexey Zinoviev >Priority: Blocker > Labels: await > Fix For: 2.8 > > Attachments: ignite-spark-patch-new.diff > > Time Spent: 20m > Remaining Estimate: 0h > > Users can't use APIs that are already available in Spark 2.4: > https://stackoverflow.com/questions/57392143/persisting-spark-dataframe-to-ignite > Let's upgrade Spark from 2.3 to 2.4 until we extract the Spark Integration as > a separate module that can support multiple Spark versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-12054) [Umbrella][Spark] Upgrade Spark module to 2.4
[ https://issues.apache.org/jira/browse/IGNITE-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981547#comment-16981547 ] Shenson Joseph commented on IGNITE-12054: - [~zaleslaw] [~nizhikov] Please see my ignite-config file below.Can you please confirm if I am missing anything here? http://www.springframework.org/schema/beans; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd;> > [Umbrella][Spark] Upgrade Spark module to 2.4 > - > > Key: IGNITE-12054 > URL: https://issues.apache.org/jira/browse/IGNITE-12054 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.8 >Reporter: Denis A. Magda >Assignee: Alexey Zinoviev >Priority: Blocker > Labels: await > Fix For: 2.8 > > Attachments: ignite-spark-patch-new.diff > > Time Spent: 20m > Remaining Estimate: 0h > > Users can't use APIs that are already available in Spark 2.4: > https://stackoverflow.com/questions/57392143/persisting-spark-dataframe-to-ignite > Let's upgrade Spark from 2.3 to 2.4 until we extract the Spark Integration as > a separate module that can support multiple Spark versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-12054) [Umbrella][Spark] Upgrade Spark module to 2.4
[ https://issues.apache.org/jira/browse/IGNITE-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981543#comment-16981543 ] Shenson Joseph commented on IGNITE-12054: - [~zaleslaw] [~nizhikov] Could you please provide a tentative release date for ignite- 2.8 & 3? > [Umbrella][Spark] Upgrade Spark module to 2.4 > - > > Key: IGNITE-12054 > URL: https://issues.apache.org/jira/browse/IGNITE-12054 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.8 >Reporter: Denis A. Magda >Assignee: Alexey Zinoviev >Priority: Blocker > Labels: await > Fix For: 2.8 > > Attachments: ignite-spark-patch-new.diff > > Time Spent: 20m > Remaining Estimate: 0h > > Users can't use APIs that are already available in Spark 2.4: > https://stackoverflow.com/questions/57392143/persisting-spark-dataframe-to-ignite > Let's upgrade Spark from 2.3 to 2.4 until we extract the Spark Integration as > a separate module that can support multiple Spark versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-12054) [Umbrella][Spark] Upgrade Spark module to 2.4
[ https://issues.apache.org/jira/browse/IGNITE-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981540#comment-16981540 ] Shenson Joseph commented on IGNITE-12054: - [~zaleslaw] [~nizhikov] I believe above mentioned exceptions happens inside ignite-spark library.We are using spark-2.4.4 for kubernetes support.We have made some changes in ignite-spark library locally to support spark-2.4 It would be nice if you can provide us a patch and we are currently blocked. Please see the attached diff file for your reference.[^ignite-spark-patch-new.diff] > [Umbrella][Spark] Upgrade Spark module to 2.4 > - > > Key: IGNITE-12054 > URL: https://issues.apache.org/jira/browse/IGNITE-12054 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.8 >Reporter: Denis A. Magda >Assignee: Alexey Zinoviev >Priority: Blocker > Labels: await > Fix For: 2.8 > > Attachments: ignite-spark-patch-new.diff > > Time Spent: 20m > Remaining Estimate: 0h > > Users can't use APIs that are already available in Spark 2.4: > https://stackoverflow.com/questions/57392143/persisting-spark-dataframe-to-ignite > Let's upgrade Spark from 2.3 to 2.4 until we extract the Spark Integration as > a separate module that can support multiple Spark versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12054) [Umbrella][Spark] Upgrade Spark module to 2.4
[ https://issues.apache.org/jira/browse/IGNITE-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shenson Joseph updated IGNITE-12054: Attachment: ignite-spark-patch-new.diff > [Umbrella][Spark] Upgrade Spark module to 2.4 > - > > Key: IGNITE-12054 > URL: https://issues.apache.org/jira/browse/IGNITE-12054 > Project: Ignite > Issue Type: New Feature > Components: spark >Affects Versions: 2.8 >Reporter: Denis A. Magda >Assignee: Alexey Zinoviev >Priority: Blocker > Labels: await > Fix For: 2.8 > > Attachments: ignite-spark-patch-new.diff > > Time Spent: 20m > Remaining Estimate: 0h > > Users can't use APIs that are already available in Spark 2.4: > https://stackoverflow.com/questions/57392143/persisting-spark-dataframe-to-ignite > Let's upgrade Spark from 2.3 to 2.4 until we extract the Spark Integration as > a separate module that can support multiple Spark versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12394) Confusing message and thread dumps about ignored failures
[
https://issues.apache.org/jira/browse/IGNITE-12394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mirza Aliev updated IGNITE-12394:
-
Description:
Currently {{FailureProcessor}} shows the following message in logs
Critical system error detected. Will be handled accordingly to configured
handler
even if a failure type is configured in {{AbstractFailureHandler}} as ignored.
The message should be changed in cases when failure types are ignored.
*Solution:*
* Change current message (but save corresponding handler and context info as
it was before ) to
{{Possible failure suppressed accordingly to a configured handler}}
for cases when failures are ignored and show the message on {{quiete mode}} and
{{warn}} logging levels
* Show original message on error level when failure is not from ignored set
Thread dump should also be logged in level accordingly to the failure processor
message log level.
was:
Currently FailureProcessor shows the following message in logs
Critical system error detected. Will be handled accordingly to configured
handler
even if a failure type is configured in \{{AbstractFailureHandler}}as ignored.
The message should be changed in cases when failure types are ignored.
*Solution:*
* Change current message (but save corresponding handler and context info as
it was before ) to
Possible failure suppressed accordingly to a configured handler
for cases when failures are ignored and show the message on quiete mode and
warn logging levels
* Show original message on error level when failure is not from ignored set
Thread dump should also be logged in level accordingly to the failure processor
message log level.
> Confusing message and thread dumps about ignored failures
> -
>
> Key: IGNITE-12394
> URL: https://issues.apache.org/jira/browse/IGNITE-12394
> Project: Ignite
> Issue Type: Improvement
>Affects Versions: 2.7
>Reporter: Mirza Aliev
>Assignee: Mirza Aliev
>Priority: Major
> Labels: failure-handling
> Fix For: 2.8
>
>
> Currently {{FailureProcessor}} shows the following message in logs
> Critical system error detected. Will be handled accordingly to configured
> handler
> even if a failure type is configured in {{AbstractFailureHandler}} as ignored.
> The message should be changed in cases when failure types are ignored.
> *Solution:*
> * Change current message (but save corresponding handler and context info as
> it was before ) to
> {{Possible failure suppressed accordingly to a configured handler}}
> for cases when failures are ignored and show the message on {{quiete mode}}
> and {{warn}} logging levels
> * Show original message on error level when failure is not from ignored set
> Thread dump should also be logged in level accordingly to the failure
> processor message log level.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (IGNITE-12394) Confusing message and thread dumps about ignored failures
[
https://issues.apache.org/jira/browse/IGNITE-12394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mirza Aliev reassigned IGNITE-12394:
Assignee: Mirza Aliev
> Confusing message and thread dumps about ignored failures
> -
>
> Key: IGNITE-12394
> URL: https://issues.apache.org/jira/browse/IGNITE-12394
> Project: Ignite
> Issue Type: Improvement
>Affects Versions: 2.7
>Reporter: Mirza Aliev
>Assignee: Mirza Aliev
>Priority: Major
> Labels: failure-handling
> Fix For: 2.8
>
>
> Currently FailureProcessor shows the following message in logs
> Critical system error detected. Will be handled accordingly to configured
> handler
> even if a failure type is configured in \{{AbstractFailureHandler}}as ignored.
> The message should be changed in cases when failure types are ignored.
> *Solution:*
> * Change current message (but save corresponding handler and context info as
> it was before ) to
> Possible failure suppressed accordingly to a configured handler
> for cases when failures are ignored and show the message on quiete mode and
> warn logging levels
> * Show original message on error level when failure is not from ignored set
> Thread dump should also be logged in level accordingly to the failure
> processor message log level.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12394) Confusing message and thread dumps about ignored failures
Mirza Aliev created IGNITE-12394:
Summary: Confusing message and thread dumps about ignored failures
Key: IGNITE-12394
URL: https://issues.apache.org/jira/browse/IGNITE-12394
Project: Ignite
Issue Type: Improvement
Affects Versions: 2.7
Reporter: Mirza Aliev
Fix For: 2.8
Currently FailureProcessor shows the following message in logs
Critical system error detected. Will be handled accordingly to configured
handler
even if a failure type is configured in \{{AbstractFailureHandler}}as ignored.
The message should be changed in cases when failure types are ignored.
*Solution:*
* Change current message (but save corresponding handler and context info as
it was before ) to
Possible failure suppressed accordingly to a configured handler
for cases when failures are ignored and show the message on quiete mode and
warn logging levels
* Show original message on error level when failure is not from ignored set
Thread dump should also be logged in level accordingly to the failure processor
message log level.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-12393) Thread pool queue system view
Nikolay Izhikov created IGNITE-12393:
Summary: Thread pool queue system view
Key: IGNITE-12393
URL: https://issues.apache.org/jira/browse/IGNITE-12393
Project: Ignite
Issue Type: Sub-task
Reporter: Nikolay Izhikov
When in the production environment exist some cluster performance issues
usually it leads to the large striped executor queue size.
The number of tasks in the queue can observe by
{StripedExecutorMXBean#getTotalQueueSize} metric. In the case queue size
becomes large it's useful to have the ability to know what tasks are waiting
for execution in the thread pool.
Especially, for dealing with failover scenarios.
We should create a system views to expose information about striped executor
services queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (IGNITE-11075) Index rebuild procedure over cache partition file
[
https://issues.apache.org/jira/browse/IGNITE-11075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981517#comment-16981517
]
Sergey Kalashnikov edited comment on IGNITE-11075 at 11/25/19 12:41 PM:
Implemented the following solution (PR
[https://github.com/apache/ignite/pull/7070):]
Goals:
- Restart failed attempts to rebuild the indexes (due to node crash).
- Minimize the scope of recovery rebuilds to those caches and partitions that
have not been able to complete the rebuild before the crash.
- Provide ability (and API) to rebuild arbitrarily selected partitions of
cache indexes.
Design:
1) A set of partition rebuild markers is kept inside {{index.bin}} file (i.e.
persisted).
For that purpose, the new {{"IndexRebuildMarkers"}} tree is introduced.
Item size for shared cache group: 6 bytes (4 for cacheId and 2 for partition)
Item size for single cache: 2 bytes (just partition)
So, for a node with 2000 local partitions: it takes 6(shared-group) or
1(single-cache) additional page(s) per cache.
For an extreme case of 65500 local partitions per node: it takes 194 or 64
pages per cache.
However, this tree is normally empty (only requires 1 page) and only takes
space when the index rebuild is in progress.
2) Before the index rebuild start:
- Store the partition ids that will be rebuilt into {{index.bin}}.
- Log a new WAL record {{START_BUILD_INDEX_RECORD}} to protect the new
information from the crash before the first checkpoint.
3) After successful completion of each partition rebuild:
- Remove the partition id from the \{{"IndexRebuildMarkers"}} tree.
4) On memory recovery:
- If during logical records recovery we happen to meet
{{START_BUILD_INDEX_RECORD}}, store partitions from the record into the
{{index.bin}} unless the file was removed.
5) On cache start:
- Check if {{index.bin}} exists for a cache-group and then retrieve partition
build markers from the {{"IndexRebuildMarkers"}} tree.
- Start index-rebuild for the marked partitions.
6) New API is provided for use by P2P rebalance:
{{public IgniteInternalFuture rebuildIndexesByPartition(CacheGroupContext
grp, int partId);}}
was (Author: skalashnikov):
Implemented the following solution (PR
https://github.com/apache/ignite/pull/7070):
Goals:
- Restart failed attempts to rebuild the indexes (due to node crash).
- Minimize the scope of recovery rebuilds to those caches and partitions that
have not been able to complete the rebuild before the crash.
- Provide ability (and API) to rebuild arbitrarily selected partitions of cache
indexes.
Design:
1) A set of partition rebuild markers is kept inside {{index.bin}} file (i.e.
persisted).
For that purpose, the new {{"IndexRebuildMarkers"}} tree is introduced.
Item size for shared cache group: 6 bytes (4 for cacheId and 2 for partition)
Item size for single cache: 2 bytes (just partition)
So, for a node with 2000 local partitions: it takes 6(shared-group) or
1(single-cache) additional page(s) per cache.
For an extreme case of 65500 local partitions per node: it takes 194 or 64
pages per cache.
However, this tree is normally empty (only requires 1 page) and only takes
space when the index rebuild is in progress.
2) Before the index rebuild start:
- Store the partition ids that will be rebuilt into {{index.bin}}.
- Log a new WAL record {{START_BUILD_INDEX_RECORD}} to protect the new
information from the crash before the first checkpoint.
3) After successful completion of each partition rebuild:
- Remove the partition id from the {{"IndexRebuildMarkers"}}tree.
4) On memory recovery:
- If during logical records recovery we happen to meet
{{START_BUILD_INDEX_RECORD}}, store partitions from the record into the
{{index.bin}} unless the file was removed.
5) On cache start:
- Check if {{index.bin}} exists for a cache-group and then retrieve partition
build markers from the {{"IndexRebuildMarkers"}} tree.
- Start index-rebuild for the marked partitions.
6) New API is provided for use by P2P rebalance:
{{public IgniteInternalFuture rebuildIndexesByPartition(CacheGroupContext
grp, int partId);}}
> Index rebuild procedure over cache partition file
> -
>
> Key: IGNITE-11075
> URL: https://issues.apache.org/jira/browse/IGNITE-11075
> Project: Ignite
> Issue Type: Sub-task
>Reporter: Maxim Muzafarov
>Assignee: Sergey Kalashnikov
>Priority: Major
> Labels: iep-28
> Fix For: 2.9
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The node can own partition when partition data is rebalanced and cache
> indexes are ready. For the message-based cluster rebalancing, approach
> indexes are rebuilding simultaneously with cache data loading. For the
> file-based rebalancing approach, the index rebuild
[jira] [Commented] (IGNITE-11075) Index rebuild procedure over cache partition file
[
https://issues.apache.org/jira/browse/IGNITE-11075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981517#comment-16981517
]
Sergey Kalashnikov commented on IGNITE-11075:
-
Implemented the following solution (PR
https://github.com/apache/ignite/pull/7070):
Goals:
- Restart failed attempts to rebuild the indexes (due to node crash).
- Minimize the scope of recovery rebuilds to those caches and partitions that
have not been able to complete the rebuild before the crash.
- Provide ability (and API) to rebuild arbitrarily selected partitions of cache
indexes.
Design:
1) A set of partition rebuild markers is kept inside {{index.bin}} file (i.e.
persisted).
For that purpose, the new {{"IndexRebuildMarkers"}} tree is introduced.
Item size for shared cache group: 6 bytes (4 for cacheId and 2 for partition)
Item size for single cache: 2 bytes (just partition)
So, for a node with 2000 local partitions: it takes 6(shared-group) or
1(single-cache) additional page(s) per cache.
For an extreme case of 65500 local partitions per node: it takes 194 or 64
pages per cache.
However, this tree is normally empty (only requires 1 page) and only takes
space when the index rebuild is in progress.
2) Before the index rebuild start:
- Store the partition ids that will be rebuilt into {{index.bin}}.
- Log a new WAL record {{START_BUILD_INDEX_RECORD}} to protect the new
information from the crash before the first checkpoint.
3) After successful completion of each partition rebuild:
- Remove the partition id from the {{"IndexRebuildMarkers"}}tree.
4) On memory recovery:
- If during logical records recovery we happen to meet
{{START_BUILD_INDEX_RECORD}}, store partitions from the record into the
{{index.bin}} unless the file was removed.
5) On cache start:
- Check if {{index.bin}} exists for a cache-group and then retrieve partition
build markers from the {{"IndexRebuildMarkers"}} tree.
- Start index-rebuild for the marked partitions.
6) New API is provided for use by P2P rebalance:
{{public IgniteInternalFuture rebuildIndexesByPartition(CacheGroupContext
grp, int partId);}}
> Index rebuild procedure over cache partition file
> -
>
> Key: IGNITE-11075
> URL: https://issues.apache.org/jira/browse/IGNITE-11075
> Project: Ignite
> Issue Type: Sub-task
>Reporter: Maxim Muzafarov
>Assignee: Sergey Kalashnikov
>Priority: Major
> Labels: iep-28
> Fix For: 2.9
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The node can own partition when partition data is rebalanced and cache
> indexes are ready. For the message-based cluster rebalancing, approach
> indexes are rebuilding simultaneously with cache data loading. For the
> file-based rebalancing approach, the index rebuild procedure must be finished
> before the partition state is set to the OWNING state.
> We need to rebuild local SQL indexes (the {{index.bin}} file) when partition
> file has been received. Crash-recovery guarantees must be supported by a node
> since index-rebuild performs on the node in the topology.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (IGNITE-12392) Faster transaction rolled back when one of backup node failed
[
https://issues.apache.org/jira/browse/IGNITE-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Tkalenko updated IGNITE-12392:
-
Description:
In case of massive prepared transactions roll back, when node fail, have a
linearizable behavior:
{code:java}
2019-09-26
18:48:21.034[ERROR][sys-stripe-16-#17%GridNodeName%[o.a.i.s.c.tcp.TcpCommunicationSpi]
Failed to send message to remote node [node=TcpDiscoveryNode
[id=1dc0c76a-8e72-48e7-9718-b157eea1b812, addrs=ArrayList [10.0.0.127],
sockAddrs=HashSet [node/10.0.0.127:47500], discPort=47500, order=524,
intOrder=311, lastExchangeTime=1569430937898, loc=false,
ver=2.5.1#20190327-sha1:6edfea1b, isClient=false], msg=GridIoMessage [plc=2,
topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false,
msg=GridCacheIdMessage [cacheId=0]GridDistributedBaseMessage
[ver=GridCacheVersion [topVer=176921134, order=1634060411645, nodeOrder=1],
committedVers=EmptyList [], rolledbackVers=EmptyList [], cnt=0,
super=]GridDistributedTxFinishRequest [topVer=AffinityTopologyVersion
[topVer=524, minorTopVer=2],
futId=fb44a686e61-9a074a8c-dca4--84fe-e9a93818fbd2, threadId=2098,
commitVer=GridCacheVersion [topVer=176921134, order=1634060411645, nodeOrder=1],
org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed to
send message (node left topology): TcpDiscoveryNode
[id=1dc0c76a-8e72-48e7-9718-b157eea1b812, addrs=ArrayList [10.0.0.127],
sockAddrs=HashSet [node/10.0.0.127:47500], discPort=47500, order=524,
intOrder=311, lastExchangeTime=1569430937898, loc=false,
ver=2.5.1#20190327-sha1:6edfea1b, isClient=false]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3276)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2998)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2878)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2721)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2680)
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1643)
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1715)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1177)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxFinishFuture.finish(GridDhtTxFinishFuture.java:462)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxFinishFuture.finish(GridDhtTxFinishFuture.java:291)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:495)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.rollbackDhtLocalAsync(GridDhtTxLocal.java:571)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1005)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:876)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:832)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:101)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:193)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:191)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1061)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:385)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:311)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:300)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:496)
at
[jira] [Commented] (IGNITE-12248) Apache Calcite based query execution engine
[ https://issues.apache.org/jira/browse/IGNITE-12248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981497#comment-16981497 ] Andrey Mashenkov commented on IGNITE-12248: --- [~gvvinblade], I've left my 50 cent. Please, see comments in the PR. > Apache Calcite based query execution engine > --- > > Key: IGNITE-12248 > URL: https://issues.apache.org/jira/browse/IGNITE-12248 > Project: Ignite > Issue Type: Task > Components: sql >Reporter: Igor Seliverstov >Assignee: Igor Seliverstov >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > Currently used H2 based query execution engine has a number of critical > limitations Which do not allow to execute an arbitrary query. > The ticket aims to show potential of a new, Calcite based, execution engine > which may act not worse than current one on co-located queries, provide a > boost for queries, using distributed joins, and provide an ability to execute > arbitrary queries, requiring more than one map-reduce step in execution flow. > [IEP > link|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130028084] > [Dev list > thread|http://apache-ignite-developers.2346864.n4.nabble.com/New-SQL-execution-engine-td43724.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-12392) Faster transaction rolled back when one of backup node failed
[
https://issues.apache.org/jira/browse/IGNITE-12392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Tkalenko updated IGNITE-12392:
-
Description:
In case of massive prepared transactions roll back, when node fail, have a
linearizable behavior:
{code:java}
2019-09-26
18:48:21.034[ERROR][sys-stripe-16-#17%DPL_GRID%DplGridNodeName%[o.a.i.s.c.tcp.TcpCommunicationSpi]
Failed to send message to remote node [node=TcpDiscoveryNode
[id=1dc0c76a-8e72-48e7-9718-b157eea1b812, addrs=ArrayList [10.124.133.201],
sockAddrs=HashSet [marica63.ca.sbrf.ru/10.124.133.201:47500], discPort=47500,
order=524, intOrder=311, lastExchangeTime=1569430937898, loc=false,
ver=2.5.1#20190327-sha1:6edfea1b, isClient=false], msg=GridIoMessage [plc=2,
topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false,
msg=GridCacheIdMessage [cacheId=0]GridDistributedBaseMessage
[ver=GridCacheVersion [topVer=176921134, order=1634060411645, nodeOrder=1],
committedVers=EmptyList [], rolledbackVers=EmptyList [], cnt=0,
super=]GridDistributedTxFinishRequest [topVer=AffinityTopologyVersion
[topVer=524, minorTopVer=2],
futId=fb44a686e61-9a074a8c-dca4--84fe-e9a93818fbd2, threadId=2098,
commitVer=GridCacheVersion [topVer=176921134, order=1634060411645, nodeOrder=1],
org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed to
send message (node left topology): TcpDiscoveryNode
[id=1dc0c76a-8e72-48e7-9718-b157eea1b812, addrs=ArrayList [10.124.133.201],
sockAddrs=HashSet [marica63.ca.sbrf.ru/10.124.133.201:47500], discPort=47500,
order=524, intOrder=311, lastExchangeTime=1569430937898, loc=false,
ver=2.5.1#20190327-sha1:6edfea1b, isClient=false]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3276)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2998)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2878)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2721)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2680)
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1643)
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1715)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1177)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxFinishFuture.finish(GridDhtTxFinishFuture.java:462)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxFinishFuture.finish(GridDhtTxFinishFuture.java:291)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:495)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.rollbackDhtLocalAsync(GridDhtTxLocal.java:571)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1005)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:876)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:832)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:101)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:193)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:191)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1061)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:385)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:311)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:300)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:496)
[jira] [Created] (IGNITE-12392) Faster transaction rolled back when one of backup node failed
Kirill Tkalenko created IGNITE-12392:
Summary: Faster transaction rolled back when one of backup node
failed
Key: IGNITE-12392
URL: https://issues.apache.org/jira/browse/IGNITE-12392
Project: Ignite
Issue Type: Improvement
Reporter: Kirill Tkalenko
Assignee: Kirill Tkalenko
Fix For: 2.8
In case of massive prepared transactions roll back, when node fail, have a
linearizable behavior:
{noformat}2019-09-26
18:48:21.034[ERROR][sys-stripe-16-#17%DPL_GRID%DplGridNodeName%[o.a.i.s.c.tcp.TcpCommunicationSpi]
Failed to send message to remote node [node=TcpDiscoveryNode
[id=1dc0c76a-8e72-48e7-9718-b157eea1b812, addrs=ArrayList [10.124.133.201],
sockAddrs=HashSet [marica63.ca.sbrf.ru/10.124.133.201:47500], discPort=47500,
order=524, intOrder=311, lastExchangeTime=1569430937898, loc=false,
ver=2.5.1#20190327-sha1:6edfea1b, isClient=false], msg=GridIoMessage [plc=2,
topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false,
msg=GridCacheIdMessage [cacheId=0]GridDistributedBaseMessage
[ver=GridCacheVersion [topVer=176921134, order=1634060411645, nodeOrder=1],
committedVers=EmptyList [], rolledbackVers=EmptyList [], cnt=0,
super=]GridDistributedTxFinishRequest [topVer=AffinityTopologyVersion
[topVer=524, minorTopVer=2],
futId=fb44a686e61-9a074a8c-dca4--84fe-e9a93818fbd2, threadId=2098,
commitVer=GridCacheVersion [topVer=176921134, order=1634060411645, nodeOrder=1],
org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed to
send message (node left topology): TcpDiscoveryNode
[id=1dc0c76a-8e72-48e7-9718-b157eea1b812, addrs=ArrayList [10.124.133.201],
sockAddrs=HashSet [marica63.ca.sbrf.ru/10.124.133.201:47500], discPort=47500,
order=524, intOrder=311, lastExchangeTime=1569430937898, loc=false,
ver=2.5.1#20190327-sha1:6edfea1b, isClient=false]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3276)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2998)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2878)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2721)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2680)
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1643)
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1715)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1177)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxFinishFuture.finish(GridDhtTxFinishFuture.java:462)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxFinishFuture.finish(GridDhtTxFinishFuture.java:291)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:495)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.rollbackDhtLocalAsync(GridDhtTxLocal.java:571)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:1005)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:876)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:832)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:101)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:193)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:191)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1061)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:385)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:311)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:300)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
[jira] [Commented] (IGNITE-11987) [IEP-35] Add ability to configure metrics
[ https://issues.apache.org/jira/browse/IGNITE-11987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981445#comment-16981445 ] Andrey N. Gura commented on IGNITE-11987: - [~nizhikov] Thanks! I'll take a look. > [IEP-35] Add ability to configure metrics > - > > Key: IGNITE-11987 > URL: https://issues.apache.org/jira/browse/IGNITE-11987 > Project: Ignite > Issue Type: Improvement >Reporter: Nikolay Izhikov >Assignee: Nikolay Izhikov >Priority: Major > Labels: IEP-35 > Fix For: 2.8 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Ignite should be able to: > * Configure Histogram metrics > * Configure HitRate metrics. > We should provide 2 ways to configure metric: > 1. -Configuration file.- Discussed on dev-list. Agreed to go with the > simplest solution - JMX method. > 2. JMX method. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-12248) Apache Calcite based query execution engine
[
https://issues.apache.org/jira/browse/IGNITE-12248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981405#comment-16981405
]
Roman Kondakov commented on IGNITE-12248:
-
[~gvvinblade], I've taken a look to your approach and overall it seems to be
very good.
I just have a list of minor questions:
# Why do we need spring dependencies in calcite/pom.xml?
# Are we going to support {{_key}} and {{_val}} columns in the new engine? Why?
# Why do we use java serialization in calcite.serialize package?
# What is the purpose of {{CalciteSchemaHolder}}? Why don't use just
{{RootSchema}}? I haven't got the idea.
# Why do we need {{IgnitePlanner}}? It is almost certain copy of
{{org.apache.calcite.prepare.PlannerImpl}}
# Why do we wrap everything into {{Context}}? Even query itself. Do we really
need it? See {{CalciteQueryProcessor#context()}}. Excessive use of
wrap/unwrap/chaining/etc methods affects code readablity.
# Looks like {{RelOp}} interface is redundant. Also as usage of {{BiFunction}}
in {{Commons#transformSubset}}. IMO we need to avoid excessive using of lambdas
and functions. It affects readability too.
# Why do we need to have {{DistributionRegistry}}? Aren't tables aware of
their distributions?
# IgniteMdDistribution#join - there is a list of differnet types of joins and
there is also a statement that "others are impossible". Why broadcast LEFT JOIN
hash is impossible?
# What is the difference between implementor and visitor? See
{{RelImplementor}}. If there are no differences, why don't use the word
{{Visitor}} instead of {{Implementor}} just because it is the well known
pattern?
> Apache Calcite based query execution engine
> ---
>
> Key: IGNITE-12248
> URL: https://issues.apache.org/jira/browse/IGNITE-12248
> Project: Ignite
> Issue Type: Task
> Components: sql
>Reporter: Igor Seliverstov
>Assignee: Igor Seliverstov
>Priority: Major
>
> Currently used H2 based query execution engine has a number of critical
> limitations Which do not allow to execute an arbitrary query.
> The ticket aims to show potential of a new, Calcite based, execution engine
> which may act not worse than current one on co-located queries, provide a
> boost for queries, using distributed joins, and provide an ability to execute
> arbitrary queries, requiring more than one map-reduce step in execution flow.
> [IEP
> link|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=130028084]
> [Dev list
> thread|http://apache-ignite-developers.2346864.n4.nabble.com/New-SQL-execution-engine-td43724.html]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
