[jira] [Created] (IGNITE-18035) C++ 3.0: Add support for all native data types
Igor Sapego created IGNITE-18035: Summary: C++ 3.0: Add support for all native data types Key: IGNITE-18035 URL: https://issues.apache.org/jira/browse/IGNITE-18035 Project: Ignite Issue Type: Improvement Components: thin client Reporter: Igor Sapego Assignee: Igor Sapego Fix For: 3.0.0-beta2 Add support for all data types in C++ client: Date, Time, etc - see -org.apache.ignite.client.proto.ClientDataType- *org.apache.ignite.internal.schema.NativeTypes*. * We now use *BinaryTuple* format to exchange row data => .NET *BinaryTuple* implementation should be updated to support all missing types. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18003) ItTablesApiTest#testCreateDropTable became flaky
[
https://issues.apache.org/jira/browse/IGNITE-18003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denis Chudov updated IGNITE-18003:
--
Description:
*Motivation:*
Seems that index create listener in IndexManager gets table from TableManager
without using causality token, and sometimes it appears that table is not
created yet. Using the causality token should gurantee the presence of table in
table manager for the index manager.
This causes unstability of the test ItTablesApiTest#testCreateDropTable:
{code:java}
[16:41:04][org.apache.ignite:ignite-runner] 2022-10-27 19:41:04:009 +0300
[INFO][main][ItTablesApiTest] Dropping the table ignored.
[16:41:04][org.apache.ignite:ignite-runner]
org.apache.ignite.lang.TableNotFoundException: IGN-TBL-2
TraceId:4ecb38fa-7440-448f-8d13-4d351df39ee7 The table does not exist
[name="PUBLIC"."TBL1"]
[16:41:04][org.apache.ignite:ignite-runner] at
org.apache.ignite.internal.table.distributed.TableManager.lambda$dropTableAsyncInternal$57(TableManager.java:1368)
[16:41:04][org.apache.ignite:ignite-runner] at
java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753)
[16:41:04][org.apache.ignite:ignite-runner] at
java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731)
[16:41:04][org.apache.ignite:ignite-runner] at
java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108)
[16:41:04][org.apache.ignite:ignite-runner] at
org.apache.ignite.internal.table.distributed.TableManager.dropTableAsyncInternal(TableManager.java:1364)
[16:41:04][org.apache.ignite:ignite-runner] at
org.apache.ignite.internal.table.distributed.TableManager.dropTableAsync(TableManager.java:1354)
[16:41:04][org.apache.ignite:ignite-runner] at
org.apache.ignite.internal.runner.app.ItTablesApiTest.dropTableIfExists(ItTablesApiTest.java:517)
[16:41:04][org.apache.ignite:ignite-runner] at
org.apache.ignite.internal.runner.app.ItTablesApiTest.testCreateDropTable(ItTablesApiTest.java:451)
[16:41:04][org.apache.ignite:ignite-runner] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[16:41:04][org.apache.ignite:ignite-runner] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[16:41:04][org.apache.ignite:ignite-runner] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[16:41:04][org.apache.ignite:ignite-runner] at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
[16:41:04][org.apache.ignite:ignite-runner] at
[jira] [Updated] (IGNITE-18003) ItTablesApiTest#testCreateDropTable became flaky
[
https://issues.apache.org/jira/browse/IGNITE-18003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denis Chudov updated IGNITE-18003:
--
Description:
*Motivation:*
Seems that index create listener in IndexManager gets table from TableManager
without using causality token, and sometimes it appears that table is not
created yet. Using the causality token should gurantee the presence of table in
table manager for the index manager.
This causes unstability of the test ItTablesApiTest#testCreateDropTable:
{code:java}
[16:41:04][org.apache.ignite:ignite-runner] 2022-10-27 19:41:04:009 +0300
[INFO][main][ItTablesApiTest] Dropping the table ignored.
[16:41:04][org.apache.ignite:ignite-runner]
org.apache.ignite.lang.TableNotFoundException: IGN-TBL-2
TraceId:4ecb38fa-7440-448f-8d13-4d351df39ee7 The table does not exist
[name="PUBLIC"."TBL1"]
[16:41:04][org.apache.ignite:ignite-runner] at
org.apache.ignite.internal.table.distributed.TableManager.lambda$dropTableAsyncInternal$57(TableManager.java:1368)
[16:41:04][org.apache.ignite:ignite-runner] at
java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753)
[16:41:04][org.apache.ignite:ignite-runner] at
java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731)
[16:41:04][org.apache.ignite:ignite-runner] at
java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108)
[16:41:04][org.apache.ignite:ignite-runner] at
org.apache.ignite.internal.table.distributed.TableManager.dropTableAsyncInternal(TableManager.java:1364)
[16:41:04][org.apache.ignite:ignite-runner] at
org.apache.ignite.internal.table.distributed.TableManager.dropTableAsync(TableManager.java:1354)
[16:41:04][org.apache.ignite:ignite-runner] at
org.apache.ignite.internal.runner.app.ItTablesApiTest.dropTableIfExists(ItTablesApiTest.java:517)
[16:41:04][org.apache.ignite:ignite-runner] at
org.apache.ignite.internal.runner.app.ItTablesApiTest.testCreateDropTable(ItTablesApiTest.java:451)
[16:41:04][org.apache.ignite:ignite-runner] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[16:41:04][org.apache.ignite:ignite-runner] at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[16:41:04][org.apache.ignite:ignite-runner] at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[16:41:04][org.apache.ignite:ignite-runner] at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
[16:41:04][org.apache.ignite:ignite-runner] at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
[16:41:04][org.apache.ignite:ignite-runner] at
[jira] [Updated] (IGNITE-17989) JVM memory usage metric source
[ https://issues.apache.org/jira/browse/IGNITE-17989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Gusakov updated IGNITE-17989: Epic Link: IGNITE-17353 > JVM memory usage metric source > -- > > Key: IGNITE-17989 > URL: https://issues.apache.org/jira/browse/IGNITE-17989 > Project: Ignite > Issue Type: Task >Reporter: Kirill Gusakov >Assignee: Kirill Gusakov >Priority: Major > Labels: ignite-3 > > h3. Motivation > It will be useful to have builtin metric source with jvm memory usage stats. > h3. Definition of Done > It must expose the following metrics as a separate gauge metric each: > * heap memory usage (init/used/commited/max) > * non-heap memory usage (init/used/commited/max) > * name of metrics should have the general prefix "jvm." to group all future > jvm metrics together > Also, it needs to check, if calculation of these stats is heavy and we need > to cache it for some time windows. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17989) JVM memory usage metric source
[ https://issues.apache.org/jira/browse/IGNITE-17989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-17989: - Description: h3. Motivation It will be useful to have builtin metric source with jvm memory usage stats. h3. Definition of Done It must expose the following metrics as a separate gauge metric each: * heap memory usage (init/used/commited/max) * non-heap memory usage (init/used/commited/max) * name of metrics should have the general prefix "jvm." to group all future jvm metrics together Also, it needs to check, if calculation of these stats is heavy and we need to cache it for some time windows. was: It will be useful to have builtin metric source with jvm memory usage stats. It must expose the following metrics as a separate gauge metric each: * heap memory usage (init/used/commited/max) * non-heap memory usage (init/used/commited/max) * name of metrics should have the general prefix "jvm." to group all future jvm metrics together Also, it needs to check, if calculation of these stats is heavy and we need to cache it for some time windows. > JVM memory usage metric source > -- > > Key: IGNITE-17989 > URL: https://issues.apache.org/jira/browse/IGNITE-17989 > Project: Ignite > Issue Type: Task >Reporter: Kirill Gusakov >Assignee: Kirill Gusakov >Priority: Major > Labels: ignite-3 > > h3. Motivation > It will be useful to have builtin metric source with jvm memory usage stats. > h3. Definition of Done > It must expose the following metrics as a separate gauge metric each: > * heap memory usage (init/used/commited/max) > * non-heap memory usage (init/used/commited/max) > * name of metrics should have the general prefix "jvm." to group all future > jvm metrics together > Also, it needs to check, if calculation of these stats is heavy and we need > to cache it for some time windows. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17357) JMX metric exporter for Ignite 3
[ https://issues.apache.org/jira/browse/IGNITE-17357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-17357: - Fix Version/s: 3.0.0-beta2 > JMX metric exporter for Ignite 3 > > > Key: IGNITE-17357 > URL: https://issues.apache.org/jira/browse/IGNITE-17357 > Project: Ignite > Issue Type: New Feature >Reporter: Denis Chudov >Assignee: Kirill Gusakov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 50m > Remaining Estimate: 0h > > Metrics should be able to be exported via JMX as a first stage of metrics > exposing. > Exporter implementation must provide the following behavior: > * for each MetricSource we need to provide separate MBean with attribute per > metric > * each MBean attribute must have the same name as corresponding metric > * on enable/disable event for MetricSource corresponding MBean must be > registered/unregistered -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18034) Address CVE-2022-39135 by upgrading calcite-core to 1.32.0
Sander created IGNITE-18034: --- Summary: Address CVE-2022-39135 by upgrading calcite-core to 1.32.0 Key: IGNITE-18034 URL: https://issues.apache.org/jira/browse/IGNITE-18034 Project: Ignite Issue Type: Bug Components: ignite-3 Affects Versions: 2.14 Reporter: Sander Hello, We have recently upgraded to ignite version 2.14.0 to take advantage of the new calcite SQL engine. However there is a critical vulnerability with the current version of calcite-core 1.30.0. Calcite-core version 1.30.0 has a critical vulnerability [https://nvd.nist.gov/vuln/detail/CVE-2022-39135] This vulnerability is resolved in calcite-core version 1.32.0. However if we force this package in our build. There are issues running sql queries against ignite with the error: ``` java.lang.AbstractMethodError: org.apache.calcite.sql.parser.SqlAbstractParserImpl.setTimeUnitCodes(Ljava/util/Map;)V at org.apache.calcite.sql.parser.SqlParser.(SqlParser.java:73) ~[calcite-core-1.32.0.jar:1.32.0] at org.apache.calcite.sql.parser.SqlParser.create(SqlParser.java:126) ~[calcite-core-1.32.0.jar:1.32.0] at org.apache.ignite.internal.processors.query.calcite.util.Commons.parse(Commons.java:220) ~[ignite-calcite-2.14.0.jar:2.14.0] at org.apache.ignite.internal.processors.query.calcite.util.Commons.parse(Commons.java:204) ~[ignite-calcite-2.14.0.jar:2.14.0] at org.apache.ignite.internal.processors.query.calcite.CalciteQueryProcessor.query(CalciteQueryProcessor.java:345) ~[ignite-calcite-2.14.0.jar:2.14.0] at org.apache.ignite.internal.processors.query.GridQueryProcessor$2.applyx(GridQueryProcessor.java:3092) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.processors.query.GridQueryProcessor$2.applyx(GridQueryProcessor.java:3074) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:36) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:3751) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.processors.query.GridQueryProcessor.lambda$querySqlFields$3(GridQueryProcessor.java:3118) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuerySafe(GridQueryProcessor.java:3190) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:3070) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:3024) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandler.querySqlFields(JdbcRequestHandler.java:773) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandler.executeQuery(JdbcRequestHandler.java:641) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandler.doHandle(JdbcRequestHandler.java:311) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.processors.odbc.jdbc.JdbcRequestHandler.handle(JdbcRequestHandler.java:251) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:204) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:55) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.body(GridNioAsyncNotifyFilter.java:97) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125) ~[ignite-core-2.14.0.jar:2.14.0] at org.apache.ignite.internal.util.worker.GridWorkerPool$1.run(GridWorkerPool.java:70) [ignite-core-2.14.0.jar:2.14.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_301] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_301] at java.lang.Thread.run(Unknown Source) [na:1.8.0_301]``` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17993) .NET: Thin 3.0: SqlTests.TestCreateTableExistsThrowsException is flaky
[
https://issues.apache.org/jira/browse/IGNITE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626737#comment-17626737
]
Pavel Tupitsyn commented on IGNITE-17993:
-
Merged to main: 8d7c589e5637cb6cf395c70dc46e6235f095cdba
> .NET: Thin 3.0: SqlTests.TestCreateTableExistsThrowsException is flaky
> --
>
> Key: IGNITE-17993
> URL: https://issues.apache.org/jira/browse/IGNITE-17993
> Project: Ignite
> Issue Type: Bug
> Components: platforms, thin client
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Minor
> Labels: .NET, ignite-3
> Fix For: 3.0.0-beta2
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Looks like a race condition, *TestEventListener* may receive events with a
> delay.
> {code}
> Expected: 742
> But was: 743
> TearDown : NUnit.Framework.AssertionException : Expected: 742
> But was: 743
>at Apache.Ignite.Tests.IgniteTestsBase.TearDown() in
> /opt/buildagent/work/b8d4df1365f1f1e5/modules/platforms/dotnet/Apache.Ignite.Tests/IgniteTestsBase.cs:line
> 89
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-17993) .NET: Thin 3.0: SqlTests.TestCreateTableExistsThrowsException is flaky
[
https://issues.apache.org/jira/browse/IGNITE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626727#comment-17626727
]
Igor Sapego commented on IGNITE-17993:
--
[~ptupitsyn] looks good to me
> .NET: Thin 3.0: SqlTests.TestCreateTableExistsThrowsException is flaky
> --
>
> Key: IGNITE-17993
> URL: https://issues.apache.org/jira/browse/IGNITE-17993
> Project: Ignite
> Issue Type: Bug
> Components: platforms, thin client
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Minor
> Labels: .NET, ignite-3
> Fix For: 3.0.0-beta2
>
>
> Looks like a race condition, *TestEventListener* may receive events with a
> delay.
> {code}
> Expected: 742
> But was: 743
> TearDown : NUnit.Framework.AssertionException : Expected: 742
> But was: 743
>at Apache.Ignite.Tests.IgniteTestsBase.TearDown() in
> /opt/buildagent/work/b8d4df1365f1f1e5/modules/platforms/dotnet/Apache.Ignite.Tests/IgniteTestsBase.cs:line
> 89
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-18032) Do not implement AutoCloseable on classes instances of which are not meant to be used in try-with-resources
[ https://issues.apache.org/jira/browse/IGNITE-18032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-18032: --- Summary: Do not implement AutoCloseable on classes instances of which are not meant to be used in try-with-resources (was: Do not implement AutoCloseable on classes which instances are not meant to be used in try-with-resources) > Do not implement AutoCloseable on classes instances of which are not meant to > be used in try-with-resources > --- > > Key: IGNITE-18032 > URL: https://issues.apache.org/jira/browse/IGNITE-18032 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3, tech-debt > Fix For: 3.0.0-beta2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (IGNITE-16667) AFFINITY_COLUMN & PK column in the TABLE_COLUMNS view are always FALSE.
[ https://issues.apache.org/jira/browse/IGNITE-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YuJue Li closed IGNITE-16667. - Ignite Flags: (was: Docs Required,Release Notes Required) > AFFINITY_COLUMN & PK column in the TABLE_COLUMNS view are always FALSE. > --- > > Key: IGNITE-16667 > URL: https://issues.apache.org/jira/browse/IGNITE-16667 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.12 >Reporter: YuJue Li >Priority: Critical > Fix For: 2.14 > > > execute the sql: > CREATE TABLE City ( > ID INT, > Name VARCHAR, > CountryCode CHAR(3), > District VARCHAR, > Population INT, > PRIMARY KEY (ID, CountryCode) > ); > then select * from sys.table_column where table_name = 'CITY'; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-16667) AFFINITY_COLUMN & PK column in the TABLE_COLUMNS view are always FALSE.
[ https://issues.apache.org/jira/browse/IGNITE-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626641#comment-17626641 ] YuJue Li commented on IGNITE-16667: --- the issue has been fixed in 2.14. > AFFINITY_COLUMN & PK column in the TABLE_COLUMNS view are always FALSE. > --- > > Key: IGNITE-16667 > URL: https://issues.apache.org/jira/browse/IGNITE-16667 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.12 >Reporter: YuJue Li >Priority: Critical > Fix For: 2.14 > > > execute the sql: > CREATE TABLE City ( > ID INT, > Name VARCHAR, > CountryCode CHAR(3), > District VARCHAR, > Population INT, > PRIMARY KEY (ID, CountryCode) > ); > then select * from sys.table_column where table_name = 'CITY'; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (IGNITE-16667) AFFINITY_COLUMN & PK column in the TABLE_COLUMNS view are always FALSE.
[ https://issues.apache.org/jira/browse/IGNITE-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YuJue Li resolved IGNITE-16667. --- Resolution: Fixed > AFFINITY_COLUMN & PK column in the TABLE_COLUMNS view are always FALSE. > --- > > Key: IGNITE-16667 > URL: https://issues.apache.org/jira/browse/IGNITE-16667 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.12 >Reporter: YuJue Li >Priority: Critical > Fix For: 2.14 > > > execute the sql: > CREATE TABLE City ( > ID INT, > Name VARCHAR, > CountryCode CHAR(3), > District VARCHAR, > Population INT, > PRIMARY KEY (ID, CountryCode) > ); > then select * from sys.table_column where table_name = 'CITY'; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-16667) AFFINITY_COLUMN & PK column in the TABLE_COLUMNS view are always FALSE.
[ https://issues.apache.org/jira/browse/IGNITE-16667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YuJue Li updated IGNITE-16667: -- Fix Version/s: 2.14 > AFFINITY_COLUMN & PK column in the TABLE_COLUMNS view are always FALSE. > --- > > Key: IGNITE-16667 > URL: https://issues.apache.org/jira/browse/IGNITE-16667 > Project: Ignite > Issue Type: Bug > Components: sql >Affects Versions: 2.12 >Reporter: YuJue Li >Priority: Critical > Fix For: 2.14 > > > execute the sql: > CREATE TABLE City ( > ID INT, > Name VARCHAR, > CountryCode CHAR(3), > District VARCHAR, > Population INT, > PRIMARY KEY (ID, CountryCode) > ); > then select * from sys.table_column where table_name = 'CITY'; -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18032) Do not implement AutoCloseable on classes which instances are not meant to be used in try-with-resources
[ https://issues.apache.org/jira/browse/IGNITE-18032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-18032: --- Labels: ignite-3 tech-debt (was: ignite-3) > Do not implement AutoCloseable on classes which instances are not meant to be > used in try-with-resources > > > Key: IGNITE-18032 > URL: https://issues.apache.org/jira/browse/IGNITE-18032 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3, tech-debt > Fix For: 3.0.0-beta2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18033) Implement cooperative GC of MV data during RAFT commands execution
Ivan Bessonov created IGNITE-18033: -- Summary: Implement cooperative GC of MV data during RAFT commands execution Key: IGNITE-18033 URL: https://issues.apache.org/jira/browse/IGNITE-18033 Project: Ignite Issue Type: Improvement Reporter: Ivan Bessonov Please refer to the Epic for the basic description. Please also refer to IGNITE-18031 for naive implementation details and general thoughts. Technically, there is a possibility that the background GC process wouldn't catch up if there's too much data being loaded into the system. Scanning through the entire partition takes time, and only a small subset of data could be under a constant stream of modification. To account for that, each update can be preceded with the manual GC of that row. In this case, there's less work for the background processor, and there's an empirical sense that frequently updated data will be just as frequently vacuumed, thus not allowing too much garbage to appear in the first place. Now that's not a guarantee that there would be no problems at all, so we should also think about other ways of cooperative GC. Of course, it would be nice to have a queue of "old" rows ready for you to clean. How feasible is it? It would have to be supported by every engine. That's extra data and extra time for management. Looks pretty much exactly like a TTL manager in Ignite 2.x. I assume that we don't implement anything like that right now, starting with more naive approaches. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18009) Fix gradle build
[ https://issues.apache.org/jira/browse/IGNITE-18009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Lukyanov updated IGNITE-18009: Fix Version/s: 3.0.0-beta2 (was: 3.0.0-beta1) > Fix gradle build > > > Key: IGNITE-18009 > URL: https://issues.apache.org/jira/browse/IGNITE-18009 > Project: Ignite > Issue Type: Bug > Components: build >Reporter: Mikhail Pochatkin >Assignee: Mikhail Pochatkin >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-18009) Fix gradle build
[ https://issues.apache.org/jira/browse/IGNITE-18009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626593#comment-17626593 ] Stanislav Lukyanov commented on IGNITE-18009: - It looks like this is not required in beta1 since the fix that caused this isn't in beta1. > Fix gradle build > > > Key: IGNITE-18009 > URL: https://issues.apache.org/jira/browse/IGNITE-18009 > Project: Ignite > Issue Type: Bug > Components: build >Reporter: Mikhail Pochatkin >Assignee: Mikhail Pochatkin >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta1 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17265) Support RAFT snapshot streaming for PageMemory storage
[ https://issues.apache.org/jira/browse/IGNITE-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626592#comment-17626592 ] Stanislav Lukyanov commented on IGNITE-17265: - I see that there is a pattern of setting "fixed by" links in Ignite Jira - let's do that. I've set the link, no more actions are required. > Support RAFT snapshot streaming for PageMemory storage > -- > > Key: IGNITE-17265 > URL: https://issues.apache.org/jira/browse/IGNITE-17265 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta1 > > > IGNITE-17254 API needs to be implemented for PageMemory storage -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-17979) Windows support for ZIP package
[ https://issues.apache.org/jira/browse/IGNITE-17979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vadim Pakhnushev reassigned IGNITE-17979: - Assignee: Vadim Pakhnushev > Windows support for ZIP package > --- > > Key: IGNITE-17979 > URL: https://issues.apache.org/jira/browse/IGNITE-17979 > Project: Ignite > Issue Type: Bug > Components: build >Reporter: Aleksandr >Assignee: Vadim Pakhnushev >Priority: Major > Labels: ignite-3 > > Cli package could be started from the windows terminal but db could not. Now > there is no way to start the ignite3db process on Windows. > We have to add the support of the bat script to ignite3db zip package. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17865) Disable partition ranges in log messages about rebalance or PME
[
https://issues.apache.org/jira/browse/IGNITE-17865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626589#comment-17626589
]
Ignite TC Bot commented on IGNITE-17865:
{panel:title=Branch: [pull/10352/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/10352/head] Base: [master] : New Tests
(12)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}Basic 1{color} [[tests
6|https://ci.ignite.apache.org/viewLog.html?buildId=6878169]]
* {color:#013220}IgniteBasicTestSuite:
GridToStringBuilderSelfTest.testCompactPartitionsDisabledByDefault -
PASSED{color}
* {color:#013220}IgniteBasicTestSuite:
EvictPartitionInLogTest.testEvictPartByRentingState[printRanges=true] -
PASSED{color}
* {color:#013220}IgniteBasicTestSuite:
EvictPartitionInLogTest.testEvictPartByMovingState[printRanges=false] -
PASSED{color}
* {color:#013220}IgniteBasicTestSuite:
EvictPartitionInLogTest.testEvictPartByMovingState[printRanges=true] -
PASSED{color}
* {color:#013220}IgniteBasicTestSuite:
EvictPartitionInLogTest.testEvictPartByRentingState[printRanges=false] -
PASSED{color}
* {color:#013220}IgniteBasicTestSuite:
GridToStringBuilderSelfTest.testCompactPartitions - PASSED{color}
{color:#8b}PDS 2{color} [[tests
6|https://ci.ignite.apache.org/viewLog.html?buildId=6878233]]
* {color:#013220}IgnitePdsTestSuite2:
IgniteWalRebalanceLoggingTest.testHistoricalRebalanceLogMsg[printRanges=true] -
PASSED{color}
* {color:#013220}IgnitePdsTestSuite2:
IgniteWalRebalanceLoggingTest.testFullRebalanceLogMsgs[printRanges=false] -
PASSED{color}
* {color:#013220}IgnitePdsTestSuite2:
IgniteWalRebalanceLoggingTest.testFullRebalanceLogMsgs[printRanges=true] -
PASSED{color}
* {color:#013220}IgnitePdsTestSuite2:
IgniteWalRebalanceLoggingTest.testFullRebalanceWithShortCpHistoryLogMsgs[printRanges=true]
- PASSED{color}
* {color:#013220}IgnitePdsTestSuite2:
IgniteWalRebalanceLoggingTest.testFullRebalanceWithShortCpHistoryLogMsgs[printRanges=false]
- PASSED{color}
* {color:#013220}IgnitePdsTestSuite2:
IgniteWalRebalanceLoggingTest.testHistoricalRebalanceLogMsg[printRanges=false]
- PASSED{color}
{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=6878279buildTypeId=IgniteTests24Java8_RunAll]
> Disable partition ranges in log messages about rebalance or PME
> ---
>
> Key: IGNITE-17865
> URL: https://issues.apache.org/jira/browse/IGNITE-17865
> Project: Ignite
> Issue Type: Sub-task
>Reporter: Ilya Shishkov
>Assignee: Ilya Shishkov
>Priority: Minor
> Labels: ise, newbie
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When you need to analyze cases of partitions inconsistency, full list of
> partitions is needed, but there is an optimization which replaces list of
> consecutive partitions with ranges. So, as you can see below, we can not find
> partition 2 by simple text search:
> {quote}2021-08-11 23:12:11.338 [WARN
> ][sys-#213|#213][org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl]
> Partitions have been scheduled for rebalancing due to outdated update
> counter [grp=group, readyTopVer=AffinityTopologyVersion [topVer=33,
> minorTopVer=0], topVer=AffinityTopologyVersion [topVer=33, minorTopVer=0],
> nodeId=8e05997a-4ff7-4fdb-9480-cfbcefa8bbf5,
> partsFull=[{color:#ff}*1-3*{color}, ...], partsHistorical=[]]
> {quote}
> {quote}2021-08-11 23:12:11.338 [WARN
> ][sys-#213|#213][org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture]
> Partitions weren't present in any history reservation: [[[grp=grp2
> part=[[{color:#ff}*1-3*{color}]]]
> {quote}
> We need to remove this optimization, because it can lead to mistakes in logs
> analysis.
> Partition ranges are formed in _GridToStringBuilder#compact_ method, which is
> used to log of partition lists (except one place with exception and tests).
> Below are such places (without usages in tests):
> # {_}GridClientPartitionTopology#resetOwners{_}:
> [L1311|https://github.com/apache/ignite/blob/8ec5d50896a2b5f2d008d0bb8f67f7f3d7cdf584/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/topology/GridClientPartitionTopology.java#L1311],
>
> [L1312|https://github.com/apache/ignite/blob/8ec5d50896a2b5f2d008d0bb8f67f7f3d7cdf584/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/topology/GridClientPartitionTopology.java#L1312]
> (WARN)
> # {_}GridDhtPartitionDemander#handleSupplyMessage{_}:
>
[jira] [Created] (IGNITE-18032) Do not implement AutoCloseable on classes which instances are not meant to be used in try-with-resources
Roman Puchkovskiy created IGNITE-18032: -- Summary: Do not implement AutoCloseable on classes which instances are not meant to be used in try-with-resources Key: IGNITE-18032 URL: https://issues.apache.org/jira/browse/IGNITE-18032 Project: Ignite Issue Type: Improvement Reporter: Roman Puchkovskiy Assignee: Roman Puchkovskiy Fix For: 3.0.0-beta2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18025) wrong version reference in libs.toml for jackson-core
[ https://issues.apache.org/jira/browse/IGNITE-18025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Lukyanov updated IGNITE-18025: Fix Version/s: 3.0.0-beta1 > wrong version reference in libs.toml for jackson-core > - > > Key: IGNITE-18025 > URL: https://issues.apache.org/jira/browse/IGNITE-18025 > Project: Ignite > Issue Type: Bug > Components: build >Reporter: Aleksandr >Assignee: Aleksandr >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta1 > > Time Spent: 10m > Remaining Estimate: 0h > > jackson-core is referenced with wrong version that causes broken published > pom for sql-engine module -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (IGNITE-17265) Support RAFT snapshot streaming for PageMemory storage
[ https://issues.apache.org/jira/browse/IGNITE-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626582#comment-17626582 ] Roman Puchkovskiy edited comment on IGNITE-17265 at 10/31/22 1:09 PM: -- Strictly speaking, not a full duplicate: this one turned out to be a part of another one. [~slukyanov] Should we reopen it to close it properly? was (Author: rpuch): Strictly speaking, not a full duplicate: this one turned out to be a part of another one. Should we reopen it to close it properly? > Support RAFT snapshot streaming for PageMemory storage > -- > > Key: IGNITE-17265 > URL: https://issues.apache.org/jira/browse/IGNITE-17265 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta1 > > > IGNITE-17254 API needs to be implemented for PageMemory storage -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17265) Support RAFT snapshot streaming for PageMemory storage
[ https://issues.apache.org/jira/browse/IGNITE-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626582#comment-17626582 ] Roman Puchkovskiy commented on IGNITE-17265: Strictly speaking, not a full duplicate: this one turned out to be a part of another one. Should we reopen it to close it properly? > Support RAFT snapshot streaming for PageMemory storage > -- > > Key: IGNITE-17265 > URL: https://issues.apache.org/jira/browse/IGNITE-17265 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta1 > > > IGNITE-17254 API needs to be implemented for PageMemory storage -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17813) Sql. Introduce sorted reducer for IndexScanNode
[ https://issues.apache.org/jira/browse/IGNITE-17813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626581#comment-17626581 ] Yury Gerzhedovich commented on IGNITE-17813: [~slukyanov], Calcite is pretty smart and your suggested WA will not work due to Calcite will rewrite the query. So, the ticket is still actually for 3.0.0-beta1 > Sql. Introduce sorted reducer for IndexScanNode > --- > > Key: IGNITE-17813 > URL: https://issues.apache.org/jira/browse/IGNITE-17813 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Konstantin Orlov >Assignee: Pavel Pereslegin >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > This task is derived from IGNITE-17655 to reduce its scope. > Currently, IgniteScanNode reads partition after partition which, obviously, > breaks the sorting order. We need to introduce wrapping cursor, which accepts > open cursors for every partition, and produce the sorted output. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-16760) Performance degradation of IgniteTables#tables after configuration changes
[
https://issues.apache.org/jira/browse/IGNITE-16760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626580#comment-17626580
]
Stanislav Lukyanov commented on IGNITE-16760:
-
The change was committed with an incorrect Jira ID (IGNITE-16670):
[https://github.com/apache/ignite-3/commit/bb268f4c00b240edfa637d8b0a6e397d17472fc6]
Nothing we can do about this now.
> Performance degradation of IgniteTables#tables after configuration changes
> --
>
> Key: IGNITE-16760
> URL: https://issues.apache.org/jira/browse/IGNITE-16760
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha4
>Reporter: Taras Ledkov
>Assignee: Denis Chudov
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Performance degradation of configuration changes:
> Steps to reproduce:
> 1. Start cluster with 3 nodes
> 2. Run in the loop
> {code}
> CREATE TABLE TEST(ID INTEGER PRIMARY KEY, V INTEGER)
> for (Table t : ign.tables().tables()) {
> ;
> }
> {code}
> On begin {{IgniteTables#tables}} takes ~ 0.7 sec.
> The time of the operation is grown.
> The time after ~100 iteration is about 20 sec.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-17265) Support RAFT snapshot streaming for PageMemory storage
[ https://issues.apache.org/jira/browse/IGNITE-17265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626573#comment-17626573 ] Stanislav Lukyanov commented on IGNITE-17265: - [~rpuch] Should this be "Duplicate" instead of "Resolved"? > Support RAFT snapshot streaming for PageMemory storage > -- > > Key: IGNITE-17265 > URL: https://issues.apache.org/jira/browse/IGNITE-17265 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta1 > > > IGNITE-17254 API needs to be implemented for PageMemory storage -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-15780) ITDistributedConfigurationPropertiesTest#testFallingBehindDistributedStorageValue is flaky
[ https://issues.apache.org/jira/browse/IGNITE-15780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Lukyanov updated IGNITE-15780: Fix Version/s: (was: 3.0.0-beta1) > ITDistributedConfigurationPropertiesTest#testFallingBehindDistributedStorageValue > is flaky > -- > > Key: IGNITE-15780 > URL: https://issues.apache.org/jira/browse/IGNITE-15780 > Project: Ignite > Issue Type: Bug >Reporter: Ivan Bessonov >Priority: Major > Labels: ignite-3 > Attachments: ITDistributedConfigurationPropertiesTest.log > > > Test fails locally after a few runs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17904) Fix flaky VolatilePageMemorySortedIndexStorageTest#testBoundsAndOrder
[ https://issues.apache.org/jira/browse/IGNITE-17904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Lukyanov updated IGNITE-17904: Fix Version/s: (was: 3.0.0-beta1) > Fix flaky VolatilePageMemorySortedIndexStorageTest#testBoundsAndOrder > - > > Key: IGNITE-17904 > URL: https://issues.apache.org/jira/browse/IGNITE-17904 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > > https://ci.ignite.apache.org/viewLog.html?buildId=6831695=ignite3_Test_RunUnitTests=true -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17208) Change storage engine names based on PageMemory
[ https://issues.apache.org/jira/browse/IGNITE-17208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Lukyanov updated IGNITE-17208: Fix Version/s: (was: 3.0.0-beta1) > Change storage engine names based on PageMemory > --- > > Key: IGNITE-17208 > URL: https://issues.apache.org/jira/browse/IGNITE-17208 > Project: Ignite > Issue Type: Improvement >Reporter: Kirill Tkalenko >Assignee: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > > PageMemory based storage engine (*pagememory*) has been split into two > *aimem* (in-memory) and *aipersist* (persistent), these names don't seem > convenient and memorable like *rocksdb*, we need something similar for them. > What needs to be changed in the code: > * > *org.apache.ignite.internal.storage.pagememory.VolatilePageMemoryStorageEngine#ENGINE_NAME* > * > *org.apache.ignite.internal.storage.pagememory.PersistentPageMemoryStorageEngine#ENGINE_NAME* > * > *org.apache.ignite.configuration.schemas.table.TablesConfigurationSchema#defaultDataStorage* > * *org.apache.ignite.example.storage.PersistentPageMemoryStorageExample* > * *org.apache.ignite.example.storage.VolatilePageMemoryStorageExample* > * Fallen tests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17280) ItComputeTest.executesColocatedByClassNameWithTupleKey failed on TC
[ https://issues.apache.org/jira/browse/IGNITE-17280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Lukyanov updated IGNITE-17280: Fix Version/s: (was: 3.0.0-beta1) > ItComputeTest.executesColocatedByClassNameWithTupleKey failed on TC > --- > > Key: IGNITE-17280 > URL: https://issues.apache.org/jira/browse/IGNITE-17280 > Project: Ignite > Issue Type: Bug > Components: compute >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > Attachments: _Integration_Tests_Module_Runner_3992.log.zip > > > [https://ci.ignite.apache.org/buildConfiguration/ignite3_Test_IntegrationTests_ModuleRunner/6655808] > > ava.lang.AssertionError > java.lang.AssertionError: Raft groups are still running > [b839ce7f-370c-4553-882e-34b471029c9c_part_0, > b839ce7f-370c-4553-882e-34b471029c9c_part_19, > b839ce7f-370c-4553-882e-34b471029c9c_part_8, > b839ce7f-370c-4553-882e-34b471029c9c_part_11, > b839ce7f-370c-4553-882e-34b471029c9c_part_4, > b839ce7f-370c-4553-882e-34b471029c9c_part_15, > b839ce7f-370c-4553-882e-34b471029c9c_part_13, > b839ce7f-370c-4553-882e-34b471029c9c_part_3, > b839ce7f-370c-4553-882e-34b471029c9c_part_14] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-16939) ClientComputeTest.testClientSendsComputeJobToTargetNodeWhenDirectConnectionExists is flaky
[
https://issues.apache.org/jira/browse/IGNITE-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stanislav Lukyanov updated IGNITE-16939:
Fix Version/s: (was: 3.0.0-beta1)
> ClientComputeTest.testClientSendsComputeJobToTargetNodeWhenDirectConnectionExists
> is flaky
> --
>
> Key: IGNITE-16939
> URL: https://issues.apache.org/jira/browse/IGNITE-16939
> Project: Ignite
> Issue Type: Bug
> Components: clients
>Reporter: Roman Puchkovskiy
>Assignee: Pavel Tupitsyn
>Priority: Major
> Labels: ignite-3
>
> See
> [https://ci.ignite.apache.org/viewLog.html?buildId=6560290=ignite3_Test_RunUnitTests]
> {code:java}
> org.opentest4j.AssertionFailedError
> org.opentest4j.AssertionFailedError: expected: but was:
> at
> org.apache.ignite.client.ClientComputeTest.testClientSendsComputeJobToTargetNodeWhenDirectConnectionExists(ClientComputeTest.java:68)
> {code}
> The branch is main with one commit which disables one test, so it's
> equivalent to main branch in everything that concerns the failed test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17551) Add the ability to work with the entire BinaryTuple ByteBuffer
[ https://issues.apache.org/jira/browse/IGNITE-17551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Lukyanov updated IGNITE-17551: Fix Version/s: (was: 3.0.0-beta1) > Add the ability to work with the entire BinaryTuple ByteBuffer > -- > > Key: IGNITE-17551 > URL: https://issues.apache.org/jira/browse/IGNITE-17551 > Project: Ignite > Issue Type: Improvement >Reporter: Kirill Tkalenko >Assignee: Kirill Tkalenko >Priority: Major > Labels: iep-92, ignite-3 > > When implementing PageMemory based indexes, it was discovered that a > *BinaryTuple* does not have a method to get its *ByteBuffer* so that it can > be saved (eg to a file). > Also, we need to create a *BinaryTuple* after reading the *ByteBuffer* (eg > from a file). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17006) Add protection against arbitrary page memory links in LinkRowId
[ https://issues.apache.org/jira/browse/IGNITE-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Lukyanov updated IGNITE-17006: Fix Version/s: (was: 3.0.0-beta1) > Add protection against arbitrary page memory links in LinkRowId > --- > > Key: IGNITE-17006 > URL: https://issues.apache.org/jira/browse/IGNITE-17006 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Roman Puchkovskiy >Assignee: Roman Puchkovskiy >Priority: Major > Labels: ignite-3 > > It's theoretically possible to pass an arbitrary page memory link (via > LinkRowId) which might cause troubles: > # If pageId exceeds page memory limit, the JVM might crash > # If the page with this pageId was never initialized, an attempt to read > will fail with an internal assertion (because lock state will be 0) > A possibility for item ID to be invalid is already handled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-16642) [Native Persistence 3.0] Support indexes for B+Tree-based storage
[ https://issues.apache.org/jira/browse/IGNITE-16642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Lukyanov updated IGNITE-16642: Fix Version/s: (was: 3.0.0-beta1) > [Native Persistence 3.0] Support indexes for B+Tree-based storage > - > > Key: IGNITE-16642 > URL: https://issues.apache.org/jira/browse/IGNITE-16642 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Assignee: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > > Need to support indexes for B+Tree-based storage . -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-16356) .NET: Thin 3.0: Customizable user type mapping
[ https://issues.apache.org/jira/browse/IGNITE-16356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626567#comment-17626567 ] Pavel Tupitsyn commented on IGNITE-16356: - Merged to main: 14b5464ed85d8eb86c90641371569c8fbcdc919c > .NET: Thin 3.0: Customizable user type mapping > -- > > Key: IGNITE-16356 > URL: https://issues.apache.org/jira/browse/IGNITE-16356 > Project: Ignite > Issue Type: Improvement > Components: platforms, thin client >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: .NET, ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 10m > Remaining Estimate: 0h > > IGNITE-16227 introduced basic POCO mapping when table column names match user > type field names exactly. Provide APIs to customize the mapping. Some > possibilities are: > * Attributes on properties and fields to alter column name > * Pluggable mapper or serializer interface that gives full control to the > user (we require schema order, so that may be complicated) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17199) Improve the usability of the abstract configuration interface
[
https://issues.apache.org/jira/browse/IGNITE-17199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stanislav Lukyanov updated IGNITE-17199:
Fix Version/s: (was: 3.0.0-beta1)
> Improve the usability of the abstract configuration interface
> -
>
> Key: IGNITE-17199
> URL: https://issues.apache.org/jira/browse/IGNITE-17199
> Project: Ignite
> Issue Type: Improvement
>Reporter: Kirill Tkalenko
>Priority: Major
> Labels: iep-55, ignite-3
>
> *Problem*
> Consider an example of generating configuration interfaces (**Configuration*)
> for an abstract configuration.
> Configuration schemas:
> {code:java}
> @AbstractConfiguration
> public class BaseConfigurationSchema {
> @Value
> public int size;
> }
> @Config
> public class VolatileConfigurationSchema extends BaseConfigurationSchema {
> @Value
> public double evictionThreshold;
> }
> {code}
> Configuration interfaces:
> {code:java}
> public interface BaseConfiguration BaseChange> extends ConfigurationTree {
> ConfigurationValue size();
> }
> public interface VolatileConfiguration extends
> BaseConfiguration {
> ConfigurationValue size();
> }
> {code}
> This implementation allows us to work with the inheritors of the abstract
> configuration as with a regular configuration (as if
> *VolatileConfigurationSchema* did not extend *BaseConfigurationSchema*), but
> when working with the abstract configuration itself, it creates
> inconvenience.
> For example, to get a view of the abstract configuration, we will need to
> write the following code:
> {code:java}
> BaseConfiguration baseConfig0 = ...;
> BaseConfiguration baseConfig1 = ...;
>
> BaseView baseView0 = (BasePageMemoryDataRegionView) baseConfig0.value();
> BaseView baseView1 = baseConfig1.value();
> {code}
> Which is not convenient and I would like us to be able to work in the same
> way as with the *VolatileConfiguration*.
> *Possible implementations*
> * Simplest is to leave it as is;
> * Creates an additional configuration interface that will be similar to
> *BaseConfiguration*, for example *BaseConfigurationTree*, but it will be
> extended by *BaseConfiguration* and all its inheritors like
> *VolatileConfiguration*, then there may be confusion about whether to use
> *BaseConfiguration* or *BaseConfigurationTree* in the end, so we need to
> decide how to create a name for such an interface;
> ** *BaseConfigurationTree*;
> ** *AbstractBaseConfigurationTree*;
> ** other.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IGNITE-18021) Creating an API for full rebalance without losing user data in MvPartitionStorage on receiver
[ https://issues.apache.org/jira/browse/IGNITE-18021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko reassigned IGNITE-18021: Assignee: Kirill Tkalenko > Creating an API for full rebalance without losing user data in > MvPartitionStorage on receiver > - > > Key: IGNITE-18021 > URL: https://issues.apache.org/jira/browse/IGNITE-18021 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Assignee: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > We need to create an API for a complete rebalance without losing user data in > class *MvPartitionStorage* and *MvTableStorage* on receiver. > And also to make a test implementation and basic tests. > The main idea is described in IGNITE-17990. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18031) Implement background GC process for MV partition storages
Ivan Bessonov created IGNITE-18031:
--
Summary: Implement background GC process for MV partition storages
Key: IGNITE-18031
URL: https://issues.apache.org/jira/browse/IGNITE-18031
Project: Ignite
Issue Type: Improvement
Reporter: Ivan Bessonov
Please refer to Epic for more details. Here I only describe thoughts about
background GC process.
h3. General thoughts
Basic algorithm is the following:
{code:java}
RowId rowId = null;
while (!partitionStopped) {
if (rowId == null) rowId = minRowId(partId);
rowId = partition.closestRowId(rowId);
partition.gc(rowId);
rowId = rodId.increment();
}{code}
Here I ignore a lot of technicalities and only show the main loop. We _could_
implement it literally the same way, but that would be a very bad decision. Why:
* it fully utilizes a single thread. That would mean a thread per each
partition, which is unacceptable
* it constantly reads the entire partition over and over again. I don't like
that, it's a waste of resources, we'd rather prioritize reading data that user
needs. There should be pauses between full runs, at least
To address all of these issues, the job should be split into small batches
(like we do in many other places) and every new batch should be put into a pool
only when the current one is completed. This allows multiple partitions to
utilize the same pool without a possibility of starvation.
The part with the _pause_ should probably involve a scheduled pool. That seems
enough for the first implementation.
Other ideas and approaches will be discussed in other issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-18026) Rework HLC propagation in parts related to Raft
[ https://issues.apache.org/jira/browse/IGNITE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denis Chudov updated IGNITE-18026: -- Description: *Motivation:* As for now, HLC propagates via replica messages and Raft appendEntries messages. The latter required changes in jraft code which was done under IGNITE-17221 . Replica safe time (see IGNITE-17263 ) also uses HLC timestamps coming from leader, but these timestamps should be applied only on commiting index, so the whole mechanism consists of two steps: creating safe time candidate on processing of appendEntries request, and appying it on index commit. All of this have grown up into significant amount of invasive code in jraft which is not related to Raft protocol itself. It can be reworked by moving the logic related to HLC and safe time propagation to ignite state machine listener working on top of Raft. *Definition of done:* There is no code related to HLC and safe time left in ignite-raft module. *Implementation notes:* Raft write commands can be enriched with HybridTimestamps. Idle HLC adjustment could be done using SafeTimeSyncCommand (it should be renamed in this case). Safe time candidates will no longer be needed. was: *Motivation:* As for now, HLC propagates via replica messages and Raft appendEntries messages. The latter required changes in jraft code which was done under IGNITE-17221 . Replica safe time (see IGNITE-17263 ) also uses HLC timestamps coming from leader, but these timestamps should be applied only on commiting index, so the whole mechanism consists of two steps: creating safe time candidate on processing of appendEntries request, and appying it on index commit. All of this have grown up into significant amount of invasive code in jraft which is not related to Raft protocol itself. It can be reworked by moving the logic replated to HLC and safe time propagation to ignite state machine listener working on top of Raft. *Definition of done:* There is no code related to HLC and safe time left in ignite-raft module. *Implementation notes:* Raft write commands can be enriched with HybridTimestamps. Idle HLC adjustment could be done using SafeTimeSyncCommand (it should be renamed in this case). Safe time candidates will no longer be needed. > Rework HLC propagation in parts related to Raft > --- > > Key: IGNITE-18026 > URL: https://issues.apache.org/jira/browse/IGNITE-18026 > Project: Ignite > Issue Type: Improvement >Reporter: Denis Chudov >Priority: Major > Labels: ignite-3 > > *Motivation:* > As for now, HLC propagates via replica messages and Raft appendEntries > messages. The latter required changes in jraft code which was done under > IGNITE-17221 . Replica safe time (see IGNITE-17263 ) also uses HLC timestamps > coming from leader, but these timestamps should be applied only on commiting > index, so the whole mechanism consists of two steps: creating safe time > candidate on processing of appendEntries request, and appying it on index > commit. > All of this have grown up into significant amount of invasive code in jraft > which is not related to Raft protocol itself. It can be reworked by moving > the logic related to HLC and safe time propagation to ignite state machine > listener working on top of Raft. > *Definition of done:* > There is no code related to HLC and safe time left in ignite-raft module. > *Implementation notes:* > Raft write commands can be enriched with HybridTimestamps. Idle HLC > adjustment could be done using SafeTimeSyncCommand (it should be renamed in > this case). Safe time candidates will no longer be needed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18026) Rework HLC propagation in parts related to Raft
[ https://issues.apache.org/jira/browse/IGNITE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denis Chudov updated IGNITE-18026: -- Description: *Motivation:* As for now, HLC propagates via replica messages and Raft appendEntries messages. The latter required changes in jraft code which was done under IGNITE-17221 . Replica safe time (see IGNITE-17263 ) also uses HLC timestamps coming from leader, but these timestamps should be applied only on commiting index, so the whole mechanism consists of two steps: creating safe time candidate on processing of appendEntries request, and appying it on index commit. All of this have grown up into significant amount of invasive code in jraft which is not related to Raft protocol itself. It can be reworked by moving the logic replated to HLC and safe time propagation to ignite state machine listener working on top of Raft. *Definition of done:* There is no code related to HLC and safe time left in ignite-raft module. *Implementation notes:* Raft write commands can be enriched with HybridTimestamps. Idle HLC adjustment could be done using SafeTimeSyncCommand (it should be renamed in this case). Safe time candidates will no longer be needed. was: *Motivation:* As for now, HLC propagates via replica messages and Raft appendEntries messages. The latter required changes in jraft code which was done under IGNITE-17221 . Replica safe time (see IGNITE-17263 ) also uses HLC timestamps coming from leader, but these timestamps should be applied only on commiting index, so the whole mechanism consists of two steps: creating safe time candidate on processing of appendEntries request, and appying it on index commit. All of this have grown up into significant amount of invasive code in jraft which is not related to Raft protocol itself. It can be reworked by moving the loginc replated to HLC and safe time propagation to ignite state machine listener working on top of Raft. *Definition of done:* There is no code related to HLC and safe time left in ignite-raft module. *Implementation notes:* Raft write commands can be enriched with HybridTimestamps. Idle HLC adjustment could be done using SafeTimeSyncCommand (it should be renamed in this case). Safe time candidates will no longer be needed. > Rework HLC propagation in parts related to Raft > --- > > Key: IGNITE-18026 > URL: https://issues.apache.org/jira/browse/IGNITE-18026 > Project: Ignite > Issue Type: Improvement >Reporter: Denis Chudov >Priority: Major > Labels: ignite-3 > > *Motivation:* > As for now, HLC propagates via replica messages and Raft appendEntries > messages. The latter required changes in jraft code which was done under > IGNITE-17221 . Replica safe time (see IGNITE-17263 ) also uses HLC timestamps > coming from leader, but these timestamps should be applied only on commiting > index, so the whole mechanism consists of two steps: creating safe time > candidate on processing of appendEntries request, and appying it on index > commit. > All of this have grown up into significant amount of invasive code in jraft > which is not related to Raft protocol itself. It can be reworked by moving > the logic replated to HLC and safe time propagation to ignite state machine > listener working on top of Raft. > *Definition of done:* > There is no code related to HLC and safe time left in ignite-raft module. > *Implementation notes:* > Raft write commands can be enriched with HybridTimestamps. Idle HLC > adjustment could be done using SafeTimeSyncCommand (it should be renamed in > this case). Safe time candidates will no longer be needed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-16356) .NET: Thin 3.0: Customizable user type mapping
[ https://issues.apache.org/jira/browse/IGNITE-16356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626559#comment-17626559 ] Igor Sapego commented on IGNITE-16356: -- [~ptupitsyn] looks good to me. > .NET: Thin 3.0: Customizable user type mapping > -- > > Key: IGNITE-16356 > URL: https://issues.apache.org/jira/browse/IGNITE-16356 > Project: Ignite > Issue Type: Improvement > Components: platforms, thin client >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: .NET, ignite-3 > Fix For: 3.0.0-beta2 > > > IGNITE-16227 introduced basic POCO mapping when table column names match user > type field names exactly. Provide APIs to customize the mapping. Some > possibilities are: > * Attributes on properties and fields to alter column name > * Pluggable mapper or serializer interface that gives full control to the > user (we require schema order, so that may be complicated) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18030) Integration of the new full rebalance API with IncomingSnapshotCopier
[ https://issues.apache.org/jira/browse/IGNITE-18030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko updated IGNITE-18030: - Description: We need to integrate the new full rebalance API without losing user data with *IncomingSnapshotCopier*, additional tests may be needed. The main idea is described in IGNITE-17990. API should be in ticket IGNITE-18021 and IGNITE-18022. was: We need to integrate the new full rebalance API without losing user data with *IncomingSnapshotCopier*, additional tests may be needed. The main idea is described in IGNITE-17990. > Integration of the new full rebalance API with IncomingSnapshotCopier > - > > Key: IGNITE-18030 > URL: https://issues.apache.org/jira/browse/IGNITE-18030 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > We need to integrate the new full rebalance API without losing user data with > *IncomingSnapshotCopier*, additional tests may be needed. > The main idea is described in IGNITE-17990. > API should be in ticket IGNITE-18021 and IGNITE-18022. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18030) Integration of the new full rebalance API with IncomingSnapshotCopier
Kirill Tkalenko created IGNITE-18030: Summary: Integration of the new full rebalance API with IncomingSnapshotCopier Key: IGNITE-18030 URL: https://issues.apache.org/jira/browse/IGNITE-18030 Project: Ignite Issue Type: Task Reporter: Kirill Tkalenko Fix For: 3.0.0-beta2 We need to integrate the new full rebalance API without losing user data with *IncomingSnapshotCopier*, additional tests may be needed. The main idea is described in IGNITE-17990. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18029) Implementation of a full rebalance without loss of user data for PersistentPageMemoryMvPartitionStorage on receiver
[ https://issues.apache.org/jira/browse/IGNITE-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko updated IGNITE-18029: - Description: We need to implement a full rebalance without losing user data for *PersistentPageMemoryMvPartitionStorage* and *PersistentPageMemoryTableStorage* on the recipient, do not forget about the tests. The main idea is described in IGNITE-17990. API should be in ticket IGNITE-18021. > Implementation of a full rebalance without loss of user data for > PersistentPageMemoryMvPartitionStorage on receiver > --- > > Key: IGNITE-18029 > URL: https://issues.apache.org/jira/browse/IGNITE-18029 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > We need to implement a full rebalance without losing user data for > *PersistentPageMemoryMvPartitionStorage* and > *PersistentPageMemoryTableStorage* on the recipient, do not forget about the > tests. > The main idea is described in IGNITE-17990. > API should be in ticket IGNITE-18021. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18029) Implementation of a full rebalance without loss of user data for PersistentPageMemoryMvPartitionStorage on receiver
Kirill Tkalenko created IGNITE-18029: Summary: Implementation of a full rebalance without loss of user data for PersistentPageMemoryMvPartitionStorage on receiver Key: IGNITE-18029 URL: https://issues.apache.org/jira/browse/IGNITE-18029 Project: Ignite Issue Type: Task Reporter: Kirill Tkalenko Fix For: 3.0.0-beta2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18028) Implementation of a full rebalance without loss of user data for VolatilePageMemoryMvPartitionStorage on receiver
Kirill Tkalenko created IGNITE-18028: Summary: Implementation of a full rebalance without loss of user data for VolatilePageMemoryMvPartitionStorage on receiver Key: IGNITE-18028 URL: https://issues.apache.org/jira/browse/IGNITE-18028 Project: Ignite Issue Type: Task Reporter: Kirill Tkalenko Fix For: 3.0.0-beta2 We need to implement a full rebalance without losing user data for *VolatilePageMemoryMvPartitionStorage* and *VolatilePageMemoryTableStorage* on the recipient, do not forget about the tests. The main idea is described in IGNITE-17990. API should be in ticket IGNITE-18021. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18027) Implementation of a full rebalance without loss of user data for RocksDbMvPartitionStorageon on receiver
[ https://issues.apache.org/jira/browse/IGNITE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko updated IGNITE-18027: - Summary: Implementation of a full rebalance without loss of user data for RocksDbMvPartitionStorageon on receiver (was: Implementation of a full rebalance without loss of user data for RocksDbMvPartitionStorageon receiver) > Implementation of a full rebalance without loss of user data for > RocksDbMvPartitionStorageon on receiver > > > Key: IGNITE-18027 > URL: https://issues.apache.org/jira/browse/IGNITE-18027 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > We need to implement a full rebalance without losing user data for > *RocksDbMvPartitionStorage* and *RocksDbTableStorage* on the recipient, do > not forget about the tests. > The main idea is described in IGNITE-17990. > API should be in ticket IGNITE-18021. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18026) Rework HLC propagation in parts related to Raft
[ https://issues.apache.org/jira/browse/IGNITE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denis Chudov updated IGNITE-18026: -- Issue Type: Improvement (was: Bug) > Rework HLC propagation in parts related to Raft > --- > > Key: IGNITE-18026 > URL: https://issues.apache.org/jira/browse/IGNITE-18026 > Project: Ignite > Issue Type: Improvement >Reporter: Denis Chudov >Priority: Major > Labels: ignite-3 > > *Motivation:* > As for now, HLC propagates via replica messages and Raft appendEntries > messages. The latter required changes in jraft code which was done under > IGNITE-17221 . Replica safe time (see IGNITE-17263 ) also uses HLC timestamps > coming from leader, but these timestamps should be applied only on commiting > index, so the whole mechanism consists of two steps: creating safe time > candidate on processing of appendEntries request, and appying it on index > commit. > All of this have grown up into significant amount of invasive code in jraft > which is not related to Raft protocol itself. It can be reworked by moving > the loginc replated to HLC and safe time propagation to ignite state machine > listener working on top of Raft. > *Definition of done:* > There is no code related to HLC and safe time left in ignite-raft module. > *Implementation notes:* > Raft write commands can be enriched with HybridTimestamps. Idle HLC > adjustment could be done using SafeTimeSyncCommand (it should be renamed in > this case). Safe time candidates will no longer be needed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18027) Implementation of a full rebalance without loss of user data for RocksDbMvPartitionStorageon receiver
Kirill Tkalenko created IGNITE-18027: Summary: Implementation of a full rebalance without loss of user data for RocksDbMvPartitionStorageon receiver Key: IGNITE-18027 URL: https://issues.apache.org/jira/browse/IGNITE-18027 Project: Ignite Issue Type: Task Reporter: Kirill Tkalenko Fix For: 3.0.0-beta2 We need to implement a full rebalance without losing user data for *RocksDbMvPartitionStorage* and *RocksDbTableStorage* on the recipient, do not forget about the tests. The main idea is described in IGNITE-17990. API should be in ticket IGNITE-18021. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18026) Rework HLC propagation in parts related to Raft
[ https://issues.apache.org/jira/browse/IGNITE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denis Chudov updated IGNITE-18026: -- Description: *Motivation:* As for now, HLC propagates via replica messages and Raft appendEntries messages. The latter required changes in jraft code which was done under IGNITE-17221 . Replica safe time (see IGNITE-17263 ) also uses HLC timestamps coming from leader, but these timestamps should be applied only on commiting index, so the whole mechanism consists of two steps: creating safe time candidate on processing of appendEntries request, and appying it on index commit. All of this have grown up into significant amount of invasive code in jraft which is not related to Raft protocol itself. It can be reworked by moving the loginc replated to HLC and safe time propagation to ignite state machine listener working on top of Raft. *Definition of done:* There is no code related to HLC and safe time left in ignite-raft module. *Implementation notes:* Raft write commands can be enriched with HybridTimestamps. Idle HLC adjustment could be done using SafeTimeSyncCommand (it should be renamed in this case). Safe time candidates will no longer be needed. was: Motivation: As for now, HLC propagates via replica messages and Raft appendEntries messages. The latter required changes in jraft code which was done under IGNITE-17221 . Replica safe time (see IGNITE-17263 ) also uses HLC timestamps coming from leader, but these timestamps should be applied only on commiting index, so the whole mechanism consists of two steps: creating safe time candidate on processing of appendEntries request, and appying it on index commit. All of this have grown up into significant amount of invasive code in jraft which is not related to Raft protocol itself. It can be reworked by moving the loginc replated to HLC and safe time propagation to ignite state machine listener working on top of Raft. Definition of done: There is no code related to HLC and safe time left in ignite-raft module. Implementation notes: Raft write commands can be enriched with HybridTimestamps. Idle HLC adjustment could be done using SafeTimeSyncCommand (it should be renamed in this case). Safe time candidates will no longer be needed. > Rework HLC propagation in parts related to Raft > --- > > Key: IGNITE-18026 > URL: https://issues.apache.org/jira/browse/IGNITE-18026 > Project: Ignite > Issue Type: Bug >Reporter: Denis Chudov >Priority: Major > Labels: ignite-3 > > *Motivation:* > As for now, HLC propagates via replica messages and Raft appendEntries > messages. The latter required changes in jraft code which was done under > IGNITE-17221 . Replica safe time (see IGNITE-17263 ) also uses HLC timestamps > coming from leader, but these timestamps should be applied only on commiting > index, so the whole mechanism consists of two steps: creating safe time > candidate on processing of appendEntries request, and appying it on index > commit. > All of this have grown up into significant amount of invasive code in jraft > which is not related to Raft protocol itself. It can be reworked by moving > the loginc replated to HLC and safe time propagation to ignite state machine > listener working on top of Raft. > *Definition of done:* > There is no code related to HLC and safe time left in ignite-raft module. > *Implementation notes:* > Raft write commands can be enriched with HybridTimestamps. Idle HLC > adjustment could be done using SafeTimeSyncCommand (it should be renamed in > this case). Safe time candidates will no longer be needed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18024) Implementation of a full rebalance without loss of user data for TxStateRocksDbStorage on receiver
[ https://issues.apache.org/jira/browse/IGNITE-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko updated IGNITE-18024: - Description: We need to implement a full rebalance without losing user data for *TxStateRocksDbStorage* and *TxStateRocksDbTableStorage* on the recipient, do not forget about the tests. The main idea is described in IGNITE-17990. API should be in ticket IGNITE-18022. was: We need to implement a full rebalance without losing user data for *TxStateRocksDbStorage* and *TxStateRocksDbTableStorage* on the recipient, do not forget about the tests. The main idea is described in IGNITE-17990. Api should be in ticket IGNITE-18022. > Implementation of a full rebalance without loss of user data for > TxStateRocksDbStorage on receiver > -- > > Key: IGNITE-18024 > URL: https://issues.apache.org/jira/browse/IGNITE-18024 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > We need to implement a full rebalance without losing user data for > *TxStateRocksDbStorage* and *TxStateRocksDbTableStorage* on the recipient, do > not forget about the tests. > The main idea is described in IGNITE-17990. > API should be in ticket IGNITE-18022. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18026) Rework HLC propagation in parts related to Raft
Denis Chudov created IGNITE-18026: - Summary: Rework HLC propagation in parts related to Raft Key: IGNITE-18026 URL: https://issues.apache.org/jira/browse/IGNITE-18026 Project: Ignite Issue Type: Bug Reporter: Denis Chudov Motivation: As for now, HLC propagates via replica messages and Raft appendEntries messages. The latter required changes in jraft code which was done under IGNITE-17221 . Replica safe time (see IGNITE-17263 ) also uses HLC timestamps coming from leader, but these timestamps should be applied only on commiting index, so the whole mechanism consists of two steps: creating safe time candidate on processing of appendEntries request, and appying it on index commit. All of this have grown up into significant amount of invasive code in jraft which is not related to Raft protocol itself. It can be reworked by moving the loginc replated to HLC and safe time propagation to ignite state machine listener working on top of Raft. Definition of done: There is no code related to HLC and safe time left in ignite-raft module. Implementation notes: Raft write commands can be enriched with HybridTimestamps. Idle HLC adjustment could be done using SafeTimeSyncCommand (it should be renamed in this case). Safe time candidates will no longer be needed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18024) Implementation of a full rebalance without loss of user data for TxStateRocksDbStorage on receiver
[ https://issues.apache.org/jira/browse/IGNITE-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko updated IGNITE-18024: - Summary: Implementation of a full rebalance without loss of user data for TxStateRocksDbStorage on receiver (was: Implementation of a full rebalance without loss of user data for TxStateRocksDbStorage on the receiver) > Implementation of a full rebalance without loss of user data for > TxStateRocksDbStorage on receiver > -- > > Key: IGNITE-18024 > URL: https://issues.apache.org/jira/browse/IGNITE-18024 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > We need to implement a full rebalance without losing user data for > *TxStateRocksDbStorage* and *TxStateRocksDbTableStorage* on the recipient, do > not forget about the tests. > The main idea is described in IGNITE-17990. > Api should be in ticket IGNITE-18022. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18024) Implementation of a full rebalance without loss of user data for TxStateRocksDbStorage on the receiver
[ https://issues.apache.org/jira/browse/IGNITE-18024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko updated IGNITE-18024: - Description: We need to implement a full rebalance without losing user data for *TxStateRocksDbStorage* and *TxStateRocksDbTableStorage* on the recipient, do not forget about the tests. The main idea is described in IGNITE-17990. Api should be in ticket IGNITE-18022. was: We need to implement a full rebalance without losing user data for *TxStateRocksDbStorage* and *TxStateRocksDbTableStorage* on the recipient, do not forget about the tests. The main idea is described in IGNITE-17990. > Implementation of a full rebalance without loss of user data for > TxStateRocksDbStorage on the receiver > -- > > Key: IGNITE-18024 > URL: https://issues.apache.org/jira/browse/IGNITE-18024 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > We need to implement a full rebalance without losing user data for > *TxStateRocksDbStorage* and *TxStateRocksDbTableStorage* on the recipient, do > not forget about the tests. > The main idea is described in IGNITE-17990. > Api should be in ticket IGNITE-18022. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18025) wrong version reference in libs.toml for jackson-core
Aleksandr created IGNITE-18025: -- Summary: wrong version reference in libs.toml for jackson-core Key: IGNITE-18025 URL: https://issues.apache.org/jira/browse/IGNITE-18025 Project: Ignite Issue Type: Bug Components: build Reporter: Aleksandr Assignee: Aleksandr jackson-core is referenced with wrong version that causes broken published pom for sql-engine module -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18016) Fix javadocs for public IgniteTables
[ https://issues.apache.org/jira/browse/IGNITE-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yury Gerzhedovich updated IGNITE-18016: --- Fix Version/s: 3.0.0-beta1 > Fix javadocs for public IgniteTables > - > > Key: IGNITE-18016 > URL: https://issues.apache.org/jira/browse/IGNITE-18016 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Yury Gerzhedovich >Assignee: Yury Gerzhedovich >Priority: Major > Fix For: 3.0.0-beta1 > > > We mention using SQL schemas in javadocs at > org.apache.ignite.table.manager.IgniteTables. Due to schemas are not > supported now need to rewrite these javadocs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18024) Implementation of a full rebalance without loss of user data for TxStateRocksDbStorage on the receiver
Kirill Tkalenko created IGNITE-18024: Summary: Implementation of a full rebalance without loss of user data for TxStateRocksDbStorage on the receiver Key: IGNITE-18024 URL: https://issues.apache.org/jira/browse/IGNITE-18024 Project: Ignite Issue Type: Task Reporter: Kirill Tkalenko Fix For: 3.0.0-beta2 We need to implement a full rebalance without losing user data for *TxStateRocksDbStorage* and *TxStateRocksDbTableStorage* on the recipient, do not forget about the tests. The main idea is described in IGNITE-17990. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18022) Creating an API for full rebalance without losing user data in TxStateStorage on receiver
[ https://issues.apache.org/jira/browse/IGNITE-18022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko updated IGNITE-18022: - Summary: Creating an API for full rebalance without losing user data in TxStateStorage on receiver (was: Creating an API for full rebalance without losing user data in TxStateStorage) > Creating an API for full rebalance without losing user data in TxStateStorage > on receiver > - > > Key: IGNITE-18022 > URL: https://issues.apache.org/jira/browse/IGNITE-18022 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > We need to create an API for a complete rebalance without losing user data in > class *TxStateStorage* and *TxStateTableStorage* on receiver. > And also to make a test implementation and basic tests. > The main idea is described in IGNITE-17990. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18022) Creating an API for full rebalance without losing user data in TxStateStorage
[ https://issues.apache.org/jira/browse/IGNITE-18022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko updated IGNITE-18022: - Description: We need to create an API for a complete rebalance without losing user data in class *TxStateStorage* and *TxStateTableStorage* on receiver. And also to make a test implementation and basic tests. The main idea is described in IGNITE-17990. was: We need to create an API for a complete rebalance without losing user data in class *TxStateStorage* and *TxStateTableStorage*. And also to make a test implementation and basic tests. The main idea is described in IGNITE-17990. > Creating an API for full rebalance without losing user data in TxStateStorage > - > > Key: IGNITE-18022 > URL: https://issues.apache.org/jira/browse/IGNITE-18022 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > We need to create an API for a complete rebalance without losing user data in > class *TxStateStorage* and *TxStateTableStorage* on receiver. > And also to make a test implementation and basic tests. > The main idea is described in IGNITE-17990. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18021) Creating an API for full rebalance without losing user data in MvPartitionStorage on receiver
[ https://issues.apache.org/jira/browse/IGNITE-18021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko updated IGNITE-18021: - Description: We need to create an API for a complete rebalance without losing user data in class *MvPartitionStorage* and *MvTableStorage* on receiver. And also to make a test implementation and basic tests. The main idea is described in IGNITE-17990. was: We need to create an API for a complete rebalance without losing user data in class *MvPartitionStorage* and *MvTableStorage*. And also to make a test implementation and basic tests. The main idea is described in IGNITE-17990. > Creating an API for full rebalance without losing user data in > MvPartitionStorage on receiver > - > > Key: IGNITE-18021 > URL: https://issues.apache.org/jira/browse/IGNITE-18021 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > We need to create an API for a complete rebalance without losing user data in > class *MvPartitionStorage* and *MvTableStorage* on receiver. > And also to make a test implementation and basic tests. > The main idea is described in IGNITE-17990. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18021) Creating an API for full rebalance without losing user data in MvPartitionStorage on receiver
[ https://issues.apache.org/jira/browse/IGNITE-18021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko updated IGNITE-18021: - Summary: Creating an API for full rebalance without losing user data in MvPartitionStorage on receiver (was: Creating an API for full rebalance without losing user data in MvPartitionStorage) > Creating an API for full rebalance without losing user data in > MvPartitionStorage on receiver > - > > Key: IGNITE-18021 > URL: https://issues.apache.org/jira/browse/IGNITE-18021 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > We need to create an API for a complete rebalance without losing user data in > class *MvPartitionStorage* and *MvTableStorage*. > And also to make a test implementation and basic tests. > The main idea is described in IGNITE-17990. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17975) Unmute ItIgniteNodeRestartTest and ItIgniteInMemoryNodeRestartTest
[ https://issues.apache.org/jira/browse/IGNITE-17975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-17975: - Reviewer: Alexander Lapin > Unmute ItIgniteNodeRestartTest and ItIgniteInMemoryNodeRestartTest > -- > > Key: IGNITE-17975 > URL: https://issues.apache.org/jira/browse/IGNITE-17975 > Project: Ignite > Issue Type: Task >Reporter: Mirza Aliev >Assignee: Mirza Aliev >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 20m > Remaining Estimate: 0h > > h4. *Motivation* > After implementation of IGNITE-15081 some test were muted because of new > managers that are supposed to be initiated in test nodes. We need to enable > ItIgniteNodeRestartTest and ItIgniteInMemoryNodeRestartTest. > h4. *Definition of Done* > All tests that are not muted with different cause must pass. > *Implementation notes* > We need to add ReplicaManager to the node flow in test, also we need to add > dummy index restore after restart of node -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17975) Unmute ItIgniteNodeRestartTest and ItIgniteInMemoryNodeRestartTest
[ https://issues.apache.org/jira/browse/IGNITE-17975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-17975: - Fix Version/s: 3.0.0-beta2 > Unmute ItIgniteNodeRestartTest and ItIgniteInMemoryNodeRestartTest > -- > > Key: IGNITE-17975 > URL: https://issues.apache.org/jira/browse/IGNITE-17975 > Project: Ignite > Issue Type: Task >Reporter: Mirza Aliev >Assignee: Mirza Aliev >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 20m > Remaining Estimate: 0h > > h4. *Motivation* > After implementation of IGNITE-15081 some test were muted because of new > managers that are supposed to be initiated in test nodes. We need to enable > ItIgniteNodeRestartTest and ItIgniteInMemoryNodeRestartTest. > h4. *Definition of Done* > All tests that are not muted with different cause must pass. > *Implementation notes* > We need to add ReplicaManager to the node flow in test, also we need to add > dummy index restore after restart of node -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18022) Creating an API for full rebalance without losing user data in TxStateStorage
[ https://issues.apache.org/jira/browse/IGNITE-18022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko updated IGNITE-18022: - Description: We need to create an API for a complete rebalance without losing user data in class *TxStateStorage* and *TxStateTableStorage*. And also to make a test implementation and basic tests. The main idea is described in IGNITE-17990. was: We need to create an API for a complete rebalance without losing user data in class *TxStateStorage* and *TxStateTableStorage*. The main idea is described in IGNITE-17990. > Creating an API for full rebalance without losing user data in TxStateStorage > - > > Key: IGNITE-18022 > URL: https://issues.apache.org/jira/browse/IGNITE-18022 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > We need to create an API for a complete rebalance without losing user data in > class *TxStateStorage* and *TxStateTableStorage*. > And also to make a test implementation and basic tests. > The main idea is described in IGNITE-17990. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18021) Creating an API for full rebalance without losing user data in MvPartitionStorage
[ https://issues.apache.org/jira/browse/IGNITE-18021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko updated IGNITE-18021: - Description: We need to create an API for a complete rebalance without losing user data in class *MvPartitionStorage* and *MvTableStorage*. And also to make a test implementation and basic tests. The main idea is described in IGNITE-17990. was: We need to create an API for a complete rebalance without losing user data in class *MvPartitionStorage* and *MvTableStorage*. The main idea is described in IGNITE-17990. > Creating an API for full rebalance without losing user data in > MvPartitionStorage > - > > Key: IGNITE-18021 > URL: https://issues.apache.org/jira/browse/IGNITE-18021 > Project: Ignite > Issue Type: Task >Reporter: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > We need to create an API for a complete rebalance without losing user data in > class *MvPartitionStorage* and *MvTableStorage*. > And also to make a test implementation and basic tests. > The main idea is described in IGNITE-17990. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18023) Implement GC API in page Memory based MV partition storages
Ivan Bessonov created IGNITE-18023: -- Summary: Implement GC API in page Memory based MV partition storages Key: IGNITE-18023 URL: https://issues.apache.org/jira/browse/IGNITE-18023 Project: Ignite Issue Type: Improvement Reporter: Ivan Bessonov Please refer to Epic for details. What I assume should happen: we probably need a hard synchronization on RowIds. And by that I mean a striped lock for reading/writing from/into a chain. I see no other options, but if there are better ideas - go ahead. All tests should pass, maybe more tests added if there are problems discovered. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17357) JMX metric exporter for Ignite 3
[ https://issues.apache.org/jira/browse/IGNITE-17357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626543#comment-17626543 ] Denis Chudov commented on IGNITE-17357: --- [~kgusakov] LGTM. > JMX metric exporter for Ignite 3 > > > Key: IGNITE-17357 > URL: https://issues.apache.org/jira/browse/IGNITE-17357 > Project: Ignite > Issue Type: New Feature >Reporter: Denis Chudov >Assignee: Kirill Gusakov >Priority: Major > Labels: ignite-3 > Time Spent: 40m > Remaining Estimate: 0h > > Metrics should be able to be exported via JMX as a first stage of metrics > exposing. > Exporter implementation must provide the following behavior: > * for each MetricSource we need to provide separate MBean with attribute per > metric > * each MBean attribute must have the same name as corresponding metric > * on enable/disable event for MetricSource corresponding MBean must be > registered/unregistered -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18022) Creating an API for full rebalance without losing user data in TxStateStorage
Kirill Tkalenko created IGNITE-18022: Summary: Creating an API for full rebalance without losing user data in TxStateStorage Key: IGNITE-18022 URL: https://issues.apache.org/jira/browse/IGNITE-18022 Project: Ignite Issue Type: Task Reporter: Kirill Tkalenko Fix For: 3.0.0-beta2 We need to create an API for a complete rebalance without losing user data in class *TxStateStorage* and *TxStateTableStorage*. The main idea is described in IGNITE-17990. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18021) Creating an API for full rebalance without losing user data in MvPartitionStorage
Kirill Tkalenko created IGNITE-18021: Summary: Creating an API for full rebalance without losing user data in MvPartitionStorage Key: IGNITE-18021 URL: https://issues.apache.org/jira/browse/IGNITE-18021 Project: Ignite Issue Type: Task Reporter: Kirill Tkalenko Fix For: 3.0.0-beta2 We need to create an API for a complete rebalance without losing user data in class *MvPartitionStorage* and *MvTableStorage*. The main idea is described in IGNITE-17990. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18020) Implement GC API in RocksDB-based MV partition storage
Ivan Bessonov created IGNITE-18020: -- Summary: Implement GC API in RocksDB-based MV partition storage Key: IGNITE-18020 URL: https://issues.apache.org/jira/browse/IGNITE-18020 Project: Ignite Issue Type: Improvement Reporter: Ivan Bessonov Please refer to the Epic description for details. All tests should pass, maybe more tests added if there are problems discovered. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-17690) Thin 3.0: Get and verify cluster id on connection
[ https://issues.apache.org/jira/browse/IGNITE-17690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Tupitsyn reassigned IGNITE-17690: --- Assignee: Pavel Tupitsyn > Thin 3.0: Get and verify cluster id on connection > - > > Key: IGNITE-17690 > URL: https://issues.apache.org/jira/browse/IGNITE-17690 > Project: Ignite > Issue Type: Improvement >Reporter: Igor Sapego >Assignee: Pavel Tupitsyn >Priority: Major > Labels: iep-90, ignite-3 > > Client need to get and verify cluster ID on handshake as described in > [IEP-90|https://cwiki.apache.org/confluence/display/IGNITE/IEP-90+Client+Lifecycle] > to make sure all servers it connects to are from the same cluster. This is > especially important during connection restoration. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18019) Add MV partition storage GC API
[ https://issues.apache.org/jira/browse/IGNITE-18019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Bessonov updated IGNITE-18019: --- Description: Please refer to the Epic for detailed description. In this issue I expect: * a new method in MvPartitionStorage that performs GC on a single row * a set of tests for this method, including concurrent scan and GC, a situation that may happen in real life and can cause real problems * implementation of the method in the test storage More on tests: * I would prefer a separate class, regular abstract partition test is too bloated * concurrent scenario that I mean. In parallel with GC trying to update a row: ** user may add a write intent *** old write intent was there *** old write intent wasn't there ** user may commit existing write intent ** user may abort existing write intent ** user may read row ** user may scan index that uses this row to resolve the validity of the value What can go wrong in these scenarios? # Version chain's head is updated # Something is removed from the chain while other thread reads it I don't think that there will be any issues in the test storage. Data is not deleted, all we need is a bunch of null-checks. But when it comes to others - my concerns are listed in the Epic. was: Please refer to the Epic for detailed description. In this issue I expect: * a new method in MvPartitionStorage that performs GC on a single row * a set of tests for this method, including concurrent scan and GC, a situation that may happen in real life and can cause real problems * implementation of the method in the test storage More on tests: * I would prefer a separate class, regular abstract partition test is too bloated * concurrent scenario that I mean. In parallel with GC trying to update a row: ** user may add a write intent *** old write intent was there *** old write intent wasn't there ** user may commit existing write intent ** user may abort existing write intent ** user may read row ** user may scan index that uses this row to resolve the validity of the value What can go wrong in these scenarios? # Version chain's head is updated # Something is removed from the chain while other thread reads it I don't think that there will be any issues in the test storage. But when it comes to others - my concerns are listed in the Epic. > Add MV partition storage GC API > --- > > Key: IGNITE-18019 > URL: https://issues.apache.org/jira/browse/IGNITE-18019 > Project: Ignite > Issue Type: Improvement >Reporter: Ivan Bessonov >Priority: Major > Labels: ignite-3 > > Please refer to the Epic for detailed description. > In this issue I expect: > * a new method in MvPartitionStorage that performs GC on a single row > * a set of tests for this method, including concurrent scan and GC, a > situation that may happen in real life and can cause real problems > * implementation of the method in the test storage > More on tests: > * I would prefer a separate class, regular abstract partition test is too > bloated > * concurrent scenario that I mean. In parallel with GC trying to update a > row: > ** user may add a write intent > *** old write intent was there > *** old write intent wasn't there > ** user may commit existing write intent > ** user may abort existing write intent > ** user may read row > ** user may scan index that uses this row to resolve the validity of the > value > What can go wrong in these scenarios? > # Version chain's head is updated > # Something is removed from the chain while other thread reads it > I don't think that there will be any issues in the test storage. Data is not > deleted, all we need is a bunch of null-checks. > But when it comes to others - my concerns are listed in the Epic. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17935) Finish implementation of streaming RAFT snapshot sender
[
https://issues.apache.org/jira/browse/IGNITE-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Puchkovskiy updated IGNITE-17935:
---
Summary: Finish implementation of streaming RAFT snapshot sender (was:
Finish implementation of RAFT streaming snapshot sender)
> Finish implementation of streaming RAFT snapshot sender
> ---
>
> Key: IGNITE-17935
> URL: https://issues.apache.org/jira/browse/IGNITE-17935
> Project: Ignite
> Issue Type: Improvement
> Components: persistence
>Reporter: Roman Puchkovskiy
>Assignee: Roman Puchkovskiy
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> Implementation is started in IGNITE-17083, but the task turned out to be too
> big for one PR.
> What has to be done:
> # Serve requests to get snapshot metadata
> # Switch from sending {{persistedIndex}} as {{lastAppliedIndex}} to sending
> {{lastAppliedIndex}} itself
> # When sending {{{}lastAppliedIndex{}}}, take into consideration
> {{lastAppliedIndex}} of both MV and TX storages
> # Resolve all TODOs mentioning this issue
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17935) Finish implementation of RAFT streaming snapshot sender
[
https://issues.apache.org/jira/browse/IGNITE-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Puchkovskiy updated IGNITE-17935:
---
Summary: Finish implementation of RAFT streaming snapshot sender (was:
Finish implementation of RAFT snapshot streamer reader)
> Finish implementation of RAFT streaming snapshot sender
> ---
>
> Key: IGNITE-17935
> URL: https://issues.apache.org/jira/browse/IGNITE-17935
> Project: Ignite
> Issue Type: Improvement
> Components: persistence
>Reporter: Roman Puchkovskiy
>Assignee: Roman Puchkovskiy
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> Implementation is started in IGNITE-17083, but the task turned out to be too
> big for one PR.
> What has to be done:
> # Serve requests to get snapshot metadata
> # Switch from sending {{persistedIndex}} as {{lastAppliedIndex}} to sending
> {{lastAppliedIndex}} itself
> # When sending {{{}lastAppliedIndex{}}}, take into consideration
> {{lastAppliedIndex}} of both MV and TX storages
> # Resolve all TODOs mentioning this issue
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17571) Implement GC for MV storages
[
https://issues.apache.org/jira/browse/IGNITE-17571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Bessonov updated IGNITE-17571:
---
Epic Name: GC for MV storages
> Implement GC for MV storages
>
>
> Key: IGNITE-17571
> URL: https://issues.apache.org/jira/browse/IGNITE-17571
> Project: Ignite
> Issue Type: Epic
>Reporter: Ivan Bessonov
>Priority: Major
> Labels: ignite-3
>
> h3. Basics
> Current implementation can only work with infinite storage space. This is
> because the storage works in appen-only mode (except for transactions
> rollbacks).
> There's a value in the system called "low watermark". It's guaranteed, that
> no new RO transactions will be started at the time earlier then LW. Existence
> of such value allows us to safely delete all versions before it. We will not
> discuss the mechanics of acquiring such value, assuming that it is simply
> given.
> It's expected that deletion will be performed in background workers, without
> holding any transactions locks. Process is independent on each RAFT group
> member. GC shouldn't affect transactions in any way, if possible.
> h3. API
> Original intended design looks like following:
> {code:java}
> cleanupCompletedVersions(@Nullable byte[] lower, boolean fromInclusive,
> @Nullable byte[] upper, boolean toInclusive, Timestamp timestamp){code}
> Now, I don't think that this is necessarily a good design. Main problem with
> it is the existence of bounds:
> * First of all, why not just have inclusive lower bound and exclusive upper
> bound, like almost all methods with bounds in existence.
> * Second, I believe that this API has been proposed in assumption that row
> ids will have a uniform distribution and every "range" cleanup will result in
> somewhat equal amount of job executed. This is simply not true in current
> implementation.
> RowId is a timestamp based value that exist in a very narrow time slice,
> making most of ranges empty and meaningless.
> Then, the way they're stored is actually up to the storage. There's no
> restrictions on byte order when physically storing RowId objects.
> h3. Problems
> There's one thing that worries me: indexes.
> Because storages are unaware of table schemas, it's impossible to cleanup
> indexes. This gets me thinking. API should be flexible enough so that indexes
> cleanup can be performed as an external operation over the storage. This will
> also reduce the amount of job that we need to do for the implementation.
> To be specific, it feels like the method would look like this:
> {code:java}
> RowId cleanup(HybridTimestamp threshold, RowId startId, int numRows,
> BiConsumer indexCleaner);{code}
> Explanation is required.
> * timestamp represents the same thing as before - low watermark.
> * startId - the row that should be first to iterate in current batch.
> * numRows - number of rows that should be cleaned in current batch. By this
> I mean individual versions, not chains.
> * cleaner closure must be used to clean indexes after every individual
> version removal. Right now it doesn't look optimal to me, but I struggle to
> find a good solution on efficient indexes cleanup.
> * next rowId is returned, that should be used a startId in next call. "null"
> if cleanup is complete. In this case it can be started from the beginning or
> simply postponed until new low watermark value is available.
> How to operate it:
> * numRows has two strategic purposes:
> ** to control the size of batches in "runConsistently" closures.
> ** to be able to run many cleanups in parallel avoiding pools starvation.
> Every task is split into small chunks.
> * cleanup should start from the smallest possible row id.
> * low watermark value can be changed in-between calls. This is fine and even
> preferable.
> h3. Alternative
> Some might find given API too complicated. There's another way that involves
> less actions from the "client".
> {code:java}
> void cleanup(HybridTimestamp threshold, RowId startId, BiConsumer BinaryRow> indexCleaner);{code}
> In a way, both of these methods are equivalent, the difference is minuscule.
> One can be transformed into the other with relative ease. If anything, this
> one is even preferable as a first implementation, because it's smaller and
> there's less room for errors.
> h3. Different engines
> This is the part where I express my thoughts of what can (and will) go wrong
> in different storages.
> Generally speaking, right now there are three implementations that must be
> compatible in behavior:
> # test storage that uses a ConcurrentSkipListMap and actual chains,
> represented with singly-linked lists
> # B+Tree-based storages, two of them. They use trees with, also, chains,
> also represented with singly-linked lists, but off-heap this time
> #
[jira] [Created] (IGNITE-18019) Add MV partition storage GC API
Ivan Bessonov created IGNITE-18019: -- Summary: Add MV partition storage GC API Key: IGNITE-18019 URL: https://issues.apache.org/jira/browse/IGNITE-18019 Project: Ignite Issue Type: Improvement Reporter: Ivan Bessonov Please refer to the Epic for detailed description. In this issue I expect: * a new method in MvPartitionStorage that performs GC on a single row * a set of tests for this method, including concurrent scan and GC, a situation that may happen in real life and can cause real problems * implementation of the method in the test storage More on tests: * I would prefer a separate class, regular abstract partition test is too bloated * concurrent scenario that I mean. In parallel with GC trying to update a row: ** user may add a write intent *** old write intent was there *** old write intent wasn't there ** user may commit existing write intent ** user may abort existing write intent ** user may read row ** user may scan index that uses this row to resolve the validity of the value What can go wrong in these scenarios? # Version chain's head is updated # Something is removed from the chain while other thread reads it I don't think that there will be any issues in the test storage. But when it comes to others - my concerns are listed in the Epic. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18018) Handle copy-on-write queue overflow in OutgoingSnapshot
Roman Puchkovskiy created IGNITE-18018: -- Summary: Handle copy-on-write queue overflow in OutgoingSnapshot Key: IGNITE-18018 URL: https://issues.apache.org/jira/browse/IGNITE-18018 Project: Ignite Issue Type: Improvement Components: persistence Reporter: Roman Puchkovskiy Fix For: 3.0.0-beta2 Currently, an OutgoingSnapshot does not restrict size of its internal queue of out-of-order rows to be sent to the receiver and does not provide any backpressure mechanism. This poses a threat of consuming too much memory (currently, it's heap memory) if the receiver is reading MV data slower than writes produce out-of-order data (or if the receiver has suddenly died and we still don't know about this). One idea is to throttle writes, but this can only work fine if the created backpressure crosses the boundary created by the RAFT log. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-17990) Download RAFT snapshot without deleting original partition data
[
https://issues.apache.org/jira/browse/IGNITE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Tkalenko reassigned IGNITE-17990:
Assignee: (was: Kirill Tkalenko)
> Download RAFT snapshot without deleting original partition data
> ---
>
> Key: IGNITE-17990
> URL: https://issues.apache.org/jira/browse/IGNITE-17990
> Project: Ignite
> Issue Type: Improvement
>Reporter: Ivan Bessonov
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> h3. Description
> In first design, full rebalance is implemented this way:
> * we drop partition data
> * we download partition data from the leader
> * we're done
> There's a problem with this approach - if download part failed, we lost one
> follower. This is bad, because technically new leader may have more data in
> the log and it could have uploaded it the follower, but now it makes no sense.
> Not only can it lead to hard-to-catch errors and introducing custom code to
> JRaft, it's also an unconditional data deletion without neither explicit
> user approval nor a copy of the data preserved durably.
> Such implementation is fine for POC and some tests, but it cannot be allowed
> in the release version of the product.
> h3. New proposed solution
> As trivial as it may seem, new solution is to _not deleting data_ before
> snapshot is fully downloaded and ready for swap. Why is it trivial? Because
> this is literally what RAFT demands to be done.
> Of course, there's a {*}but{*}. Snapshot application, when it's downloaded,
> should be {{O(1)}} when it comes to the number of rows in the partition and a
> number of transactions in a tx state store. This may not be fully achievable,
> depending on the implementation that we chose, more on that later.
> Following sections will describe all my concerns and possible
> implementations. Some sections can be skipped while reading. For example, if
> you're not interested in a specific storage engine, but want to read
> everything else.
> h3. TX state storage
> There's one really good thing about TX state storage. It has no storage
> engine, there's only a single RocksDB-based implementation. This makes
> possible the following approach:
> * when we stream data, we can write it into a SST file, almost like in
> snapshots of meta-storage ans CMG storages
> * once snapshot is downloaded, we ingest it into a storage
> What I like about this solution is that it's very simple. But, there are
> concerns:
> * ingesting leads to additional implicit data flush. Maybe it can be
> avoided, more on that later
> * it's not clear whether RocksDB creates a copy of SST file or not. I would
> assume that it does, because the file might be in other folder or on another
> device, for example. Although copying files is fast, it still takes time. Add
> to this a time required for the flush and we see a problem - operation may
> become unnecessarily long
> For these reasons, I don't think that such solution should be implemented.
> The point of this description was to show, that I thought about this
> alternative and consciously decided to use another one.
> I believe that TX state storage should use the same approach as a
> RocksDB-based partition storage. Its description can be found later in this
> issue.
> h3. MV storage - Test engine
> Test uses concurrent skip-list map for MV data and a bunch of other maps for
> indexes. While snapshots is being downloaded, we should insert all data into
> new maps, that have the same structure. In the end, we should have two
> versions of the partition: old and new.
> {{onSnapshotLoad}} should just swap all objects. After that, old partition
> data can be cleaned by the garbage collector.
> This is a good place to start implementation. I assume that some new API will
> be introduced. I have thoughts about it as well, they are described later in
> *API* section.
> h3. MV storage - RocksDB engine
> SST-based approach is described in a *TX state storage* section. There I
> describe why I don't think that this is a good solution. Same reasoning can
> be applied here just as effectively. This means that we should write data in
> the same RocksDB instance. This is a little bit tricky.
> The reason is that all stored data is merged together, and Columns Families
> are shared between different partitions. This makes it harder to find a place
> to write partition data while old partition data persists. As a reminder and
> an example, let's take a look at how data is stored in row storage:
> {code:java}
> +---+-+---+
> |Partition 0| ... | Partition MAX |
> +---+-+---+
> | Row1 | ... | RowN | ... | Row1 | ... | RowN |
>
[jira] [Assigned] (IGNITE-17990) Download RAFT snapshot without deleting original partition data
[
https://issues.apache.org/jira/browse/IGNITE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Tkalenko reassigned IGNITE-17990:
Assignee: (was: Kirill Tkalenko)
> Download RAFT snapshot without deleting original partition data
> ---
>
> Key: IGNITE-17990
> URL: https://issues.apache.org/jira/browse/IGNITE-17990
> Project: Ignite
> Issue Type: Improvement
>Reporter: Ivan Bessonov
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> h3. Description
> In first design, full rebalance is implemented this way:
> * we drop partition data
> * we download partition data from the leader
> * we're done
> There's a problem with this approach - if download part failed, we lost one
> follower. This is bad, because technically new leader may have more data in
> the log and it could have uploaded it the follower, but now it makes no sense.
> Not only can it lead to hard-to-catch errors and introducing custom code to
> JRaft, it's also an unconditional data deletion without neither explicit
> user approval nor a copy of the data preserved durably.
> Such implementation is fine for POC and some tests, but it cannot be allowed
> in the release version of the product.
> h3. New proposed solution
> As trivial as it may seem, new solution is to _not deleting data_ before
> snapshot is fully downloaded and ready for swap. Why is it trivial? Because
> this is literally what RAFT demands to be done.
> Of course, there's a {*}but{*}. Snapshot application, when it's downloaded,
> should be {{O(1)}} when it comes to the number of rows in the partition and a
> number of transactions in a tx state store. This may not be fully achievable,
> depending on the implementation that we chose, more on that later.
> Following sections will describe all my concerns and possible
> implementations. Some sections can be skipped while reading. For example, if
> you're not interested in a specific storage engine, but want to read
> everything else.
> h3. TX state storage
> There's one really good thing about TX state storage. It has no storage
> engine, there's only a single RocksDB-based implementation. This makes
> possible the following approach:
> * when we stream data, we can write it into a SST file, almost like in
> snapshots of meta-storage ans CMG storages
> * once snapshot is downloaded, we ingest it into a storage
> What I like about this solution is that it's very simple. But, there are
> concerns:
> * ingesting leads to additional implicit data flush. Maybe it can be
> avoided, more on that later
> * it's not clear whether RocksDB creates a copy of SST file or not. I would
> assume that it does, because the file might be in other folder or on another
> device, for example. Although copying files is fast, it still takes time. Add
> to this a time required for the flush and we see a problem - operation may
> become unnecessarily long
> For these reasons, I don't think that such solution should be implemented.
> The point of this description was to show, that I thought about this
> alternative and consciously decided to use another one.
> I believe that TX state storage should use the same approach as a
> RocksDB-based partition storage. Its description can be found later in this
> issue.
> h3. MV storage - Test engine
> Test uses concurrent skip-list map for MV data and a bunch of other maps for
> indexes. While snapshots is being downloaded, we should insert all data into
> new maps, that have the same structure. In the end, we should have two
> versions of the partition: old and new.
> {{onSnapshotLoad}} should just swap all objects. After that, old partition
> data can be cleaned by the garbage collector.
> This is a good place to start implementation. I assume that some new API will
> be introduced. I have thoughts about it as well, they are described later in
> *API* section.
> h3. MV storage - RocksDB engine
> SST-based approach is described in a *TX state storage* section. There I
> describe why I don't think that this is a good solution. Same reasoning can
> be applied here just as effectively. This means that we should write data in
> the same RocksDB instance. This is a little bit tricky.
> The reason is that all stored data is merged together, and Columns Families
> are shared between different partitions. This makes it harder to find a place
> to write partition data while old partition data persists. As a reminder and
> an example, let's take a look at how data is stored in row storage:
> {code:java}
> +---+-+---+
> |Partition 0| ... | Partition MAX |
> +---+-+---+
> | Row1 | ... | RowN | ... | Row1 | ... | RowN |
>
[jira] [Assigned] (IGNITE-17990) Download RAFT snapshot without deleting original partition data
[
https://issues.apache.org/jira/browse/IGNITE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Tkalenko reassigned IGNITE-17990:
Assignee: Kirill Tkalenko
> Download RAFT snapshot without deleting original partition data
> ---
>
> Key: IGNITE-17990
> URL: https://issues.apache.org/jira/browse/IGNITE-17990
> Project: Ignite
> Issue Type: Improvement
>Reporter: Ivan Bessonov
>Assignee: Kirill Tkalenko
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> h3. Description
> In first design, full rebalance is implemented this way:
> * we drop partition data
> * we download partition data from the leader
> * we're done
> There's a problem with this approach - if download part failed, we lost one
> follower. This is bad, because technically new leader may have more data in
> the log and it could have uploaded it the follower, but now it makes no sense.
> Not only can it lead to hard-to-catch errors and introducing custom code to
> JRaft, it's also an unconditional data deletion without neither explicit
> user approval nor a copy of the data preserved durably.
> Such implementation is fine for POC and some tests, but it cannot be allowed
> in the release version of the product.
> h3. New proposed solution
> As trivial as it may seem, new solution is to _not deleting data_ before
> snapshot is fully downloaded and ready for swap. Why is it trivial? Because
> this is literally what RAFT demands to be done.
> Of course, there's a {*}but{*}. Snapshot application, when it's downloaded,
> should be {{O(1)}} when it comes to the number of rows in the partition and a
> number of transactions in a tx state store. This may not be fully achievable,
> depending on the implementation that we chose, more on that later.
> Following sections will describe all my concerns and possible
> implementations. Some sections can be skipped while reading. For example, if
> you're not interested in a specific storage engine, but want to read
> everything else.
> h3. TX state storage
> There's one really good thing about TX state storage. It has no storage
> engine, there's only a single RocksDB-based implementation. This makes
> possible the following approach:
> * when we stream data, we can write it into a SST file, almost like in
> snapshots of meta-storage ans CMG storages
> * once snapshot is downloaded, we ingest it into a storage
> What I like about this solution is that it's very simple. But, there are
> concerns:
> * ingesting leads to additional implicit data flush. Maybe it can be
> avoided, more on that later
> * it's not clear whether RocksDB creates a copy of SST file or not. I would
> assume that it does, because the file might be in other folder or on another
> device, for example. Although copying files is fast, it still takes time. Add
> to this a time required for the flush and we see a problem - operation may
> become unnecessarily long
> For these reasons, I don't think that such solution should be implemented.
> The point of this description was to show, that I thought about this
> alternative and consciously decided to use another one.
> I believe that TX state storage should use the same approach as a
> RocksDB-based partition storage. Its description can be found later in this
> issue.
> h3. MV storage - Test engine
> Test uses concurrent skip-list map for MV data and a bunch of other maps for
> indexes. While snapshots is being downloaded, we should insert all data into
> new maps, that have the same structure. In the end, we should have two
> versions of the partition: old and new.
> {{onSnapshotLoad}} should just swap all objects. After that, old partition
> data can be cleaned by the garbage collector.
> This is a good place to start implementation. I assume that some new API will
> be introduced. I have thoughts about it as well, they are described later in
> *API* section.
> h3. MV storage - RocksDB engine
> SST-based approach is described in a *TX state storage* section. There I
> describe why I don't think that this is a good solution. Same reasoning can
> be applied here just as effectively. This means that we should write data in
> the same RocksDB instance. This is a little bit tricky.
> The reason is that all stored data is merged together, and Columns Families
> are shared between different partitions. This makes it harder to find a place
> to write partition data while old partition data persists. As a reminder and
> an example, let's take a look at how data is stored in row storage:
> {code:java}
> +---+-+---+
> |Partition 0| ... | Partition MAX |
> +---+-+---+
> | Row1 | ... | RowN | ... | Row1 | ... |
[jira] [Assigned] (IGNITE-18016) Fix javadocs for public IgniteTables
[ https://issues.apache.org/jira/browse/IGNITE-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yury Gerzhedovich reassigned IGNITE-18016: -- Assignee: Yury Gerzhedovich > Fix javadocs for public IgniteTables > - > > Key: IGNITE-18016 > URL: https://issues.apache.org/jira/browse/IGNITE-18016 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Yury Gerzhedovich >Assignee: Yury Gerzhedovich >Priority: Major > > We mention using SQL schemas in javadocs at > org.apache.ignite.table.manager.IgniteTables. Due to schemas are not > supported now need to rewrite these javadocs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17980) ./gradlew clean build -x test fails
[ https://issues.apache.org/jira/browse/IGNITE-17980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Lukyanov updated IGNITE-17980: Fix Version/s: 3.0.0-beta1 (was: 3.0.0-beta2) > ./gradlew clean build -x test fails > --- > > Key: IGNITE-17980 > URL: https://issues.apache.org/jira/browse/IGNITE-17980 > Project: Ignite > Issue Type: Bug > Components: build >Reporter: Aleksandr >Assignee: Aleksandr >Priority: Critical > Labels: ignite-3 > Fix For: 3.0.0-beta1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > On the latest main branch, the following fails: ./gradlew clean build -x test -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18009) Fix gradle build
[ https://issues.apache.org/jira/browse/IGNITE-18009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Lukyanov updated IGNITE-18009: Fix Version/s: 3.0.0-beta1 (was: 3.0.0-beta2) > Fix gradle build > > > Key: IGNITE-18009 > URL: https://issues.apache.org/jira/browse/IGNITE-18009 > Project: Ignite > Issue Type: Bug > Components: build >Reporter: Mikhail Pochatkin >Assignee: Mikhail Pochatkin >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta1 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17965) Enable remote build cache for Gradle
[ https://issues.apache.org/jira/browse/IGNITE-17965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Lukyanov updated IGNITE-17965: Fix Version/s: 3.0.0-beta1 (was: 3.0.0-beta2) > Enable remote build cache for Gradle > > > Key: IGNITE-17965 > URL: https://issues.apache.org/jira/browse/IGNITE-17965 > Project: Ignite > Issue Type: Improvement > Components: build >Reporter: Mikhail Pochatkin >Assignee: Mikhail Pochatkin >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta1 > > Time Spent: 20m > Remaining Estimate: 0h > > Need to enable remote build cache feature for Gradle on CI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17966) Fix problem with stuck Gradle processes in .NET tests
[ https://issues.apache.org/jira/browse/IGNITE-17966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Lukyanov updated IGNITE-17966: Fix Version/s: 3.0.0-beta1 (was: 3.0.0-beta2) > Fix problem with stuck Gradle processes in .NET tests > - > > Key: IGNITE-17966 > URL: https://issues.apache.org/jira/browse/IGNITE-17966 > Project: Ignite > Issue Type: Bug > Components: build >Reporter: Mikhail Pochatkin >Assignee: Mikhail Pochatkin >Priority: Blocker > Labels: ignite-3 > Fix For: 3.0.0-beta1 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, when .NET test suite finished some Gradle processes will alive and > after that CI agents start be in inconsistance state. The main problem is > using ports and next build will be failed because of unreachable port. So, > need to fix this problem and kill all processes which producing in build > time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18017) Add tests for binary tuple (C++)
[ https://issues.apache.org/jira/browse/IGNITE-18017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Semyon Danilov updated IGNITE-18017: Description: Right now C++ implementation of the binary tuple is not tested. > Add tests for binary tuple (C++) > > > Key: IGNITE-18017 > URL: https://issues.apache.org/jira/browse/IGNITE-18017 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > > Right now C++ implementation of the binary tuple is not tested. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-18017) Add tests for binary tuple (C++)
Semyon Danilov created IGNITE-18017: --- Summary: Add tests for binary tuple (C++) Key: IGNITE-18017 URL: https://issues.apache.org/jira/browse/IGNITE-18017 Project: Ignite Issue Type: Improvement Reporter: Semyon Danilov Assignee: Semyon Danilov Fix For: 3.0.0-beta2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18005) ItDataTypesTest#testUnicodeStrings fails with IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/IGNITE-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stanislav Lukyanov updated IGNITE-18005:
Fix Version/s: 3.0.0-beta1
> ItDataTypesTest#testUnicodeStrings fails with IndexOutOfBoundsException
> ---
>
> Key: IGNITE-18005
> URL: https://issues.apache.org/jira/browse/IGNITE-18005
> Project: Ignite
> Issue Type: Bug
>Reporter: Alexander Lapin
>Assignee: Vladislav Pyatkov
>Priority: Major
> Fix For: 3.0.0-beta1
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {code:java}
> Caused by: java.lang.IndexOutOfBoundsException: 4
> at java.base/java.nio.HeapByteBuffer.get(HeapByteBuffer.java:166)
> at org.apache.ignite.internal.schema.row.Row.findColumn(Row.java:516)
> at org.apache.ignite.internal.schema.row.Row.stringValue(Row.java:288)
> at
> org.apache.ignite.internal.schema.NativeTypeSpec$9.objectValue(NativeTypeSpec.java:134)
> at org.apache.ignite.internal.schema.row.Row.value(Row.java:122)
> at
> org.apache.ignite.internal.sql.engine.schema.IgniteTableImpl.toRow(IgniteTableImpl.java:284)
> at
> org.apache.ignite.internal.sql.engine.exec.rel.TableScanNode.convert(TableScanNode.java:279)
> at
> org.apache.ignite.internal.sql.engine.exec.rel.TableScanNode$SubscriberImpl.onNext(TableScanNode.java:246)
> at
> org.apache.ignite.internal.sql.engine.exec.rel.TableScanNode$SubscriberImpl.onNext(TableScanNode.java:233)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
> at
> org.apache.ignite.internal.table.distributed.storage.InternalTableImpl$PartitionScanPublisher$PartitionScanSubscription.lambda$scanBatch$2(InternalTableImpl.java:1182)
> at
> java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714)
> ... 11 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IGNITE-17640) Update Ignite Spring Data Dependency Snippet
[
https://issues.apache.org/jira/browse/IGNITE-17640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Gusev reassigned IGNITE-17640:
---
Assignee: Igor Gusev
> Update Ignite Spring Data Dependency Snippet
>
>
> Key: IGNITE-17640
> URL: https://issues.apache.org/jira/browse/IGNITE-17640
> Project: Ignite
> Issue Type: Bug
> Components: documentation
>Reporter: Igor Belyakov
>Assignee: Igor Gusev
>Priority: Major
>
> Since "org.apache.ignite:ignite-spring-data-ext:2.0.0" was released:
>
> [https://search.maven.org/artifact/org.apache.ignite/ignite-spring-data-ext/2.0.0/jar]
> a dependency snippet on the Spring Data page should be updated:
>
> [https://ignite.apache.org/docs/extensions/spring/spring-data#maven-configuration]
> as well as the "Apache Ignite Spring Data extension version" in the
> compatibility table.
> Also, the "org.springframework:spring-tx" dependency should be added to the
> dependency snippet:
> {code:java}
>
> org.springframework
> spring-tx
> ${spring.version}
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-18016) Fix javadocs for public IgniteTables
Yury Gerzhedovich created IGNITE-18016: -- Summary: Fix javadocs for public IgniteTables Key: IGNITE-18016 URL: https://issues.apache.org/jira/browse/IGNITE-18016 Project: Ignite Issue Type: Improvement Components: sql Reporter: Yury Gerzhedovich We mention using SQL schemas in javadocs at org.apache.ignite.table.manager.IgniteTables. Due to schemas are not supported now need to rewrite these javadocs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17975) Unmute ItIgniteNodeRestartTest and ItIgniteInMemoryNodeRestartTest
[ https://issues.apache.org/jira/browse/IGNITE-17975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626486#comment-17626486 ] Alexander Lapin commented on IGNITE-17975: -- [~maliev] LGTM, thanks! > Unmute ItIgniteNodeRestartTest and ItIgniteInMemoryNodeRestartTest > -- > > Key: IGNITE-17975 > URL: https://issues.apache.org/jira/browse/IGNITE-17975 > Project: Ignite > Issue Type: Task >Reporter: Mirza Aliev >Assignee: Mirza Aliev >Priority: Major > Labels: ignite-3 > Time Spent: 10m > Remaining Estimate: 0h > > h4. *Motivation* > After implementation of IGNITE-15081 some test were muted because of new > managers that are supposed to be initiated in test nodes. We need to enable > ItIgniteNodeRestartTest and ItIgniteInMemoryNodeRestartTest. > h4. *Definition of Done* > All tests that are not muted with different cause must pass. > *Implementation notes* > We need to add ReplicaManager to the node flow in test, also we need to add > dummy index restore after restart of node -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-17899) Properly stop resources in TableManager
[
https://issues.apache.org/jira/browse/IGNITE-17899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Uttsel reassigned IGNITE-17899:
--
Assignee: Sergey Uttsel
> Properly stop resources in TableManager
> ---
>
> Key: IGNITE-17899
> URL: https://issues.apache.org/jira/browse/IGNITE-17899
> Project: Ignite
> Issue Type: Task
>Reporter: Mirza Aliev
>Assignee: Sergey Uttsel
>Priority: Major
> Labels: ignite-3
>
> It's required to add sort of stopAll logic that will try to stop all
> resources even if some of them throws an exception on stop in
> {{org.apache.ignite.internal.table.distributed.TableManager#cleanUpTablesResources}}.
> Currently, if some components throws exception during `TableManager` stop,
> other components won't be stopped.
> I'd rather use something similar to
> {{org.apache.ignite.internal.util.IgniteUtils#closeAll(java.util.Collection extends java.lang.AutoCloseable>)}}.
> Also `MvTableStorage` must be `AutoClosable`
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-18013) Unmute tests after IGNITE-17968
[ https://issues.apache.org/jira/browse/IGNITE-18013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626480#comment-17626480 ] Alexander Lapin commented on IGNITE-18013: -- [~sdanilov] LGTM, thanks! > Unmute tests after IGNITE-17968 > --- > > Key: IGNITE-18013 > URL: https://issues.apache.org/jira/browse/IGNITE-18013 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 10m > Remaining Estimate: 0h > > Need to unmute tests muted for IGNITE-17968 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-18013) Unmute tests after IGNITE-17968
[ https://issues.apache.org/jira/browse/IGNITE-18013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin updated IGNITE-18013: - Reviewer: Alexander Lapin > Unmute tests after IGNITE-17968 > --- > > Key: IGNITE-18013 > URL: https://issues.apache.org/jira/browse/IGNITE-18013 > Project: Ignite > Issue Type: Improvement >Reporter: Semyon Danilov >Assignee: Semyon Danilov >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 10m > Remaining Estimate: 0h > > Need to unmute tests muted for IGNITE-17968 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (IGNITE-17970) NPE during check error code in SessionImpl
[
https://issues.apache.org/jira/browse/IGNITE-17970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yury Gerzhedovich reopened IGNITE-17970:
> NPE during check error code in SessionImpl
> --
>
> Key: IGNITE-17970
> URL: https://issues.apache.org/jira/browse/IGNITE-17970
> Project: Ignite
> Issue Type: Bug
> Components: sql
>Reporter: Yury Gerzhedovich
>Priority: Major
> Labels: ignite-3
>
> At line 179 in SessionImpl class we compare Integer result of
> IgniteExceptionUtils.getIgniteErrorCode invocation with int, but result
> invocation could be null.
> Need to fix NPE for the case
> {code:java}
> if (IgniteExceptionUtils.getIgniteErrorCode(th) == SESSION_NOT_FOUND_ERR) {
> closeInternal();
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (IGNITE-17970) NPE during check error code in SessionImpl
[
https://issues.apache.org/jira/browse/IGNITE-17970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yury Gerzhedovich resolved IGNITE-17970.
Resolution: Duplicate
> NPE during check error code in SessionImpl
> --
>
> Key: IGNITE-17970
> URL: https://issues.apache.org/jira/browse/IGNITE-17970
> Project: Ignite
> Issue Type: Bug
> Components: sql
>Reporter: Yury Gerzhedovich
>Priority: Major
> Labels: ignite-3
>
> At line 179 in SessionImpl class we compare Integer result of
> IgniteExceptionUtils.getIgniteErrorCode invocation with int, but result
> invocation could be null.
> Need to fix NPE for the case
> {code:java}
> if (IgniteExceptionUtils.getIgniteErrorCode(th) == SESSION_NOT_FOUND_ERR) {
> closeInternal();
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (IGNITE-17970) NPE during check error code in SessionImpl
[
https://issues.apache.org/jira/browse/IGNITE-17970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yury Gerzhedovich resolved IGNITE-17970.
Resolution: Invalid
> NPE during check error code in SessionImpl
> --
>
> Key: IGNITE-17970
> URL: https://issues.apache.org/jira/browse/IGNITE-17970
> Project: Ignite
> Issue Type: Bug
> Components: sql
>Reporter: Yury Gerzhedovich
>Priority: Major
> Labels: ignite-3
>
> At line 179 in SessionImpl class we compare Integer result of
> IgniteExceptionUtils.getIgniteErrorCode invocation with int, but result
> invocation could be null.
> Need to fix NPE for the case
> {code:java}
> if (IgniteExceptionUtils.getIgniteErrorCode(th) == SESSION_NOT_FOUND_ERR) {
> closeInternal();
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17571) Implement GC for MV storages
[
https://issues.apache.org/jira/browse/IGNITE-17571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Bessonov updated IGNITE-17571:
---
Issue Type: Epic (was: Improvement)
> Implement GC for MV storages
>
>
> Key: IGNITE-17571
> URL: https://issues.apache.org/jira/browse/IGNITE-17571
> Project: Ignite
> Issue Type: Epic
>Reporter: Ivan Bessonov
>Priority: Major
> Labels: ignite-3
>
> h3. Basics
> Current implementation can only work with infinite storage space. This is
> because the storage works in appen-only mode (except for transactions
> rollbacks).
> There's a value in the system called "low watermark". It's guaranteed, that
> no new RO transactions will be started at the time earlier then LW. Existence
> of such value allows us to safely delete all versions before it. We will not
> discuss the mechanics of acquiring such value, assuming that it is simply
> given.
> It's expected that deletion will be performed in background workers, without
> holding any transactions locks. Process is independent on each RAFT group
> member. GC shouldn't affect transactions in any way, if possible.
> h3. API
> Original intended design looks like following:
> {code:java}
> cleanupCompletedVersions(@Nullable byte[] lower, boolean fromInclusive,
> @Nullable byte[] upper, boolean toInclusive, Timestamp timestamp){code}
> Now, I don't think that this is necessarily a good design. Main problem with
> it is the existence of bounds:
> * First of all, why not just have inclusive lower bound and exclusive upper
> bound, like almost all methods with bounds in existence.
> * Second, I believe that this API has been proposed in assumption that row
> ids will have a uniform distribution and every "range" cleanup will result in
> somewhat equal amount of job executed. This is simply not true in current
> implementation.
> RowId is a timestamp based value that exist in a very narrow time slice,
> making most of ranges empty and meaningless.
> Then, the way they're stored is actually up to the storage. There's no
> restrictions on byte order when physically storing RowId objects.
> h3. Problems
> There's one thing that worries me: indexes.
> Because storages are unaware of table schemas, it's impossible to cleanup
> indexes. This gets me thinking. API should be flexible enough so that indexes
> cleanup can be performed as an external operation over the storage. This will
> also reduce the amount of job that we need to do for the implementation.
> To be specific, it feels like the method would look like this:
> {code:java}
> RowId cleanup(HybridTimestamp threshold, RowId startId, int numRows,
> BiConsumer indexCleaner);{code}
> Explanation is required.
> * timestamp represents the same thing as before - low watermark.
> * startId - the row that should be first to iterate in current batch.
> * numRows - number of rows that should be cleaned in current batch. By this
> I mean individual versions, not chains.
> * cleaner closure must be used to clean indexes after every individual
> version removal. Right now it doesn't look optimal to me, but I struggle to
> find a good solution on efficient indexes cleanup.
> * next rowId is returned, that should be used a startId in next call. "null"
> if cleanup is complete. In this case it can be started from the beginning or
> simply postponed until new low watermark value is available.
> How to operate it:
> * numRows has two strategic purposes:
> ** to control the size of batches in "runConsistently" closures.
> ** to be able to run many cleanups in parallel avoiding pools starvation.
> Every task is split into small chunks.
> * cleanup should start from the smallest possible row id.
> * low watermark value can be changed in-between calls. This is fine and even
> preferable.
> h3. Alternative
> Some might find given API too complicated. There's another way that involves
> less actions from the "client".
> {code:java}
> void cleanup(HybridTimestamp threshold, RowId startId, BiConsumer BinaryRow> indexCleaner);{code}
> In a way, both of these methods are equivalent, the difference is minuscule.
> One can be transformed into the other with relative ease. If anything, this
> one is even preferable as a first implementation, because it's smaller and
> there's less room for errors.
> h3. Different engines
> This is the part where I express my thoughts of what can (and will) go wrong
> in different storages.
> Generally speaking, right now there are three implementations that must be
> compatible in behavior:
> # test storage that uses a ConcurrentSkipListMap and actual chains,
> represented with singly-linked lists
> # B+Tree-based storages, two of them. They use trees with, also, chains,
> also represented with singly-linked lists, but off-heap this time
> #
