[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='true'.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: (was: bench_results_defaultIsolated_longer.txt)

> Datastreamer may consume heap with allowOverwtire=='true'.
> --
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='true'.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: bench_results_defaultIsolated_longer.txt

> Datastreamer may consume heap with allowOverwtire=='true'.
> --
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_results_defaultIsolated_longer.txt
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17937) Confusing folder appears after using test framework for non Ignite projects.

[Mikhail Petrov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Petrov updated IGNITE-17937:

Summary: Confusing folder appears after using test framework for non Ignite 
projects.  (was: [Extensions] Confusing folder appears after running Ignite 
Extension tests.)

> Confusing folder appears after using test framework for non Ignite projects.
> 
>
> Key: IGNITE-17937
> URL: https://issues.apache.org/jira/browse/IGNITE-17937
> Project: Ignite
>  Issue Type: Bug
>Reporter: Mikhail Petrov
>Assignee: Mikhail Petrov
>Priority: Minor
> Fix For: 2.15
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Confusing folder with name sys:IGNITE_HOME appears after running Ignite 
> Extension tests.
> It happens because GridTestUtils#initTestProjectHome method which is 
> responsible for initializing IGNITE_HOME system property for non Ignite tests 
> does not work corrctly.
> First invocation of U.getIgniteHome() method tries to resolve Ignite Home 
> path and in case of a failure initializes it with null.
> U.setIgniteHome(...) does actually nothing if the previous attempt to resolve 
> Ignite Home was performed (even if it fails and null was set).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='true'.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: (was: bench_persistent_results_Isolated_pc1.txt)

> Datastreamer may consume heap with allowOverwtire=='true'.
> --
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='true'.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: (was: bench_persistent_results_Individual_pc1.txt)

> Datastreamer may consume heap with allowOverwtire=='true'.
> --
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17937) [Extensions] Confusing folder appears after running Ignite Extension tests.

[Mikhail Petrov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Petrov updated IGNITE-17937:

Fix Version/s: 2.15

> [Extensions] Confusing folder appears after running Ignite Extension tests.
> ---
>
> Key: IGNITE-17937
> URL: https://issues.apache.org/jira/browse/IGNITE-17937
> Project: Ignite
>  Issue Type: Bug
>Reporter: Mikhail Petrov
>Assignee: Mikhail Petrov
>Priority: Minor
> Fix For: 2.15
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Confusing folder with name sys:IGNITE_HOME appears after running Ignite 
> Extension tests.
> It happens because GridTestUtils#initTestProjectHome method which is 
> responsible for initializing IGNITE_HOME system property for non Ignite tests 
> does not work corrctly.
> First invocation of U.getIgniteHome() method tries to resolve Ignite Home 
> path and in case of a failure initializes it with null.
> U.setIgniteHome(...) does actually nothing if the previous attempt to resolve 
> Ignite Home was performed (even if it fails and null was set).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='true'.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: (was: bench_persistent_full_Individual_pc1.txt)

> Datastreamer may consume heap with allowOverwtire=='true'.
> --
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, 
> bench_persistent_results_Individual_pc1.txt, 
> bench_persistent_results_Isolated_pc1.txt
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='true'.

[Vladimir Steshin (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607891#comment-17607891
 ] 

Vladimir Steshin edited comment on IGNITE-17735 at 10/28/22 5:18 AM:
-

Datastreamer with 'allowOverwrite==true' and PRIMARY_SYNC persistent cache may 
cause heap issue or consume increased heap amount with persistent caches.

There is related fixing 'perNodeParallelOperations()' setting. But I met this 
issue with trivial research like `HeapConsumptionDataStreamerTest.src`.
Default parallel batches amount for persistent caches looks too much. The 
setting is historically for in-memory caches. 

What happens: streamer keep sending more and more streamer batches to process 
while receiving node collects backup updates futures, requests. 
Similarly, backup node accumulates incoming update requests stucking at disk 
writes. See 'DS_heap_consumption.png' for example.

Suggestion.
We might bring reduced default parallel batches number for persistent caches 
`IgniteDataStreamer#DFLT_PARALLEL_OPS_PERSISTENT_MULTIPLIER`(PR #10343).
Why sending more? Helps a lot, reduces heap utilization even if there is no 
OOMe. Better solution would be a backpressure. Not sure it worth the case.

Did some benchmarks. For persistent caches `CPUs x 2` seems enough.


was (Author: vladsz83):
Datastreamer with 'allowOverwrite==true' and PRIMARY_SYNC persistent cache may 
cause heap issue or consume increased heap amount.

There is related 'perNodeParallelOperations()' setting. What discouraged, I met 
this issue with trivial research like few servers, simple cache and just trying 
data streaming with various persistence and loading settings (like 
`HeapConsumptionDataStreamerTest.src`). Think user may meet the same. The 
default value might be adjusted for persistant caches.

Streamer node may not wait for backup updates. And keep sending more and more 
streamer batches to process. The receiving node collects related to backup 
updates futures, requests. 
The same happens on backup node: collecting update incoming update requests 
stucking at disk writes. See 'DS_heap_consumption.png' for example.

Suggestion: bring reduced default parallel batches number for persistent caches 
`IgniteDataStreamer#DFLT_PARALLEL_OPS_PERSISTENT_MULTIPLIER` (PR #10343).

Did estimation benchmarks. Even in-memory benchmarks (like 
'bench_inmem_isolated_pc2.txt') shows 2 or may be 4 batches per threads seems 
enough. 

For persistent caches, `CPUs x 2` seems enough. See 
`bench_persistent_results_Isolated_pc1.txt` and 
`bench_persistent_results_Individual_pc1.txt`

> Datastreamer may consume heap with allowOverwtire=='true'.
> --
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, 
> bench_persistent_results_Individual_pc1.txt, 
> bench_persistent_results_Isolated_pc1.txt
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='true'.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: (was: bench_inmem_isolated_pc2.txt)

> Datastreamer may consume heap with allowOverwtire=='true'.
> --
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, 
> bench_persistent_full_Individual_pc1.txt, 
> bench_persistent_results_Individual_pc1.txt, 
> bench_persistent_results_Isolated_pc1.txt
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='true'.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: (was: bench_inmem_individual_pc2.txt)

> Datastreamer may consume heap with allowOverwtire=='true'.
> --
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_isolated_pc2.txt, 
> bench_persistent_full_Individual_pc1.txt, 
> bench_persistent_results_Individual_pc1.txt, 
> bench_persistent_results_Isolated_pc1.txt
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-18004) Fix compilation after merging safe time propagation

[Alexander Lapin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-18004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin updated IGNITE-18004:
-
Reviewer: Vladislav Pyatkov

> Fix compilation after merging safe time propagation
> ---
>
> Key: IGNITE-18004
> URL: https://issues.apache.org/jira/browse/IGNITE-18004
> Project: Ignite
>  Issue Type: Bug
>Reporter: Alexander Lapin
>Assignee: Alexander Lapin
>Priority: Major
> Fix For: 3.0.0-beta1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> {code:java}
> [ERROR] 
> /home/alapin/workspace/ignite-3/modules/sql-engine/src/main/java/org/apache/ignite/internal/sql/engine/SqlQueryProcessor.java:[393,110]
>  org.apache.ignite.internal.hlc.HybridClock is abstract; cannot be 
> instantiated
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (IGNITE-18004) Fix compilation after merging safe time propagation

[Vladislav Pyatkov (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-18004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625328#comment-17625328
 ] 

Vladislav Pyatkov commented on IGNITE-18004:


LGTM

> Fix compilation after merging safe time propagation
> ---
>
> Key: IGNITE-18004
> URL: https://issues.apache.org/jira/browse/IGNITE-18004
> Project: Ignite
>  Issue Type: Bug
>Reporter: Alexander Lapin
>Assignee: Alexander Lapin
>Priority: Major
> Fix For: 3.0.0-beta1
>
>
> {code:java}
> [ERROR] 
> /home/alapin/workspace/ignite-3/modules/sql-engine/src/main/java/org/apache/ignite/internal/sql/engine/SqlQueryProcessor.java:[393,110]
>  org.apache.ignite.internal.hlc.HybridClock is abstract; cannot be 
> instantiated
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-18004) Fix compilation after merging safe time propagation

[Alexander Lapin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-18004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin updated IGNITE-18004:
-
Fix Version/s: 3.0.0-beta1

> Fix compilation after merging safe time propagation
> ---
>
> Key: IGNITE-18004
> URL: https://issues.apache.org/jira/browse/IGNITE-18004
> Project: Ignite
>  Issue Type: Bug
>Reporter: Alexander Lapin
>Assignee: Alexander Lapin
>Priority: Major
> Fix For: 3.0.0-beta1
>
>
> {code:java}
> [ERROR] 
> /home/alapin/workspace/ignite-3/modules/sql-engine/src/main/java/org/apache/ignite/internal/sql/engine/SqlQueryProcessor.java:[393,110]
>  org.apache.ignite.internal.hlc.HybridClock is abstract; cannot be 
> instantiated
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (IGNITE-18004) Fix compilation after merging safe time propagation

[Alexander Lapin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-18004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin reassigned IGNITE-18004:


Assignee: Alexander Lapin

> Fix compilation after merging safe time propagation
> ---
>
> Key: IGNITE-18004
> URL: https://issues.apache.org/jira/browse/IGNITE-18004
> Project: Ignite
>  Issue Type: Bug
>Reporter: Alexander Lapin
>Assignee: Alexander Lapin
>Priority: Major
>
> {code:java}
> [ERROR] 
> /home/alapin/workspace/ignite-3/modules/sql-engine/src/main/java/org/apache/ignite/internal/sql/engine/SqlQueryProcessor.java:[393,110]
>  org.apache.ignite.internal.hlc.HybridClock is abstract; cannot be 
> instantiated
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (IGNITE-18005) ItDataTypesTest#testUnicodeStrings fails with IndexOutOfBoundsException

[Alexander Lapin (Jira)]

Alexander Lapin created IGNITE-18005:


 Summary: ItDataTypesTest#testUnicodeStrings fails with 
IndexOutOfBoundsException
 Key: IGNITE-18005
 URL: https://issues.apache.org/jira/browse/IGNITE-18005
 Project: Ignite
  Issue Type: Bug
Reporter: Alexander Lapin


{code:java}
Caused by: java.lang.IndexOutOfBoundsException: 4
    at java.base/java.nio.HeapByteBuffer.get(HeapByteBuffer.java:166)
    at org.apache.ignite.internal.schema.row.Row.findColumn(Row.java:516)
    at org.apache.ignite.internal.schema.row.Row.stringValue(Row.java:288)
    at 
org.apache.ignite.internal.schema.NativeTypeSpec$9.objectValue(NativeTypeSpec.java:134)
    at org.apache.ignite.internal.schema.row.Row.value(Row.java:122)
    at 
org.apache.ignite.internal.sql.engine.schema.IgniteTableImpl.toRow(IgniteTableImpl.java:284)
    at 
org.apache.ignite.internal.sql.engine.exec.rel.TableScanNode.convert(TableScanNode.java:279)
    at 
org.apache.ignite.internal.sql.engine.exec.rel.TableScanNode$SubscriberImpl.onNext(TableScanNode.java:246)
    at 
org.apache.ignite.internal.sql.engine.exec.rel.TableScanNode$SubscriberImpl.onNext(TableScanNode.java:233)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
    at 
org.apache.ignite.internal.table.distributed.storage.InternalTableImpl$PartitionScanPublisher$PartitionScanSubscription.lambda$scanBatch$2(InternalTableImpl.java:1182)
    at 
java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714)
    ... 11 more
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17988) Wrong version of jackson-core dependency resolved in ml module

[Vladimir Kornyshev (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Kornyshev updated IGNITE-17988:

Affects Version/s: 2.13
   2.12

> Wrong version of jackson-core dependency resolved in ml module
> --
>
> Key: IGNITE-17988
> URL: https://issues.apache.org/jira/browse/IGNITE-17988
> Project: Ignite
>  Issue Type: Bug
>  Components: build, ml
>Affects Versions: 2.12, 2.13, 2.14
>Reporter: Vladimir Kornyshev
>Assignee: Vladimir Kornyshev
>Priority: Major
>  Labels: newbie
> Fix For: 2.15
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> While building ml module resolved version of jackson-core is 2.7.4 instead of 
> necessary 2.12.7, because there are transitive dependency from 
> com.dropbox.core:dropbox-core-sdk. That leads to copying wrong artifact 
> jackson-core-2.7.4.jar to modules\ml\target\libs folder.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17743) Use existing streamer batched receivers/updaters.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17743:
--
Summary: Use existing streamer batched receivers/updaters.  (was: Use 
existing batched receivers/updaters.)

> Use existing streamer batched receivers/updaters.
> -
>
> Key: IGNITE-17743
> URL: https://issues.apache.org/jira/browse/IGNITE-17743
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Minor
>  Labels: ise
>
> There are couple of useful streamer receivers ('DataStreamerCacheUpdaters'): 
> Batched, BatchedOrted. 
> Batched could be used instead of individual for atomic caches whe 
> 'allowOverwrite' is set to 'true' by default. Looks faster than Individual. 
> BatchedOrdered looks useful for transactional caches. Might be documented. 
> But they are hidden from user.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-18004) Fix compilation after merging safe time propagation

[Alexander Lapin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-18004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin updated IGNITE-18004:
-
Description: 
{code:java}
[ERROR] 
/home/alapin/workspace/ignite-3/modules/sql-engine/src/main/java/org/apache/ignite/internal/sql/engine/SqlQueryProcessor.java:[393,110]
 org.apache.ignite.internal.hlc.HybridClock is abstract; cannot be instantiated
 {code}

> Fix compilation after merging safe time propagation
> ---
>
> Key: IGNITE-18004
> URL: https://issues.apache.org/jira/browse/IGNITE-18004
> Project: Ignite
>  Issue Type: Bug
>Reporter: Alexander Lapin
>Priority: Major
>
> {code:java}
> [ERROR] 
> /home/alapin/workspace/ignite-3/modules/sql-engine/src/main/java/org/apache/ignite/internal/sql/engine/SqlQueryProcessor.java:[393,110]
>  org.apache.ignite.internal.hlc.HybridClock is abstract; cannot be 
> instantiated
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (IGNITE-18004) Fix compilation after merging safe time propagation

[Alexander Lapin (Jira)]

Alexander Lapin created IGNITE-18004:


 Summary: Fix compilation after merging safe time propagation
 Key: IGNITE-18004
 URL: https://issues.apache.org/jira/browse/IGNITE-18004
 Project: Ignite
  Issue Type: Bug
Reporter: Alexander Lapin






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17743) Use existing batched receivers/updaters.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17743:
--
Description: 
There are couple of useful streamer receivers ('DataStreamerCacheUpdaters'): 
Batched, BatchedOrted. 
Batched could be used instead of individual for atomic caches whe 
'allowOverwrite' is set to 'true' by default. Looks faster than Individual. 
BatchedOrdered looks useful for transactional caches. Might be documented. But 
they are hidden from user.

  was:
There are couple of useful streamer receivers ('DataStreamerCacheUpdaters'): 
Batched, BatchedOrted. 
Batched could be used instead of individual for atomic caches whe 
'allowOverwrite' is set to 'true' by default. Looks faster than Individual. 
BatchedOrdered looks useful for transactional caches. Might be documented. But 
they are hidden from user,


> Use existing batched receivers/updaters.
> 
>
> Key: IGNITE-17743
> URL: https://issues.apache.org/jira/browse/IGNITE-17743
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Minor
>  Labels: ise
>
> There are couple of useful streamer receivers ('DataStreamerCacheUpdaters'): 
> Batched, BatchedOrted. 
> Batched could be used instead of individual for atomic caches whe 
> 'allowOverwrite' is set to 'true' by default. Looks faster than Individual. 
> BatchedOrdered looks useful for transactional caches. Might be documented. 
> But they are hidden from user.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17743) Use existing batched receivers/updaters.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17743:
--
Summary: Use existing batched receivers/updaters.  (was: Expose 
datastreamer receivers/updaters.)

> Use existing batched receivers/updaters.
> 
>
> Key: IGNITE-17743
> URL: https://issues.apache.org/jira/browse/IGNITE-17743
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Minor
>  Labels: ise
>
> There are couple of useful streamer receivers ('DataStreamerCacheUpdaters'): 
> Batched, BatchedOrted. 
> Batched could be used instead of individual for atomic caches whe 
> 'allowOverwrite' is set to 'true' by default. Looks faster than Individual. 
> BatchedOrdered looks useful for transactional caches. Might be documented. 
> But they are hidden from user,



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17743) Expose datastreamer receivers/updaters.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17743:
--
Description: 
There are couple of useful streamer receivers ('DataStreamerCacheUpdaters'): 
Batched, BatchedOrted. 
Batched could be used instead of individual for atomic caches whe 
'allowOverwrite' is set to 'true' by default. Looks faster than Individual. 
BatchedOrdered looks useful for transactional caches. Might be documented. But 
they are hidden from user,

  was:
There are several prepared streamer receivers: batched, batchedsorted, 
individual, isolated. 
Batched could be used instead of individual for atomic caches whe 
allowOverwrite is set to True.
Datastreamer allows user to choose one. But there is no docs, examples.
Also, allowOverwrite option depends on the receiver. It should be moved or be 
bound to receiver.


> Expose datastreamer receivers/updaters.
> ---
>
> Key: IGNITE-17743
> URL: https://issues.apache.org/jira/browse/IGNITE-17743
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Minor
>  Labels: ise
>
> There are couple of useful streamer receivers ('DataStreamerCacheUpdaters'): 
> Batched, BatchedOrted. 
> Batched could be used instead of individual for atomic caches whe 
> 'allowOverwrite' is set to 'true' by default. Looks faster than Individual. 
> BatchedOrdered looks useful for transactional caches. Might be documented. 
> But they are hidden from user,



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (IGNITE-17988) Wrong version of jackson-core dependency resolved in ml module

[Vladimir Kornyshev (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Kornyshev reassigned IGNITE-17988:
---

Assignee: Vladimir Kornyshev

> Wrong version of jackson-core dependency resolved in ml module
> --
>
> Key: IGNITE-17988
> URL: https://issues.apache.org/jira/browse/IGNITE-17988
> Project: Ignite
>  Issue Type: Bug
>  Components: build, ml
>Affects Versions: 2.14
>Reporter: Vladimir Kornyshev
>Assignee: Vladimir Kornyshev
>Priority: Major
>  Labels: newbie
> Fix For: 2.15
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> While building ml module resolved version of jackson-core is 2.7.4 instead of 
> necessary 2.12.7, because there are transitive dependency from 
> com.dropbox.core:dropbox-core-sdk. That leads to copying wrong artifact 
> jackson-core-2.7.4.jar to modules\ml\target\libs folder.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-18003) ItTablesApiTest#testCreateDropTable became flaky

[Denis Chudov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-18003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Chudov updated IGNITE-18003:
--
Ignite Flags:   (was: Docs Required,Release Notes Required)

> ItTablesApiTest#testCreateDropTable became flaky
> 
>
> Key: IGNITE-18003
> URL: https://issues.apache.org/jira/browse/IGNITE-18003
> Project: Ignite
>  Issue Type: Bug
>Reporter: Denis Chudov
>Assignee: Denis Chudov
>Priority: Major
>  Labels: ignite-3
>
> Seems that index create listener in IndexManager gets table from TableManager 
> without using causality token, and sometimes it appears that table is not 
> created yet. Using the causality token should gurantee the presence of table 
> in table manager for the index manager.
> {code:java}
> [16:41:04][org.apache.ignite:ignite-runner] 2022-10-27 19:41:04:009 +0300 
> [INFO][main][ItTablesApiTest] Dropping the table ignored.
> [16:41:04][org.apache.ignite:ignite-runner] 
> org.apache.ignite.lang.TableNotFoundException: IGN-TBL-2 
> TraceId:4ecb38fa-7440-448f-8d13-4d351df39ee7 The table does not exist 
> [name="PUBLIC"."TBL1"]
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.apache.ignite.internal.table.distributed.TableManager.lambda$dropTableAsyncInternal$57(TableManager.java:1368)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.apache.ignite.internal.table.distributed.TableManager.dropTableAsyncInternal(TableManager.java:1364)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.apache.ignite.internal.table.distributed.TableManager.dropTableAsync(TableManager.java:1354)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.apache.ignite.internal.runner.app.ItTablesApiTest.dropTableIfExists(ItTablesApiTest.java:517)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.apache.ignite.internal.runner.app.ItTablesApiTest.testCreateDropTable(ItTablesApiTest.java:451)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> 

[jira] [Comment Edited] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='true'.

[Vladimir Steshin (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607891#comment-17607891
 ] 

Vladimir Steshin edited comment on IGNITE-17735 at 10/27/22 7:10 PM:
-

Datastreamer with 'allowOverwrite==true' and PRIMARY_SYNC persistent cache may 
cause heap issue or consume increased heap amount.

There is related 'perNodeParallelOperations()' setting. What discouraged, I met 
this issue with trivial research like few servers, simple cache and just trying 
data streaming with various persistence and loading settings (like 
`HeapConsumptionDataStreamerTest.src`). Think user may meet the same. The 
default value might be adjusted for persistant caches.

Streamer node may not wait for backup updates. And keep sending more and more 
streamer batches to process. The receiving node collects related to backup 
updates futures, requests. 
The same happens on backup node: collecting update incoming update requests 
stucking at disk writes. See 'DS_heap_consumption.png' for example.

Suggestion: bring reduced default parallel batches number for persistent caches 
`IgniteDataStreamer#DFLT_PARALLEL_OPS_PERSISTENT_MULTIPLIER` (PR #10343).

Did estimation benchmarks. Even in-memory benchmarks (like 
'bench_inmem_isolated_pc2.txt') shows 2 or may be 4 batches per threads seems 
enough. 

For persistent caches, `CPUs x 2` seems enough. See 
`bench_persistent_results_Isolated_pc1.txt` and 
`bench_persistent_results_Individual_pc1.txt`


was (Author: vladsz83):
Datastreamer with 'allowOverwrite==true' and PRIMARY_SYNC persistent cache may 
cause heap issue or consume increased heap amount.

There is related 'perNodeParallelOperations()' setting. What discouraged, I met 
this issue with trivial research like few servers, simple cache and just trying 
data streaming with various persistence and loading settings (like 
`HeapConsumptionDataStreamerTest.src`). Think user may meet the same. But the 
default value might be improved for persistence.

Streamer node may not wait for backup updates depending on streamer receiver, 
setting 'allowOverwrite' and cache sync mode.
And keep sending more and more streamer batches to process. The receiving node 
collects related to backup updates futures, requests. 
The same happens on backup node: collecting update incoming update requests 
stucking at disk writes. See 'DS_heap_consumption.png' for example.

Suggestion: bring reduced default parallel batches number for persistent caches 
`IgniteDataStreamer#DFLT_PARALLEL_OPS_PERSISTENT_MULTIPLIER` (PR #10343).

Did estimation benchmarks. Even in-memory benchmarks (like 
'bench_inmem_isolated_pc2.txt') shows 2 or may be 4 batches per threads seems 
enough. 

For persistent caches, `CPUs x 2` seems enough. See 
`bench_persistent_results_Isolated_pc1.txt` and 
`bench_persistent_results_Individual_pc1.txt`

> Datastreamer may consume heap with allowOverwtire=='true'.
> --
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_individual_pc2.txt, 
> bench_inmem_isolated_pc2.txt, bench_persistent_full_Individual_pc1.txt, 
> bench_persistent_results_Individual_pc1.txt, 
> bench_persistent_results_Isolated_pc1.txt
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (IGNITE-18003) ItTablesApiTest#testCreateDropTable became flaky

[Denis Chudov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-18003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Chudov reassigned IGNITE-18003:
-

Assignee: Denis Chudov

> ItTablesApiTest#testCreateDropTable became flaky
> 
>
> Key: IGNITE-18003
> URL: https://issues.apache.org/jira/browse/IGNITE-18003
> Project: Ignite
>  Issue Type: Bug
>Reporter: Denis Chudov
>Assignee: Denis Chudov
>Priority: Major
>  Labels: ignite-3
>
> Seems that index create listener in IndexManager gets table from TableManager 
> without using causality token, and sometimes it appears that table is not 
> created yet. Using the causality token should gurantee the presence of table 
> in table manager for the index manager.
> {code:java}
> [16:41:04][org.apache.ignite:ignite-runner] 2022-10-27 19:41:04:009 +0300 
> [INFO][main][ItTablesApiTest] Dropping the table ignored.
> [16:41:04][org.apache.ignite:ignite-runner] 
> org.apache.ignite.lang.TableNotFoundException: IGN-TBL-2 
> TraceId:4ecb38fa-7440-448f-8d13-4d351df39ee7 The table does not exist 
> [name="PUBLIC"."TBL1"]
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.apache.ignite.internal.table.distributed.TableManager.lambda$dropTableAsyncInternal$57(TableManager.java:1368)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.apache.ignite.internal.table.distributed.TableManager.dropTableAsyncInternal(TableManager.java:1364)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.apache.ignite.internal.table.distributed.TableManager.dropTableAsync(TableManager.java:1354)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.apache.ignite.internal.runner.app.ItTablesApiTest.dropTableIfExists(ItTablesApiTest.java:517)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.apache.ignite.internal.runner.app.ItTablesApiTest.testCreateDropTable(ItTablesApiTest.java:451)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> 

[jira] [Comment Edited] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='true'.

[Vladimir Steshin (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607891#comment-17607891
 ] 

Vladimir Steshin edited comment on IGNITE-17735 at 10/27/22 6:59 PM:
-

Datastreamer with 'allowOverwrite==true' and PRIMARY_SYNC persistent cache may 
cause heap issue or consume increased heap amount.

There is related 'perNodeParallelOperations()' setting. What discouraged, I met 
this issue with trivial research like few servers, simple cache and just trying 
data streaming with various persistence and loading settings (like 
`HeapConsumptionDataStreamerTest.src`). Think user may meet the same. But the 
default value might be improved for persistence.

Streamer node may not wait for backup updates depending on streamer receiver, 
setting 'allowOverwrite' and cache sync mode.
And keep sending more and more streamer batches to process. The receiving node 
collects related to backup updates futures, requests. 
The same happens on backup node: collecting update incoming update requests 
stucking at disk writes. See 'DS_heap_consumption.png' for example.

Suggestion: bring reduced default parallel batches number for persistent caches 
`IgniteDataStreamer#DFLT_PARALLEL_OPS_PERSISTENT_MULTIPLIER` (PR #10343).

Did estimation benchmarks. Even in-memory benchmarks (like 
'bench_inmem_isolated_pc2.txt') shows 2 or may be 4 batches per threads seems 
enough. 

For persistent caches, `CPUs x 2` seems enough. See 
`bench_persistent_results_Isolated_pc1.txt` and 
`bench_persistent_results_Individual_pc1.txt`


was (Author: vladsz83):
Datastreamer with 'allowOverwrite==true' and PRIMARY_SYNC persistent cache may 
cause heap issue or consume increased heap amount.

There is related 'perNodeParallelOperations()' setting. What discouraged, I met 
this issue with trivial research like few servers, simple cache and just trying 
data streaming with various persistence and loading settings (like 
`HeapConsumptionDataStreamerTest.src`). Think user may meet the same. But the 
default value might be improved for persistence.

Streamer node may not wait for backup updates depending on streamer receiver, 
setting 'allowOverwrite' and cache sync mode.
And keep sending more and more streamer batches to process. The receiving node 
collects related to backup updates futures, requests. 
The same happens on backup node: collecting update incoming update requests 
stucking at disk writes. See 'DS_heap_consumption.png' for example.

Suggestion: bring reduced default parallel batches number for persistent caches 
`IgniteDataStreamer#DFLT_PARALLEL_OPS_PERSISTENT_MULTIPLIER` (PR #10343).
Or use per-internal-receiver setting 
`InternalUpdater#perNodeParallelOperations()` (PR #10351)

Did estimation benchmarks. Even in-memory benchmarks (like 
'bench_inmem_isolated_pc2.txt') shows 2 or may be 4 batches per threads seems 
enough. 

For persistent caches, `CPUs x 2` seems enough. See 
`bench_persistent_results_Isolated_pc1.txt` and 
`bench_persistent_results_Individual_pc1.txt`

> Datastreamer may consume heap with allowOverwtire=='true'.
> --
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_individual_pc2.txt, 
> bench_inmem_isolated_pc2.txt, bench_persistent_full_Individual_pc1.txt, 
> bench_persistent_results_Individual_pc1.txt, 
> bench_persistent_results_Isolated_pc1.txt
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17983) Fix work with the last applied index in RAFT

[Kirill Tkalenko (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirill Tkalenko updated IGNITE-17983:
-
Reviewer: Ivan Bessonov

> Fix work with the last applied index in RAFT
> 
>
> Key: IGNITE-17983
> URL: https://issues.apache.org/jira/browse/IGNITE-17983
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
>  Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> At the moment, *MvPartitionStorage* and *TxStateStorage* are used for a 
> partition raft group, while only *MvPartitionStorage#persistedIndex* of 
> *MvPartitionStorage#lastAppliedIndex* is used, which is not correct.
> On recovery, we need to use a minimum of 
> *MvPartitionStorage#lastAppliedIndex* and *TxStateStorage#lastAppliedIndex* 
> so as not to lose data for one of the storage.
> When taking a snapshot for a full rebalance, we should use the maximum of 
> *MvPartitionStorage#lastAppliedIndex* and *TxStateStorage#lastAppliedIndex* 
> so that we can load up-to-date data from the leader.
> Etc.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='true'.

[Vladimir Steshin (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607891#comment-17607891
 ] 

Vladimir Steshin edited comment on IGNITE-17735 at 10/27/22 6:58 PM:
-

Datastreamer with 'allowOverwrite==true' and PRIMARY_SYNC persistent cache may 
cause heap issue or consume increased heap amount.

There is related 'perNodeParallelOperations()' setting. What discouraged, I met 
this issue with trivial research like few servers, simple cache and just trying 
data streaming with various persistence and loading settings (like 
`HeapConsumptionDataStreamerTest.src`). Think user may meet the same. But the 
default value might be improved for persistence.

Streamer node may not wait for backup updates depending on streamer receiver, 
setting 'allowOverwrite' and cache sync mode.
And keep sending more and more streamer batches to process. The receiving node 
collects related to backup updates futures, requests. 
The same happens on backup node: collecting update incoming update requests 
stucking at disk writes. See 'DS_heap_consumption.png' for example.

Suggestion: bring reduced default parallel batches number for persistent caches 
`IgniteDataStreamer#DFLT_PARALLEL_OPS_PERSISTENT_MULTIPLIER` (PR #10343).
Or use per-internal-receiver setting 
`InternalUpdater#perNodeParallelOperations()` (PR #10351)

Did estimation benchmarks. Even in-memory benchmarks (like 
'bench_inmem_isolated_pc2.txt') shows 2 or may be 4 batches per threads seems 
enough. 

For persistent caches, `CPUs x 2` seems enough. See 
`bench_persistent_results_Isolated_pc1.txt` and 
`bench_persistent_results_Individual_pc1.txt`


was (Author: vladsz83):
Datastreamer with 'allowOverwrite==true' and PRIMARY_SYNC persistent cache may 
cause heap issue or consume increased heap amount.

There is related 'perNodeParallelOperations()' setting. Probably not an issue 
at all. What discouraged, I met this issue with trivial research like few 
servers, simple cache and just trying data streaming with various persistence 
and loading settings (like `HeapConsumptionDataStreamerTest.src`). Think user 
may meet the same. But the default value might be improved for persistence.

Streamer node may not wait for backup updates depending on streamer receiver, 
setting 'allowOverwrite' and cache sync mode.
And keep sending more and more streamer batches to process. The receiving node 
collects related to backup updates futures, requests. 
The same happens on backup node: collecting update incoming update requests 
stucking at disk writes. See 'DS_heap_consumption.png' for example.

Suggestion: bring reduced default parallel batches number for persistent caches 
`IgniteDataStreamer#DFLT_PARALLEL_OPS_PERSISTENT_MULTIPLIER` (PR #10343).
Or use per-internal-receiver setting 
`InternalUpdater#perNodeParallelOperations()` (PR #10351)

Did estimation benchmarks. Even in-memory benchmarks (like 
'bench_inmem_isolated_pc2.txt') shows 2 or may be 4 batches per threads seems 
enough. 

For persistent caches, `CPUs x 2` seems enough. See 
`bench_persistent_results_Isolated_pc1.txt` and 
`bench_persistent_results_Individual_pc1.txt`

> Datastreamer may consume heap with allowOverwtire=='true'.
> --
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_individual_pc2.txt, 
> bench_inmem_isolated_pc2.txt, bench_persistent_full_Individual_pc1.txt, 
> bench_persistent_results_Individual_pc1.txt, 
> bench_persistent_results_Isolated_pc1.txt
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='true'.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Summary: Datastreamer may consume heap with allowOverwtire=='true'.  (was: 
Datastreamer may consume heap with allowOverwtire=='false'.)

> Datastreamer may consume heap with allowOverwtire=='true'.
> --
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_individual_pc2.txt, 
> bench_inmem_isolated_pc2.txt, bench_persistent_full_Individual_pc1.txt, 
> bench_persistent_results_Individual_pc1.txt, 
> bench_persistent_results_Isolated_pc1.txt
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='false'.

[Vladimir Steshin (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607891#comment-17607891
 ] 

Vladimir Steshin edited comment on IGNITE-17735 at 10/27/22 6:37 PM:
-

Datastreamer with 'allowOverwrite==true' and PRIMARY_SYNC persistent cache may 
cause heap issue or consume increased heap amount.

There is related 'perNodeParallelOperations()' setting. Probably not an issue 
at all. What discouraged, I met this issue with trivial research like few 
servers, simple cache and just trying data streaming with various persistence 
and loading settings (like `HeapConsumptionDataStreamerTest.src`). Think user 
may meet the same. But the default value might be improved for persistence.

Streamer node may not wait for backup updates depending on streamer receiver, 
setting 'allowOverwrite' and cache sync mode.
And keep sending more and more streamer batches to process. The receiving node 
collects related to backup updates futures, requests. 
The same happens on backup node: collecting update incoming update requests 
stucking at disk writes. See 'DS_heap_consumption.png' for example.

Suggestion: bring reduced default parallel batches number for persistent caches 
`IgniteDataStreamer#DFLT_PARALLEL_OPS_PERSISTENT_MULTIPLIER` (PR #10343).
Or use per-internal-receiver setting 
`InternalUpdater#perNodeParallelOperations()` (PR #10351)

Did estimation benchmarks. Even in-memory benchmarks (like 
'bench_inmem_isolated_pc2.txt') shows 2 or may be 4 batches per threads seems 
enough. 

For persistent caches, `CPUs x 2` seems enough. See 
`bench_persistent_results_Isolated_pc1.txt` and 
`bench_persistent_results_Individual_pc1.txt`


was (Author: vladsz83):
Datastreamer with 'allowOverwrite==true' and PRIMARY_SYNC persistent cache may 
cause heap issue or consume increased heap amount.

Streamer node may not wait for backup updates depending on streamer receiver, 
setting 'allowOverwrite' and cache sync mode.
And keep sending more and more streamer batches to process. The receiving node 
collects related to backup updates futures, requests. 
The same happens on backup node: collecting update incoming update requests 
stucking at disk writes. See 'DS_heap_consumption.png' for example.

There is related 'perNodeParallelOperations()' setting. Probably not an issue 
at all. What discouraged, I met this issue with trivial research like few 
servers, simple cache and just trying data streaming with various persistence 
and loading settings (like `HeapConsumptionDataStreamerTest.src`). Think user 
may meet the same. But the default value might be improved for persistence.

Suggestion: bring reduced default parallel batches number for persistent caches 
`IgniteDataStreamer#DFLT_PARALLEL_OPS_PERSISTENT_MULTIPLIER` (PR #10343).
Or use per-internal-receiver setting 
`InternalUpdater#perNodeParallelOperations()` (PR #10351)

Did estimation benchmarks. Even in-memory benchmarks (like 
'bench_inmem_isolated_pc2.txt') shows 2 or may be 4 batches per threads seems 
enough. 

For persistent caches, `CPUs x 2` seems enough. See 
`bench_persistent_results_Isolated_pc1.txt` and 
`bench_persistent_results_Individual_pc1.txt`

> Datastreamer may consume heap with allowOverwtire=='false'.
> ---
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_individual_pc2.txt, 
> bench_inmem_isolated_pc2.txt, bench_persistent_full_Individual_pc1.txt, 
> bench_persistent_results_Individual_pc1.txt, 
> bench_persistent_results_Isolated_pc1.txt
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='false'.

[Vladimir Steshin (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17607891#comment-17607891
 ] 

Vladimir Steshin edited comment on IGNITE-17735 at 10/27/22 6:34 PM:
-

Datastreamer with 'allowOverwrite==true' and PRIMARY_SYNC persistent cache may 
cause heap issue or consume increased heap amount.

Streamer node may not wait for backup updates depending on streamer receiver, 
setting 'allowOverwrite' and cache sync mode.
And keep sending more and more streamer batches to process. The receiving node 
collects related to backup updates futures, requests. 
The same happens on backup node: collecting update incoming update requests 
stucking at disk writes. See 'DS_heap_consumption.png' for example.

There is related 'perNodeParallelOperations()' setting. Probably not an issue 
at all. What discouraged, I met this issue with trivial research like few 
servers, simple cache and just trying data streaming with various persistence 
and loading settings (like `HeapConsumptionDataStreamerTest.src`). Think user 
may meet the same. But the default value might be improved for persistence.

Suggestion: bring reduced default parallel batches number for persistent caches 
`IgniteDataStreamer#DFLT_PARALLEL_OPS_PERSISTENT_MULTIPLIER` (PR #10343).
Or use per-internal-receiver setting 
`InternalUpdater#perNodeParallelOperations()` (PR #10351)

Did estimation benchmarks. Even in-memory benchmarks (like 
'bench_inmem_isolated_pc2.txt') shows 2 or may be 4 batches per threads seems 
enough. 

For persistent caches, `CPUs x 2` seems enough. See 
`bench_persistent_results_Isolated_pc1.txt` and 
`bench_persistent_results_Individual_pc1.txt`


was (Author: vladsz83):
Datastreamer with '_allowOverwrite==true_' and _ATOMIC/PRIMARY_SYNC_ persistent 
cache may consume heap. 

The streamer had been created before the persistence. It's default setting are 
still for in-memory caches. Streamer decides how many data send to a node based 
on CPU number. Probably it's not best approach for persistent caches.

There is related 'perNodeParallelOperations()' setting. But the defaults might 
be adjusted for persistence.

Suggestion: reduce default max unresponded streamer batches for persistent 
caches. There is no reason to send more than 4-8-16 unresponded batches because 
they stuck at disk writes, WAL writes, page replacements, WAL rolling, GCs and 
so on. 

The problem is that certain streamer receiver might not wait for backup updates 
on loading node and keep sending update batches again and again. Default 
_Individual_ receiver (when _allowOverwrite_ if _true_) uses _cache.put()_. 
Every put creates additional backup requests. But current streamer batch 
request is already responded to. Next batch updates is accepted. Nodes start 
accumulating related to records update structures in the heap. Some JFR screens 
attached.

 See `DataStreamProcessorSelfTest.testAtomicPrimarySyncStability()`, 
`JmhStreamerReceiverBenchmark`.

> Datastreamer may consume heap with allowOverwtire=='false'.
> ---
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_individual_pc2.txt, 
> bench_inmem_isolated_pc2.txt, bench_persistent_full_Individual_pc1.txt, 
> bench_persistent_results_Individual_pc1.txt, 
> bench_persistent_results_Isolated_pc1.txt
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='false'.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: bench_persistent_full_Individual_pc1.txt

> Datastreamer may consume heap with allowOverwtire=='false'.
> ---
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_individual_pc2.txt, 
> bench_inmem_isolated_pc2.txt, bench_persistent_full_Individual_pc1.txt, 
> bench_persistent_results_Individual_pc1.txt, 
> bench_persistent_results_Isolated_pc1.txt
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-18002) Revert accident changes to project files

[Konstantin Orlov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-18002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Orlov updated IGNITE-18002:
--
Fix Version/s: 3.0.0-beta2

> Revert accident changes to project files
> 
>
> Key: IGNITE-18002
> URL: https://issues.apache.org/jira/browse/IGNITE-18002
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Konstantin Orlov
>Assignee: Konstantin Orlov
>Priority: Major
>  Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> Need to revert changes to files Project.xml, codeStyleConfig.xml, and 
> Project_Default.xml, which were accidentally merged within IGNITE-17859
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (IGNITE-17330) Support RO TX by SQL

[Evgeny Stanilovsky (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Evgeny Stanilovsky resolved IGNITE-17330.
-
Resolution: Fixed

> Support RO TX by SQL
> 
>
> Key: IGNITE-17330
> URL: https://issues.apache.org/jira/browse/IGNITE-17330
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Reporter: Yury Gerzhedovich
>Assignee: Evgeny Stanilovsky
>Priority: Major
>  Labels: ignite-3
> Fix For: 3.0.0-beta1
>
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> SQL use RW transaction mode. It's not efficient in case we need just read 
> consistent data.
> After implementing IGNITE-17260 we should add support RO transaction for SQL 
> queries. Seems required just use transaction method with timestamp 
> (org.apache.ignite.hlc.HybridClock#now) as parameter.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (IGNITE-17263) Implement leader to replica safe time propagation

[Alexander Lapin (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-17263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625230#comment-17625230
 ] 

Alexander Lapin commented on IGNITE-17263:
--

[~Denis Chudov] Looks good. Some follow-up tickets are required, however core 
implementation definitely should be merged. Thank you!

> Implement leader to replica safe time propagation
> -
>
> Key: IGNITE-17263
> URL: https://issues.apache.org/jira/browse/IGNITE-17263
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexander Lapin
>Assignee: Denis Chudov
>Priority: Major
>  Labels: ignite-3, transaction3_ro
> Fix For: 3.0.0-beta1
>
> Attachments: Screenshot from 2022-07-06 16-48-30.png, Screenshot from 
> 2022-07-06 16-48-41.png
>
>
> In order to perform replica reads, it's required either to use read index or 
> check the safe time. Let's recall corresponding section from tx design 
> document.
> RO transactions can be executed on non-primary replicas. write intent 
> resolution doesn’t help because a write intent for a committed transaction 
> may not be yet replicated to the replica. To mitigate this issue, it’s enough 
> to run readIndex on each mapped partition leader, fetch the commit index and 
> wait on a replica until it’s applied. This will guarantee that all required 
> write intents are replicated and present locally. After that the normal write 
> intern resolution should do the job.
> There is a second option, which doesn’t require the network RTT. We can use a 
> special low watermark timestamp (safeTs) per replication group, which 
> corresponds to the apply index of a replicated entry, so then an apply index 
> is advanced during the replication, then the safeTs is monotonically 
> incremented too. The HLC used for safeTs advancing is assigned to a 
> replicated entry in an ordered way.
> Special measures are needed to periodically advance the safeTs if no updates 
> are happening. It’s enough to use a special replication command for this 
> purpose.
> All we need during RO txn is to wait until a safeTs advances past the RO txn 
> readTs. 
>  !Screenshot from 2022-07-06 16-48-30.png! 
> In the picture we have two concurrent transactions mapped to the same 
> partition: T1 and T2.
> OpReq(w1(x)) and OpReq(w2(x)) are received concurrently. Each write intent is 
> assigned a timestamp in a monotonic order consistent with the replication 
> order. This can be for example done when replication entries are dequeued for 
> processing by replication protocol (we assume entries are replicated 
> successively.
> It’s not enough only to wait for safeTs - it may never happen due to absence 
> of activity in the partition. Consider the next diagram:
>  !Screenshot from 2022-07-06 16-48-41.png! 
> We need an additional safeTsSync command to propagate a safeTs event in case 
> there are no updates in the partition.
> We need to linerialize safe time updates in all cases including leader 
> change. So we need a guarantee that safe time on non-primary replicas never 
> will be greater than HLC on leader (as we assume that primary replica is 
> colocated with leader). We are going to solve this problem by associating 
> every potential value of safeTime (propagated to the replica from leader via 
> appendEntries) with some log index, and this value (safe time candidate) 
> should be applied as new safe time value at the moment when corresponding 
> index is committed.
> Hence, the safeTimeSyncCommand also should be a Raft write command.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='false'.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: (was: bench_persistent_full_Individual_pc1.txt)

> Datastreamer may consume heap with allowOverwtire=='false'.
> ---
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_individual_pc2.txt, 
> bench_inmem_isolated_pc2.txt, bench_persistent_results_Individual_pc1.txt, 
> bench_persistent_results_Isolated_pc1.txt
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='false'.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: (was: bench_inmem_batched_pc2.txt)

> Datastreamer may consume heap with allowOverwtire=='false'.
> ---
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_individual_pc2.txt, 
> bench_inmem_isolated_pc2.txt, bench_persistent_results_Individual_pc1.txt, 
> bench_persistent_results_Isolated_pc1.txt
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with allowOverwtire=='false'.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Summary: Datastreamer may consume heap with allowOverwtire=='false'.  (was: 
Datastreamer may consume heap with default settings.)

> Datastreamer may consume heap with allowOverwtire=='false'.
> ---
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_batched_pc2.txt, 
> bench_inmem_individual_pc2.txt, bench_inmem_isolated_pc2.txt, 
> bench_persistent_full_Individual_pc1.txt, 
> bench_persistent_results_Individual_pc1.txt, 
> bench_persistent_results_Isolated_pc1.txt
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-18003) ItTablesApiTest#testCreateDropTable became flaky

[Denis Chudov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-18003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Chudov updated IGNITE-18003:
--
Description: 
Seems that index create listener in IndexManager gets table from TableManager 
without using causality token, and sometimes it appears that table is not 
created yet. Using the causality token should gurantee the presence of table in 
table manager for the index manager.
{code:java}
[16:41:04][org.apache.ignite:ignite-runner] 2022-10-27 19:41:04:009 +0300 
[INFO][main][ItTablesApiTest] Dropping the table ignored.
[16:41:04][org.apache.ignite:ignite-runner] 
org.apache.ignite.lang.TableNotFoundException: IGN-TBL-2 
TraceId:4ecb38fa-7440-448f-8d13-4d351df39ee7 The table does not exist 
[name="PUBLIC"."TBL1"]
[16:41:04][org.apache.ignite:ignite-runner] at 
org.apache.ignite.internal.table.distributed.TableManager.lambda$dropTableAsyncInternal$57(TableManager.java:1368)
[16:41:04][org.apache.ignite:ignite-runner] at 
java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753)
[16:41:04][org.apache.ignite:ignite-runner] at 
java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731)
[16:41:04][org.apache.ignite:ignite-runner] at 
java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.apache.ignite.internal.table.distributed.TableManager.dropTableAsyncInternal(TableManager.java:1364)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.apache.ignite.internal.table.distributed.TableManager.dropTableAsync(TableManager.java:1354)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.apache.ignite.internal.runner.app.ItTablesApiTest.dropTableIfExists(ItTablesApiTest.java:517)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.apache.ignite.internal.runner.app.ItTablesApiTest.testCreateDropTable(ItTablesApiTest.java:451)
[16:41:04][org.apache.ignite:ignite-runner] at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[16:41:04][org.apache.ignite:ignite-runner] at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[16:41:04][org.apache.ignite:ignite-runner] at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[16:41:04][org.apache.ignite:ignite-runner] at 
java.base/java.lang.reflect.Method.invoke(Method.java:566)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
[16:41:04][org.apache.ignite:ignite-runner] at 

[jira] [Created] (IGNITE-18003) ItTablesApiTest#testCreateDropTable became flaky

[Denis Chudov (Jira)]

Denis Chudov created IGNITE-18003:
-

 Summary: ItTablesApiTest#testCreateDropTable became flaky
 Key: IGNITE-18003
 URL: https://issues.apache.org/jira/browse/IGNITE-18003
 Project: Ignite
  Issue Type: Bug
Reporter: Denis Chudov



{code:java}
[16:41:04][org.apache.ignite:ignite-runner] 2022-10-27 19:41:04:009 +0300 
[INFO][main][ItTablesApiTest] Dropping the table ignored.
[16:41:04][org.apache.ignite:ignite-runner] 
org.apache.ignite.lang.TableNotFoundException: IGN-TBL-2 
TraceId:4ecb38fa-7440-448f-8d13-4d351df39ee7 The table does not exist 
[name="PUBLIC"."TBL1"]
[16:41:04][org.apache.ignite:ignite-runner] at 
org.apache.ignite.internal.table.distributed.TableManager.lambda$dropTableAsyncInternal$57(TableManager.java:1368)
[16:41:04][org.apache.ignite:ignite-runner] at 
java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753)
[16:41:04][org.apache.ignite:ignite-runner] at 
java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731)
[16:41:04][org.apache.ignite:ignite-runner] at 
java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.apache.ignite.internal.table.distributed.TableManager.dropTableAsyncInternal(TableManager.java:1364)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.apache.ignite.internal.table.distributed.TableManager.dropTableAsync(TableManager.java:1354)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.apache.ignite.internal.runner.app.ItTablesApiTest.dropTableIfExists(ItTablesApiTest.java:517)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.apache.ignite.internal.runner.app.ItTablesApiTest.testCreateDropTable(ItTablesApiTest.java:451)
[16:41:04][org.apache.ignite:ignite-runner] at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[16:41:04][org.apache.ignite:ignite-runner] at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[16:41:04][org.apache.ignite:ignite-runner] at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[16:41:04][org.apache.ignite:ignite-runner] at 
java.base/java.lang.reflect.Method.invoke(Method.java:566)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
[16:41:04][org.apache.ignite:ignite-runner] at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
[16:41:04][org.apache.ignite:ignite-runner] at 

[jira] [Updated] (IGNITE-18003) ItTablesApiTest#testCreateDropTable became flaky

[Denis Chudov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-18003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denis Chudov updated IGNITE-18003:
--
Labels: ignite-3  (was: )

> ItTablesApiTest#testCreateDropTable became flaky
> 
>
> Key: IGNITE-18003
> URL: https://issues.apache.org/jira/browse/IGNITE-18003
> Project: Ignite
>  Issue Type: Bug
>Reporter: Denis Chudov
>Priority: Major
>  Labels: ignite-3
>
> {code:java}
> [16:41:04][org.apache.ignite:ignite-runner] 2022-10-27 19:41:04:009 +0300 
> [INFO][main][ItTablesApiTest] Dropping the table ignored.
> [16:41:04][org.apache.ignite:ignite-runner] 
> org.apache.ignite.lang.TableNotFoundException: IGN-TBL-2 
> TraceId:4ecb38fa-7440-448f-8d13-4d351df39ee7 The table does not exist 
> [name="PUBLIC"."TBL1"]
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.apache.ignite.internal.table.distributed.TableManager.lambda$dropTableAsyncInternal$57(TableManager.java:1368)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.apache.ignite.internal.table.distributed.TableManager.dropTableAsyncInternal(TableManager.java:1364)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.apache.ignite.internal.table.distributed.TableManager.dropTableAsync(TableManager.java:1354)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.apache.ignite.internal.runner.app.ItTablesApiTest.dropTableIfExists(ItTablesApiTest.java:517)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.apache.ignite.internal.runner.app.ItTablesApiTest.testCreateDropTable(ItTablesApiTest.java:451)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> [16:41:04][org.apache.ignite:ignite-runner]   at 
> 

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with default settings.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: bench_persistent_results_Isolated_pc1.txt

> Datastreamer may consume heap with default settings.
> 
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_batched_pc2.txt, 
> bench_inmem_individual_pc2.txt, bench_inmem_isolated_pc2.txt, 
> bench_persistent_full_Individual_pc1.txt, 
> bench_persistent_results_Individual_pc1.txt, 
> bench_persistent_results_Isolated_pc1.txt
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-18002) Revert accident changes to project files

[Yury Gerzhedovich (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-18002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yury Gerzhedovich updated IGNITE-18002:
---
Labels: ignite-3  (was: )

> Revert accident changes to project files
> 
>
> Key: IGNITE-18002
> URL: https://issues.apache.org/jira/browse/IGNITE-18002
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Konstantin Orlov
>Assignee: Konstantin Orlov
>Priority: Major
>  Labels: ignite-3
>
> Need to revert changes to files Project.xml, codeStyleConfig.xml, and 
> Project_Default.xml, which were accidentally merged within IGNITE-17859
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (IGNITE-15629) Cluster Management API

[Nikolay Izhikov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-15629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikolay Izhikov reassigned IGNITE-15629:


Assignee: Nikolay Izhikov  (was: Maxim Muzafarov)

> Cluster Management API
> --
>
> Key: IGNITE-15629
> URL: https://issues.apache.org/jira/browse/IGNITE-15629
> Project: Ignite
>  Issue Type: Task
>Reporter: Maxim Muzafarov
>Assignee: Nikolay Izhikov
>Priority: Major
>  Labels: iep-81, important, ise
>
> Ignite must provide cluster management internal API in that way the other 
> available user APIs like REST, CLI or JMX won't require the source code 
> changes.
> The following features must be available out of the box:
> - man pages and documentation autogeneration
> - registering new commands from plugins



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-18002) Revert accident changes to project files

[Konstantin Orlov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-18002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Orlov updated IGNITE-18002:
--
Summary: Revert accident changes to project files  (was: Revert accident 
changes to a project files)

> Revert accident changes to project files
> 
>
> Key: IGNITE-18002
> URL: https://issues.apache.org/jira/browse/IGNITE-18002
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Konstantin Orlov
>Assignee: Konstantin Orlov
>Priority: Major
>
> Need to revert changes to files Project.xml, codeStyleConfig.xml, and 
> Project_Default.xml, which were accidentally merged within IGNITE-17859
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (IGNITE-18002) Revert accident changes to a project files

[Konstantin Orlov (Jira)]

Konstantin Orlov created IGNITE-18002:
-

 Summary: Revert accident changes to a project files
 Key: IGNITE-18002
 URL: https://issues.apache.org/jira/browse/IGNITE-18002
 Project: Ignite
  Issue Type: Improvement
Reporter: Konstantin Orlov
Assignee: Konstantin Orlov


Need to revert changes to files Project.xml, codeStyleConfig.xml, and 
Project_Default.xml, which were accidentally merged within IGNITE-17859

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17865) Disable partition ranges in log messages about rebalance or PME

[Ilya Shishkov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ilya Shishkov updated IGNITE-17865:
---
Description: 
When you need to analyze cases of partitions inconsistency, full list of 
partitions is needed, but there is an optimization which replaces list of 
consecutive partitions with ranges. So, as you can see below, we can not find 
partition 2 by simple text search:
{quote}2021-08-11 23:12:11.338 [WARN 
][sys-#213|#213][org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl]
 Partitions have been scheduled for rebalancing due to outdated update counter 
[grp=group, readyTopVer=AffinityTopologyVersion [topVer=33, minorTopVer=0], 
topVer=AffinityTopologyVersion [topVer=33, minorTopVer=0], 
nodeId=8e05997a-4ff7-4fdb-9480-cfbcefa8bbf5, 
partsFull=[{color:#ff}*1-3*{color}, ...], partsHistorical=[]]
{quote}
{quote}2021-08-11 23:12:11.338 [WARN 
][sys-#213|#213][org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture]
 Partitions weren't present in any history reservation: [[[grp=grp2 
part=[[{color:#ff}*1-3*{color}]]]
{quote}
We need to remove this optimization, because it can lead to mistakes in logs 
analysis.

Partition ranges are formed in _GridToStringBuilder#compact_ method, which is 
used to log of partition lists (except one place with exception and tests). 
Below are such places (without usages in tests):
 # {_}GridClientPartitionTopology#resetOwners{_}: 
[L1311|https://github.com/apache/ignite/blob/8ec5d50896a2b5f2d008d0bb8f67f7f3d7cdf584/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/topology/GridClientPartitionTopology.java#L1311],
 
[L1312|https://github.com/apache/ignite/blob/8ec5d50896a2b5f2d008d0bb8f67f7f3d7cdf584/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/topology/GridClientPartitionTopology.java#L1312]
 (WARN)
 # {_}GridDhtPartitionDemander#handleSupplyMessage{_}: 
[L561|https://github.com/apache/ignite/blob/147e03177aeadee03cb4925649c03633ce6be192/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPartitionDemander.java#L561]
 (ERROR)
 # {_}GridDhtPartitionDemander.RebalanceFuture#requestPartitions0{_}: 
[L1434|https://github.com/apache/ignite/blob/147e03177aeadee03cb4925649c03633ce6be192/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPartitionDemander.java#L1434],
 
[L1435|https://github.com/apache/ignite/blob/147e03177aeadee03cb4925649c03633ce6be192/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPartitionDemander.java#L1435]
 (INFO)
 # {_}GridDhtPartitionsExchangeFuture#printPartitionRebalancingFully{_}: 
[L4282|https://github.com/apache/ignite/blob/bc843a5b40a6da0e2bfcb77857bea499ab9a4512/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPartitionsExchangeFuture.java#L4282]
 (INFO)
 # {_}GridDhtPartitionSupplier#handleDemandMessage{_}: 
[L276|https://github.com/apache/ignite/blob/00988d20af19485585e98e885c610a704640c083/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPartitionSupplier.java#L276]
 (DEBUG)
 # {_}GridDhtPartitionTopologyImpl#detectLostPartitions{_}: 
[L2281|https://github.com/apache/ignite/blob/8ec5d50896a2b5f2d008d0bb8f67f7f3d7cdf584/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/topology/GridDhtPartitionTopologyImpl.java#L2281]
 (WARN)
 # {_}GridDhtPartitionTopologyImpl#resetOwners{_}: 
[L2445|https://github.com/apache/ignite/blob/8ec5d50896a2b5f2d008d0bb8f67f7f3d7cdf584/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/topology/GridDhtPartitionTopologyImpl.java#L2445]
 (WARN)
 # {_}PartitionsEvictManager{_}: called in _#toString_ at 
[L254|https://github.com/apache/ignite/blob/9021f783e9453375482c9b255a42ca827e091daa/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/topology/PartitionsEvictManager.java#L254],
 result used in _#showProgress_ at 
[L199|https://github.com/apache/ignite/blob/9021f783e9453375482c9b255a42ca827e091daa/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/topology/PartitionsEvictManager.java#L199]
 (INFO)
 # {_}SnapshotFutureTask#onMarkCheckpointBegin{_}: 
[L400|https://github.com/apache/ignite/blob/52121c2e4d4792e92748f15f0d58bc22b7f2259e/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/snapshot/SnapshotFutureTask.java#L400]
 (in exception message), 

[jira] [Resolved] (IGNITE-17205) Need to create primary index implementation

[Alexander Lapin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin resolved IGNITE-17205.
--
Resolution: Won't Fix

Was implemented within https://issues.apache.org/jira/browse/IGNITE-17859

> Need to create primary index implementation
> ---
>
> Key: IGNITE-17205
> URL: https://issues.apache.org/jira/browse/IGNITE-17205
> Project: Ignite
>  Issue Type: Task
>Reporter: Sergey Uttsel
>Priority: Major
>  Labels: ignite-3
>
> Now VersionedRowStore use a dummy primary index to operate with data in 
> storage.
> Need to create a primary index implementation to replace it.
> By the way we need to think about use cases when we don't have a key and need 
> to do a scan operation, for example sql queries. May be we don't need a 
> primary index in VersionedRowStore.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (IGNITE-17996) Strange behavior of SQL CASE expression

[Konstantin Orlov (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-17996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625171#comment-17625171
 ] 

Konstantin Orlov commented on IGNITE-17996:
---

Please see the comments to the IGNITE-17997

> Strange behavior of SQL CASE expression
> ---
>
> Key: IGNITE-17996
> URL: https://issues.apache.org/jira/browse/IGNITE-17996
> Project: Ignite
>  Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Andrey Khitrin
>Priority: Major
>
> I observe strange behavior in the next scenario:
>  
> {code:java}
> sql-cli> create table xx (f1 int primary key);
> Updated 0 rows.
> sql-cli> insert into xx values (1);
> Updated 1 rows.
> sql-cli> insert into xx values (2);
> Updated 1 rows.
> sql-cli> select f1, case when f1 < 2 then 'foo' else 'barbar' end as s, 
> length(case when f1 < 2 then 'foo' else 'barbar' end) as ls from xx;
> ╔╤╤╗
> ║ F1 │ S      │ LS ║
> ╠╪╪╣
> ║ 2  │ barbar │ 6  ║
> ╟┼┼╢
> ║ 1  │ foo    │ 6  ║
> ╚╧╧╝
>  {code}
> I expect `CASE` to return 'foo' value, but de-facto it returns 'foo   ' 
> ('foo' with 3 whitespaces at the end).  Seems like this should be fixed.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (IGNITE-17997) Whitespaces at the end are ignored during string comparison

[Konstantin Orlov (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-17997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625170#comment-17625170
 ] 

Konstantin Orlov edited comment on IGNITE-17997 at 10/27/22 3:06 PM:
-

Yes in IGNITE-17996 the same rule about type inference is applied. You could 
make an explicit cast to 

varchar, though. This will do the trick.

 

Well, right now we don't have any plans to support different/custom collations, 
but this may be subject to change in the future.


was (Author: korlov):
Yes in IGNITE-17996 the same rule about type inference is applied. You could 
make an explicit cast to 

varchar, though. This will do the trick.

> Whitespaces at the end are ignored during string comparison
> ---
>
> Key: IGNITE-17997
> URL: https://issues.apache.org/jira/browse/IGNITE-17997
> Project: Ignite
>  Issue Type: Bug
>  Components: sql
>Affects Versions: 3.0.0-beta1
>Reporter: Andrey Khitrin
>Priority: Major
>
> In 3.0.0-Beta-1:
> {code:java}
> sql-cli> select 'a' = 'a' as t1, 'a' = 'b' as t2, 'a' = 'a   ' as t3, 'a' = ' 
>   a' as t4;
> ╔══╤═══╤══╤═══╗
> ║ T1   │ T2    │ T3   │ T4    ║
> ╠══╪═══╪══╪═══╣
> ║ true │ false │ true │ false ║
> ╚══╧═══╧══╧═══╝
> {code}
> Tests T1, T2, and T4 show correct behavior. But in test T2 we see that string 
> 'a' is considered being equal to string 'a   '  (same string but with 
> arbitrary amount of whitespaces at the end). This is incorrect behavior.
> This issue may have the same nature as IGNITE-17996, but it's just a 
> hypothesis.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (IGNITE-17997) Whitespaces at the end are ignored during string comparison

[Konstantin Orlov (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-17997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625170#comment-17625170
 ] 

Konstantin Orlov commented on IGNITE-17997:
---

Yes in IGNITE-17996 the same rule about type inference is applied. You could 
make an explicit cast to 

varchar, though. This will do the trick.

> Whitespaces at the end are ignored during string comparison
> ---
>
> Key: IGNITE-17997
> URL: https://issues.apache.org/jira/browse/IGNITE-17997
> Project: Ignite
>  Issue Type: Bug
>  Components: sql
>Affects Versions: 3.0.0-beta1
>Reporter: Andrey Khitrin
>Priority: Major
>
> In 3.0.0-Beta-1:
> {code:java}
> sql-cli> select 'a' = 'a' as t1, 'a' = 'b' as t2, 'a' = 'a   ' as t3, 'a' = ' 
>   a' as t4;
> ╔══╤═══╤══╤═══╗
> ║ T1   │ T2    │ T3   │ T4    ║
> ╠══╪═══╪══╪═══╣
> ║ true │ false │ true │ false ║
> ╚══╧═══╧══╧═══╝
> {code}
> Tests T1, T2, and T4 show correct behavior. But in test T2 we see that string 
> 'a' is considered being equal to string 'a   '  (same string but with 
> arbitrary amount of whitespaces at the end). This is incorrect behavior.
> This issue may have the same nature as IGNITE-17996, but it's just a 
> hypothesis.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (IGNITE-18001) Ignite node may become unavailable after some play with SQL

[Andrey Khitrin (Jira)]

Andrey Khitrin created IGNITE-18001:
---

 Summary: Ignite node may become unavailable after some play with 
SQL
 Key: IGNITE-18001
 URL: https://issues.apache.org/jira/browse/IGNITE-18001
 Project: Ignite
  Issue Type: Bug
Affects Versions: 3.0.0-beta1
Reporter: Andrey Khitrin
 Attachments: ignite3db-0.log, ignite3db-1.log

Steps to reproduce:
 # Start AI3 node, init cluster
 # Connect to node via Ignite3 CLI
 # Open SQL console in Ignite3 CLI
 # Play with SQL queries: create tables, invoke select queries, drop tables, so 
on. Probably, this step is not needed. Probably, it may be important to perform 
some queries with errors.
 # Wait for some time (30-60 mins) having SQL console open.
 # Try to execute new query after the pause. It {*}hangs{*}.

In DB log, a lot of the following errors occur (some of them could occur even 
before step 6 above):
{code}
2022-10-27 18:15:57:391 +0400 
[WARNING][%defaultNode%Raft-Group-Client-11][RaftGroupServiceImpl] Recoverable 
error duri
ng the request type=ActionRequestImpl occurred (will be retried on the randomly 
selected node):
java.util.concurrent.TimeoutException
at 
java.base/java.util.concurrent.CompletableFuture$Timeout.run(CompletableFuture.java:2792)
at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at 
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecut
or.java:304)
at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-10-27 18:17:21:376 +0400 
[INFO][%defaultNode%checkpoint-thread-1][Checkpoint] Skipping checkpoint (no 
pages were m
odified) [checkpointBeforeWriteLockTime=0ms, checkpointWriteLockWait=0ms, 
checkpointListenersExecuteTime=0ms, checkpoin
tWriteLockHoldTime=0ms, reason='timeout']
{code}

After attempt to restart node (`./bin/ignite3db.sh stop && ./bin/ignite3db.sh 
start`) another stacktrace occurs in log:
{code}2022-10-27 18:24:06:489 +0400 
[INFO][ForkJoinPool.commonPool-worker-9][ClientHandlerModule] Thin client 
protocol started successfully[port=10800]
2022-10-27 18:24:06:490 +0400 
[INFO][ForkJoinPool.commonPool-worker-9][IgniteImpl] Components started, 
performing recovery
2022-10-27 18:24:06:868 +0400 
[INFO][ForkJoinPool.commonPool-worker-9][ConfigurationRegistry] Failed to 
notify configuration listener
java.util.NoSuchElementException: 
table.tables.ee9c42e0-9b96-4164-b13b-8bec99d3171a.assignments
at 
org.apache.ignite.internal.configuration.util.ConfigurationUtil.findEx(ConfigurationUtil.java:852)
at 
org.apache.ignite.internal.configuration.ConfigurationChanger.getLatest(ConfigurationChanger.java:439)
at 
org.apache.ignite.internal.configuration.direct.DirectPropertyProxy.value(DirectPropertyProxy.java:65)
at 
org.apache.ignite.internal.table.distributed.TableManager.updateAssignmentInternal(TableManager.java:652)
at 
org.apache.ignite.internal.table.distributed.TableManager.onUpdateAssignments(TableManager.java:616)
at 
org.apache.ignite.internal.configuration.notifications.ConfigurationNotifier.notifyPublicListeners(ConfigurationNotifier.java:492)
at 
org.apache.ignite.internal.configuration.notifications.ConfigurationNotifier$2.visitLeafNode(ConfigurationNotifier.java:374)
at 
org.apache.ignite.internal.configuration.notifications.ConfigurationNotifier$2.visitLeafNode(ConfigurationNotifier.java:370)
at 
org.apache.ignite.internal.schema.configuration.TableNode.traverseChildren(Unknown
 Source)
at 
org.apache.ignite.internal.configuration.notifications.ConfigurationNotifier.notifyListeners(ConfigurationNotifier.java:370)
at 
org.apache.ignite.internal.configuration.notifications.ConfigurationNotifier$2.visitNamedListNode(ConfigurationNotifier.java:460)
at 
org.apache.ignite.internal.configuration.notifications.ConfigurationNotifier$2.visitNamedListNode(ConfigurationNotifier.java:370)
at 
org.apache.ignite.internal.schema.configuration.TablesNode.traverseChildren(Unknown
 Source)
at 
org.apache.ignite.internal.configuration.notifications.ConfigurationNotifier.notifyListeners(ConfigurationNotifier.java:370)
at 
org.apache.ignite.internal.configuration.notifications.ConfigurationNotifier.notifyListeners(ConfigurationNotifier.java:88)
at 
org.apache.ignite.internal.configuration.ConfigurationRegistry$2.visitInnerNode(ConfigurationRegistry.java:310)
at 
org.apache.ignite.internal.configuration.ConfigurationRegistry$2.visitInnerNode(ConfigurationRegistry.java:292)
at 

[jira] [Created] (IGNITE-18000) Sql. Unmute SqlLogicTest

[Konstantin Orlov (Jira)]

Konstantin Orlov created IGNITE-18000:
-

 Summary: Sql. Unmute SqlLogicTest
 Key: IGNITE-18000
 URL: https://issues.apache.org/jira/browse/IGNITE-18000
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Konstantin Orlov


Due to the high instability of the ItSqlLogicTest, these tests were disabled 
until the reason will be found and fixed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (IGNITE-17999) Incorrect handling of TxStateReplicaRequest

[Vladislav Pyatkov (Jira)]

Vladislav Pyatkov created IGNITE-17999:
--

 Summary: Incorrect handling of TxStateReplicaRequest
 Key: IGNITE-17999
 URL: https://issues.apache.org/jira/browse/IGNITE-17999
 Project: Ignite
  Issue Type: Bug
Reporter: Vladislav Pyatkov


After observation, the following points were highlighting to fix:
# The request has a property named is commitTimestamp, but it is not a time to 
commit. This property should be renamed to readTimestamp, because it is a time 
with which a transaction tries to read another transaction state (to read an 
entry).
# The request should to be handled by a primary replica (presently, it is 
handled by leader, that can be a different node).
# TxMata should be read through RAFT, because the leader can be changed, and 
the actual data will be store on the a new one.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17990) Download RAFT snapshot without deleting original partition data

[Ivan Bessonov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Bessonov updated IGNITE-17990:
---
Description: 
h3. Description

In first design, full rebalance is implemented this way:
 * we drop partition data
 * we download partition data from the leader
 * we're done

There's a problem with this approach - if download part failed, we lost one 
follower. This is bad, because technically new leader may have more data in the 
log and it could have uploaded it the follower, but now it makes no sense.

Not only can it lead to hard-to-catch errors and introducing custom code to 
JRaft, it's also an unconditional data deletion without neither explicit  user 
approval nor a copy of the data preserved durably.

Such implementation is fine for POC and some tests, but it cannot be allowed in 
the release version of the product.
h3. New proposed solution

As trivial as it may seem, new solution is to _not deleting data_ before 
snapshot is fully downloaded and ready for swap. Why is it trivial? Because 
this is literally what RAFT demands to be done.

Of course, there's a {*}but{*}. Snapshot application, when it's downloaded, 
should be {{O(1)}} when it comes to the number of rows in the partition and a 
number of transactions in a tx state store. This may not be fully achievable, 
depending on the implementation that we chose, more on that later.

Following sections will describe all my concerns and possible implementations. 
Some sections can be skipped while reading. For example, if you're not 
interested in a specific storage engine, but want to read everything else.
h3. TX state storage

There's one really good thing about TX state storage. It has no storage engine, 
there's only a single RocksDB-based implementation. This makes possible the 
following approach:
 * when we stream data, we can write it into a SST file, almost like in 
snapshots of meta-storage ans CMG storages
 * once snapshot is downloaded, we ingest it into a storage

What I like about this solution is that it's very simple. But, there are 
concerns:
 * ingesting leads to additional implicit data flush. Maybe it can be avoided, 
more on that later
 * it's not clear whether RocksDB creates a copy of SST file or not. I would 
assume that it does, because the file might be in other folder or on another 
device, for example. Although copying files is fast, it still takes time. Add 
to this a time required for the flush and we see a problem - operation may 
become unnecessarily long

For these reasons, I don't think that such solution should be implemented. The 
point of this description was to show, that I thought about this alternative 
and consciously decided to use another one.

I believe that TX state storage should use the same approach as a RocksDB-based 
partition storage. Its description can be found later in this issue.
h3. MV storage - Test engine

Test uses concurrent skip-list map for MV data and a bunch of other maps for 
indexes. While snapshots is being downloaded, we should insert all data into 
new maps, that have the same structure. In the end, we should have two versions 
of the partition: old and new.

{{onSnapshotLoad}} should just swap all objects. After that, old partition data 
can be cleaned by the garbage collector.

This is a good place to start implementation. I assume that some new API will 
be introduced. I have thoughts about it as well, they are described later in 
*API* section.
h3. MV storage - RocksDB engine

SST-based approach is described in a *TX state storage* section. There I 
describe why I don't think that this is a good solution. Same reasoning can be 
applied here just as effectively. This means that we should write data in the 
same RocksDB instance. This is a little bit tricky.

The reason is that all stored data is merged together, and Columns Families are 
shared between different partitions. This makes it harder to find a place to 
write partition data while old partition data persists. As a reminder and an 
example, let's take a look at how data is stored in row storage:
{code:java}
+---+-+---+
|Partition 0| ... |   Partition MAX   |
+---+-+---+
| Row1 | ... | RowN | ... | Row1 | ... | RowN |
+---+-+---+{code}
Logically, CF is split into a different "blocks", and each block represents a 
partition. Currently, each partition block is associated with an 2-bytes 
identifier that matches a partition number in Big Endian.

 

We could add new CF with similar structure and write snapshot data in it, but 
then the snapshot load operation would require us to move data from one CF to 
another. The only option that I know of, that can do this, is SST ingestion. 
And I already explained why I don't like it.

This leaves us with the necessity to write data into the same column family. 

[jira] [Commented] (IGNITE-17997) Whitespaces at the end are ignored during string comparison

[Andrey Khitrin (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-17997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625149#comment-17625149
 ] 

Andrey Khitrin commented on IGNITE-17997:
-

Thank you. I've forgot about collations. As I can see, the same behavior could 
be achieved in other database engines, depending on data type and/or query 
options. E.g., in [sqlite3|https://www.sqlite.org/datatype3.html#collation]:
{code:sql}
sqlite> SELECT 'a' = 'a ';
0-- false
sqlite> SELECT 'a' = 'a ' COLLATE RTRIM;
1-- true
{code}
In that case, I hardly can treat this behavior as a bug. At most, it's a 
different default, not obvious though.

[~korlov] what would you say about IGNITE-17996? Does this behavior obey the 
same rules?

Also, are there plans to support different collation schemes in the future?

> Whitespaces at the end are ignored during string comparison
> ---
>
> Key: IGNITE-17997
> URL: https://issues.apache.org/jira/browse/IGNITE-17997
> Project: Ignite
>  Issue Type: Bug
>  Components: sql
>Affects Versions: 3.0.0-beta1
>Reporter: Andrey Khitrin
>Priority: Major
>
> In 3.0.0-Beta-1:
> {code:java}
> sql-cli> select 'a' = 'a' as t1, 'a' = 'b' as t2, 'a' = 'a   ' as t3, 'a' = ' 
>   a' as t4;
> ╔══╤═══╤══╤═══╗
> ║ T1   │ T2    │ T3   │ T4    ║
> ╠══╪═══╪══╪═══╣
> ║ true │ false │ true │ false ║
> ╚══╧═══╧══╧═══╝
> {code}
> Tests T1, T2, and T4 show correct behavior. But in test T2 we see that string 
> 'a' is considered being equal to string 'a   '  (same string but with 
> arbitrary amount of whitespaces at the end). This is incorrect behavior.
> This issue may have the same nature as IGNITE-17996, but it's just a 
> hypothesis.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with default settings.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: bench_persistent_full_Individual_pc1.txt

> Datastreamer may consume heap with default settings.
> 
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_batched_pc2.txt, 
> bench_inmem_individual_pc2.txt, bench_inmem_isolated_pc2.txt, 
> bench_persistent_full_Individual_pc1.txt, 
> bench_persistent_results_Individual_pc1.txt
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with default settings.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: bench_persistent_results_Individual_pc1.txt

> Datastreamer may consume heap with default settings.
> 
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_batched_pc2.txt, 
> bench_inmem_individual_pc2.txt, bench_inmem_isolated_pc2.txt, 
> bench_persistent_full_Individual_pc1.txt, 
> bench_persistent_results_Individual_pc1.txt
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with default settings.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: .behch_persistent_full_benchIndividual_pc1.txt.kate-swp

> Datastreamer may consume heap with default settings.
> 
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_batched_pc2.txt, 
> bench_inmem_individual_pc2.txt, bench_inmem_isolated_pc2.txt
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with default settings.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: (was: 
.behch_persistent_full_benchIndividual_pc1.txt.kate-swp)

> Datastreamer may consume heap with default settings.
> 
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_batched_pc2.txt, 
> bench_inmem_individual_pc2.txt, bench_inmem_isolated_pc2.txt
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with default settings.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: behch_persistent_full_benchIndividual_pc1.txt

> Datastreamer may consume heap with default settings.
> 
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_batched_pc2.txt, 
> bench_inmem_individual_pc2.txt, bench_inmem_isolated_pc2.txt
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with default settings.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: (was: behch_persistent_full_benchIndividual_pc1.txt)

> Datastreamer may consume heap with default settings.
> 
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_batched_pc2.txt, 
> bench_inmem_individual_pc2.txt, bench_inmem_isolated_pc2.txt
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (IGNITE-17330) Support RO TX by SQL

[Yury Gerzhedovich (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-17330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625114#comment-17625114
 ] 

Yury Gerzhedovich commented on IGNITE-17330:


[~zstan] , LGTM. Thanks for your efforts!

> Support RO TX by SQL
> 
>
> Key: IGNITE-17330
> URL: https://issues.apache.org/jira/browse/IGNITE-17330
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Reporter: Yury Gerzhedovich
>Assignee: Evgeny Stanilovsky
>Priority: Major
>  Labels: ignite-3
> Fix For: 3.0.0-beta1
>
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> SQL use RW transaction mode. It's not efficient in case we need just read 
> consistent data.
> After implementing IGNITE-17260 we should add support RO transaction for SQL 
> queries. Seems required just use transaction method with timestamp 
> (org.apache.ignite.hlc.HybridClock#now) as parameter.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with default settings.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: DS_heap_consumption_2.png
DS_heap_consumption.png

> Datastreamer may consume heap with default settings.
> 
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_batched_pc2.txt, 
> bench_inmem_individual_pc2.txt, bench_inmem_isolated_pc2.txt
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with default settings.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: (was: DS_heap_no_events_no_wal.png)

> Datastreamer may consume heap with default settings.
> 
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_batched_pc2.txt, 
> bench_inmem_individual_pc2.txt, bench_inmem_isolated_pc2.txt
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with default settings.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: (was: DS_heap_no_events_no_wal_2.png)

> Datastreamer may consume heap with default settings.
> 
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_consumption.png, DS_heap_consumption_2.png, 
> HeapConsumptionDataStreamerTest.src, bench_inmem_batched_pc2.txt, 
> bench_inmem_individual_pc2.txt, bench_inmem_isolated_pc2.txt
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17859) Update indexes on data modifications

[Alexander Lapin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Lapin updated IGNITE-17859:
-
Reviewer: Alexander Lapin

> Update indexes on data modifications
> 
>
> Key: IGNITE-17859
> URL: https://issues.apache.org/jira/browse/IGNITE-17859
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexander Lapin
>Assignee: Konstantin Orlov
>Priority: Major
>  Labels: ignite-3
> Fix For: 3.0.0-beta1
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> h3. Motivation
> For the sake of better performance and common sense, it is necessary to 
> integrate indexes into data manipulation flow, which implies
>  * Integrating indexes into the process of efficient evaluation keys to 
> rowsIds both within pure read/scan operations and as part of modification 
> operations in order to find proper rowId of a row to be modified.
>  * Integrating indexes into the process of data modification itself, meaning 
> that not only data should be updated but also all corresponding indexes 
> should be populated with updated entries along with cleanup on transaction 
> rollback.
> Given Jira issue is about second part.
> *Definition* *of Done*
>  * All indexes with relevant schema version are populated with modified rows.
>  * All pending index entries are removed on tx rollback.
> h3. Implementation Notes
>  # Seems, that it has sense to introduce new Index abstractions that will 
> manage update logic internally. Something like following:
>  ** Index
>  ** HashIndex extends Index
>  ** HashUniqueIndex extends HashIndex
>  ** SortedIndex extneds Index
>  ** SorteUniquedIndex extneds SortedIndex
>  # In order to define which indexes to update both during update itself or 
> during rollback it'll be useful to add extra parameter _schemaVersion_ to 
> each operation enlisted into transaction. All in all, that will allow to 
> select only proper set of indexes with relevant schema versions.
>  # During transaction rollback it'll be necessary  to cleanup outdated 
> pending entries that were touched during tx lifetime. 
> h4. More details about *first* item.
> Index itself may declare update() and other methods that will have 
> index-type-specific lock management and uniqueness processing logic, e.g. for 
> HashIndex.update following is suggested:
> {code:java}
> @Override
> public CompletableFuture update(UUID txId, TxState txState, Tuple oldRow, 
> Tuple newRow, VersionChain rowId) {
> Tuple oldVal = oldRow == Tuple.TOMBSTONE ? Tuple.TOMBSTONE : 
> oldRow.select(col);
> Tuple newVal = newRow == Tuple.TOMBSTONE ? Tuple.TOMBSTONE : 
> newRow.select(col);
> List futs = new ArrayList<>();
> if (!oldVal.equals(newVal)) {
> if (oldVal.length() > 0) {
> Lock lock0 = lockTable.getOrAddEntry(oldVal);
> txState.addLock(lock0);
> futs.add(lock0.acquire(txId, LockMode.IX));
> // Do not remove bookmarks due to multi-versioning.
> }
> if (newVal.length() > 0) {
> Lock lock0 = lockTable.getOrAddEntry(newVal);
> txState.addLock(lock0);
> futs.add(lock0.acquire(txId, LockMode.IX).thenAccept(ignored0 -> {
> if (index.insert(newVal, rowId)) {
> txState.addUndo(() -> index.remove(newVal, rowId));
> }
> }));
> }
> }
> return CompletableFuture.allOf(futs.toArray(new CompletableFuture[0]));
> } {code}
> Further details could be found in 
> [https://github.com/ascherbakoff/ai3-txn-mvp]
> Detailed lock management design is described in  [IEP-91-Locking 
> model|https://cwiki.apache.org/confluence/display/IGNITE/IEP-91%3A+Transaction+protocol#IEP91:Transactionprotocol-Lockingmodel]
>  See index related sections, e.g. for HashIndex following lock flow is 
> suggested:
> {code:java}
> Non-unique locks
>   // scan
>   S_commit(key)
>   // insert
>   IX_commit(key)
>   // delete
>   IX_commit(key) {code}
> h4. More details about *third* item.
> Please see cleanup flow described in 
> https://issues.apache.org/jira/browse/IGNITE-17673
>  
> It might have sense to consider all three items as separate tickets.
> It also worth to mention that index modification is triggered from 
> PartitionListener
>  * handleUpdateCommand
>  * handleUpdateAllCommand
>  * handleTxCleanupCommand



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (IGNITE-17983) Fix work with the last applied index in RAFT

[Kirill Tkalenko (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirill Tkalenko reassigned IGNITE-17983:


Assignee: Kirill Tkalenko

> Fix work with the last applied index in RAFT
> 
>
> Key: IGNITE-17983
> URL: https://issues.apache.org/jira/browse/IGNITE-17983
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Kirill Tkalenko
>Assignee: Kirill Tkalenko
>Priority: Major
>  Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> At the moment, *MvPartitionStorage* and *TxStateStorage* are used for a 
> partition raft group, while only *MvPartitionStorage#persistedIndex* of 
> *MvPartitionStorage#lastAppliedIndex* is used, which is not correct.
> On recovery, we need to use a minimum of 
> *MvPartitionStorage#lastAppliedIndex* and *TxStateStorage#lastAppliedIndex* 
> so as not to lose data for one of the storage.
> When taking a snapshot for a full rebalance, we should use the maximum of 
> *MvPartitionStorage#lastAppliedIndex* and *TxStateStorage#lastAppliedIndex* 
> so that we can load up-to-date data from the leader.
> Etc.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17735) Datastreamer may consume heap with default settings.

[Vladimir Steshin (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Steshin updated IGNITE-17735:
--
Attachment: bench_inmem_batched_pc2.txt
bench_inmem_individual_pc2.txt
bench_inmem_isolated_pc2.txt

> Datastreamer may consume heap with default settings.
> 
>
> Key: IGNITE-17735
> URL: https://issues.apache.org/jira/browse/IGNITE-17735
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Vladimir Steshin
>Assignee: Vladimir Steshin
>Priority: Major
>  Labels: ise
> Attachments: DS_heap_no_events_no_wal.png, 
> DS_heap_no_events_no_wal_2.png, HeapConsumptionDataStreamerTest.src, 
> bench_inmem_batched_pc2.txt, bench_inmem_individual_pc2.txt, 
> bench_inmem_isolated_pc2.txt
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (IGNITE-17859) Update indexes on data modifications

[Alexander Lapin (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-17859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625076#comment-17625076
 ] 

Alexander Lapin commented on IGNITE-17859:
--

[~korlov] Looks fine as a skeleton implementation. With an assumption of 
several follow up tickets definitely worth merging. Great job actually!

> Update indexes on data modifications
> 
>
> Key: IGNITE-17859
> URL: https://issues.apache.org/jira/browse/IGNITE-17859
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Alexander Lapin
>Assignee: Konstantin Orlov
>Priority: Major
>  Labels: ignite-3
> Fix For: 3.0.0-beta1
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> h3. Motivation
> For the sake of better performance and common sense, it is necessary to 
> integrate indexes into data manipulation flow, which implies
>  * Integrating indexes into the process of efficient evaluation keys to 
> rowsIds both within pure read/scan operations and as part of modification 
> operations in order to find proper rowId of a row to be modified.
>  * Integrating indexes into the process of data modification itself, meaning 
> that not only data should be updated but also all corresponding indexes 
> should be populated with updated entries along with cleanup on transaction 
> rollback.
> Given Jira issue is about second part.
> *Definition* *of Done*
>  * All indexes with relevant schema version are populated with modified rows.
>  * All pending index entries are removed on tx rollback.
> h3. Implementation Notes
>  # Seems, that it has sense to introduce new Index abstractions that will 
> manage update logic internally. Something like following:
>  ** Index
>  ** HashIndex extends Index
>  ** HashUniqueIndex extends HashIndex
>  ** SortedIndex extneds Index
>  ** SorteUniquedIndex extneds SortedIndex
>  # In order to define which indexes to update both during update itself or 
> during rollback it'll be useful to add extra parameter _schemaVersion_ to 
> each operation enlisted into transaction. All in all, that will allow to 
> select only proper set of indexes with relevant schema versions.
>  # During transaction rollback it'll be necessary  to cleanup outdated 
> pending entries that were touched during tx lifetime. 
> h4. More details about *first* item.
> Index itself may declare update() and other methods that will have 
> index-type-specific lock management and uniqueness processing logic, e.g. for 
> HashIndex.update following is suggested:
> {code:java}
> @Override
> public CompletableFuture update(UUID txId, TxState txState, Tuple oldRow, 
> Tuple newRow, VersionChain rowId) {
> Tuple oldVal = oldRow == Tuple.TOMBSTONE ? Tuple.TOMBSTONE : 
> oldRow.select(col);
> Tuple newVal = newRow == Tuple.TOMBSTONE ? Tuple.TOMBSTONE : 
> newRow.select(col);
> List futs = new ArrayList<>();
> if (!oldVal.equals(newVal)) {
> if (oldVal.length() > 0) {
> Lock lock0 = lockTable.getOrAddEntry(oldVal);
> txState.addLock(lock0);
> futs.add(lock0.acquire(txId, LockMode.IX));
> // Do not remove bookmarks due to multi-versioning.
> }
> if (newVal.length() > 0) {
> Lock lock0 = lockTable.getOrAddEntry(newVal);
> txState.addLock(lock0);
> futs.add(lock0.acquire(txId, LockMode.IX).thenAccept(ignored0 -> {
> if (index.insert(newVal, rowId)) {
> txState.addUndo(() -> index.remove(newVal, rowId));
> }
> }));
> }
> }
> return CompletableFuture.allOf(futs.toArray(new CompletableFuture[0]));
> } {code}
> Further details could be found in 
> [https://github.com/ascherbakoff/ai3-txn-mvp]
> Detailed lock management design is described in  [IEP-91-Locking 
> model|https://cwiki.apache.org/confluence/display/IGNITE/IEP-91%3A+Transaction+protocol#IEP91:Transactionprotocol-Lockingmodel]
>  See index related sections, e.g. for HashIndex following lock flow is 
> suggested:
> {code:java}
> Non-unique locks
>   // scan
>   S_commit(key)
>   // insert
>   IX_commit(key)
>   // delete
>   IX_commit(key) {code}
> h4. More details about *third* item.
> Please see cleanup flow described in 
> https://issues.apache.org/jira/browse/IGNITE-17673
>  
> It might have sense to consider all three items as separate tickets.
> It also worth to mention that index modification is triggered from 
> PartitionListener
>  * handleUpdateCommand
>  * handleUpdateAllCommand
>  * handleTxCleanupCommand



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17998) ItSqlAsynchronousApiTest#closeSession is unstable.

[Yury Gerzhedovich (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yury Gerzhedovich updated IGNITE-17998:
---
Labels: ignite-3  (was: )

> ItSqlAsynchronousApiTest#closeSession is unstable.
> --
>
> Key: IGNITE-17998
> URL: https://issues.apache.org/jira/browse/IGNITE-17998
> Project: Ignite
>  Issue Type: Bug
>  Components: sql
>Affects Versions: 3.0.0-alpha5
>Reporter: Evgeny Stanilovsky
>Priority: Major
>  Labels: ignite-3
>
> ItSqlAsynchronousApiTest#closeSession is unstable.
> {noformat}
> org.opentest4j.AssertionFailedError: Exception is neither of a specified 
> class, nor has a cause of the specified class: class 
> org.apache.ignite.sql.SqlException
>   at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:43)
>   at org.junit.jupiter.api.Assertions.fail(Assertions.java:146)
>   at 
> org.apache.ignite.internal.testframework.IgniteTestUtils.assertThrowsWithCause(IgniteTestUtils.java:284)
>   at 
> org.apache.ignite.internal.testframework.IgniteTestUtils.assertThrowsWithCause(IgniteTestUtils.java:264)
>   at 
> org.apache.ignite.internal.sql.api.ItSqlAsynchronousApiTest.closeSession(ItSqlAsynchronousApiTest.java:541)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (IGNITE-17998) ItSqlAsynchronousApiTest#closeSession is unstable.

[Evgeny Stanilovsky (Jira)]

Evgeny Stanilovsky created IGNITE-17998:
---

 Summary: ItSqlAsynchronousApiTest#closeSession is unstable.
 Key: IGNITE-17998
 URL: https://issues.apache.org/jira/browse/IGNITE-17998
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 3.0.0-alpha5
Reporter: Evgeny Stanilovsky


ItSqlAsynchronousApiTest#closeSession is unstable.

{noformat}
org.opentest4j.AssertionFailedError: Exception is neither of a specified class, 
nor has a cause of the specified class: class org.apache.ignite.sql.SqlException

at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:43)
at org.junit.jupiter.api.Assertions.fail(Assertions.java:146)
at 
org.apache.ignite.internal.testframework.IgniteTestUtils.assertThrowsWithCause(IgniteTestUtils.java:284)
at 
org.apache.ignite.internal.testframework.IgniteTestUtils.assertThrowsWithCause(IgniteTestUtils.java:264)
at 
org.apache.ignite.internal.sql.api.ItSqlAsynchronousApiTest.closeSession(ItSqlAsynchronousApiTest.java:541)
{noformat}




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17980) ./gradlew clean build -x test fails

[Kirill Tkalenko (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirill Tkalenko updated IGNITE-17980:
-
Fix Version/s: 3.0.0-beta2

> ./gradlew clean build -x test fails
> ---
>
> Key: IGNITE-17980
> URL: https://issues.apache.org/jira/browse/IGNITE-17980
> Project: Ignite
>  Issue Type: Bug
>  Components: build
>Reporter: Aleksandr
>Assignee: Aleksandr
>Priority: Critical
>  Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> On the latest main branch, the following fails: ./gradlew clean build -x test



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17980) ./gradlew clean build -x test fails

[Kirill Tkalenko (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirill Tkalenko updated IGNITE-17980:
-
Ignite Flags:   (was: Docs Required,Release Notes Required)

> ./gradlew clean build -x test fails
> ---
>
> Key: IGNITE-17980
> URL: https://issues.apache.org/jira/browse/IGNITE-17980
> Project: Ignite
>  Issue Type: Bug
>  Components: build
>Reporter: Aleksandr
>Assignee: Aleksandr
>Priority: Critical
>  Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> On the latest main branch, the following fails: ./gradlew clean build -x test



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (IGNITE-17997) Whitespaces at the end are ignored during string comparison

[Konstantin Orlov (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-17997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625070#comment-17625070
 ] 

Konstantin Orlov commented on IGNITE-17997:
---

Hi [~akhitrin]! According to SQL Standard (in standard 1999 it's chapter 8.2 
``, p 3.b of general rules): 

{code:java}
If the length in characters of X is not equal to the length in characters of Y, 
then the shorter string is effectively replaced, for the purposes of 
comparison, with a copy of itself that has been extended to the length of the 
longer string by concatenation on the right of one or more pad characters, 
where the pad character is chosen based on CS. If CS has the NO PAD 
characteristic, then the pad character is an implementation-dependent character 
different from any character in the character set of X and Y that collates less 
than any string under CS. Otherwise, the pad character is a . {code}
 

Type inference for literal is working in a way to find the most restrictive 
type. In your example, these will be CHAR(1) and CHAR(4) accordingly. With that 
said, I consider the result of your example rather valid.

> Whitespaces at the end are ignored during string comparison
> ---
>
> Key: IGNITE-17997
> URL: https://issues.apache.org/jira/browse/IGNITE-17997
> Project: Ignite
>  Issue Type: Bug
>  Components: sql
>Affects Versions: 3.0.0-beta1
>Reporter: Andrey Khitrin
>Priority: Major
>
> In 3.0.0-Beta-1:
> {code:java}
> sql-cli> select 'a' = 'a' as t1, 'a' = 'b' as t2, 'a' = 'a   ' as t3, 'a' = ' 
>   a' as t4;
> ╔══╤═══╤══╤═══╗
> ║ T1   │ T2    │ T3   │ T4    ║
> ╠══╪═══╪══╪═══╣
> ║ true │ false │ true │ false ║
> ╚══╧═══╧══╧═══╝
> {code}
> Tests T1, T2, and T4 show correct behavior. But in test T2 we see that string 
> 'a' is considered being equal to string 'a   '  (same string but with 
> arbitrary amount of whitespaces at the end). This is incorrect behavior.
> This issue may have the same nature as IGNITE-17996, but it's just a 
> hypothesis.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17997) Whitespaces at the end are ignored during string comparison

[Evgeny Stanilovsky (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Evgeny Stanilovsky updated IGNITE-17997:

Ignite Flags:   (was: Docs Required,Release Notes Required)

> Whitespaces at the end are ignored during string comparison
> ---
>
> Key: IGNITE-17997
> URL: https://issues.apache.org/jira/browse/IGNITE-17997
> Project: Ignite
>  Issue Type: Bug
>  Components: sql
>Affects Versions: 3.0.0-beta1
>Reporter: Andrey Khitrin
>Priority: Major
>
> In 3.0.0-Beta-1:
> {code:java}
> sql-cli> select 'a' = 'a' as t1, 'a' = 'b' as t2, 'a' = 'a   ' as t3, 'a' = ' 
>   a' as t4;
> ╔══╤═══╤══╤═══╗
> ║ T1   │ T2    │ T3   │ T4    ║
> ╠══╪═══╪══╪═══╣
> ║ true │ false │ true │ false ║
> ╚══╧═══╧══╧═══╝
> {code}
> Tests T1, T2, and T4 show correct behavior. But in test T2 we see that string 
> 'a' is considered being equal to string 'a   '  (same string but with 
> arbitrary amount of whitespaces at the end). This is incorrect behavior.
> This issue may have the same nature as IGNITE-17996, but it's just a 
> hypothesis.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17997) Whitespaces at the end are ignored during string comparison

[Evgeny Stanilovsky (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Evgeny Stanilovsky updated IGNITE-17997:

Component/s: sql

> Whitespaces at the end are ignored during string comparison
> ---
>
> Key: IGNITE-17997
> URL: https://issues.apache.org/jira/browse/IGNITE-17997
> Project: Ignite
>  Issue Type: Bug
>  Components: sql
>Affects Versions: 3.0.0-beta1
>Reporter: Andrey Khitrin
>Priority: Major
>
> In 3.0.0-Beta-1:
> {code:java}
> sql-cli> select 'a' = 'a' as t1, 'a' = 'b' as t2, 'a' = 'a   ' as t3, 'a' = ' 
>   a' as t4;
> ╔══╤═══╤══╤═══╗
> ║ T1   │ T2    │ T3   │ T4    ║
> ╠══╪═══╪══╪═══╣
> ║ true │ false │ true │ false ║
> ╚══╧═══╧══╧═══╝
> {code}
> Tests T1, T2, and T4 show correct behavior. But in test T2 we see that string 
> 'a' is considered being equal to string 'a   '  (same string but with 
> arbitrary amount of whitespaces at the end). This is incorrect behavior.
> This issue may have the same nature as IGNITE-17996, but it's just a 
> hypothesis.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (IGNITE-17997) Whitespaces at the end are ignored during string comparison

[Andrey Khitrin (Jira)]

Andrey Khitrin created IGNITE-17997:
---

 Summary: Whitespaces at the end are ignored during string 
comparison
 Key: IGNITE-17997
 URL: https://issues.apache.org/jira/browse/IGNITE-17997
 Project: Ignite
  Issue Type: Bug
Affects Versions: 3.0.0-beta1
Reporter: Andrey Khitrin


In 3.0.0-Beta-1:
{code:java}
sql-cli> select 'a' = 'a' as t1, 'a' = 'b' as t2, 'a' = 'a   ' as t3, 'a' = '   
a' as t4;
╔══╤═══╤══╤═══╗
║ T1   │ T2    │ T3   │ T4    ║
╠══╪═══╪══╪═══╣
║ true │ false │ true │ false ║
╚══╧═══╧══╧═══╝
{code}
Tests T1, T2, and T4 show correct behavior. But in test T2 we see that string 
'a' is considered being equal to string 'a   '  (same string but with arbitrary 
amount of whitespaces at the end). This is incorrect behavior.

This issue may have the same nature as IGNITE-17996, but it's just a hypothesis.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (IGNITE-17996) Strange behavior of SQL CASE expression

[Andrey Khitrin (Jira)]

Andrey Khitrin created IGNITE-17996:
---

 Summary: Strange behavior of SQL CASE expression
 Key: IGNITE-17996
 URL: https://issues.apache.org/jira/browse/IGNITE-17996
 Project: Ignite
  Issue Type: Bug
Affects Versions: 3.0.0-beta1
Reporter: Andrey Khitrin


I observe strange behavior in the next scenario:

 
{code:java}
sql-cli> create table xx (f1 int primary key);
Updated 0 rows.
sql-cli> insert into xx values (1);
Updated 1 rows.
sql-cli> insert into xx values (2);
Updated 1 rows.
sql-cli> select f1, case when f1 < 2 then 'foo' else 'barbar' end as s, 
length(case when f1 < 2 then 'foo' else 'barbar' end) as ls from xx;
╔╤╤╗
║ F1 │ S      │ LS ║
╠╪╪╣
║ 2  │ barbar │ 6  ║
╟┼┼╢
║ 1  │ foo    │ 6  ║
╚╧╧╝
 {code}
I expect `CASE` to return 'foo' value, but de-facto it returns 'foo   ' ('foo' 
with 3 whitespaces at the end).  Seems like this should be fixed.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (IGNITE-17994) Use toByteArray for colocation hash of BigDecimal and BigInteger

[Pavel Tupitsyn (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625028#comment-17625028
 ] 

Pavel Tupitsyn commented on IGNITE-17994:
-

Merged to main: ecc88efd5bc178702301bacc5c0c0bfe3ee7ea4a

> Use toByteArray for colocation hash of BigDecimal and BigInteger
> 
>
> Key: IGNITE-17994
> URL: https://issues.apache.org/jira/browse/IGNITE-17994
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
>  Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> [HashCalculator converts BigDecimal and BigInteger to double to compute 
> hash|https://github.com/apache/ignite-3/blob/fa79ad532b18c3a9cf277d09b4fd0e8cd25775e0/modules/core/src/main/java/org/apache/ignite/internal/util/HashCalculator.java#L152].
>  Under the hood this involves weird and complicated logic - bit magic for 
> BigInteger, string conversion for BigDecimal.
> This logic is too hard to replicate in non-Java clients.
> Use toByteArray()  instead to compute the hash. Byte representation of those 
> types is used in BinaryTuple already, so clients know how to work with that.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17935) Finish implementation of RAFT snapshot streamer reader

[Roman Puchkovskiy (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Puchkovskiy updated IGNITE-17935:
---
Summary: Finish implementation of RAFT snapshot streamer reader  (was: 
Finish implementation of RAFT snapshot streamer)

> Finish implementation of RAFT snapshot streamer reader
> --
>
> Key: IGNITE-17935
> URL: https://issues.apache.org/jira/browse/IGNITE-17935
> Project: Ignite
>  Issue Type: Improvement
>  Components: persistence
>Reporter: Roman Puchkovskiy
>Assignee: Roman Puchkovskiy
>Priority: Major
>  Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> Implementation is started in IGNITE-17083, but the task turned out to be too 
> big for one PR.
> What has to be done:
>  # Serve requests to get snapshot metadata
>  # Switch from sending {{persistedIndex}} as {{lastAppliedIndex}} to sending 
> {{lastAppliedIndex}} itself
>  # When sending {{{}lastAppliedIndex{}}}, take into consideration 
> {{lastAppliedIndex}} of both MV and TX storages
>  # Resolve all TODOs mentioning this issue



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (IGNITE-17994) Use toByteArray for colocation hash of BigDecimal and BigInteger

[Igor Sapego (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625021#comment-17625021
 ] 

Igor Sapego commented on IGNITE-17994:
--

[~ptupitsyn] looks good to me

> Use toByteArray for colocation hash of BigDecimal and BigInteger
> 
>
> Key: IGNITE-17994
> URL: https://issues.apache.org/jira/browse/IGNITE-17994
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
>  Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> [HashCalculator converts BigDecimal and BigInteger to double to compute 
> hash|https://github.com/apache/ignite-3/blob/fa79ad532b18c3a9cf277d09b4fd0e8cd25775e0/modules/core/src/main/java/org/apache/ignite/internal/util/HashCalculator.java#L152].
>  Under the hood this involves weird and complicated logic - bit magic for 
> BigInteger, string conversion for BigDecimal.
> This logic is too hard to replicate in non-Java clients.
> Use toByteArray()  instead to compute the hash. Byte representation of those 
> types is used in BinaryTuple already, so clients know how to work with that.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17990) Download RAFT snapshot without deleting original partition data

[Ivan Bessonov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Bessonov updated IGNITE-17990:
---
Description: 
h3. Description

In first design, full rebalance is implemented this way:
 * we drop partition data
 * we download partition data from the leader
 * we're done

There's a problem with this approach - if download part failed, we lost one 
follower. This is bad, because technically new leader may have more data in the 
log and it could have uploaded it the follower, but now it makes no sense.

Not only can it lead to hard-to-catch errors and introducing custom code to 
JRaft, it's also an unconditional data deletion without neither explicit  user 
approval nor a copy of the data preserved durably.

Such implementation is fine for POC and some tests, but it cannot be allowed in 
the release version of the product.
h3. New proposed solution

As trivial as it may seem, new solution is to _not deleting data_ before 
snapshot is fully downloaded and ready for swap. Why is it trivial? Because 
this is literally what RAFT demands to be done.

Of course, there's a {*}but{*}. Snapshot application, when it's downloaded, 
should be {{O(1)}} when it comes to the number of rows in the partition and a 
number of transactions in a tx state store. This may not be fully achievable, 
depending on the implementation that we chose, more on that later.

Following sections will describe all my concerns and possible implementations. 
Some sections can be skipped while reading. For example, if you're not 
interested in a specific storage engine, but want to read everything else.
h3. TX state storage

There's one really good thing about TX state storage. It has no storage engine, 
there's only a single RocksDB-based implementation. This makes possible the 
following approach:
 * when we stream data, we can write it into a SST file, almost like in 
snapshots of meta-storage ans CMG storages
 * once snapshot is downloaded, we ingest it into a storage

What I like about this solution is that it's very simple. But, there are 
concerns:
 * ingesting leads to additional implicit data flush. Maybe it can be avoided, 
more on that later
 * it's not clear whether RocksDB creates a copy of SST file or not. I would 
assume that it does, because the file might be in other folder or on another 
device, for example. Although copying files is fast, it still takes time. Add 
to this a time required for the flush and we see a problem - operation may 
become unnecessarily long

For these reasons, I don't think that such solution should be implemented. The 
point of this description was to show, that I thought about this alternative 
and consciously decided to use another one.

I believe that TX state storage should use the same approach as a RocksDB-based 
partition storage. Its description can be found later in this issue.
h3. MV storage - Test engine

Test uses concurrent skip-list map for MV data and a bunch of other maps for 
indexes. While snapshots is being downloaded, we should insert all data into 
new maps, that have the same structure. In the end, we should have two versions 
of the partition: old and new.

{{onSnapshotLoad}} should just swap all objects. After that, old partition data 
can be cleaned by the garbage collector.

This is a good place to start implementation. I assume that some new API will 
be introduced. I have thoughts about it as well, they are described later in 
*API* section.
h3. MV storage - RocksDB engine

SST-based approach is described in a *TX state storage* section. There I 
describe why I don't think that this is a good solution. Same reasoning can be 
applied here just as effectively. This means that we should write data in the 
same RocksDB instance. This is a little bit tricky.

The reason is that all stored data is merged together, and Columns Families are 
shared between different partitions. This makes it harder to find a place to 
write partition data while old partition data persists. As a reminder and an 
example, let's take a look at how data is stored in row storage:
{code:java}
+---+-+---+
|Partition 0| ... |   Partition MAX   |
+---+-+---+
| Row1 | ... | RowN | ... | Row1 | ... | RowN |
+---+-+---+{code}
Logically, CF is split into a different "blocks", and each block represents a 
partition. Currently, each partition block is associated with an 2-bytes 
identifier that matches a partition number in Big Endian.

 

We could add new CF with similar structure and write snapshot data in it, but 
then the snapshot load operation would require us to move data from one CF to 
another. The only option that I know of, that can do this, is SST ingestion. 
And I already explained why I don't like it.

This leaves us with the necessity to write data into the same column family. 

[jira] [Updated] (IGNITE-17786) Add --verbose option to all commands

[Vadim Pakhnushev (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vadim Pakhnushev updated IGNITE-17786:
--
Epic Link: IGNITE-16970

> Add --verbose option to all commands
> 
>
> Key: IGNITE-17786
> URL: https://issues.apache.org/jira/browse/IGNITE-17786
> Project: Ignite
>  Issue Type: Improvement
>  Components: cli
>Reporter: Aleksandr
>Assignee: Vadim Pakhnushev
>Priority: Major
>  Labels: ignite-3
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Users of CLI should have the ability to go deeper if they face issues with 
> CLI.  
> The goal of this ticket is to add {{--verbose}} option to all commands and 
> print additional debug information to the terminal if this flag is set to 
> true.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17990) Download RAFT snapshot without deleting original partition data

[Ivan Bessonov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Bessonov updated IGNITE-17990:
---
Description: 
h3. Description

In first design, full rebalance is implemented this way:
 * we drop partition data
 * we download partition data from the leader
 * we're done

There's a problem with this approach - if download part failed, we lost one 
follower. This is bad, because technically new leader may have more data in the 
log and it could have uploaded it the follower, but now it makes no sense.

Not only can it lead to hard-to-catch errors and introducing custom code to 
JRaft, it's also an unconditional data deletion without neither explicit  user 
approval nor a copy of the data preserved durably.

Such implementation is fine for POC and some tests, but it cannot be allowed in 
the release version of the product.
h3. New proposed solution

As trivial as it may seem, new solution is to _not deleting data_ before 
snapshot is fully downloaded and ready for swap. Why is it trivial? Because 
this is literally what RAFT demands to be done.

Of course, there's a {*}but{*}. Snapshot application, when it's downloaded, 
should be {{O(1)}} when it comes to the number of rows in the partition and a 
number of transactions in a tx state store. This may not be fully achievable, 
depending on the implementation that we chose, more on that later.

Following sections will describe all my concerns and possible implementations. 
Some sections can be skipped while reading. For example, if you're not 
interested in a specific storage engine, but want to read everything else.
h3. TX state storage

There's one really good thing about TX state storage. It has no storage engine, 
there's only a single RocksDB-based implementation. This makes possible the 
following approach:
 * when we stream data, we can write it into a SST file, almost like in 
snapshots of meta-storage ans CMG storages
 * once snapshot is downloaded, we ingest it into a storage

What I like about this solution is that it's very simple. But, there are 
concerns:
 * ingesting leads to additional implicit data flush. Maybe it can be avoided, 
more on that later
 * it's not clear whether RocksDB creates a copy of SST file or not. I would 
assume that it does, because the file might be in other folder or on another 
device, for example. Although copying files is fast, it still takes time. Add 
to this a time required for the flush and we see a problem - operation may 
become unnecessarily long

For these reasons, I don't think that such solution should be implemented. The 
point of this description was to show, that I thought about this alternative 
and consciously decided to use another one.

I believe that TX state storage should use the same approach as a RocksDB-based 
partition storage. Its description can be found later in this issue.
h3. MV storage - Test engine

Test uses concurrent skip-list map for MV data and a bunch of other maps for 
indexes. While snapshots is being downloaded, we should insert all data into 
new maps, that have the same structure. In the end, we should have two versions 
of the partition: old and new.

{{onSnapshotLoad}} should just swap all objects. After that, old partition data 
can be cleaned by the garbage collector.

This is a good place to start implementation. I assume that some new API will 
be introduced. I have thoughts about it as well, they are described later in 
*API* section.
h3. MV storage - RocksDB engine

SST-based approach is described in a *TX state storage* section. There I 
describe why I don't think that this is a good solution. Same reasoning can be 
applied here just as effectively. This means that we should write data in the 
same RocksDB instance. This is a little bit tricky.

The reason is that all stored data is merged together, and Columns Families are 
shared between different partitions. This makes it harder to find a place to 
write partition data while old partition data persists. As a reminder and an 
example, let's take a look at how data is stored in row storage:
{code:java}
+---+-+---+
|Partition 0| ... |   Partition MAX   |
+---+-+---+
| Row1 | ... | RowN | ... | Row1 | ... | RowN |
+---+-+---+{code}
Logically, CF is split into a different "blocks", and each block represents a 
partition. Currently, each partition block is associated with an 2-bytes 
identifier that matches a partition number in Big Endian.

 

We could add new CF with similar structure and write snapshot data in it, but 
then the snapshot load operation would require us to move data from one CF to 
another. The only option that I know of, that can do this, is SST ingestion. 
And I already explained why I don't like it.

This leaves us with the necessity to write data into the same column family. 

[jira] [Updated] (IGNITE-17990) Download RAFT snapshot without deleting original partition data

[Ivan Bessonov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Bessonov updated IGNITE-17990:
---
Description: 
h3. Description

In first design, full rebalance is implemented this way:
 * we drop partition data
 * we download partition data from the leader
 * we're done

There's a problem with this approach - if download part failed, we lost one 
follower. This is bad, because technically new leader may have more data in the 
log and it could have uploaded it the follower, but now it makes no sense.

Not only can it lead to hard-to-catch errors and introducing custom code to 
JRaft, it's also an unconditional data deletion without neither explicit  user 
approval nor a copy of the data preserved durably.

Such implementation is fine for POC and some tests, but it cannot be allowed in 
the release version of the product.
h3. New proposed solution

As trivial as it may seem, new solution is to _not deleting data_ before 
snapshot is fully downloaded and ready for swap. Why is it trivial? Because 
this is literally what RAFT demands to be done.

Of course, there's a {*}but{*}. Snapshot application, when it's downloaded, 
should be {{O(1)}} when it comes to the number of rows in the partition and a 
number of transactions in a tx state store. This may not be fully achievable, 
depending on the implementation that we chose, more on that later.

Following sections will describe all my concerns and possible implementations. 
Some sections can be skipped while reading. For example, if you're not 
interested in a specific storage engine, but want to read everything else.
h3. TX state storage

There's one really good thing about TX state storage. It has no storage engine, 
there's only a single RocksDB-based implementation. This makes possible the 
following approach:
 * when we stream data, we can write it into a SST file, almost like in 
snapshots of meta-storage ans CMG storages
 * once snapshot is downloaded, we ingest it into a storage

What I like about this solution is that it's very simple. But, there are 
concerns:
 * ingesting leads to additional implicit data flush. Maybe it can be avoided, 
more on that later
 * it's not clear whether RocksDB creates a copy of SST file or not. I would 
assume that it does, because the file might be in other folder or on another 
device, for example. Although copying files is fast, it still takes time. Add 
to this a time required for the flush and we see a problem - operation may 
become unnecessarily long

For these reasons, I don't think that such solution should be implemented. The 
point of this description was to show, that I thought about this alternative 
and consciously decided to use another one.

I believe that TX state storage should use the same approach as a RocksDB-based 
partition storage. Its description can be found later in this issue.
h3. MV storage - Test engine

Test uses concurrent skip-list map for MV data and a bunch of other maps for 
indexes. While snapshots is being downloaded, we should insert all data into 
new maps, that have the same structure. In the end, we should have two versions 
of the partition: old and new.

{{onSnapshotLoad}} should just swap all objects. After that, old partition data 
can be cleaned by the garbage collector.

This is a good place to start implementation. I assume that some new API will 
be introduced. I have thoughts about it as well, they are described later in 
*API* section.
h3. MV storage - RocksDB engine

SST-based approach is described in a *TX state storage* section. There I 
describe why I don't think that this is a good solution. Same reasoning can be 
applied here just as effectively. This means that we should write data in the 
same RocksDB instance. This is a little bit tricky.

The reason is that all stored data is merged together, and Columns Families are 
shared between different partitions. This makes it harder to find a place to 
write partition data while old partition data persists. As a reminder and an 
example, let's take a look at how data is stored in row storage:
{code:java}
+---+-+---+
|Partition 0| ... |   Partition MAX   |
+---+-+---+
| Row1 | ... | RowN | ... | Row1 | ... | RowN |
+---+-+---+{code}
Logically, CF is split into a different "blocks", and each block represents a 
partition. Currently, each partition block is associated with an 2-bytes 
identifier that matches a partition number in Big Endian.

 

We could add new CF with similar structure and write snapshot data in it, but 
then the snapshot load operation would require us to move data from one CF to 
another. The only option that I know of, that can do this, is SST ingestion. 
And I already explained why I don't like it.

This leaves us with the necessity to write data into the same column family. 

[jira] [Updated] (IGNITE-17990) Download RAFT snapshot without deleting original partition data

[Ivan Bessonov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Bessonov updated IGNITE-17990:
---
Description: 
h3. Description

In first design, full rebalance is implemented this way:
 * we drop partition data
 * we download partition data from the leader
 * we're done

There's a problem with this approach - if download part failed, we lost one 
follower. This is bad, because technically new leader may have more data in the 
log and it could have uploaded it the follower, but now it makes no sense.

Not only can it lead to hard-to-catch errors and introducing custom code to 
JRaft, it's also an unconditional data deletion without neither explicit  user 
approval nor a copy of the data preserved durably.

Such implementation is fine for POC and some tests, but it cannot be allowed in 
the release version of the product.
h3. New proposed solution

As trivial as it may seem, new solution is to _not deleting data_ before 
snapshot is fully downloaded and ready for swap. Why is it trivial? Because 
this is literally what RAFT demands to be done.

Of course, there's a {*}but{*}. Snapshot application, when it's downloaded, 
should be {{O(1)}} when it comes to the number of rows in the partition and a 
number of transactions in a tx state store. This may not be fully achievable, 
depending on the implementation that we chose, more on that later.

Following sections will describe all my concerns and possible implementations. 
Some sections can be skipped while reading. For example, if you're not 
interested in a specific storage engine, but want to read everything else.
h3. TX state storage

There's one really good thing about TX state storage. It has no storage engine, 
there's only a single RocksDB-based implementation. This makes possible the 
following approach:
 * when we stream data, we can write it into a SST file, almost like in 
snapshots of meta-storage ans CMG storages
 * once snapshot is downloaded, we ingest it into a storage

What I like about this solution is that it's very simple. But, there are 
concerns:
 * ingesting leads to additional implicit data flush. Maybe it can be avoided, 
more on that later
 * it's not clear whether RocksDB creates a copy of SST file or not. I would 
assume that it does, because the file might be in other folder or on another 
device, for example. Although copying files is fast, it still takes time. Add 
to this a time required for the flush and we see a problem - operation may 
become unnecessarily long

For these reasons, I don't think that such solution should be implemented. The 
point of this description was to show, that I thought about this alternative 
and consciously decided to use another one.

I believe that TX state storage should use the same approach as a RocksDB-based 
partition storage. Its description can be found later in this issue.
h3. MV storage - Test engine

Test uses concurrent skip-list map for MV data and a bunch of other maps for 
indexes. While snapshots is being downloaded, we should insert all data into 
new maps, that have the same structure. In the end, we should have two versions 
of the partition: old and new.

{{onSnapshotLoad}} should just swap all objects. After that, old partition data 
can be cleaned by the garbage collector.

This is a good place to start implementation. I assume that some new API will 
be introduced. I have thoughts about it as well, they are described later in 
*API* section.
h3. MV storage - RocksDB engine

SST-based approach is described in a *TX state storage* section. There I 
describe why I don't think that this is a good solution. Same reasoning can be 
applied here just as effectively. This means that we should write data in the 
same RocksDB instance. This is a little bit tricky.

The reason is that all stored data is merged together, and Columns Families are 
shared between different partitions. This makes it harder to find a place to 
write partition data while old partition data persists. As a reminder and an 
example, let's take a look at how data is stored in row storage:

 
{code:java}
+---+-+---+
|Partition 0| ... |   Partition MAX   |
+---+-+---+
| Row1 | ... | RowN | ... | Row1 | ... | RowN |
+---+-+---+{code}
Logically, CF is split into a different "blocks", and each block represents a 
partition. Currently, each partition block is associated with an 2-bytes 
identifier that matches a partition number in Big Endian.

 

We could add new CF with similar structure and write snapshot data in it, but 
then the snapshot load operation would require us to move data from one CF to 
another. The only option that I know of, that can do this, is SST ingestion. 
And I already explained why I don't like it.

This leaves us with the necessity to write data into the same column 

[jira] [Assigned] (IGNITE-17995) Idle_verify must print broken partitions list

[Ilya Shishkov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ilya Shishkov reassigned IGNITE-17995:
--

Assignee: Ilya Shishkov

> Idle_verify must print broken partitions list
> -
>
> Key: IGNITE-17995
> URL: https://issues.apache.org/jira/browse/IGNITE-17995
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Anton Vinogradov
>Assignee: Ilya Shishkov
>Priority: Major
>  Labels: ise
>
> Currently, idle_verify provides per partition information, but not just a set 
> of broken partitions.
> So, additional actions is necessary to get the list required by consistency 
> repair tool.
> {noformat}
>  idle_verify task was executed with the following args: caches=[], 
> excluded=[], cacheFilter=[DEFAULT] 
>  idle_verify check has finished, found XXX conflict partitions: 
> [counterConflicts=0, hashConflicts=XXX] 
>  Hash conflicts: 
>  Conflict partition: PartitionKeyV2 [grpId=CacheGroup1, 
> grpName=CacheGroup1Cache, partId=600]
>  ...
>  Control utility [ver. XXX] 
>  2021 Copyright(C) Apache Software Foundation 
>  User: xxx 
>  Time: XXX
>  
> 
>  
>  Command [CACHE] finished with code: 0 
>  Control utility has completed execution at: XXX 
>  Execution time: XXX ms 
> {noformat}
> Let's, in addition to detailed view, provide such lists.
> Something like:
> {noformat}
> Total:
> CacheGroup1 (18/1024)
> 1,7,42,987,99
> CacheGroup2 (15/1024)
> 5,14,17,850,900
> ...
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17995) Idle_verify must print broken partitions list

[Anton Vinogradov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Vinogradov updated IGNITE-17995:
--
Labels: ise  (was: )

> Idle_verify must print broken partitions list
> -
>
> Key: IGNITE-17995
> URL: https://issues.apache.org/jira/browse/IGNITE-17995
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Anton Vinogradov
>Priority: Major
>  Labels: ise
>
> Currently, idle_verify provides per partition information, but not just a set 
> of broken partitions.
> So, additional actions is necessary to get the list required by consistency 
> repair tool.
> {noformat}
>  idle_verify task was executed with the following args: caches=[], 
> excluded=[], cacheFilter=[DEFAULT] 
>  idle_verify check has finished, found XXX conflict partitions: 
> [counterConflicts=0, hashConflicts=XXX] 
>  Hash conflicts: 
>  Conflict partition: PartitionKeyV2 [grpId=CacheGroup1, 
> grpName=CacheGroup1Cache, partId=600]
>  ...
>  Control utility [ver. XXX] 
>  2021 Copyright(C) Apache Software Foundation 
>  User: xxx 
>  Time: XXX
>  
> 
>  
>  Command [CACHE] finished with code: 0 
>  Control utility has completed execution at: XXX 
>  Execution time: XXX ms 
> {noformat}
> Let's, in addition to detailed view, provide such lists.
> Something like:
> {noformat}
> Total:
> CacheGroup1 (18/1024)
> 1,7,42,987,99
> CacheGroup2 (15/1024)
> 5,14,17,850,900
> ...
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17995) Idle_verify must print broken partitions list

[Anton Vinogradov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Vinogradov updated IGNITE-17995:
--
Description: 
Currently, idle_verify provides per partition information, but not just a set 
of broken partitions.
So, additional actions is necessary to get the list required by consistency 
repair tool.
{noformat}
 idle_verify task was executed with the following args: caches=[], excluded=[], 
cacheFilter=[DEFAULT] 
 idle_verify check has finished, found XXX conflict partitions: 
[counterConflicts=0, hashConflicts=XXX] 
 Hash conflicts: 
 Conflict partition: PartitionKeyV2 [grpId=CacheGroup1, 
grpName=CacheGroup1Cache, partId=600]
 ...

 Control utility [ver. XXX] 
 2021 Copyright(C) Apache Software Foundation 
 User: xxx 
 Time: XXX
 

 
 Command [CACHE] finished with code: 0 
 Control utility has completed execution at: XXX 
 Execution time: XXX ms 
{noformat}
Let's, in addition to detailed view, provide such lists.
Something like:
{noformat}
Total:
CacheGroup1 (18/1024)
1,7,42,987,99

CacheGroup2 (15/1024)
5,14,17,850,900

...
{noformat}

  was:
Currently, idle_verify provides per partition information, but not just a set 
of broken partitions.
So, additional actions is necessary to get the list required by consistency 
repair tool.

Let's, in addition to detailed view, provide such lists.
Something like:
{noformat}
Total:
CacheGroup1 (18/1024)
1,7,42,987,99

CacheGroup2 (18/1024)
5,14,17,850,900

...
{noformat}


> Idle_verify must print broken partitions list
> -
>
> Key: IGNITE-17995
> URL: https://issues.apache.org/jira/browse/IGNITE-17995
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Anton Vinogradov
>Priority: Major
>
> Currently, idle_verify provides per partition information, but not just a set 
> of broken partitions.
> So, additional actions is necessary to get the list required by consistency 
> repair tool.
> {noformat}
>  idle_verify task was executed with the following args: caches=[], 
> excluded=[], cacheFilter=[DEFAULT] 
>  idle_verify check has finished, found XXX conflict partitions: 
> [counterConflicts=0, hashConflicts=XXX] 
>  Hash conflicts: 
>  Conflict partition: PartitionKeyV2 [grpId=CacheGroup1, 
> grpName=CacheGroup1Cache, partId=600]
>  ...
>  Control utility [ver. XXX] 
>  2021 Copyright(C) Apache Software Foundation 
>  User: xxx 
>  Time: XXX
>  
> 
>  
>  Command [CACHE] finished with code: 0 
>  Control utility has completed execution at: XXX 
>  Execution time: XXX ms 
> {noformat}
> Let's, in addition to detailed view, provide such lists.
> Something like:
> {noformat}
> Total:
> CacheGroup1 (18/1024)
> 1,7,42,987,99
> CacheGroup2 (15/1024)
> 5,14,17,850,900
> ...
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (IGNITE-17995) Idle_verify must print broken partitions list

[Anton Vinogradov (Jira)]

Anton Vinogradov created IGNITE-17995:
-

 Summary: Idle_verify must print broken partitions list
 Key: IGNITE-17995
 URL: https://issues.apache.org/jira/browse/IGNITE-17995
 Project: Ignite
  Issue Type: Sub-task
Reporter: Anton Vinogradov


Currently, idle_verify provides per partition information, but not just a set 
of broken partitions.
So, additional actions is necessary to get the list required by consistency 
repair tool.

Let's, in addition to detailed view, provide such lists.
Something like:
{noformat}
Total:
CacheGroup1 (18/1024)
1,7,42,987,99

CacheGroup2 (18/1024)
5,14,17,850,900

...
{noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (IGNITE-17990) Download RAFT snapshot without deleting original partition data

[Ivan Bessonov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Bessonov updated IGNITE-17990:
---
Description: 
h3. Description

In first design, full rebalance is implemented this way:
 * we drop partition data
 * we download partition data from the leader
 * we're done

There's a problem with this approach - if download part failed, we lost one 
follower. This is bad, because technically new leader may have more data in the 
log and it could have uploaded it the follower, but now it makes no sense.

Not only can it lead to hard-to-catch errors and introducing custom code to 
JRaft, it's also an unconditional data deletion without neither explicit  user 
approval nor a copy of the data preserved durably.

Such implementation is fine for POC and some tests, but it cannot be allowed in 
the release version of the product.
h3. New proposed solution

As trivial as it may seem, new solution is to _not deleting data_ before 
snapshot is fully downloaded and ready for swap. Why is it trivial? Because 
this is literally what RAFT demands to be done.

Of course, there's a {*}but{*}. Snapshot application, when it's downloaded, 
should be {{O(1)}} when it comes to the number of rows in the partition and a 
number of transactions in a tx state store. This may not be fully achievable, 
depending on the implementation that we chose, more on that later.

Following sections will describe all my concerns and possible implementations. 
Some sections can be skipped while reading. For example, if you're not 
interested in a specific storage engine, but want to read everything else.
h3. TX state storage

There's one really good thing about TX state storage. It has no storage engine, 
there's only a single RocksDB-based implementation. This makes possible the 
following approach:
 * when we stream data, we can write it into a SST file, almost like in 
snapshots of meta-storage ans CMG storages
 * once snapshot is downloaded, we ingest it into a storage

What I like about this solution is that it's very simple. But, there are 
concerns:
 * ingesting leads to implicit data flush. Technically, RocksDB could implement 
a "load" operation that doesn't require that
 * it's not clear whether RocksDB creates a copy of SST file or not. I would 
assume that it does, because the file might be in other folder or on another 
device, for example. Although copying files is fast, it still takes time. Add 
to this a time required for the flush and we see a problem - operation may 
become unnecessarily long

For these reasons, I don't think that such solution should be implemented. The 
point of this description was to show, that I thought about this alternative 
and consciously decided to use another one.

I believe that TX state storage should use the same approach as a RocksDB-based 
partition storage. Its description can be found later in this issue.
h3. MV storage - Test engine

Test uses concurrent skip-list map for MV data and a bunch of other maps for 
indexes. While snapshots is being downloaded, we should insert all data into 
new maps, that have the same structure. In the end, we should have two versions 
of the partition: old and new.

{{onSnapshotLoad}} should just swap all objects. After that, old partition data 
can be cleaned by the garbage collector.

This is a good place to start implementation. I assume that some new API will 
be introduced. I have thoughts about it as well, they are described later in 
*API* section.
h3. MV storage - RocksDB engine

SST-based approach is described in a *TX state storage* section. There I 
describe why I don't think that this is a good solution. Same reasoning can be 
applied here just as effectively. This means that we should write data in the 
same RocksDB instance. This is a little bit tricky.

The reason is that all stored data is merged together, and Columns Families are 
shared between different partitions. This makes it harder to find a place to 
write partition data while old partition data persists. As a reminder and an 
example, let's take a look at how data is stored in row storage:

 
{code:java}
+---+-+---+
|Partition 0| ... |   Partition MAX   |
+---+-+---+
| Row1 | ... | RowN | ... | Row1 | ... | RowN |
+---+-+---+{code}
Logically, CF is split into a different "blocks", and each block represents a 
partition. Currently, each partition block is associated with an 2-bytes 
identifier that matches a partition number in Big Endian.

 

We could add new CF with similar structure and write snapshot data in it, but 
then the snapshot load operation would require us to move data from one CF to 
another. The only option that I know of, that can do this, is SST ingestion. 
And I already explained why I don't like it.

This leaves us with the necessity to write data 

[jira] [Assigned] (IGNITE-16092) Read Repair should print all object's fields to the consistency.log

[Anton Vinogradov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-16092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Vinogradov reassigned IGNITE-16092:
-

Assignee: (was: Anton Vinogradov)

> Read Repair should print all object's fields to the consistency.log
> ---
>
> Key: IGNITE-16092
> URL: https://issues.apache.org/jira/browse/IGNITE-16092
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Anton Vinogradov
>Priority: Major
>  Labels: iep-31
>
> We should make sure that random non-primitive types are logged not as 
> Object$535235234, but as fields with fields with fields and so on.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (IGNITE-16064) Missed values should be logged by Read Repair

[Anton Vinogradov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-16064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Vinogradov reassigned IGNITE-16064:
-

Assignee: (was: Anton Vinogradov)

> Missed values should be logged by Read Repair
> -
>
> Key: IGNITE-16064
> URL: https://issues.apache.org/jira/browse/IGNITE-16064
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Anton Vinogradov
>Priority: Major
>  Labels: iep-31
>
> Events should contain info about missed values.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (IGNITE-16289) Consistency check should make sure that each found key belongs to the partition where it was found.

[Anton Vinogradov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-16289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Vinogradov reassigned IGNITE-16289:
-

Assignee: (was: Anton Vinogradov)

> Consistency check should make sure that each found key belongs to the 
> partition where it was found.
> ---
>
> Key: IGNITE-16289
> URL: https://issues.apache.org/jira/browse/IGNITE-16289
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Anton Vinogradov
>Priority: Major
>  Labels: iep-31
>
> We must have some check (maybe at regular get operation) that the key's 
> hashCode belongs to the partitions.
> At least, at consistency check, we must check each key found using an 
> off-heap iterator.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (IGNITE-16224) Read Repair attempts must be limited to avoid stack overflow

[Anton Vinogradov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-16224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Vinogradov reassigned IGNITE-16224:
-

Assignee: (was: Anton Vinogradov)

> Read Repair attempts must be limited to avoid stack overflow
> 
>
> Key: IGNITE-16224
> URL: https://issues.apache.org/jira/browse/IGNITE-16224
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Anton Vinogradov
>Priority: Major
>
> Currently, we perform `repairAsync` on every consistency violation.
> {noformat}
> protected Map repairableGetAll(
> Collection keys,
> boolean deserializeBinary,
> boolean needVer,
> boolean recovery,
> ReadRepairStrategy readRepairStrategy) throws IgniteCheckedException {
> try {
> return getAll(keys, deserializeBinary, needVer, recovery, 
> readRepairStrategy);
> }
> catch (IgniteIrreparableConsistencyViolationException e) {
> throw e;
> }
> catch (IgniteConsistencyViolationException e) {
> repairAsync(keys, ctx.operationContextPerCall(), false).get();
> return repairableGetAll(keys, deserializeBinary, needVer, 
> recovery, readRepairStrategy);
> }
> }
> {noformat}
> In case of concurrent modifications and/or some bugs, stack overflow is 
> possible here.
> Rechecks and repairs should be limited.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (IGNITE-16931) Read Repair should support unstable topology

[Anton Vinogradov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-16931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Vinogradov reassigned IGNITE-16931:
-

Assignee: (was: Anton Vinogradov)

> Read Repair should support unstable topology
> 
>
> Key: IGNITE-16931
> URL: https://issues.apache.org/jira/browse/IGNITE-16931
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Anton Vinogradov
>Priority: Major
>  Labels: iep-31
>
> Currently RR does not support unstable topology (when not all owners are 
> located by affinity) and this can be fixed.
> As a start point, it's a good idea to consider a replacement
> {noformat}
> for (KeyCacheObject key : keys) {
> List nodes = ctx.affinity().nodesByKey(key, 
> topVer); // affinity
> primaryNodes.put(key, nodes.get(0));
> ...
> {noformat}
> to
> {noformat}
> for (KeyCacheObject key : keys) {
> List nodes = 
> ctx.topology().nodes(key.partition(), topVer); // topology
> primaryNodes.put(key, nodes.get(0));
> ...
> {noformat}
> at 
> {{{}org.apache.ignite.internal.processors.cache.distributed.near.consistency.GridNearReadRepairAbstractFuture#map{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (IGNITE-15606) Consistency repair ducktest to check the performance

[Anton Vinogradov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-15606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Vinogradov reassigned IGNITE-15606:
-

Assignee: (was: Anton Vinogradov)

> Consistency repair ducktest to check the performance
> 
>
> Key: IGNITE-15606
> URL: https://issues.apache.org/jira/browse/IGNITE-15606
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Anton Vinogradov
>Priority: Major
>  Labels: iep-31
>
> Goal is to check/tune the performance. Make consistency repair as fast as 
> possible.
> - tune the default getAll items number.
> - ...



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (IGNITE-15316) Read Repair may see inconsistent entry when it is consistent but updated right before the check

[Anton Vinogradov (Jira)]

 [ 
https://issues.apache.org/jira/browse/IGNITE-15316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Vinogradov reassigned IGNITE-15316:
-

Assignee: (was: Anton Vinogradov)

> Read Repair may see inconsistent entry when it is consistent but updated 
> right before the check
> ---
>
> Key: IGNITE-15316
> URL: https://issues.apache.org/jira/browse/IGNITE-15316
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Anton Vinogradov
>Priority: Major
>  Labels: iep-31
>
> Even at FULL_SYNC mode stale reads are possible from backups after the lock 
> is obtained by "Read Repair" tx.
> This is possible because (at previous tx) entry becomes unlocked (committed) 
> on primary before tx committed on backups.
> This is not a problem for Ignite (since backups keep locks until updated) but 
> produces false positive "inconsistency state found" events and repairs.
> As to Atomic caches, there is even no chance to lock entry before the check, 
> so, the inconsistency window is wider than in the tx case.
> This problem does not allow to use ReadRepair with concurrent modifications, 
> since repair may happen because of an inconsistent read (while another 
> operation is in progress), not because of real inconsistency.
> A possible solution is to implement fake updates, which will guarantee that 
> the previous update is fully finished -> consistent read.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)