[jira] [Updated] (IGNITE-14937) Introduce index management

2022-08-03 Thread Konstantin Orlov (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-14937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Orlov updated IGNITE-14937:
--
Description: 
We need to introduce component to handle index-related manipulation commands 
(like CREATE or DROP) as well as manage indices' lifecycle.

As a first phase let's introduce index manager with functionality reduced to 
the following duties:
* accepting CREATE|DROP command and changing the schema
* handling the event from configuration and creating reduced set of 
index-related structures (index description only for now) on all necessary 
nodes  

  was:
Public index schema (required indexes) and current indexes state on the cluster 
are different.
We have to track it, store it and provide actual indexes schema state for any 
components: select query, DDL query etc..


> Introduce index management
> --
>
> Key: IGNITE-14937
> URL: https://issues.apache.org/jira/browse/IGNITE-14937
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Reporter: Taras Ledkov
>Assignee: Konstantin Orlov
>Priority: Major
>  Labels: ignite-3
>
> We need to introduce component to handle index-related manipulation commands 
> (like CREATE or DROP) as well as manage indices' lifecycle.
> As a first phase let's introduce index manager with functionality reduced to 
> the following duties:
> * accepting CREATE|DROP command and changing the schema
> * handling the event from configuration and creating reduced set of 
> index-related structures (index description only for now) on all necessary 
> nodes  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-14937) Introduce index management

2022-08-03 Thread Konstantin Orlov (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-14937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Orlov updated IGNITE-14937:
--
Summary: Introduce index management  (was: Index schema & Index management 
integration)

> Introduce index management
> --
>
> Key: IGNITE-14937
> URL: https://issues.apache.org/jira/browse/IGNITE-14937
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Reporter: Taras Ledkov
>Assignee: Konstantin Orlov
>Priority: Major
>  Labels: ignite-3
>
> Public index schema (required indexes) and current indexes state on the 
> cluster are different.
> We have to track it, store it and provide actual indexes schema state for any 
> components: select query, DDL query etc..



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (IGNITE-17457) Cluster locks after the transaction recovery procedure if the tx primary node fail

2022-08-03 Thread Ignite TC Bot (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574993#comment-17574993
 ] 

Ignite TC Bot commented on IGNITE-17457:


{panel:title=Branch: [pull/10178/head] Base: [master] : No blockers 
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/10178/head] Base: [master] : New Tests 
(1)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}Cache 12{color} [[tests 
1|https://ci.ignite.apache.org/viewLog.html?buildId=6710160]]
* {color:#013220}IgniteCacheTestSuite12: 
TxRecoveryWithConcurrentRollbackTest.testRecoveryNotDeadLockOnPrimaryFail - 
PASSED{color}

{panel}
[TeamCity *-- Run :: All* 
Results|https://ci.ignite.apache.org/viewLog.html?buildId=6710922buildTypeId=IgniteTests24Java8_RunAll]

> Cluster locks after the transaction recovery procedure if the tx primary node 
> fail
> --
>
> Key: IGNITE-17457
> URL: https://issues.apache.org/jira/browse/IGNITE-17457
> Project: Ignite
>  Issue Type: Bug
>Reporter: Sergey Korotkov
>Assignee: Sergey Korotkov
>Priority: Major
>
> Ignite cluster may be locked (all client operations would block) after the tx 
> recovery procedure executed on the tx primary node failure.
> The prepared transaction may remain un-commited on the backup node after the 
> tx recovery.  So the partition exchange wouldn't complete. So cluster would 
> be locked.
> 
> The Immediate reason is the race condition in the method:
> {code:java}
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter::markFinalizing(RECOVERY_FINISH){code}
> If 2 or more backups are configured It may be called concurrently for the 
> same transaction both from the recovery procedure:
> {code:java}
> IgniteTxManager::commitIfPrepared{code}
> and from the tx recovery request handler:
> {code:java}
> IgniteTxHandler::processCheckPreparedTxRequest{code}
> Problem occur if thread context is switched between old finalization status 
> request and status update.
> 
> The problematic sequence of events is as follows (the lock will be in the 
> node1):
> 1. Start cluster with 3 nodes (node0, node1, node2) and cache with 2 backups.
> 2. On node2 start and prepare transaction choosing key with primary partition 
> stored on node2.
> 3. Kill node2
> 4. The tx recovery procedure is started both on node0 and node1
> 5. In scope of the recovery procedure node0 sends tx recovery request to node1
> 6. The following steps are executed on the node1 in two threads ("procedure" 
> which is a system pool thread executing the tx recovery procedure and 
> "handler" which is a striped pool thread processing the tx recovery request 
> sent from node0):
>  - tx.finalization == NONE
>  - "procedure": calls markFinalizing(RECOVERY_FINISH)
>  - "handler": calls markFinalizing(RECOVERY_FINISH)
>  - "procedure": gets old tx.finlalization - it's NONE
>  - "handler": gets old tx.finalization - it's NONE
>  - "handler": updates tx.finalization - now it's RECOVERY_FINISH
>  - "procedure": tries to update tx.finalization via compareAndSet and fails 
> since compare fails.
>  - "procedure": stops transaction processing and does not try to commit it.
>  - Transaction remains not finished on node1.
> 
> Reproducer is in the pull request.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (IGNITE-17463) AssertionError: Logging at INFO level without checking if INFO level

2022-08-03 Thread Ignite TC Bot (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574975#comment-17574975
 ] 

Ignite TC Bot commented on IGNITE-17463:


{panel:title=Branch: [pull/10180/head] Base: [master] : Possible Blockers 
(1)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}
{color:#d04437}Control Utility{color} [[tests 0 TIMEOUT , Exit Code 
|https://ci.ignite.apache.org/viewLog.html?buildId=6711744]]

{panel}
{panel:title=Branch: [pull/10180/head] Base: [master] : No new tests 
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}{panel}
[TeamCity *-- Run :: All* 
Results|https://ci.ignite.apache.org/viewLog.html?buildId=6712038buildTypeId=IgniteTests24Java8_RunAll]

> AssertionError: Logging at INFO level without checking if INFO level
> 
>
> Key: IGNITE-17463
> URL: https://issues.apache.org/jira/browse/IGNITE-17463
> Project: Ignite
>  Issue Type: Bug
>Reporter: Maxim Muzafarov
>Assignee: Maxim Muzafarov
>Priority: Major
> Fix For: 2.14
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The suite failed with the follwoing stacktrace.
> {code}
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] Exception 
> in thread "shutdown-hook" java.lang.AssertionError: Logging at INFO level 
> without checking if INFO level is enabled: No verification for local node 
> leave has been received from coordinator (will stop node anyway).
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.testframework.junits.logger.GridTestLog4jLogger.info(GridTestLog4jLogger.java:522)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.testframework.ListeningTestLogger.info(ListeningTestLogger.java:157)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.internal.GridLoggerProxy.info(GridLoggerProxy.java:137)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStop0(ServerImpl.java:555)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStop(ServerImpl.java:492)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStop(TcpDiscoverySpi.java:2345)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.internal.managers.GridManagerAdapter.stopSpi(GridManagerAdapter.java:313)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.stop(GridDiscoveryManager.java:1777)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:1928)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:1806)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2340)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2163)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$4.run(IgnitionEx.java:1806)
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (IGNITE-13087) java.nio.file.FileSystemException: /home/ranger/EIIP/tools/work/db/ServerNode/cache-TOFTableCache/part-942.bin: Too many open files

2022-08-03 Thread Jeremy McMillan (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-13087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574927#comment-17574927
 ] 

Jeremy McMillan commented on IGNITE-13087:
--

This does not appear to be an Apache Ignite issue.

Please check and post `ulimit -a` settings to confirm your max open files are 
effective for the shell which launches your JVM. Maybe ulimits work differently 
than you expect?

See also:
https://unix.stackexchange.com/questions/75996/modify-ulimit-open-files-of-a-specific-process

> java.nio.file.FileSystemException: 
> /home/ranger/EIIP/tools/work/db/ServerNode/cache-TOFTableCache/part-942.bin: 
> Too many open files
> ---
>
> Key: IGNITE-13087
> URL: https://issues.apache.org/jira/browse/IGNITE-13087
> Project: Ignite
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 2.8.1
> Environment: Ignite 2.8.1
> Java 1.8.0_241
> Ubuntu 18.04
>  
>Reporter: RangerZhou
>Priority: Blocker
> Attachments: Ignite_VM_log
>
>
> I start a server and persistent, then I start a client, after a while I face 
> the exception like this:
> {code:java}
> [2020-06-04 11:04:52,769][ERROR][sys-stripe-6-#7][root] Critical system error 
> detected. Will be handled accordingly to configured handler 
> [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, 
> super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet 
> [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], 
> failureCtx=FailureContext [type=CRITICAL_ERROR, err=class 
> o.a.i.i.processors.cache.persistence.StorageException: Failed to initialize 
> partition file: 
> /home/ranger/EIIP/tools/work/db/ServerNode/cache-TOFTableCache/part-942.bin]][2020-06-04
>  11:04:52,769][ERROR][sys-stripe-6-#7][root] Critical system error detected. 
> Will be handled accordingly to configured handler 
> [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, 
> super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet 
> [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], 
> failureCtx=FailureContext [type=CRITICAL_ERROR, err=class 
> o.a.i.i.processors.cache.persistence.StorageException: Failed to initialize 
> partition file: 
> /home/ranger/EIIP/tools/work/db/ServerNode/cache-TOFTableCache/part-942.bin]]class
>  org.apache.ignite.internal.processors.cache.persistence.StorageException: 
> Failed to initialize partition file: 
> /home/ranger/EIIP/tools/work/db/ServerNode/cache-TOFTableCache/part-942.bin 
> at 
> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:553)
>  at 
> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:437)
>  at 
> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:521)
>  at 
> org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:505)
>  at 
> org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:880)
>  at 
> org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:707)
>  at 
> org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.getOrAllocatePartitionMetas(GridCacheOffheapManager.java:1909)
>  at 
> org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.init0(GridCacheOffheapManager.java:1693)
>  at 
> org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:2442)
>  at 
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:445)
>  at 
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2314)
>  at 
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2641)
>  at 
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:2102)
>  at 
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1919)
>  at 
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1719)
>  at 
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3306)
>  at 
> 

[jira] [Updated] (IGNITE-17451) Provide internal access to BinaryContext in thin client

2022-08-03 Thread Amelchev Nikita (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amelchev Nikita updated IGNITE-17451:
-
Description: Provide internal access to BinaryContext in thin client.  
(was: Implement thin client CDC streamer metadata replication)

> Provide internal access to BinaryContext in thin client
> ---
>
> Key: IGNITE-17451
> URL: https://issues.apache.org/jira/browse/IGNITE-17451
> Project: Ignite
>  Issue Type: Task
>Reporter: Amelchev Nikita
>Assignee: Amelchev Nikita
>Priority: Major
>  Labels: IEP-59, ise
> Fix For: 2.14
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Provide internal access to BinaryContext in thin client.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-17451) Provide internal access to BinaryContext in thin client

2022-08-03 Thread Amelchev Nikita (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amelchev Nikita updated IGNITE-17451:
-
Summary: Provide internal access to BinaryContext in thin client  (was: 
Implement thin client CDC streamer metadata replication)

> Provide internal access to BinaryContext in thin client
> ---
>
> Key: IGNITE-17451
> URL: https://issues.apache.org/jira/browse/IGNITE-17451
> Project: Ignite
>  Issue Type: Task
>Reporter: Amelchev Nikita
>Assignee: Amelchev Nikita
>Priority: Major
>  Labels: IEP-59, ise
> Fix For: 2.14
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Implement thin client CDC streamer metadata replication



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (IGNITE-17463) AssertionError: Logging at INFO level without checking if INFO level

2022-08-03 Thread Maxim Muzafarov (Jira)
Maxim Muzafarov created IGNITE-17463:


 Summary: AssertionError: Logging at INFO level without checking if 
INFO level
 Key: IGNITE-17463
 URL: https://issues.apache.org/jira/browse/IGNITE-17463
 Project: Ignite
  Issue Type: Bug
Reporter: Maxim Muzafarov
Assignee: Maxim Muzafarov
 Fix For: 2.14


The suite failed with the follwoing stacktrace.

{code}
[18:11:11]W: [org.apache.ignite:ignite-control-utility] Exception 
in thread "shutdown-hook" java.lang.AssertionError: Logging at INFO level 
without checking if INFO level is enabled: No verification for local node leave 
has been received from coordinator (will stop node anyway).
[18:11:11]W: [org.apache.ignite:ignite-control-utility] at 
org.apache.ignite.testframework.junits.logger.GridTestLog4jLogger.info(GridTestLog4jLogger.java:522)
[18:11:11]W: [org.apache.ignite:ignite-control-utility] at 
org.apache.ignite.testframework.ListeningTestLogger.info(ListeningTestLogger.java:157)
[18:11:11]W: [org.apache.ignite:ignite-control-utility] at 
org.apache.ignite.internal.GridLoggerProxy.info(GridLoggerProxy.java:137)
[18:11:11]W: [org.apache.ignite:ignite-control-utility] at 
org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStop0(ServerImpl.java:555)
[18:11:11]W: [org.apache.ignite:ignite-control-utility] at 
org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStop(ServerImpl.java:492)
[18:11:11]W: [org.apache.ignite:ignite-control-utility] at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStop(TcpDiscoverySpi.java:2345)
[18:11:11]W: [org.apache.ignite:ignite-control-utility] at 
org.apache.ignite.internal.managers.GridManagerAdapter.stopSpi(GridManagerAdapter.java:313)
[18:11:11]W: [org.apache.ignite:ignite-control-utility] at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.stop(GridDiscoveryManager.java:1777)
[18:11:11]W: [org.apache.ignite:ignite-control-utility] at 
org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:1928)
[18:11:11]W: [org.apache.ignite:ignite-control-utility] at 
org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:1806)
[18:11:11]W: [org.apache.ignite:ignite-control-utility] at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2340)
[18:11:11]W: [org.apache.ignite:ignite-control-utility] at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2163)
[18:11:11]W: [org.apache.ignite:ignite-control-utility] at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$4.run(IgnitionEx.java:1806)
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-17463) AssertionError: Logging at INFO level without checking if INFO level

2022-08-03 Thread Maxim Muzafarov (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maxim Muzafarov updated IGNITE-17463:
-
Ignite Flags:   (was: Docs Required,Release Notes Required)

> AssertionError: Logging at INFO level without checking if INFO level
> 
>
> Key: IGNITE-17463
> URL: https://issues.apache.org/jira/browse/IGNITE-17463
> Project: Ignite
>  Issue Type: Bug
>Reporter: Maxim Muzafarov
>Assignee: Maxim Muzafarov
>Priority: Major
> Fix For: 2.14
>
>
> The suite failed with the follwoing stacktrace.
> {code}
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] Exception 
> in thread "shutdown-hook" java.lang.AssertionError: Logging at INFO level 
> without checking if INFO level is enabled: No verification for local node 
> leave has been received from coordinator (will stop node anyway).
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.testframework.junits.logger.GridTestLog4jLogger.info(GridTestLog4jLogger.java:522)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.testframework.ListeningTestLogger.info(ListeningTestLogger.java:157)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.internal.GridLoggerProxy.info(GridLoggerProxy.java:137)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStop0(ServerImpl.java:555)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStop(ServerImpl.java:492)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStop(TcpDiscoverySpi.java:2345)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.internal.managers.GridManagerAdapter.stopSpi(GridManagerAdapter.java:313)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.stop(GridDiscoveryManager.java:1777)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:1928)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:1806)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2340)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2163)
> [18:11:11]W:   [org.apache.ignite:ignite-control-utility] at 
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$4.run(IgnitionEx.java:1806)
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-17451) [Extensions] Implement thin client CDC streamer metadata replication

2022-08-03 Thread Amelchev Nikita (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amelchev Nikita updated IGNITE-17451:
-
Ignite Flags:   (was: Docs Required,Release Notes Required)

> [Extensions] Implement thin client CDC streamer metadata replication
> 
>
> Key: IGNITE-17451
> URL: https://issues.apache.org/jira/browse/IGNITE-17451
> Project: Ignite
>  Issue Type: Task
>Reporter: Amelchev Nikita
>Assignee: Amelchev Nikita
>Priority: Major
>  Labels: IEP-59, ise
> Fix For: 2.14
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Implement thin client CDC streamer metadata replication



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-17451) Implement thin client CDC streamer metadata replication

2022-08-03 Thread Amelchev Nikita (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amelchev Nikita updated IGNITE-17451:
-
Summary: Implement thin client CDC streamer metadata replication  (was: 
[Extensions] Implement thin client CDC streamer metadata replication)

> Implement thin client CDC streamer metadata replication
> ---
>
> Key: IGNITE-17451
> URL: https://issues.apache.org/jira/browse/IGNITE-17451
> Project: Ignite
>  Issue Type: Task
>Reporter: Amelchev Nikita
>Assignee: Amelchev Nikita
>Priority: Major
>  Labels: IEP-59, ise
> Fix For: 2.14
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Implement thin client CDC streamer metadata replication



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-17451) [Extensions] Implement thin client CDC streamer metadata replication

2022-08-03 Thread Amelchev Nikita (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amelchev Nikita updated IGNITE-17451:
-
Fix Version/s: 2.14

> [Extensions] Implement thin client CDC streamer metadata replication
> 
>
> Key: IGNITE-17451
> URL: https://issues.apache.org/jira/browse/IGNITE-17451
> Project: Ignite
>  Issue Type: Task
>Reporter: Amelchev Nikita
>Assignee: Amelchev Nikita
>Priority: Major
>  Labels: IEP-59, ise
> Fix For: 2.14
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Implement thin client CDC streamer metadata replication



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (IGNITE-11486) Support Automatic modules for ignite-zookeeper: Migrate to log4j2 to resolve issues with logging packages conflict

2022-08-03 Thread Maxim Muzafarov (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-11486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574860#comment-17574860
 ] 

Maxim Muzafarov commented on IGNITE-11486:
--

Log4j was removed IGNITE-16650
I think the issue can be viewed from a new angle from now on.

> Support Automatic modules for ignite-zookeeper: Migrate to log4j2 to resolve 
> issues with logging packages conflict
> --
>
> Key: IGNITE-11486
> URL: https://issues.apache.org/jira/browse/IGNITE-11486
> Project: Ignite
>  Issue Type: Sub-task
>Reporter: Dmitry Pavlov
>Priority: Minor
>
> Usage of Ignite Zookeeper module in a modular environment failed
> {noformat}
> error: the unnamed module reads package org.apache.log4j from both 
> slf4j.log4j12 and log4j
> {noformat}
> slf4j version is updated by the build system when Ignite Zookeeper is used.
> {noformat}
> +--- org.slf4j:slf4j-api:1.7.7 -> 1.7.25
> +--- org.slf4j:slf4j-log4j12:1.7.7 -> 1.7.25
>   +--- org.slf4j:slf4j-api:1.7.25
>   \--- log4j:log4j:1.2.17
> {noformat}
> Probably we should update whole Ignite dependency for Slf4j
> Following combination seems to be compilable in a modular environment
> {code}
> compile (group: 'org.apache.ignite', name: 'ignite-zookeeper', version: 
> ignVer) {
> exclude group: "org.slf4j", module: "slf4j-log4j12"
> exclude group: "log4j", module: "log4j"
> }
> compile group: "org.slf4j", name: "slf4j-api", version: "1.7.25"
> compile group: "org.apache.logging.log4j", name: "log4j-slf4j-impl", 
> version: "2.11.2"
> {code}
> According to https://www.slf4j.org/news.html only 1.8x version is 
> modularized, but this major versions line contains only betas. See also 
> https://logging.apache.org/log4j/2.0/log4j-slf4j-impl/
> See also https://blogs.apache.org/logging/entry/moving_on_to_log4j_2
> Migration quide https://logging.apache.org/log4j/2.x/manual/migration.html



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (IGNITE-17462) Use of reflection to load NullAppender causes NullPointerException when NullAppender is removed

2022-08-03 Thread Bob (Jira)
Bob created IGNITE-17462:


 Summary: Use of reflection to load NullAppender causes 
NullPointerException when NullAppender is removed
 Key: IGNITE-17462
 URL: https://issues.apache.org/jira/browse/IGNITE-17462
 Project: Ignite
  Issue Type: Bug
  Components: general
Affects Versions: 2.13
Reporter: Bob


When using log4j 2.17, the use of Java reflection to create the NullAppender 
instance during node start causes the NullAppender to be created with 
NullAppender.name=null. Later during node start, when the NullAppender is being 
removed, a NullPointerException occurs because of the null name. While log4j 
2.18 changed the code slightly to prevent the NPE from occurring, the side 
effect is that the NullAppender never actually gets removed.

Line 8983 of 
/core/src/main/java/org/apache/ignite/internal/util/IgniteUtils.java does the 
following:

                nullApp = 
Class.forName("org.apache.log4j.varia.NullAppender").newInstance();

This creates the NullAppender with name=null. The documentaion for NullAppender 
states it should be created with NullAppender.createAppender(), which will make 
sure name is not null, which will in turn make it possible to remove the 
NullAppender later on. It also states it should NOT be created with new 
NullAppender(), which is what happens when newInstance() is called.

This issue is causing some pain to avoid the NPEs when running alongside Spark 
3.3, which pulls in Log4j 2.17.2. Basically, the only way to get this 
environment to work is to make sure none of the Log4j v1 classes are present in 
the classpath, including those provided by the log4j-1.2-api jar that gets 
bundled with log4j-api and log4j-core.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Assigned] (IGNITE-17085) Support persistent case for page-memory-based MV storage

2022-08-03 Thread Kirill Tkalenko (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirill Tkalenko reassigned IGNITE-17085:


Assignee: Kirill Tkalenko

> Support persistent case for page-memory-based MV storage
> 
>
> Key: IGNITE-17085
> URL: https://issues.apache.org/jira/browse/IGNITE-17085
> Project: Ignite
>  Issue Type: Improvement
>  Components: persistence
>Reporter: Roman Puchkovskiy
>Assignee: Kirill Tkalenko
>Priority: Major
>  Labels: ignite-3
> Fix For: 3.0.0-alpha6
>
>
> Analogously to IGNITE-16641, we need to support persistent case for 
> PageMemoryMvPartitionStorage



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (IGNITE-17461) Status command incorrectly handle invalid URLs

2022-08-03 Thread Vadim Pakhnushev (Jira)
Vadim Pakhnushev created IGNITE-17461:
-

 Summary: Status command incorrectly handle invalid URLs
 Key: IGNITE-17461
 URL: https://issues.apache.org/jira/browse/IGNITE-17461
 Project: Ignite
  Issue Type: Bug
  Components: cli
Reporter: Vadim Pakhnushev
Assignee: Vadim Pakhnushev


node/cluster status commands print "Internal error!" instead of an error 
description in case invalid URL is passed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (IGNITE-13181) Ignite support for spark-3.0.0

2022-08-03 Thread Ivan Gagarkin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Gagarkin resolved IGNITE-13181.

Resolution: Duplicate

Dup of IGNITE-17460

> Ignite support for spark-3.0.0
> --
>
> Key: IGNITE-13181
> URL: https://issues.apache.org/jira/browse/IGNITE-13181
> Project: Ignite
>  Issue Type: New Feature
>  Components: spark
>Affects Versions: 2.8.1
>Reporter: Shenson Joseph
>Priority: Blocker
>
> feature to support apache spark-3.0.0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (IGNITE-12716) Ignite support for spark-3.0.0

2022-08-03 Thread Ivan Gagarkin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-12716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Gagarkin resolved IGNITE-12716.

Resolution: Duplicate

Dup of IGNITE-17460

> Ignite support for spark-3.0.0
> --
>
> Key: IGNITE-12716
> URL: https://issues.apache.org/jira/browse/IGNITE-12716
> Project: Ignite
>  Issue Type: Bug
>  Components: spark
>Affects Versions: 2.7, 2.8, 2.7.5, 2.7.6
>Reporter: Shenson Joseph
>Priority: Blocker
>
> Ignite support for spark-3.0.0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Resolved] (IGNITE-13998) [Spark] Upgrade ignite-spark to Spark 3.0.1/Scala 2.12

2022-08-03 Thread Ivan Gagarkin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Gagarkin resolved IGNITE-13998.

Resolution: Duplicate

Dup of IGNITE-17460

> [Spark] Upgrade ignite-spark to Spark 3.0.1/Scala 2.12 
> ---
>
> Key: IGNITE-13998
> URL: https://issues.apache.org/jira/browse/IGNITE-13998
> Project: Ignite
>  Issue Type: New Feature
>  Components: spark
>Affects Versions: 2.8
>Reporter: Rajkumar Arumugham
>Assignee: Alexey Zinoviev
>Priority: Major
>
> To run Ignite 2.9+ (with Ignite-Spark) on Java 11, required to use Spark 3.0 
> with Scala 2.12
> Spark 2.x/Scala 2.11 is not supported on Java 11 as per 
> https://issues.apache.org/jira/browse/SPARK-24417



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-17460) Add support of Spark 3.2 to Ignite 2.x

2022-08-03 Thread Ivan Gagarkin (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Gagarkin updated IGNITE-17460:
---
Epic Name: spark-3.2  (was: Add support of Spark 3.2)

> Add support of Spark 3.2 to Ignite 2.x
> --
>
> Key: IGNITE-17460
> URL: https://issues.apache.org/jira/browse/IGNITE-17460
> Project: Ignite
>  Issue Type: Epic
>Reporter: Ivan Gagarkin
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (IGNITE-17460) Add support of Spark 3.2 to Ignite 2.x

2022-08-03 Thread Ivan Gagarkin (Jira)
Ivan Gagarkin created IGNITE-17460:
--

 Summary: Add support of Spark 3.2 to Ignite 2.x
 Key: IGNITE-17460
 URL: https://issues.apache.org/jira/browse/IGNITE-17460
 Project: Ignite
  Issue Type: Epic
Reporter: Ivan Gagarkin






--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-17233) Clarify cluster URL parameter name

2022-08-03 Thread Vadim Pakhnushev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vadim Pakhnushev updated IGNITE-17233:
--
Component/s: cli

> Clarify cluster URL parameter name
> --
>
> Key: IGNITE-17233
> URL: https://issues.apache.org/jira/browse/IGNITE-17233
> Project: Ignite
>  Issue Type: Task
>  Components: cli
>Reporter: Vadim Pakhnushev
>Assignee: Vadim Pakhnushev
>Priority: Major
>  Labels: ignite-3, ignite-3-cli-tool
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> [IEP-88|https://cwiki.apache.org/confluence/display/IGNITE/IEP-88%3A+CLI+Tool]
>  states that commands for cluster management use --cluster-url parameter for 
> the endpoint which is unclear. In fact --cluster-url could be a URL to any 
> node in the cluster.
> We could use --cluster-endpoint-url in cluster commands to clarify that it is 
> an endpoint to the cluster.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-17347) Add port parameters to Ignite3 CLI node start command

2022-08-03 Thread Vadim Pakhnushev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vadim Pakhnushev updated IGNITE-17347:
--
Epic Link: IGNITE-16970
   Labels: ignite-3 ignite-3-cli-tool  (was: ignite3)

> Add port parameters to Ignite3 CLI node start command
> -
>
> Key: IGNITE-17347
> URL: https://issues.apache.org/jira/browse/IGNITE-17347
> Project: Ignite
>  Issue Type: Task
>  Components: cli
>Reporter: Yury Yudin
>Assignee: Vadim Pakhnushev
>Priority: Major
>  Labels: ignite-3, ignite-3-cli-tool
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently Ignite3 CLI node start command only provides a way to set different 
> port parameters via supplying a different configuration file. This makes it 
> somewhat cumbersome to start multple nodes on the same machine.
> Let's provide --port and --rest-port parameters to the node start command to 
> make it easier.
> In order to properly test a cluster on the same machine, --join parameter is 
> also needed which will provide a list of seed nodes to join physical topology.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-17348) Add quit/exit command to Ignite3 CLI sql REPL

2022-08-03 Thread Vadim Pakhnushev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vadim Pakhnushev updated IGNITE-17348:
--
Epic Link: IGNITE-16970
   Labels: ignite-3 ignite-3-cli-tool  (was: ignite-3)

> Add quit/exit command to Ignite3 CLI sql REPL
> -
>
> Key: IGNITE-17348
> URL: https://issues.apache.org/jira/browse/IGNITE-17348
> Project: Ignite
>  Issue Type: Task
>  Components: cli
>Reporter: Yury Yudin
>Assignee: Vadim Pakhnushev
>Priority: Major
>  Labels: ignite-3, ignite-3-cli-tool
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> turns out there is no way to exit from sql REPL in ignite3 CLI other than 
> pressing Ctrl+D - while this works, a separate command to exit to the top 
> REPL would be nice to have.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Created] (IGNITE-17459) Different output from !desc "SYS".TABLES and !desc TABLE

2022-08-03 Thread Luchnikov Alexander (Jira)
Luchnikov Alexander created IGNITE-17459:


 Summary: Different output from !desc "SYS".TABLES and !desc TABLE
 Key: IGNITE-17459
 URL: https://issues.apache.org/jira/browse/IGNITE-17459
 Project: Ignite
  Issue Type: Wish
Reporter: Luchnikov Alexander


When running commands in sqlline, we get different results:
* !desc "SYS".TABLES lists view columns, their names, types, and so on
* !desc TABLE outputs the contents of the view as if !tables were executed

When running !desc "SYS".INDEXES and !desc INDEXES the result is the same.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-17233) Clarify cluster URL parameter name

2022-08-03 Thread Vadim Pakhnushev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vadim Pakhnushev updated IGNITE-17233:
--
Description: 
[IEP-88|https://cwiki.apache.org/confluence/display/IGNITE/IEP-88%3A+CLI+Tool] 
states that commands for cluster management use --cluster-url parameter for the 
endpoint which is unclear. In fact --cluster-url could be a URL to any node in 
the cluster.

We could use --cluster-endpoint-url in cluster commands to clarify that it is 
an endpoint to the cluster.

  was:
[IEP-88|https://cwiki.apache.org/confluence/display/IGNITE/IEP-88%3A+CLI+Tool] 
states that commands for cluster management use --cluster-url parameter for the 
endpoint which is unclear. In fact --cluster-url could be a URL to any node in 
the cluster.

We could use --cluster-endpoint-url in cluster commands to clarify that it is .


> Clarify cluster URL parameter name
> --
>
> Key: IGNITE-17233
> URL: https://issues.apache.org/jira/browse/IGNITE-17233
> Project: Ignite
>  Issue Type: Task
>Reporter: Vadim Pakhnushev
>Assignee: Vadim Pakhnushev
>Priority: Major
>  Labels: ignite-3, ignite-3-cli-tool
>
> [IEP-88|https://cwiki.apache.org/confluence/display/IGNITE/IEP-88%3A+CLI+Tool]
>  states that commands for cluster management use --cluster-url parameter for 
> the endpoint which is unclear. In fact --cluster-url could be a URL to any 
> node in the cluster.
> We could use --cluster-endpoint-url in cluster commands to clarify that it is 
> an endpoint to the cluster.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-17233) Clarify node and cluster URL parameter names

2022-08-03 Thread Vadim Pakhnushev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vadim Pakhnushev updated IGNITE-17233:
--
Description: 
[IEP-88|https://cwiki.apache.org/confluence/display/IGNITE/IEP-88%3A+CLI+Tool] 
states that commands for cluster management use --cluster-url parameter for the 
endpoint which is unclear. In fact --cluster-url could be a URL to any node in 
the cluster.

We could use --cluster-endpoint-url in cluster commands to clarify that it is .

  was:
[IEP-88|https://cwiki.apache.org/confluence/display/IGNITE/IEP-88%3A+CLI+Tool] 
states that commands for cluster management use --cluster-url parameter for the 
endpoint while commands specific to the node use --node-url while in fact 
--cluster-url is a URL to any node in the cluster.

We could use --node-url in cluster commands as well.


> Clarify node and cluster URL parameter names
> 
>
> Key: IGNITE-17233
> URL: https://issues.apache.org/jira/browse/IGNITE-17233
> Project: Ignite
>  Issue Type: Task
>Reporter: Vadim Pakhnushev
>Assignee: Vadim Pakhnushev
>Priority: Major
>  Labels: ignite-3, ignite-3-cli-tool
>
> [IEP-88|https://cwiki.apache.org/confluence/display/IGNITE/IEP-88%3A+CLI+Tool]
>  states that commands for cluster management use --cluster-url parameter for 
> the endpoint which is unclear. In fact --cluster-url could be a URL to any 
> node in the cluster.
> We could use --cluster-endpoint-url in cluster commands to clarify that it is 
> .



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-17233) Clarify cluster URL parameter name

2022-08-03 Thread Vadim Pakhnushev (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vadim Pakhnushev updated IGNITE-17233:
--
Summary: Clarify cluster URL parameter name  (was: Clarify node and cluster 
URL parameter names)

> Clarify cluster URL parameter name
> --
>
> Key: IGNITE-17233
> URL: https://issues.apache.org/jira/browse/IGNITE-17233
> Project: Ignite
>  Issue Type: Task
>Reporter: Vadim Pakhnushev
>Assignee: Vadim Pakhnushev
>Priority: Major
>  Labels: ignite-3, ignite-3-cli-tool
>
> [IEP-88|https://cwiki.apache.org/confluence/display/IGNITE/IEP-88%3A+CLI+Tool]
>  states that commands for cluster management use --cluster-url parameter for 
> the endpoint which is unclear. In fact --cluster-url could be a URL to any 
> node in the cluster.
> We could use --cluster-endpoint-url in cluster commands to clarify that it is 
> .



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-17457) Cluster locks after the transaction recovery procedure if the tx primary node fail

2022-08-03 Thread Sergey Korotkov (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Korotkov updated IGNITE-17457:
-
Description: 
Ignite cluster may be locked (all client operations would block) after the tx 
recovery procedure executed on the tx primary node failure.

The prepared transaction may remain un-commited on the backup node after the tx 
recovery.  So the partition exchange wouldn't complete. So cluster would be 
locked.

The Immediate reason is the race condition in the method:
{code:java}
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter::markFinalizing(RECOVERY_FINISH){code}
If 2 or more backups are configured It may be called concurrently for the same 
transaction both from the recovery procedure:
{code:java}
IgniteTxManager::commitIfPrepared{code}
and from the tx recovery request handler:
{code:java}
IgniteTxHandler::processCheckPreparedTxRequest{code}
Problem occur if thread context is switched between old finalization status 
request and status update.

The problematic sequence of events is as follows (the lock will be in the 
node1):

1. Start cluster with 3 nodes (node0, node1, node2) and cache with 2 backups.
2. On node2 start and prepare transaction choosing key with primary partition 
stored on node2.
3. Kill node2
4. The tx recovery procedure is started both on node0 and node1
5. In scope of the recovery procedure node0 sends tx recovery request to node1
6. The following steps are executed on the node1 in two threads ("procedure" 
which is a system pool thread executing the tx recovery procedure and "handler" 
which is a striped pool thread processing the tx recovery request sent from 
node0):
 - tx.finalization == NONE
 - "procedure": calls markFinalizing(RECOVERY_FINISH)
 - "handler": calls markFinalizing(RECOVERY_FINISH)
 - "procedure": gets old tx.finlalization - it's NONE
 - "handler": gets old tx.finalization - it's NONE
 - "handler": updates tx.finalization - now it's RECOVERY_FINISH
 - "procedure": tries to update tx.finalization via compareAndSet and fails 
since compare fails.
 - "procedure": stops transaction processing and does not try to commit it.
 - Transaction remains not finished on node1.


Reproducer is in the pull request.

  was:
Ignite cluster may be locked (all client operations would block) after the tx 
recovery procedure executed on the tx primary node failure.

The prepared transaction may remain un-commited on the backup node after the tx 
recovery.  So the partition exchange wouldn't complete. So cluster would be 
locked.

The Immediate reason is the race condition in the method:
{code:java}
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter::markFinalizing(RECOVERY_FINISH){code}
If 2 or more backups are configured It may be called concurrently for the same 
transaction both from the recovery procedure:
{code:java}
IgniteTxManager::commitIfPrepared{code}
and from the tx recovery request handler:
{code:java}
IgniteTxHandler::processCheckPreparedTxRequest{code}
Problem occur if thread context is switched between old finalization status 
request and status update.

The problematic sequence of events is as follows (the lock will be in the 
node1):

1. Start cluster with 3 nodes (node0, node1, node2) and cache with 2 backups.
2. On node2 start and prepare pessimistic transaction choosing key with primary 
partition stored on node2.
3. Kill node2
4. The tx recovery procedure is started both on node0 and node1
5. In scope of the recovery procedure node0 sends tx recovery request to node1
6. The following steps are executed on the node1 in two threads ("procedure" 
which is a system pool thread executing the tx recovery procedure and "handler" 
which is a striped pool thread processing the tx recovery request sent from 
node0):
 - tx.finalization == NONE
 - "procedure": calls markFinalizing(RECOVERY_FINISH)
 - "handler": calls markFinalizing(RECOVERY_FINISH)
 - "procedure": gets old tx.finlalization - it's NONE
 - "handler": gets old tx.finalization - it's NONE
 - "handler": updates tx.finalization - now it's RECOVERY_FINISH
 - "procedure": tries to update tx.finalization via compareAndSet and fails 
since compare fails.
 - "procedure": stops transaction processing and does not try to commit it.
 - Transaction remains not finished on node1.


Reproducer is in the pull request.


> Cluster locks after the transaction recovery procedure if the tx primary node 
> fail
> --
>
> Key: IGNITE-17457
> URL: https://issues.apache.org/jira/browse/IGNITE-17457
> Project: Ignite
>  Issue Type: Bug
>Reporter: Sergey Korotkov
>Assignee: Sergey Korotkov
>Priority: Major
>
> Ignite cluster may be locked (all client operations would block) after the tx 

[jira] [Updated] (IGNITE-17457) Cluster locks after the transaction recovery procedure if the tx primary node fail

2022-08-03 Thread Sergey Korotkov (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Korotkov updated IGNITE-17457:
-
Description: 
Ignite cluster may be locked (all client operations would block) after the tx 
recovery procedure executed on the tx primary node failure.

The prepared transaction may remain un-commited on the backup node after the tx 
recovery.  So the partition exchange wouldn't complete. So cluster would be 
locked.

The Immediate reason is the race condition in the method:
{code:java}
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter::markFinalizing(RECOVERY_FINISH){code}
If 2 or more backups are configured It may be called concurrently for the same 
transaction both from the recovery procedure:
{code:java}
IgniteTxManager::commitIfPrepared{code}
and from the tx recovery request handler:
{code:java}
IgniteTxHandler::processCheckPreparedTxRequest{code}
Problem occur if thread context is switched between old finalization status 
request and status update.

The problematic sequence of events is as follows (the lock will be in the 
node1):

1. Start cluster with 3 nodes (node0, node1, node2) and cache with 2 backups.
2. On node2 start and prepare pessimistic transaction choosing key with primary 
partition stored on node2.
3. Kill node2
4. The tx recovery procedure is started both on node0 and node1
5. In scope of the recovery procedure node0 sends tx recovery request to node1
6. The following steps are executed on the node1 in two threads ("procedure" 
which is a system pool thread executing the tx recovery procedure and "handler" 
which is a striped pool thread processing the tx recovery request sent from 
node0):
 - tx.finalization == NONE
 - "procedure": calls markFinalizing(RECOVERY_FINISH)
 - "handler": calls markFinalizing(RECOVERY_FINISH)
 - "procedure": gets old tx.finlalization - it's NONE
 - "handler": gets old tx.finalization - it's NONE
 - "handler": updates tx.finalization - now it's RECOVERY_FINISH
 - "procedure": tries to update tx.finalization via compareAndSet and fails 
since compare fails.
 - "procedure": stops transaction processing and does not try to commit it.
 - Transaction remains not finished on node1.


Reproducer is in the pull request.

  was:
Ignite cluster may be locked (all client operations would block) after the tx 
recovery procedure executed on the tx primary node failure.

The prepared transaction may remain un-commited on the backup node after the tx 
recovery.  So the partition exchange wouldn't complete. So cluster would be 
locked.

The Immediate reason is the race condition in the method:
{code:java}
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter::markFinalizing(RECOVERY_FINISH){code}
If 2 or more backups are configured It may be called concurrently for the same 
transaction both from the recovery procedure:
{code:java}
IgniteTxManager::commitIfPrepared{code}
and from the tx recovery request handler:
{code:java}
IgniteTxHandler::processCheckPreparedTxRequest{code}
Problem occur if thread context is switched between old finalization status 
request and status update.

The problematic sequence of events is as follows (the lock will be in the 
node1):

1. Start cluster with 3 nodes (node0, node1, node2) and cache with 2 backups.
2. On node2 start and prepare pessimistic transaction choosing key with primary 
partition stored on node2.
3. Kill node2
4. The tx recovery procedure is started both on node0 and node1
5. In scope of the recovery procedure node0 sends tx recovery request to node1
6. The following steps are executed on the node1 in two threads ("procedure" 
which is a system pool thread executing the tx recovery procedure and "handler" 
which is a striped pool thread processing the tx recovery request sent from 
node0):
 - tx.finalization == NONE
 - "procedure": calls markFinalizing(RECOVERY_FINISH)
 - "handler": calls markFinalizing(RECOVERY_FINISH)
 - "procedure": gets old tx.finlalization - it's NONE
 - "handler": gets old tx.finalization - it's NONE
 - "handler": updates tx.finalization - now it's RECOVERY_FINISH
 - "procedure": trys to update tx.finalization via compareAndSet and fails 
since compare fails.
 - "procedure": stops transaction processing and does not try to commit it.
 - Transaction remains not finished on node1.


Reproducer is in the pull request.


> Cluster locks after the transaction recovery procedure if the tx primary node 
> fail
> --
>
> Key: IGNITE-17457
> URL: https://issues.apache.org/jira/browse/IGNITE-17457
> Project: Ignite
>  Issue Type: Bug
>Reporter: Sergey Korotkov
>Assignee: Sergey Korotkov
>Priority: Major
>
> Ignite cluster may be locked (all client operations would block) 

[jira] [Updated] (IGNITE-17457) Cluster locks after the transaction recovery procedure if the tx primary node fail

2022-08-03 Thread Sergey Korotkov (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Korotkov updated IGNITE-17457:
-
Description: 
Ignite cluster may be locked (all client operations would block) after the tx 
recovery procedure executed on the tx primary node failure.

The prepared transaction may remain un-commited on the backup node after the tx 
recovery.  So the partition exchange wouldn't complete. So cluster would be 
locked.

The Immediate reason is the race condition in the method:
{code:java}
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter::markFinalizing(RECOVERY_FINISH){code}
If 2 or more backups are configured It may be called concurrently for the same 
transaction both from the recovery procedure:
{code:java}
IgniteTxManager::commitIfPrepared{code}
and from the tx recovery request handler:
{code:java}
IgniteTxHandler::processCheckPreparedTxRequest{code}
Problem occur if thread context is switched between old finalization status 
request and status update.

The problematic sequence of events is as follows (the lock will be in the 
node1):

1. Start cluster with 3 nodes (node0, node1, node2) and cache with 2 backups.
2. On node2 start and prepare pessimistic transaction choosing key with primary 
partition stored on node2.
3. Kill node2
4. The tx recovery procedure is started both on node0 and node1
5. In scope of the recovery procedure node0 sends tx recovery request to node1
6. The following steps are executed on the node1 in two threads ("procedure" 
which is a system pool thread executing the tx recovery procedure and "handler" 
which is a striped pool thread processing the tx recovery request sent from 
node0):
 - tx.finalization == NONE
 - "procedure": calls markFinalizing(RECOVERY_FINISH)
 - "handler": calls markFinalizing(RECOVERY_FINISH)
 - "procedure": gets old tx.finlalization - it's NONE
 - "handler": gets old tx.finalization - it's NONE
 - "handler": updates tx.finalization - now it's RECOVERY_FINISH
 - "procedure": trys to update tx.finalization via compareAndSet and fails 
since compare fails.
 - "procedure": stops transaction processing and does not try to commit it.
 - Transaction remains not finished on node1.


Reproducer is in the pull request.

  was:
Ignite cluster may be locked (all client operations would block) after the tx 
recovery procedure executed on the tx primary node failure.

The prepared transaction may remain un-commited on the backup node after the tx 
recovery.  So the partition exchange wouldn't complete. So cluster would be 
locked.

The Immediate reason is the race condition in the method:
{code:java}
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter::markFinalizing(RECOVERY_FINISH){code}
If 2 or more backups are configured It may be called concurrently for the same 
transaction both from the recovery procedure:
{code:java}
IgniteTxManager::commitIfPrepared{code}
and from the tx recovery request handler:
{code:java}
IgniteTxHandler::processCheckPreparedTxRequest{code}
Problem occur if thread context is switched between old finalization status 
request and status update.

The problematic sequence of events is as follows (the lock will be in the 
node1):

1. Start cluster with 3 nodes (node0, node1, node2) and cache with 2 backups.
2. On node2 start and prepare pessimistic transaction choosing key with primary 
partition stored on node2.
3. Kill node2
4. The tx recovery procedure is started both on node0 and node1
5. In scope of the recovery procedure node0 sends tx recovery request to node1
6. The following steps are executed on the node1 in two threads ("procedure" 
which is a system pool thread executing the tx recovery procedure and "handler" 
which is a striped thread processing the tx recovery request sent from node0):
 - tx.finalization == NONE
 - "procedure": calls markFinalizing(RECOVERY_FINISH)
 - "handler": calls markFinalizing(RECOVERY_FINISH)
 - "procedure": gets old tx.finlalization - it's NONE
 - "handler": gets old tx.finalization - it's NONE
 - "handler": updates tx.finalization - now it's RECOVERY_FINISH
 - "procedure": trys to update tx.finalization via compareAndSet and fails 
since compare fails.
 - "procedure": stops transaction processing and does not try to commit it.
 - Transaction remains not finished on node1.


Reproducer is in the pull request.


> Cluster locks after the transaction recovery procedure if the tx primary node 
> fail
> --
>
> Key: IGNITE-17457
> URL: https://issues.apache.org/jira/browse/IGNITE-17457
> Project: Ignite
>  Issue Type: Bug
>Reporter: Sergey Korotkov
>Assignee: Sergey Korotkov
>Priority: Major
>
> Ignite cluster may be locked (all client operations would block) after the 

[jira] [Updated] (IGNITE-17457) Cluster locks after the transaction recovery procedure if the tx primary node fail

2022-08-03 Thread Sergey Korotkov (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Korotkov updated IGNITE-17457:
-
Description: 
Ignite cluster may be locked (all client operations would block) after the tx 
recovery procedure executed on the tx primary node failure.

The prepared transaction may remain un-commited on the backup node after the tx 
recovery.  So the partition exchange wouldn't complete. So cluster would be 
locked.

The Immediate reason is the race condition in the method:
{code:java}
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter::markFinalizing(RECOVERY_FINISH){code}
If 2 or more backups are configured It may be called concurrently for the same 
transaction both from the recovery procedure:
{code:java}
IgniteTxManager::commitIfPrepared{code}
and from the tx recovery request handler:
{code:java}
IgniteTxHandler::processCheckPreparedTxRequest{code}
Problem occur if thread context is switched between old finalization status 
request and status update.

The problematic sequence of events is as follows (the lock will be in the 
node1):

1. Start cluster with 3 nodes (node0, node1, node2) and cache with 2 backups.
2. On node2 start and prepare pessimistic transaction choosing key with primary 
partition stored on node2.
3. Kill node2
4. The tx recovery procedure is started both on node0 and node1
5. In scope of the recovery procedure node0 sends tx recovery request to node1
6. The following steps are executed on the node1 in two threads ("procedure" 
which is a system pool thread executing the tx recovery procedure and "handler" 
which is a striped thread processing the tx recovery request sent from node0):
 - tx.finalization == NONE
 - "procedure": calls markFinalizing(RECOVERY_FINISH)
 - "handler": calls markFinalizing(RECOVERY_FINISH)
 - "procedure": gets old tx.finlalization - it's NONE
 - "handler": gets old tx.finalization - it's NONE
 - "handler": updates tx.finalization - now it's RECOVERY_FINISH
 - "procedure": trys to update tx.finalization via compareAndSet and fails 
since compare fails.
 - "procedure": stops transaction processing and does not try to commit it.
 - Transaction remains not finished on node1.


Reproducer is in the pull request.

  was:
Ignite cluster may be locked (all client operations would block) after the tx 
recovery procedure executed on the tx primary node failure.

The prepared transaction may remain un-commited on the backup node after the tx 
recovery.  So the partition exchange wouldn't complete. So cluster would be 
locked.

The Immediate reason is the race condition in the method:
{code:java}
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter::markFinalizing(RECOVERY_FINISH){code}
If 2 or more backups are configured It may be called concurrently for the same 
transaction both from the recovery procedure:
{code:java}
IgniteTxManager::commitIfPrepared{code}
and from the tx recovery request handler:
{code:java}
IgniteTxHandler::processCheckPreparedTxRequest{code}
Problem occur if thread context is switched between old finalization status 
request and status update.

***

{color:#172b4d}Reproducer is in the pull request.{color}


> Cluster locks after the transaction recovery procedure if the tx primary node 
> fail
> --
>
> Key: IGNITE-17457
> URL: https://issues.apache.org/jira/browse/IGNITE-17457
> Project: Ignite
>  Issue Type: Bug
>Reporter: Sergey Korotkov
>Assignee: Sergey Korotkov
>Priority: Major
>
> Ignite cluster may be locked (all client operations would block) after the tx 
> recovery procedure executed on the tx primary node failure.
> The prepared transaction may remain un-commited on the backup node after the 
> tx recovery.  So the partition exchange wouldn't complete. So cluster would 
> be locked.
> 
> The Immediate reason is the race condition in the method:
> {code:java}
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter::markFinalizing(RECOVERY_FINISH){code}
> If 2 or more backups are configured It may be called concurrently for the 
> same transaction both from the recovery procedure:
> {code:java}
> IgniteTxManager::commitIfPrepared{code}
> and from the tx recovery request handler:
> {code:java}
> IgniteTxHandler::processCheckPreparedTxRequest{code}
> Problem occur if thread context is switched between old finalization status 
> request and status update.
> 
> The problematic sequence of events is as follows (the lock will be in the 
> node1):
> 1. Start cluster with 3 nodes (node0, node1, node2) and cache with 2 backups.
> 2. On node2 start and prepare pessimistic transaction choosing key with 
> primary partition stored on node2.
> 3. Kill node2
> 4. The tx recovery procedure is 

[jira] [Updated] (IGNITE-17457) Cluster locks after the transaction recovery procedure if the tx primary node fail

2022-08-03 Thread Sergey Korotkov (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Korotkov updated IGNITE-17457:
-
Description: 
Ignite cluster may be locked (all client operations would block) after the tx 
recovery procedure executed on the tx primary node failure.

The prepared transaction may remain un-commited on the backup node after the tx 
recovery.  So the partition exchange wouldn't complete. So cluster would be 
locked.

The Immediate reason is the race condition in the method:
{code:java}
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter::markFinalizing(RECOVERY_FINISH){code}
If 2 or more backups are configured It may be called concurrently for the same 
transaction both from the recovery procedure:
{code:java}
IgniteTxManager::commitIfPrepared{code}
and from the tx recovery request handler:
{code:java}
IgniteTxHandler::processCheckPreparedTxRequest{code}
Problem occur if thread context is switched between old finalization status 
request and status update.

***

{color:#172b4d}Reproducer is in the pull request.{color}

  was:
Ignite cluster may be locked (all client operations would block) after the tx 
recovery procedure executed on the tx primary node failure.

The prepared transaction may remain un-commited on the backup node after the tx 
recovery.  So the partition exchange wouldn't complete. So cluster would be 
locked.

The Immediate reason is the race condition in the method:
{code:java}
org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter::markFinalizing(RECOVERY_FINISH){code}
It may be called concurrently for the same transaction both from the recovery 
procedure:
{code:java}
IgniteTxManager::commitIfPrepared{code}
and from the tx recovery request handler:
{code:java}
IgniteTxHandler::processCheckPreparedTxRequest{code}
 

Details  {color:#ff}TBD{color}

{color:#172b4d}Reproducer is in the pull request.{color}


> Cluster locks after the transaction recovery procedure if the tx primary node 
> fail
> --
>
> Key: IGNITE-17457
> URL: https://issues.apache.org/jira/browse/IGNITE-17457
> Project: Ignite
>  Issue Type: Bug
>Reporter: Sergey Korotkov
>Assignee: Sergey Korotkov
>Priority: Major
>
> Ignite cluster may be locked (all client operations would block) after the tx 
> recovery procedure executed on the tx primary node failure.
> The prepared transaction may remain un-commited on the backup node after the 
> tx recovery.  So the partition exchange wouldn't complete. So cluster would 
> be locked.
> The Immediate reason is the race condition in the method:
> {code:java}
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter::markFinalizing(RECOVERY_FINISH){code}
> If 2 or more backups are configured It may be called concurrently for the 
> same transaction both from the recovery procedure:
> {code:java}
> IgniteTxManager::commitIfPrepared{code}
> and from the tx recovery request handler:
> {code:java}
> IgniteTxHandler::processCheckPreparedTxRequest{code}
> Problem occur if thread context is switched between old finalization status 
> request and status update.
> ***
> {color:#172b4d}Reproducer is in the pull request.{color}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Updated] (IGNITE-17437) SQL. Investigate of support LISTAGG (aka GROUP_CONCAT, STRING_AGG) aggregate function for ignite 3

2022-08-03 Thread Yury Gerzhedovich (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yury Gerzhedovich updated IGNITE-17437:
---
Summary: SQL. Investigate of support LISTAGG (aka GROUP_CONCAT, STRING_AGG) 
aggregate function for ignite 3  (was: SQL. Support LISTAGG (aka GROUP_CONCAT, 
STRING_AGG) aggregate function for ignite 3)

> SQL. Investigate of support LISTAGG (aka GROUP_CONCAT, STRING_AGG) aggregate 
> function for ignite 3
> --
>
> Key: IGNITE-17437
> URL: https://issues.apache.org/jira/browse/IGNITE-17437
> Project: Ignite
>  Issue Type: Improvement
>  Components: sql
>Reporter: Yury Gerzhedovich
>Priority: Major
>  Labels: ignite-3
>
> LISTAGG function implemented for Ignite 2, however for Ignite 3 need to 
> rethink design of the solution.  
> TBD: reasons why we want to do it differently.
> Let's investigate possible ways to implement it.
> As a result need to implement it, create ticket to document it and unmute the 
> test_order_variable_size_payload.test_ignore test



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Assigned] (IGNITE-17077) Implement checkpointIndex for PDS

2022-08-03 Thread Ivan Bessonov (Jira)


 [ 
https://issues.apache.org/jira/browse/IGNITE-17077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Bessonov reassigned IGNITE-17077:
--

Assignee: Ivan Bessonov

> Implement checkpointIndex for PDS
> -
>
> Key: IGNITE-17077
> URL: https://issues.apache.org/jira/browse/IGNITE-17077
> Project: Ignite
>  Issue Type: Improvement
>Reporter: Ivan Bessonov
>Assignee: Ivan Bessonov
>Priority: Major
>  Labels: ignite-3
>
> Please refer to https://issues.apache.org/jira/browse/IGNITE-16907 for 
> prerequisites.
> h2. General idea
> The idea doesn't seem complicated. There will be a "setUpdateIndex" and 
> "getUpdateIndex" methods (names might be different).
>  * First one is invoked at the end of every write command, with RAFT commit 
> index being passed as a parameter. This is done right before releasing 
> checkpoint read lock (or whatever the name we will come up with). More on 
> that later.
>  * Second one is invoked at the beginning of every write command to validate 
> that update don't come out of order or with gaps. This is the way to 
> guarantee that IndexMismatchException can be thrown at the right time.
> So, the write command flow will look like this. All names here are completely 
> random.
>  
> {code:java}
> try (ConsistencyLock lock = partition.acquireConsistencyLock()) {
> long updateIndex = partition.getUpdateIndex();
> long raftIndex = writeCommand.raftIndex();
> if (raftIndex != updateIndex + 1) {
> throw new IndexMismatchException(updateIndex);
> }
> partition.write(writeCommand.row());
> for (Index index : table.indexes(partition) {
> index.index(writeCommand.row());
> }
> partition.setUpdateIndex(raftIndex);
> }{code}
>  
> Some nuances:
>  * Mismatch exception must be thrown before any data modifications. Storage 
> content must be intact, otherwise we'll just break it.
>  * Case above is the simplest one - there's a single "atomic" storage update. 
> Generally speaking, we can't or sometimes don't want to work this way. 
> Examples of operations, where atomicity this strict is not required:
>  ** Batch insert/update from the transaction.
>  ** Transaction commit might have a huge number of row ids, we can exhaust 
> the memory while committing.
>  * If we split write operation into several operations, we should externally 
> guarantee their idempotence. "setUpdateIndex" should be at the end of the 
> last "atomic" operation, so that the last command could be safely reapplied.
> h2. Implementation
> "set" method could write a value directly into partitions meta page. This 
> *will* work. But it's not quite optimal.
> Optimal solution is tightly coupled with the way checkpoint should work. This 
> may not be the right place to describe the issue, but I do it nonetheless. 
> It'll probably get split into another issue one day.
> There's a simple way to touch every meta page only once per checkpoint. We 
> just do it while holding checkpoint write lock. This way data is consistent. 
> But this solution is equally {*}bad{*}, it forces us to perform pages 
> manipulation under write lock. Flushing freelists is enough already. (NOTE: 
> we should test the performance without onheap-cache, it'll speed-up 
> checkpoint start process, thus reducing latency spikes)
> Better way to do this is not having meta pages in page memory whatsoever. 
> Maybe during the start, but that's it. It's a common practice to have a 
> pageSize being equal to 16Kb. Effective payload of partition meta page in 
> Ignite 2.x is just above 100 bytes. I expect it to be way lower in Ignite 
> 3.0. Having a loaded page for every partition is just a waste of resources, 
> all required data can be stored on-heap.
> Then, let's rely on two simple facts:
>  * If meta page date is cached on-heap, no one would need to read it from 
> disk. I should also mention that it will mostly be immutable.
>  * We can write partition meta page into every delta file even if meta has 
> not changed. In actuality, this will be very rare situation.
> Considering both of these facts, checkpointer may unconditionally write meta 
> page from heap to disk at the beginning of writing the delta file. This page 
> will become a write-only page, which is basically what we need. 
> h2. Callbacks and RAFT snapshots
> I argue against scheduled RAFT snapshots. They will produce a lot of junk 
> checkpoints. This is because checkpoint is a {*}global operation{*}. Imagine 
> RAFT triggering snapshots for 100 partitions in a row. This will result in a 
> 100 minuscule checkpoints, no one needs it. So, I'd say, we need to operation:
>  * partition.getCheckpointerUpdateIndex();
>  * partition.registerCheckpointedUpdateIndexListener(closure);
> Bot of these methods could be used by RAFT to determine whether it 

[jira] [Created] (IGNITE-17458) Revise (part 2) SQL related resolved tickets but still presented mentions in code AI3

2022-08-03 Thread Yury Gerzhedovich (Jira)
Yury Gerzhedovich created IGNITE-17458:
--

 Summary: Revise (part 2) SQL related resolved tickets but still 
presented mentions in code AI3
 Key: IGNITE-17458
 URL: https://issues.apache.org/jira/browse/IGNITE-17458
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Yury Gerzhedovich
Assignee: Yury Gerzhedovich


We have a bunch of resolved tickes which still present as TODO in AI3 code.

Let's get it right.

List of issues:

https://issues.apache.org/jira/browse/IGNITE-17219
https://issues.apache.org/jira/browse/IGNITE-14925
https://issues.apache.org/jira/browse/IGNITE-16760
https://issues.apache.org/jira/browse/IGNITE-16059



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (IGNITE-17398) Opencensus metrics do not work well with Prometheus since it do not have tags

2022-08-03 Thread Andrey N. Gura (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-17398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574579#comment-17574579
 ] 

Andrey N. Gura commented on IGNITE-17398:
-

[~sergeykad] We already know that OpenCensus metrics exporter performs bad 
enough especially on deployments with a lot of caches. And of course it affects 
Ignite node performance and stability.
[~northdragon] As I remember you wanted to add a filter in order to limit 
amount of exported metrics, right? Plain text format is not friendly for such 
data sets. Labels\tags will make this data set bigger.

Also adding tags\labels can lead to breaking changes for already existing 
users' metrics exporters. 

All these issues are important enough and can not be ignored in favor of 
convenience of using tags in Prometheus.

> Opencensus metrics do not work well with Prometheus since it do not have tags
> -
>
> Key: IGNITE-17398
> URL: https://issues.apache.org/jira/browse/IGNITE-17398
> Project: Ignite
>  Issue Type: Improvement
>Affects Versions: 2.9.1
>Reporter: Sergey Kadaner
>Priority: Major
>  Labels: metrics
> Attachments: image-2022-07-20-17-51-07-217.png
>
>
> The metrics created by the [new metrics 
> system|https://ignite.apache.org/docs/latest/monitoring-metrics/new-metrics] 
> are very inconvenient to use with Prometheus since it does not use tags.
> For example, Spring metric generated for the same cache looks like the 
> following in Prometheus and is very convenient to use:  
>  
> {noformat}
> cache_gets_total{cache="MY_CACHE_NAME",name="MY_CACHE_NAME",result="hit",} 
> 1387.0{noformat}
>  
> The native Ignite metric looks like the following: 
>  
> {noformat}
> cache_MY_CACHE_NAME_CacheHits 1387.0{noformat}
> The Spring reported statistics can be easily filtered by cache name and other 
> attributes, while the build-in Ignite metrics do not provide an easy way to 
> access cache names. The only option seems to be parsing the 
> "cache_MY_CACHE_NAME_CacheHits" strings which AFAIK is not supported by 
> Grafana.
>  
> For example with tags it is very easy to get a graph in Grafana with a cache 
> hit percentage that includes every existing cache. It automatically extracts 
> the cache name from the "cache" tag. Unfortunately, it is not possible if 
> there are no tags.
> !image-2022-07-20-17-51-07-217.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (IGNITE-17206) .NET: Thin client: Add IgniteSet

2022-08-03 Thread Pavel Tupitsyn (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-17206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574548#comment-17574548
 ] 

Pavel Tupitsyn commented on IGNITE-17206:
-

Merged to master: d7c307a6dd1b0f1874b507cf231347672ec871bb

> .NET: Thin client: Add IgniteSet
> 
>
> Key: IGNITE-17206
> URL: https://issues.apache.org/jira/browse/IGNITE-17206
> Project: Ignite
>  Issue Type: New Feature
>  Components: platforms, thin client
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
>  Labels: .NET
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Add IgniteSet data structure to .NET thin client.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)


[jira] [Commented] (IGNITE-17206) .NET: Thin client: Add IgniteSet

2022-08-03 Thread Igor Sapego (Jira)


[ 
https://issues.apache.org/jira/browse/IGNITE-17206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574539#comment-17574539
 ] 

Igor Sapego commented on IGNITE-17206:
--

[~ptupitsyn] looks good to me.

> .NET: Thin client: Add IgniteSet
> 
>
> Key: IGNITE-17206
> URL: https://issues.apache.org/jira/browse/IGNITE-17206
> Project: Ignite
>  Issue Type: New Feature
>  Components: platforms, thin client
>Reporter: Pavel Tupitsyn
>Assignee: Pavel Tupitsyn
>Priority: Major
>  Labels: .NET
>
> Add IgniteSet data structure to .NET thin client.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)