[jira] [Updated] (IGNITE-17328) Support explicit transactions interactions with SQL.
[ https://issues.apache.org/jira/browse/IGNITE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Evgeny Stanilovsky updated IGNITE-17328: Ignite Flags: (was: Docs Required,Release Notes Required) > Support explicit transactions interactions with SQL. > > > Key: IGNITE-17328 > URL: https://issues.apache.org/jira/browse/IGNITE-17328 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Yury Gerzhedovich >Assignee: Evgeny Stanilovsky >Priority: Major > Labels: ignite-3 > Time Spent: 1h 50m > Remaining Estimate: 0h > > All SQL queries should be executed transactionally, for now - explicit tx are > not forwarded into inner sql execution layer, this need to be fixed. > This issue will close only dml modification\insertion case. > All others : transactional ddl, RO tx scan, need to be implemented under > other issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-13612) Ignite Docs: Duplicate document
[ https://issues.apache.org/jira/browse/IGNITE-13612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YuJue Li updated IGNITE-13612: -- Fix Version/s: 2.14 (was: 2.13) > Ignite Docs: Duplicate document > --- > > Key: IGNITE-13612 > URL: https://issues.apache.org/jira/browse/IGNITE-13612 > Project: Ignite > Issue Type: Task > Components: documentation >Affects Versions: 2.9 >Reporter: YuJue Li >Assignee: YuJue Li >Priority: Minor > Fix For: 2.14 > > Attachments: 0001-IGNITE-13612-Ignite-Docs-Duplicate-document.patch > > Time Spent: 10m > Remaining Estimate: 0h > > The following two documents are duplicate. Keep one. > [https://ignite.apache.org/docs/latest/data-structures/atomic-sequence] > [https://ignite.apache.org/docs/latest/data-structures/id-generator] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-11404) Document CREATE TABLE "parallelism" option
[
https://issues.apache.org/jira/browse/IGNITE-11404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
YuJue Li updated IGNITE-11404:
--
Fix Version/s: 2.14
(was: 2.13)
> Document CREATE TABLE "parallelism" option
> --
>
> Key: IGNITE-11404
> URL: https://issues.apache.org/jira/browse/IGNITE-11404
> Project: Ignite
> Issue Type: Task
> Components: documentation, sql
>Reporter: Vladimir Ozerov
>Assignee: YuJue Li
>Priority: Major
> Fix For: 2.14
>
> Attachments:
> 0001-IGNITE-11404-Document-CREATE-TABLE-parallelism-optio.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We added new {{PARALLELISM}} option:
> {code}
> CREATE TABLE ... WITH "parallelism = 4"
> {code}
> This option affect query parallelism which is otherwise set from
> {{CacheConfiguration.queryParallelism}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IGNITE-17432) ignitevisorcmd can't run in jdk17 env
[ https://issues.apache.org/jira/browse/IGNITE-17432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YuJue Li reassigned IGNITE-17432: - Assignee: YuJue Li > ignitevisorcmd can't run in jdk17 env > - > > Key: IGNITE-17432 > URL: https://issues.apache.org/jira/browse/IGNITE-17432 > Project: Ignite > Issue Type: Bug > Components: cli >Affects Versions: 2.13 >Reporter: YuJue Li >Assignee: YuJue Li >Priority: Minor > Fix For: 2.14 > > > Unrecognized VM option 'MaxPermSize=128M' > Error: Could not create the Java Virtual Machine. > Error: A fatal exception has occurred. Program will exit. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17432) ignitevisorcmd can't run in jdk17 env
YuJue Li created IGNITE-17432: - Summary: ignitevisorcmd can't run in jdk17 env Key: IGNITE-17432 URL: https://issues.apache.org/jira/browse/IGNITE-17432 Project: Ignite Issue Type: Bug Components: cli Affects Versions: 2.13 Reporter: YuJue Li Fix For: 2.14 Unrecognized VM option 'MaxPermSize=128M' Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-16668) Design in-memory raft group reconfiguration on node failure
[
https://issues.apache.org/jira/browse/IGNITE-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Semyon Danilov updated IGNITE-16668:
Description:
If a node storing a partition of an in-memory table fails and leaves the
cluster all data it had is lost. From the point of view of the partition it
looks like as the node is left forever.
Although Raft protocol tolerates leaving some amount of nodes composing Raft
group (partition); for in-memory caches we cannot restore replica factor
because of in-memory nature of the table.
It means that we need to detect failures of each node owning a partition and
recalculate assignments for the table without keeping replica factor.
h4. Upd 1:
h4. Problem
By design raft has several persisted segments, e.g. raft meta
(term/committedIndex) and stable raft log. So, by converting common raft to
in-memory one it’s possible to break some of it’s invariants. For example Node
C could vote for Candidate A before self-restart and vote then for Candidate B
after one. As a result two leaders will be elected which is illegal.
!Screenshot from 2022-04-19 11-11-05.png!
h4. Solution
In order to solve the problem mentioned above it’s possible to remove and then
return back the restarting node from the peers of the corresponding raft group.
The peer-removal process should be finished before the restarting of the
corresponding raft server node.
!Screenshot from 2022-04-19 11-12-55.png!
The process of removing and then returning back the restarting node is however
itself tricky. And to answer why it’s non-trivial action, it’s necessary to
reveal the main ideas of the rebalance protocol.
Reconfiguration of the raft group - is a process driven by the fact of changing
the assignments. Each partition has three corresponding sets of assignments
stored in the metastore:
# assignments.stable - current distribution
# assignments.pending - partition distribution for an ongoing rebalance if any
# assignments.planned - in some cases it’s not possible to cancel or merge
pending rebalance with new one. In that case newly calculated assignments will
be stored explicitly with corresponding assignments.planned key. It's worth
noting that it doesn't make sense to keep more than one planned rebalance. Any
new scheduled one will overwrite already existing.
However such idea of overwriting the assignments.planned key wont work within
the context of an in-memory raft restart, because it’s not valid to overwrite
the reduction of assignments. Let's illustrate this problem with the following
example.
# In-memory partition p1 is hosted on nodes A, B and C, meaning that
p1.assignments.stable=[A,B,C]
# Let's say that the baseline was changed, resulting in a rebalance on
assignments.pending=[A,B,C,*D*]
# During the non-cancelable phase of [A,B,C]->[A,B,C,D], node C fails and
returns back, meaning that we should plan [A,B,D] and [A,B,C,D] assignments.
Both must be recorded in the only assignments.planned key meaning that
[A,B,C,D] will overwrite reduction [A,B,D], so no actual raft reconfiguration
will take place, which is not acceptable.
In order to overcome given issue, let’s introduce two new keys
_assignments.switch.reduce_ that will hold nodes that should be removed and
_assignments.switch.append_ that will hold nodes that should be returned back
and run following actions:
h5. On in-memory partition restart (or on partition start with cleaned-up PDS)
within retry loop add current node to assignments.switch.reduce set:
{code:java}
do {
retrievedAssignmentsSwitchReduce =
metastorage.read(assignments.switch.reduce);
calculatedAssignmetnsSwitchReduce =
union(retrievedAssignmentsSwitchReduce.value, currentNode);
if (retrievedAssignmentsSwitchReduce.isEmpty()) {
invokeRes = metastoreInvoke:
if empty(assignments.switch.reduce)
assignments.switch.reduce = calculatedAssignmentsSwitchReduce
} else {
invokeRes = metastoreInvoke:
eq(revision(assignments.switch.reduce),
retrievedAssignmentsSwitchReduce.revision)
assignments.switch.reduce = calculatedAssignmentsSwitchReduce
}
} while (!invokeRes);{code}
h5. On assignments.switch.reduce change on corresponding partition leader
Within watch listener on assignments.switch.reduce key on corresponding
partition leader we trigger new rebalance if there is no pending one.
{code:java}
calculatedAssignments = substract(calcPartAssighments(),
assignments.switch.reduce);
metastoreInvoke:
if empty(partition.assignments.change.trigger.revision) ||
partition.assignments.change.trigger.revision < event.revision
if empty(assignments.pending)
assignments.pending = calculatedAssignments
partition.assignments.change.trigger.revision = event.revision
{code}
h5. On rebalance done
changePeers() calles onRebalanceDone
[jira] [Commented] (IGNITE-17221) Need to propagate HLC with RAFT events
[ https://issues.apache.org/jira/browse/IGNITE-17221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17572018#comment-17572018 ] Sergey Uttsel commented on IGNITE-17221: Merged into [https://github.com/gridgain/apache-ignite-3/tree/ignite3_tx] > Need to propagate HLC with RAFT events > -- > > Key: IGNITE-17221 > URL: https://issues.apache.org/jira/browse/IGNITE-17221 > Project: Ignite > Issue Type: Improvement >Reporter: Sergey Uttsel >Assignee: Sergey Uttsel >Priority: Major > Labels: ignite-3, transaction3_ro > > RAFT events help to synchronize HLC between RAFT replicas. All RAFT > communications are initiated by a leader, and only one leader can exist at a > time. This enforces monotonic growth of HLC on raft group replicas. Heartbeat > AppendEntries RPC calls are enriched with sender’s HLC. The HLC update rules > are applied on receiving messages. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17221) Need to propagate HLC with RAFT events
[ https://issues.apache.org/jira/browse/IGNITE-17221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Uttsel updated IGNITE-17221: --- Description: RAFT events help to synchronize HLC between RAFT replicas. All RAFT communications are initiated by a leader, and only one leader can exist at a time. This enforces monotonic growth of HLC on raft group replicas. Heartbeat AppendEntries RPC calls are enriched with sender’s HLC. The HLC update rules are applied on receiving messages. (was: RAFT events help to synchronize HLC between RAFT replicas. All RAFT communications are initiated by a leader, and only one leader can exist at a time. This enforces monotonic growth of HLC on raft group replicas. RequestVote and AppendEntries RPC calls are enriched with sender’s HLC. The HLC update rules are applied on receiving messages.) > Need to propagate HLC with RAFT events > -- > > Key: IGNITE-17221 > URL: https://issues.apache.org/jira/browse/IGNITE-17221 > Project: Ignite > Issue Type: Improvement >Reporter: Sergey Uttsel >Assignee: Sergey Uttsel >Priority: Major > Labels: ignite-3, transaction3_ro > > RAFT events help to synchronize HLC between RAFT replicas. All RAFT > communications are initiated by a leader, and only one leader can exist at a > time. This enforces monotonic growth of HLC on raft group replicas. Heartbeat > AppendEntries RPC calls are enriched with sender’s HLC. The HLC update rules > are applied on receiving messages. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17429) Sql. Extend sql grammar to support CREATE/DROP INDEX statements
[
https://issues.apache.org/jira/browse/IGNITE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yury Gerzhedovich updated IGNITE-17429:
---
Description:
We need to give an ability to manage index via DDL statements.
As a first step lets support CREATE and DROP statements only. Within this
ticket we need to extend the sql grammar file as well as provide all necessary
structures to keep parsed AST.
The proposed grammar:
{code:java}
CREATE INDEX [IF NOT EXISTS] name ON table_name [USING HASH | SORTED ]
(column_name [ASC | DESC] [NULLS {FIRST | LAST}] [, ...]);
{code}
{code:java}
DROP INDEX [IF EXISTS] name;{code}
was:
We need to give an ability to manage index via DDL statements.
As a first step lets support CREATE and DROP statements only. Within this
ticket we need to extend the sql grammar file as well as provide all necessary
structures to keep parsed AST.
The proposed grammar:
{code:java}
CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | SORTED ]
(column_name [ASC | DESC] [NULLS {FIRST | LAST}] [, ...]);
{code}
{code:java}
DROP INDEX [IF EXISTS] name;{code}
> Sql. Extend sql grammar to support CREATE/DROP INDEX statements
> ---
>
> Key: IGNITE-17429
> URL: https://issues.apache.org/jira/browse/IGNITE-17429
> Project: Ignite
> Issue Type: Improvement
> Components: sql
>Reporter: Konstantin Orlov
>Priority: Major
> Labels: ignite-3
>
> We need to give an ability to manage index via DDL statements.
> As a first step lets support CREATE and DROP statements only. Within this
> ticket we need to extend the sql grammar file as well as provide all
> necessary structures to keep parsed AST.
> The proposed grammar:
> {code:java}
> CREATE INDEX [IF NOT EXISTS] name ON table_name [USING HASH | SORTED ]
> (column_name [ASC | DESC] [NULLS {FIRST | LAST}] [, ...]);
>
> {code}
> {code:java}
> DROP INDEX [IF EXISTS] name;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IGNITE-17258) Implement ReplicaListener
[
https://issues.apache.org/jira/browse/IGNITE-17258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladislav Pyatkov reassigned IGNITE-17258:
--
Assignee: Vladislav Pyatkov
> Implement ReplicaListener
> -
>
> Key: IGNITE-17258
> URL: https://issues.apache.org/jira/browse/IGNITE-17258
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexander Lapin
>Assignee: Vladislav Pyatkov
>Priority: Major
> Labels: ignite-3, transaction3_rw
>
> For general context please check IGNITE-17252. In order to specify
> request-specific handling logic that will map particular actionRequest to
> corresponding set of operations it's required to introduce such mapping rules
> in a similar way that is used within raft listeners, in other words it's
> required to introduce a sort of state machine for replica
> As tx design document notes common flow for major tx requests is following:
> {code:java}
> On receiving OpRequest
> 1. Check primary replica lease. Return the failure if not valid.
> 2. Try to acquire a shared or exclusive lock, depending on the op type.
> 3. If failed to acquire the lock due to a conflict, return the failure.
> 4. When the lock is acquired, return an OpResponse to a coordinator with a
> value, if op type is read or read-write.The OpResponse structure is:
> opCode:int // 0 - ok, !0 - error code
> result:Array
> readLeases:Map
> timestamp:HLC
> 5. Replicate a write intent asynchronously if op type is write or read-write
> As soon as the write intent is replicated, send WriteAckResponse to the
> coordinator. The WriteAckReponse structure is:
> opCode:int // 0 - ok, !0 - error code
> opId:int
> timestamp:HLC
> 6. Return the replication ack response to a coordinator. {code}
> Given steps should be managed from within ReplicaListener. Why? Because
> concrete set of locks to acquire depends on operation type:
> {code:java}
> The required locks on the row store are the following:
> 1. Tuple get(RowId rowId, UUID txId)
> IS_commit(table) S_commit(rowId)
> 2.Tuple get(RowId rowId, @Nullable Timestamp timestamp)
> No locks. Null timestamp is used to read the latest committed value for a
> single get.
> 3.Tuple getForUpdate(RowId rowId, UUID txId)
> IX_commit(table) X_commit(rowId)
> 4. RowId insert(Tuple row, UUID txId)
> IX_commit(table)
> 5. boolean update(RowId rowId, Tuple newRow, UUID txId)
> IX_commit(table) X_commit(rowId)
> 6. Tuple remove(RowId rowId, UUID txId)
> IX_commit(table) X_commit(rowId)
> 7. void commitWrite(RowId rowId, Timestamp timestamp, UUID txId)
> 8. void abortWrite(RowId rowId, UUID txId)
> 9. Iterator scan(Predicate filter, UUID txId)
> S_commit(table) - if a predicate can produce phantom reads,
> IS_commit(table) - otherwise
> 10. Iterator scan(Predicate filter, Timestamp timestamp)
> No locks
> 11. Iterator invoke(Predicate filter, InvokeClosure clo,
> UUID txId)
> SIX_commit(table) - if a predicate can produce phantom reads,
> IX_commit(table) otherwise X_commit on each updated row. {code}
> Please check ts design for full set of required actions for lock management,
> e.g. index-based locks.
> Besides that there are some actions, like commit/abort transaction
> (replicateTxnState) that have dedicated handling logic.
> *!* Given ticket should be validated with SE_team in order to check whether
> whey are fine with proposed index managing actors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IGNITE-16961) SQL API: Add prepared statement support.
[ https://issues.apache.org/jira/browse/IGNITE-16961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Orlov reassigned IGNITE-16961: - Assignee: Konstantin Orlov > SQL API: Add prepared statement support. > > > Key: IGNITE-16961 > URL: https://issues.apache.org/jira/browse/IGNITE-16961 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Andrey Mashenkov >Assignee: Konstantin Orlov >Priority: Major > Labels: ignite-3 > > Let's add prepared statement support. > * avoid extra sending query parameters (incl. query text) after server-side > statement object has been created (unless client re-connect happened) > * avoid query planing more than once (unless client re-connect happened or > schema has been changed). > * release server resources when statement was closed or session was > invalidated. > Should be added support for the following clints: JDBC, embedded. ( thin > clients as separate ticket) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17429) Sql. Extend sql grammar to support CREATE/DROP INDEX statements
[
https://issues.apache.org/jira/browse/IGNITE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yury Gerzhedovich updated IGNITE-17429:
---
Description:
We need to give an ability to manage index via DDL statements.
As a first step lets support CREATE and DROP statements only. Within this
ticket we need to extend the sql grammar file as well as provide all necessary
structures to keep parsed AST.
The proposed grammar:
{code:java}
CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | SORTED ]
(column_name [ASC | DESC] [NULLS {FIRST | LAST}] [, ...]);
{code}
{code:java}
DROP INDEX [IF EXISTS] name;{code}
was:
We need to give an ability to manage index via DDL statements.
As a first step lets support CREATE and DROP statements only. Within this
ticket we need to extend the sql grammar file as well as provide all necessary
structures to keep parsed AST.
The proposed grammar:
{code:java}
CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | BTREE ]
(column_name [ASC | DESC] [NULLS {FIRST | LAST}] [, ...]);
{code}
{code:java}
DROP INDEX [IF EXISTS] name;{code}
> Sql. Extend sql grammar to support CREATE/DROP INDEX statements
> ---
>
> Key: IGNITE-17429
> URL: https://issues.apache.org/jira/browse/IGNITE-17429
> Project: Ignite
> Issue Type: Improvement
> Components: sql
>Reporter: Konstantin Orlov
>Priority: Major
> Labels: ignite-3
>
> We need to give an ability to manage index via DDL statements.
> As a first step lets support CREATE and DROP statements only. Within this
> ticket we need to extend the sql grammar file as well as provide all
> necessary structures to keep parsed AST.
> The proposed grammar:
> {code:java}
> CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | SORTED ]
> (column_name [ASC | DESC] [NULLS {FIRST | LAST}] [, ...]);
>
> {code}
> {code:java}
> DROP INDEX [IF EXISTS] name;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-17252) Introduce Replica, ReplicaServer(?), ReplicaService and ReplicaListener interfaces
[
https://issues.apache.org/jira/browse/IGNITE-17252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571937#comment-17571937
]
Vladislav Pyatkov commented on IGNITE-17252:
Merged to ignite3_tx
> Introduce Replica, ReplicaServer(?), ReplicaService and ReplicaListener
> interfaces
> --
>
> Key: IGNITE-17252
> URL: https://issues.apache.org/jira/browse/IGNITE-17252
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexander Lapin
>Assignee: Vladislav Pyatkov
>Priority: Major
> Labels: ignite-3, transaction3_rw
> Time Spent: 12h 10m
> Remaining Estimate: 0h
>
> h2. General context
> According to tx design document new abstraction is introduced to encapsulate
> replication engine (e.g. Raft) from business logic, called {*}primary
> replica{*}:
> {code:java}
> A primary replica is a replica which serves a special purpose in the
> transaction protocol.Only one primary replica can exist at a time. Each
> replica is identified by liveness interval (startTs, endTs). All such
> intervals are disjoint, so the new primary replica liveness interval can’t
> overlap with the previous. Timestamps used for defining the intervals must be
> comparable with timestamps assigned to committing transactions. For example,
> HLC timestamps can be used for this purpose.
> Primary replica is used to execute CC protocol (so all reads and writes go
> through it), thus maintaining serializable executions, as described in the
> next section.
> The simplest implementation would be piggy-backing to RAFT protocol for tying
> a primary replica to a RAFT leader. See the leaseholder section from the RAFT
> paper for details. For this approach a RAFT leader is identical to a primary
> replica node. The endTs is constantly extended using RAFT heart beating.
> A primary replica’s status can be voluntarily transferred to another replica.
> This is only possible after its liveness interval expires. This can be
> useful, for example, for RAFT leaders balancing. {code}
> Besides obvious lease-based disjoint replication leader detection, primary
> replica is also responsible for handling messages acting as a storage and
> replication pre-and-post-processor. It's up to replica to
> * acquire, release and await locks
> * propagate requests to storage directly
> * convert message to an appropriate replication(Raft) command and propagate
> it to the replication engine.
> Let's check following example:
> *As-Is (currently):*
> {code:java}
> // client-side
> InternalTable.upsert()
> enlistInTx()
> raftService.run(upsertCommand)
> raftGroupService.sendWithRetry(ActionRequest.of(upsertCommand))
> messagingService().invoke(actionRequest)
> // server-side
> ActionRequestProcessor.handleRequest(actionRequest)
> future =
> JraftServerImpl.DelegatingStateMachine.getListener().onBeforeApply(request.command());
> // Lock management
> future.handle(actionRequest.command() instanceof WriteCommand ?
> applyWrite(actionRequest) : applyRead(actionRequest)){code}
> Please pay attention to *onBeforeApply* step. It was introduced in order to
> manage(acquire) locks with further locks awaiting *outside* the raft. It is
> critical not to occupy the linearized in-raft execution with such lengthy
> operations as waiting for locks to be released.
> It worth to mention, that such approach has several disadvantages, e.g.
> onBeforeApply step is executed before isLeader() check, so that, it might
> acquire lock on non-leader-node that is not the expected behavior.
> *To-Be (should be implemented):*
> {code:java}
> // client-side
> InternalTable.upsert()
> enlistInTx()
> replicaService.invoke(upsertRequest, primary=true)
> // server-side
> Replica.handleRequest(actionRequest)
>if (actionRequest.isPrimaryEvaluationExpected())
> checkLease(); // Return failure if not valid
>
>if (actionRequest instanceOf WriteRequest) {
> // validate writeRequest locally
>
> // acquire all locks !locally!
> fut = txManager.intentWriteLock(table);
>
> fut.handle(()->
> return
> future.of(async(replicationEngine.replicate(ReplicationCommand.of(writeRequest
> )
>}{code}
> in other word:
> * Instead of raftGroupService, replicaService should be used.
> * ReplicaService uses messages (actionRequests) instead of raft commands.
> * Within the scope of RW transactions replicaService always sends requests
> to the *primary* replica, however within the RO transactions non-primary
> replicas will also participate in requests handling, so that I believe we
> should introduce common Replica instead of strict PrimaryReplica.
> * Replica is aware of requests
[jira] [Updated] (IGNITE-17252) Introduce Replica, ReplicaServer(?), ReplicaService and ReplicaListener interfaces
[
https://issues.apache.org/jira/browse/IGNITE-17252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Lapin updated IGNITE-17252:
-
Reviewer: Alexander Lapin
> Introduce Replica, ReplicaServer(?), ReplicaService and ReplicaListener
> interfaces
> --
>
> Key: IGNITE-17252
> URL: https://issues.apache.org/jira/browse/IGNITE-17252
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexander Lapin
>Assignee: Vladislav Pyatkov
>Priority: Major
> Labels: ignite-3, transaction3_rw
> Time Spent: 11h 50m
> Remaining Estimate: 0h
>
> h2. General context
> According to tx design document new abstraction is introduced to encapsulate
> replication engine (e.g. Raft) from business logic, called {*}primary
> replica{*}:
> {code:java}
> A primary replica is a replica which serves a special purpose in the
> transaction protocol.Only one primary replica can exist at a time. Each
> replica is identified by liveness interval (startTs, endTs). All such
> intervals are disjoint, so the new primary replica liveness interval can’t
> overlap with the previous. Timestamps used for defining the intervals must be
> comparable with timestamps assigned to committing transactions. For example,
> HLC timestamps can be used for this purpose.
> Primary replica is used to execute CC protocol (so all reads and writes go
> through it), thus maintaining serializable executions, as described in the
> next section.
> The simplest implementation would be piggy-backing to RAFT protocol for tying
> a primary replica to a RAFT leader. See the leaseholder section from the RAFT
> paper for details. For this approach a RAFT leader is identical to a primary
> replica node. The endTs is constantly extended using RAFT heart beating.
> A primary replica’s status can be voluntarily transferred to another replica.
> This is only possible after its liveness interval expires. This can be
> useful, for example, for RAFT leaders balancing. {code}
> Besides obvious lease-based disjoint replication leader detection, primary
> replica is also responsible for handling messages acting as a storage and
> replication pre-and-post-processor. It's up to replica to
> * acquire, release and await locks
> * propagate requests to storage directly
> * convert message to an appropriate replication(Raft) command and propagate
> it to the replication engine.
> Let's check following example:
> *As-Is (currently):*
> {code:java}
> // client-side
> InternalTable.upsert()
> enlistInTx()
> raftService.run(upsertCommand)
> raftGroupService.sendWithRetry(ActionRequest.of(upsertCommand))
> messagingService().invoke(actionRequest)
> // server-side
> ActionRequestProcessor.handleRequest(actionRequest)
> future =
> JraftServerImpl.DelegatingStateMachine.getListener().onBeforeApply(request.command());
> // Lock management
> future.handle(actionRequest.command() instanceof WriteCommand ?
> applyWrite(actionRequest) : applyRead(actionRequest)){code}
> Please pay attention to *onBeforeApply* step. It was introduced in order to
> manage(acquire) locks with further locks awaiting *outside* the raft. It is
> critical not to occupy the linearized in-raft execution with such lengthy
> operations as waiting for locks to be released.
> It worth to mention, that such approach has several disadvantages, e.g.
> onBeforeApply step is executed before isLeader() check, so that, it might
> acquire lock on non-leader-node that is not the expected behavior.
> *To-Be (should be implemented):*
> {code:java}
> // client-side
> InternalTable.upsert()
> enlistInTx()
> replicaService.invoke(upsertRequest, primary=true)
> // server-side
> Replica.handleRequest(actionRequest)
>if (actionRequest.isPrimaryEvaluationExpected())
> checkLease(); // Return failure if not valid
>
>if (actionRequest instanceOf WriteRequest) {
> // validate writeRequest locally
>
> // acquire all locks !locally!
> fut = txManager.intentWriteLock(table);
>
> fut.handle(()->
> return
> future.of(async(replicationEngine.replicate(ReplicationCommand.of(writeRequest
> )
>}{code}
> in other word:
> * Instead of raftGroupService, replicaService should be used.
> * ReplicaService uses messages (actionRequests) instead of raft commands.
> * Within the scope of RW transactions replicaService always sends requests
> to the *primary* replica, however within the RO transactions non-primary
> replicas will also participate in requests handling, so that I believe we
> should introduce common Replica instead of strict PrimaryReplica.
> * Replica is aware of requests handling logic and process actions in a
[jira] [Commented] (IGNITE-17252) Introduce Replica, ReplicaServer(?), ReplicaService and ReplicaListener interfaces
[
https://issues.apache.org/jira/browse/IGNITE-17252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571931#comment-17571931
]
Alexander Lapin commented on IGNITE-17252:
--
[~v.pyatkov] LGTM to feature branch.
> Introduce Replica, ReplicaServer(?), ReplicaService and ReplicaListener
> interfaces
> --
>
> Key: IGNITE-17252
> URL: https://issues.apache.org/jira/browse/IGNITE-17252
> Project: Ignite
> Issue Type: Improvement
>Reporter: Alexander Lapin
>Assignee: Vladislav Pyatkov
>Priority: Major
> Labels: ignite-3, transaction3_rw
> Time Spent: 11h 50m
> Remaining Estimate: 0h
>
> h2. General context
> According to tx design document new abstraction is introduced to encapsulate
> replication engine (e.g. Raft) from business logic, called {*}primary
> replica{*}:
> {code:java}
> A primary replica is a replica which serves a special purpose in the
> transaction protocol.Only one primary replica can exist at a time. Each
> replica is identified by liveness interval (startTs, endTs). All such
> intervals are disjoint, so the new primary replica liveness interval can’t
> overlap with the previous. Timestamps used for defining the intervals must be
> comparable with timestamps assigned to committing transactions. For example,
> HLC timestamps can be used for this purpose.
> Primary replica is used to execute CC protocol (so all reads and writes go
> through it), thus maintaining serializable executions, as described in the
> next section.
> The simplest implementation would be piggy-backing to RAFT protocol for tying
> a primary replica to a RAFT leader. See the leaseholder section from the RAFT
> paper for details. For this approach a RAFT leader is identical to a primary
> replica node. The endTs is constantly extended using RAFT heart beating.
> A primary replica’s status can be voluntarily transferred to another replica.
> This is only possible after its liveness interval expires. This can be
> useful, for example, for RAFT leaders balancing. {code}
> Besides obvious lease-based disjoint replication leader detection, primary
> replica is also responsible for handling messages acting as a storage and
> replication pre-and-post-processor. It's up to replica to
> * acquire, release and await locks
> * propagate requests to storage directly
> * convert message to an appropriate replication(Raft) command and propagate
> it to the replication engine.
> Let's check following example:
> *As-Is (currently):*
> {code:java}
> // client-side
> InternalTable.upsert()
> enlistInTx()
> raftService.run(upsertCommand)
> raftGroupService.sendWithRetry(ActionRequest.of(upsertCommand))
> messagingService().invoke(actionRequest)
> // server-side
> ActionRequestProcessor.handleRequest(actionRequest)
> future =
> JraftServerImpl.DelegatingStateMachine.getListener().onBeforeApply(request.command());
> // Lock management
> future.handle(actionRequest.command() instanceof WriteCommand ?
> applyWrite(actionRequest) : applyRead(actionRequest)){code}
> Please pay attention to *onBeforeApply* step. It was introduced in order to
> manage(acquire) locks with further locks awaiting *outside* the raft. It is
> critical not to occupy the linearized in-raft execution with such lengthy
> operations as waiting for locks to be released.
> It worth to mention, that such approach has several disadvantages, e.g.
> onBeforeApply step is executed before isLeader() check, so that, it might
> acquire lock on non-leader-node that is not the expected behavior.
> *To-Be (should be implemented):*
> {code:java}
> // client-side
> InternalTable.upsert()
> enlistInTx()
> replicaService.invoke(upsertRequest, primary=true)
> // server-side
> Replica.handleRequest(actionRequest)
>if (actionRequest.isPrimaryEvaluationExpected())
> checkLease(); // Return failure if not valid
>
>if (actionRequest instanceOf WriteRequest) {
> // validate writeRequest locally
>
> // acquire all locks !locally!
> fut = txManager.intentWriteLock(table);
>
> fut.handle(()->
> return
> future.of(async(replicationEngine.replicate(ReplicationCommand.of(writeRequest
> )
>}{code}
> in other word:
> * Instead of raftGroupService, replicaService should be used.
> * ReplicaService uses messages (actionRequests) instead of raft commands.
> * Within the scope of RW transactions replicaService always sends requests
> to the *primary* replica, however within the RO transactions non-primary
> replicas will also participate in requests handling, so that I believe we
> should introduce common Replica instead of strict PrimaryReplica.
> * Replica is aware of
[jira] [Updated] (IGNITE-17429) Sql. Extend sql grammar to support CREATE/DROP INDEX statements
[
https://issues.apache.org/jira/browse/IGNITE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yury Gerzhedovich updated IGNITE-17429:
---
Description:
We need to give an ability to manage index via DDL statements.
As a first step lets support CREATE and DROP statements only. Within this
ticket we need to extend the sql grammar file as well as provide all necessary
structures to keep parsed AST.
The proposed grammar:
{code:java}
CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | BTREE ]
(column_name [ASC | DESC] [NULLS {FIRST | LAST}] [, ...]);
{code}
{code:java}
DROP INDEX [IF EXISTS] name;{code}
was:
We need to give an ability to manage index via DDL statements.
As a first step lets support CREATE and DROP statements only. Within this
ticket we need to extend the sql grammar file as well as provide all necessary
structures to keep parsed AST.
The proposed grammar:
CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | BTREE ]
(column_name [ASC | DESC] [NULLS {FIRST | LAST}] [, ...]);
DROP INDEX [IF EXISTS] name;
> Sql. Extend sql grammar to support CREATE/DROP INDEX statements
> ---
>
> Key: IGNITE-17429
> URL: https://issues.apache.org/jira/browse/IGNITE-17429
> Project: Ignite
> Issue Type: Improvement
> Components: sql
>Reporter: Konstantin Orlov
>Priority: Major
> Labels: ignite-3
>
> We need to give an ability to manage index via DDL statements.
> As a first step lets support CREATE and DROP statements only. Within this
> ticket we need to extend the sql grammar file as well as provide all
> necessary structures to keep parsed AST.
> The proposed grammar:
> {code:java}
> CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | BTREE ]
> (column_name [ASC | DESC] [NULLS {FIRST | LAST}] [, ...]);
>
> {code}
> {code:java}
> DROP INDEX [IF EXISTS] name;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17429) Sql. Extend sql grammar to support CREATE/DROP INDEX statements
[
https://issues.apache.org/jira/browse/IGNITE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yury Gerzhedovich updated IGNITE-17429:
---
Description:
We need to give an ability to manage index via DDL statements.
As a first step lets support CREATE and DROP statements only. Within this
ticket we need to extend the sql grammar file as well as provide all necessary
structures to keep parsed AST.
The proposed grammar:
CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | BTREE ]
(column_name [ASC | DESC] [NULLS {FIRST | LAST}] [, ...]);
DROP INDEX [IF EXISTS] name;
was:
We need to give an ability to manage index via DDL statements.
As a first step lets support CREATE and DROP statements only. Within this
ticket we need to extend the sql grammar file as well as provide all necessary
structures to keep parsed AST.
The proposed grammar:
CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | BTREE ]
(column_name [ASC | DESC] [NULLS \\{FIRST | LAST}] [, ...])
DROP INDEX [IF EXISTS] name
> Sql. Extend sql grammar to support CREATE/DROP INDEX statements
> ---
>
> Key: IGNITE-17429
> URL: https://issues.apache.org/jira/browse/IGNITE-17429
> Project: Ignite
> Issue Type: Improvement
> Components: sql
>Reporter: Konstantin Orlov
>Priority: Major
> Labels: ignite-3
>
> We need to give an ability to manage index via DDL statements.
> As a first step lets support CREATE and DROP statements only. Within this
> ticket we need to extend the sql grammar file as well as provide all
> necessary structures to keep parsed AST.
> The proposed grammar:
> CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | BTREE ]
> (column_name [ASC | DESC] [NULLS {FIRST | LAST}] [, ...]);
> DROP INDEX [IF EXISTS] name;
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17429) Sql. Extend sql grammar to support CREATE/DROP INDEX statements
[
https://issues.apache.org/jira/browse/IGNITE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yury Gerzhedovich updated IGNITE-17429:
---
Description:
We need to give an ability to manage index via DDL statements.
As a first step lets support CREATE and DROP statements only. Within this
ticket we need to extend the sql grammar file as well as provide all necessary
structures to keep parsed AST.
The proposed grammar:
CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | BTREE ]
(column_name [ASC | DESC] [NULLS \\{FIRST | LAST}] [, ...])
DROP INDEX [IF EXISTS] name
was:
We need to give an ability to manage index via DDL statements.
As a first step lets support CREATE and DROP statements only. Within this
ticket we need to extend the sql grammar file as well as provide all necessary
structures to keep parsed AST.
The proposed grammar:
CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | BTREE ]
(column_name [ASC | DESC] [NULLS \{FIRST | LAST}] [, ...])
> Sql. Extend sql grammar to support CREATE/DROP INDEX statements
> ---
>
> Key: IGNITE-17429
> URL: https://issues.apache.org/jira/browse/IGNITE-17429
> Project: Ignite
> Issue Type: Improvement
> Components: sql
>Reporter: Konstantin Orlov
>Priority: Major
> Labels: ignite-3
>
> We need to give an ability to manage index via DDL statements.
> As a first step lets support CREATE and DROP statements only. Within this
> ticket we need to extend the sql grammar file as well as provide all
> necessary structures to keep parsed AST.
> The proposed grammar:
> CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | BTREE ]
> (column_name [ASC | DESC] [NULLS \\{FIRST | LAST}] [, ...])
> DROP INDEX [IF EXISTS] name
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17429) Sql. Extend sql grammar to support CREATE/DROP INDEX statements
[
https://issues.apache.org/jira/browse/IGNITE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yury Gerzhedovich updated IGNITE-17429:
---
Description:
We need to give an ability to manage index via DDL statements.
As a first step lets support CREATE and DROP statements only. Within this
ticket we need to extend the sql grammar file as well as provide all necessary
structures to keep parsed AST.
The proposed grammar:
CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | BTREE ]
(column_name [ASC | DESC] [NULLS \{FIRST | LAST}] [, ...])
was:
We need to give an ability to manage index via DDL statements.
As a first step lets support CREATE and DROP statements only. Within this
ticket we need to extend the sql grammar file as well as provide all necessary
structures to keep parsed AST.
The proposed grammar:
CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | BTREE ]
(column_name [ASC | DESC] [NULLS \{FIRST | LAST}] [, ...])
> Sql. Extend sql grammar to support CREATE/DROP INDEX statements
> ---
>
> Key: IGNITE-17429
> URL: https://issues.apache.org/jira/browse/IGNITE-17429
> Project: Ignite
> Issue Type: Improvement
> Components: sql
>Reporter: Konstantin Orlov
>Priority: Major
> Labels: ignite-3
>
> We need to give an ability to manage index via DDL statements.
> As a first step lets support CREATE and DROP statements only. Within this
> ticket we need to extend the sql grammar file as well as provide all
> necessary structures to keep parsed AST.
> The proposed grammar:
> CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | BTREE ]
> (column_name [ASC | DESC] [NULLS \{FIRST | LAST}] [, ...])
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17429) Sql. Extend sql grammar to support CREATE/DROP INDEX statements
[
https://issues.apache.org/jira/browse/IGNITE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yury Gerzhedovich updated IGNITE-17429:
---
Description:
We need to give an ability to manage index via DDL statements.
As a first step lets support CREATE and DROP statements only. Within this
ticket we need to extend the sql grammar file as well as provide all necessary
structures to keep parsed AST.
The proposed grammar:
CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | BTREE ]
(column_name [ASC | DESC] [NULLS \{FIRST | LAST}] [, ...])
was:
We need to give an ability to manage index via DDL statements.
As a first step lets support CREATE and DROP statements only. Within this
ticket we need to extend the sql grammar file as well as provide all necessary
structures to keep parsed AST.
> Sql. Extend sql grammar to support CREATE/DROP INDEX statements
> ---
>
> Key: IGNITE-17429
> URL: https://issues.apache.org/jira/browse/IGNITE-17429
> Project: Ignite
> Issue Type: Improvement
> Components: sql
>Reporter: Konstantin Orlov
>Priority: Major
> Labels: ignite-3
>
> We need to give an ability to manage index via DDL statements.
> As a first step lets support CREATE and DROP statements only. Within this
> ticket we need to extend the sql grammar file as well as provide all
> necessary structures to keep parsed AST.
> The proposed grammar:
> CREATE INDEX [[IF NOT EXISTS] name] ON table_name [USING HASH | BTREE ]
> (column_name [ASC | DESC] [NULLS \{FIRST | LAST}] [, ...])
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-17328) Support explicit transactions interactions with SQL.
[ https://issues.apache.org/jira/browse/IGNITE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571901#comment-17571901 ] Konstantin Orlov commented on IGNITE-17328: --- [~zstan], LGTM! > Support explicit transactions interactions with SQL. > > > Key: IGNITE-17328 > URL: https://issues.apache.org/jira/browse/IGNITE-17328 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Yury Gerzhedovich >Assignee: Evgeny Stanilovsky >Priority: Major > Labels: ignite-3 > Time Spent: 1h 40m > Remaining Estimate: 0h > > All SQL queries should be executed transactionally, for now - explicit tx are > not forwarded into inner sql execution layer, this need to be fixed. > This issue will close only dml modification\insertion case. > All others : transactional ddl, RO tx scan, need to be implemented under > other issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-14937) Index schema & Index management integration
[ https://issues.apache.org/jira/browse/IGNITE-14937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Orlov updated IGNITE-14937: -- Ignite Flags: (was: Docs Required,Release Notes Required) > Index schema & Index management integration > --- > > Key: IGNITE-14937 > URL: https://issues.apache.org/jira/browse/IGNITE-14937 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Taras Ledkov >Priority: Major > Labels: ignite-3 > > Public index schema (required indexes) and current indexes state on the > cluster are different. > We have to track it, store it and provide actual indexes schema state for any > components: select query, DDL query etc.. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17431) Sql. Index support by optimizer
Konstantin Orlov created IGNITE-17431:
-
Summary: Sql. Index support by optimizer
Key: IGNITE-17431
URL: https://issues.apache.org/jira/browse/IGNITE-17431
Project: Ignite
Issue Type: Improvement
Components: sql
Reporter: Konstantin Orlov
We need to integrate indexes into optimisation framework. This includes
following parts:
- Integration of indexes into sql schema management: we need to provide a way
to discover indexes during optimising phase (see {{ExposeIndexRule}})
- Integration of indexes into execution: need to provide an execution node to
convert IndexScan to (see {{LogicalRelImplementor#visit(IgniteIndexScan)}})
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-17430) Sql. Provide commands and handlers for index related operations
Konstantin Orlov created IGNITE-17430: - Summary: Sql. Provide commands and handlers for index related operations Key: IGNITE-17430 URL: https://issues.apache.org/jira/browse/IGNITE-17430 Project: Ignite Issue Type: Improvement Components: sql Reporter: Konstantin Orlov After IGNITE-17429 and backend for index management will be implemented, we need to connect both parts together. For thus, we need to implement AST to Command conversion as well as handlers for new index-related commands which will delegate invocations to index manager (similar to table-related commands). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17429) Sql. Extend sql grammar to support CREATE/DROP INDEX statements
Konstantin Orlov created IGNITE-17429: - Summary: Sql. Extend sql grammar to support CREATE/DROP INDEX statements Key: IGNITE-17429 URL: https://issues.apache.org/jira/browse/IGNITE-17429 Project: Ignite Issue Type: Improvement Components: sql Reporter: Konstantin Orlov We need to give an ability to manage index via DDL statements. As a first step lets support CREATE and DROP statements only. Within this ticket we need to extend the sql grammar file as well as provide all necessary structures to keep parsed AST. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17427) Update netty version up to 4.1.79.Final
[ https://issues.apache.org/jira/browse/IGNITE-17427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Gidaspov updated IGNITE-17427: - Summary: Update netty version up to 4.1.79.Final (was: Update netty version up to 4.1.75.Final) > Update netty version up to 4.1.79.Final > --- > > Key: IGNITE-17427 > URL: https://issues.apache.org/jira/browse/IGNITE-17427 > Project: Ignite > Issue Type: Task >Reporter: Alexey Gidaspov >Assignee: Alexey Gidaspov >Priority: Minor > > Need to update netty version up to 4.1.75.Final because of > [CVE-2022-24823|https://nvd.nist.gov/vuln/detail/CVE-2022-24823] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17427) Update netty version up to 4.1.79.Final
[ https://issues.apache.org/jira/browse/IGNITE-17427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Gidaspov updated IGNITE-17427: - Description: Need to update netty version up to 4.1.79.Final because of [CVE-2022-24823|https://nvd.nist.gov/vuln/detail/CVE-2022-24823] (was: Need to update netty version up to 4.1.75.Final because of [CVE-2022-24823|https://nvd.nist.gov/vuln/detail/CVE-2022-24823]) > Update netty version up to 4.1.79.Final > --- > > Key: IGNITE-17427 > URL: https://issues.apache.org/jira/browse/IGNITE-17427 > Project: Ignite > Issue Type: Task >Reporter: Alexey Gidaspov >Assignee: Alexey Gidaspov >Priority: Minor > > Need to update netty version up to 4.1.79.Final because of > [CVE-2022-24823|https://nvd.nist.gov/vuln/detail/CVE-2022-24823] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (IGNITE-17329) Transaction exception handling for SQL
[ https://issues.apache.org/jira/browse/IGNITE-17329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yury Gerzhedovich resolved IGNITE-17329. Resolution: Invalid > Transaction exception handling for SQL > -- > > Key: IGNITE-17329 > URL: https://issues.apache.org/jira/browse/IGNITE-17329 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Yury Gerzhedovich >Priority: Major > Labels: ignite-3 > > Let's implement simple version of transaction exception handling for SQL > execution - just a abort a transaction and stop execution. It's not a good > and final solution, but enought for start. As a next step (in a separate > ticket) we should think out cases which could be retried without of aborting > a transaction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17331) Smart transaction exception handling for SQL
[ https://issues.apache.org/jira/browse/IGNITE-17331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yury Gerzhedovich updated IGNITE-17331: --- Description: As of now we have just abort transaction and SQL execution in case of any error. Let's investigate for which cases possible retry execute whole query or just a fragments behind the scene and don't notify user about a issue. As result of the task should be banch of tickets with description possible cases and ways to implement the retries logic. was: After implementing IGNITE-17329 we have just abort transaction and SQL execution in case of any error. Let's investigate for which cases possible retry execute whole query or just a fragments behind the scene and don't notify user about a issue. As result of the task should be banch of tickets with description possible cases and ways to implement the retries logic. > Smart transaction exception handling for SQL > > > Key: IGNITE-17331 > URL: https://issues.apache.org/jira/browse/IGNITE-17331 > Project: Ignite > Issue Type: Improvement >Reporter: Yury Gerzhedovich >Priority: Major > Labels: ignite-3 > > As of now we have just abort transaction and SQL execution in case of any > error. Let's investigate for which cases possible retry execute whole query > or just a fragments behind the scene and don't notify user about a issue. > As result of the task should be banch of tickets with description possible > cases and ways to implement the retries logic. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17428) Race between table create and table get
Denis Chudov created IGNITE-17428:
-
Summary: Race between table create and table get
Key: IGNITE-17428
URL: https://issues.apache.org/jira/browse/IGNITE-17428
Project: Ignite
Issue Type: Bug
Reporter: Denis Chudov
Current version of TableManager#tableAsyncInternal can possibly not detect
table that is being created while tableAsyncInternal is called. Scenario:
- tableAsyncInternal checks tablesByIdVv.latest() and there is no table
- the table creation started, table metadata appears in meta storage
- TableEvent.CREATE is fired
- tableAsyncInternal registers a listener for TableEvent.CREATE (after it is
fired for corresponding table)
- tableAsyncInternal checks tablesByIdVv.latest() once again and there still is
no table, because the table creation is not completed
- {{!isTableConfigured(id)}} condition returns *false* as the table is present
in meta storage
- {{if (tbl != null && getTblFut.complete(tbl) || !isTableConfigured(id) &&
getTblFut.complete(null))}} evaluates *false* and the future created fot
getTable never completes.
Possibly we should use VersionedValue#whenComplete instead of creating listener
for event. The table is present in map wrapped in versioned value only when the
table creation is completed, and whenComplete allows to create a callback to
check the table presence.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-17427) Update netty version up to 4.1.75.Final
Alexey Gidaspov created IGNITE-17427: Summary: Update netty version up to 4.1.75.Final Key: IGNITE-17427 URL: https://issues.apache.org/jira/browse/IGNITE-17427 Project: Ignite Issue Type: Task Reporter: Alexey Gidaspov Assignee: Alexey Gidaspov Need to update netty version up to 4.1.75.Final because of [CVE-2022-24823|https://nvd.nist.gov/vuln/detail/CVE-2022-24823] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17135) SQL API: Exceptions are different on server and client
[ https://issues.apache.org/jira/browse/IGNITE-17135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571813#comment-17571813 ] Pavel Tupitsyn commented on IGNITE-17135: - Merged to main: f00df8cf14fdebe30c6584c8a01bfed1e2cd55da > SQL API: Exceptions are different on server and client > --- > > Key: IGNITE-17135 > URL: https://issues.apache.org/jira/browse/IGNITE-17135 > Project: Ignite > Issue Type: Bug > Components: sql, thin client >Reporter: Pavel Tupitsyn >Assignee: Pavel Tupitsyn >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-alpha6 > > Time Spent: 40m > Remaining Estimate: 0h > > Server and client SQL API implementations throw different exception types > (when cursor is closed, when SQL is incorrect, etc). Check > *ItSqlClientAsynchronousApiTest* and *ItSqlAsynchronousApiTest*. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-16860) Sql. Ability to create table with autogenerated PK values
[
https://issues.apache.org/jira/browse/IGNITE-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571809#comment-17571809
]
Yury Gerzhedovich commented on IGNITE-16860:
[~korlov] , LGTM
> Sql. Ability to create table with autogenerated PK values
> -
>
> Key: IGNITE-16860
> URL: https://issues.apache.org/jira/browse/IGNITE-16860
> Project: Ignite
> Issue Type: New Feature
> Components: sql
>Reporter: Konstantin Orlov
>Assignee: Konstantin Orlov
>Priority: Major
> Labels: ignite-3
>
> To make it easier for users to migrate from systems which allow table without
> PK, let's introduce an ability to create tables with PK whose value is
> autogenerated by system function. The syntax could look like this:
> {noformat}
> CREATE TABLE my_table (id UUID DEFAULT gen_random_uuid PRIMARY KEY, columns>){noformat}
> Where {{gen_random_uuid}} is a system function returning a random UUID value.
> Within this ticket we need to update grammar for {{CREATE TABLE}} operation
> and make all sql-runtime related changes to properly handle such default
> values.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-17407) The snapshot rate limit does not work correctly when set to values greater than 100 MB per second.
[
https://issues.apache.org/jira/browse/IGNITE-17407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571795#comment-17571795
]
Ignite TC Bot commented on IGNITE-17407:
{panel:title=Branch: [pull/10167/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/10167/head] Base: [master] : New Tests
(1)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}Basic 1{color} [[tests
1|https://ci.ignite.apache.org/viewLog.html?buildId=6698151]]
* {color:#013220}IgniteBasicTestSuite:
BasicRateLimiterTest.checkSpeedLimitChangeMultithreaded - PASSED{color}
{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=6698260buildTypeId=IgniteTests24Java8_RunAll]
> The snapshot rate limit does not work correctly when set to values greater
> than 100 MB per second.
>
>
> Key: IGNITE-17407
> URL: https://issues.apache.org/jira/browse/IGNITE-17407
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.13
>Reporter: Pavel Pereslegin
>Assignee: Pavel Pereslegin
>Priority: Major
> Fix For: 2.14
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The snapshot rate limit does not work correctly when set to values greater
> than 100 MB per second.
> Currently, the snapshot transfer rate is limited by the BasicRateLimiter.
> Testing shows that limiter is unable to give more than ~119 millions permits
> per second.
> {code:java}
> long dataSize = U.GB;
> BasicRateLimiter limiter = new BasicRateLimiter(dataSize / 2);
> int blockSize =
> IgniteSnapshotManager.SNAPSHOT_LIMITED_TRANSFER_BLOCK_SIZE_BYTES;
> long start = System.currentTimeMillis();
> for (long i = 0; i < dataSize; i+=blockSize)
> limiter.acquire(blockSize);
> long totalSec =
> TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis() - start);
> System.out.println("Speed= " + (dataSize / totalSec));
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-15837) Read Repair should support binary keys
[ https://issues.apache.org/jira/browse/IGNITE-15837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571788#comment-17571788 ] Anton Vinogradov commented on IGNITE-15837: --- Merged to the master. [~NIzhikov], thanks for the review! > Read Repair should support binary keys > -- > > Key: IGNITE-15837 > URL: https://issues.apache.org/jira/browse/IGNITE-15837 > Project: Ignite > Issue Type: Sub-task >Reporter: Anton Vinogradov >Assignee: Anton Vinogradov >Priority: Major > Labels: iep-31 > Fix For: 2.14 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17408) Parsing WAL for encrypted caches fails with InvalidMarkException
[ https://issues.apache.org/jira/browse/IGNITE-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571784#comment-17571784 ] Maksim Timonin commented on IGNITE-17408: - [~NSAmelchev] [~apolovtcev] Thanks for review! PR merged to master > Parsing WAL for encrypted caches fails with InvalidMarkException > > > Key: IGNITE-17408 > URL: https://issues.apache.org/jira/browse/IGNITE-17408 > Project: Ignite > Issue Type: New Feature >Affects Versions: 2.14 >Reporter: Maksim Timonin >Assignee: Maksim Timonin >Priority: Major > Fix For: 2.14 > > Time Spent: 50m > Remaining Estimate: 0h > > RecordV2Serializer::readWithHeaders fails while extracting actualType of > record. > Reason is the `ensure()` wasn't invoked before extracting a plain record from > an encrypted one. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17408) Parsing WAL for encrypted caches fails with InvalidMarkException
[
https://issues.apache.org/jira/browse/IGNITE-17408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571783#comment-17571783
]
Ignite TC Bot commented on IGNITE-17408:
{panel:title=Branch: [pull/10168/head] Base: [master] : Possible Blockers
(1)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}
{color:#d04437}Thin client: Node.js{color} [[tests 0 Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=6699232]]
{panel}
{panel:title=Branch: [pull/10168/head] Base: [master] : No new tests
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=6696388buildTypeId=IgniteTests24Java8_RunAll]
> Parsing WAL for encrypted caches fails with InvalidMarkException
>
>
> Key: IGNITE-17408
> URL: https://issues.apache.org/jira/browse/IGNITE-17408
> Project: Ignite
> Issue Type: New Feature
>Affects Versions: 2.14
>Reporter: Maksim Timonin
>Assignee: Maksim Timonin
>Priority: Major
> Fix For: 2.14
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> RecordV2Serializer::readWithHeaders fails while extracting actualType of
> record.
> Reason is the `ensure()` wasn't invoked before extracting a plain record from
> an encrypted one.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-16676) Investigate a possibility to start raft groups in parallel
[ https://issues.apache.org/jira/browse/IGNITE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571777#comment-17571777 ] Mirza Aliev commented on IGNITE-16676: -- [~kgusakov] LGTM > Investigate a possibility to start raft groups in parallel > -- > > Key: IGNITE-16676 > URL: https://issues.apache.org/jira/browse/IGNITE-16676 > Project: Ignite > Issue Type: Task >Reporter: Mirza Aliev >Assignee: Kirill Gusakov >Priority: Major > Labels: ignite-3 > Time Spent: 40m > Remaining Estimate: 0h > > After some investigation that was made in under IGNITE-16559 (see the > [comment|https://issues.apache.org/jira/browse/IGNITE-16559?focusedCommentId=17495362=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17495362]), > it was decided that we could investigate the possibility to start several > raft groups in parallel, hence we could reduce the duration of starting a > several raft groups, as far as the initialisation of a raft group can be > time-consuming, see https://issues.apache.org/jira/browse/IGNITE-16675. > For example, the current improvement could optimise creating of several > partition raft groups. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (IGNITE-17379) Investigate main bottlenecks in YCSB benchmark
[ https://issues.apache.org/jira/browse/IGNITE-17379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vyacheslav Koptilin resolved IGNITE-17379. -- Resolution: Fixed > Investigate main bottlenecks in YCSB benchmark > -- > > Key: IGNITE-17379 > URL: https://issues.apache.org/jira/browse/IGNITE-17379 > Project: Ignite > Issue Type: Task >Reporter: Kirill Gusakov >Assignee: Kirill Gusakov >Priority: Major > Labels: ignite-3 > > We have significant performance drop comparing to ignite2 according to YCSB > benchmarks results from https://issues.apache.org/jira/browse/IGNITE-17220. > As a result of this issue we need to investigate main bottlenecks and create > separate issues for fixing it, if needed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (IGNITE-15932) Implement storage for locks
[ https://issues.apache.org/jira/browse/IGNITE-15932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Lapin resolved IGNITE-15932. -- Resolution: Won't Fix It's not necessary to implement persistent storage for locks, because we still don't know whether it's required to replicate intention locks (https://issues.apache.org/jira/browse/IGNITE-17422), so that we might use writeIntents in order to recreate X locks on primary replica switch. Besides that we are currently fine with scanning over whole partition in order to convert writeIntents to X locks. So, within the context of lock storage, it's only required to add a sort of lock index by txId in order to evaluate Iterable locks(UUIDtxId) efficiently. And it definitely worth to do it in-place, meaning within https://issues.apache.org/jira/browse/IGNITE-16892. > Implement storage for locks > --- > > Key: IGNITE-15932 > URL: https://issues.apache.org/jira/browse/IGNITE-15932 > Project: Ignite > Issue Type: Improvement >Reporter: Alexey Scherbakov >Assignee: Alexander Lapin >Priority: Major > Labels: ignite-3, transaction3_rw > > Currently locks are stored in the heap and can take unlimited amount of > memory. > Need to implement storage for a locks with a configurable limits. It can even > store locks off-heap of use disk spilling. > Lock attempt must fail if a storage is overflowed. > *Upd:* > It's required to introduce both volatile and persistent storages for locks. > Volatile storages will be used directly from replica, persistent ones will > be populated from within corresponding ParitionListeners. > Volatile storage will manager both read and write locks for all types of lock > entities: rowIds, index keys, tables, indexes, meaning that it covers both > common and intent locks. > It's an open question whether persistent storage should handle (and replicate > intent locks), so for now it will only handle common write locks. Please pay > attention that persistent storage should be fully integrated into raft > replication logic: snapshots, checkpointIndexes, etc. > To sum up, persistent storage isn't a part of transactions3_rw stable > support, however it's better to implement it from the very begging. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17328) Support explicit transactions interactions with SQL.
[ https://issues.apache.org/jira/browse/IGNITE-17328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Evgeny Stanilovsky updated IGNITE-17328: Description: All SQL queries should be executed transactionally, for now - explicit tx are not forwarded into inner sql execution layer, this need to be fixed. This issue will close only dml modification\insertion case. All others : transactional ddl, RO tx scan, need to be implemented under other issues. was:All SQL queries should be executed transactionally, so it required implement implicit start/stop transactions in case of transaction is not active. > Support explicit transactions interactions with SQL. > > > Key: IGNITE-17328 > URL: https://issues.apache.org/jira/browse/IGNITE-17328 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Yury Gerzhedovich >Assignee: Evgeny Stanilovsky >Priority: Major > Labels: ignite-3 > Time Spent: 1h 20m > Remaining Estimate: 0h > > All SQL queries should be executed transactionally, for now - explicit tx are > not forwarded into inner sql execution layer, this need to be fixed. > This issue will close only dml modification\insertion case. > All others : transactional ddl, RO tx scan, need to be implemented under > other issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17425) C++ 3.0: Basic C++ client
[ https://issues.apache.org/jira/browse/IGNITE-17425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Sapego updated IGNITE-17425: - Summary: C++ 3.0: Basic C++ client (was: C++3.0: Basic C++ client) > C++ 3.0: Basic C++ client > - > > Key: IGNITE-17425 > URL: https://issues.apache.org/jira/browse/IGNITE-17425 > Project: Ignite > Issue Type: New Feature > Components: thin client >Reporter: Igor Sapego >Assignee: Igor Sapego >Priority: Major > > Implement basic version of C++ client that will be able to get a table and > perform put and get operation on it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17426) C++ 3.0: Implement table API
Igor Sapego created IGNITE-17426: Summary: C++ 3.0: Implement table API Key: IGNITE-17426 URL: https://issues.apache.org/jira/browse/IGNITE-17426 Project: Ignite Issue Type: New Feature Components: thin client Reporter: Igor Sapego Assignee: Igor Sapego Implement basic table API -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17425) C++3.0: Basic C++ client
[ https://issues.apache.org/jira/browse/IGNITE-17425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Sapego updated IGNITE-17425: - Summary: C++3.0: Basic C++ client (was: Thin 3.0: Basic C++ client) > C++3.0: Basic C++ client > > > Key: IGNITE-17425 > URL: https://issues.apache.org/jira/browse/IGNITE-17425 > Project: Ignite > Issue Type: New Feature > Components: thin client >Reporter: Igor Sapego >Assignee: Igor Sapego >Priority: Major > > Implement basic version of C++ client that will be able to get a table and > perform put and get operation on it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17425) Thin 3.0: Basic C++ client
Igor Sapego created IGNITE-17425: Summary: Thin 3.0: Basic C++ client Key: IGNITE-17425 URL: https://issues.apache.org/jira/browse/IGNITE-17425 Project: Ignite Issue Type: New Feature Components: thin client Reporter: Igor Sapego Assignee: Igor Sapego Implement basic version of C++ client that will be able to get a table and perform put and get operation on it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17424) Thin 3.0: C++ client
Igor Sapego created IGNITE-17424: Summary: Thin 3.0: C++ client Key: IGNITE-17424 URL: https://issues.apache.org/jira/browse/IGNITE-17424 Project: Ignite Issue Type: Epic Components: thin client Reporter: Igor Sapego Assignee: Igor Sapego We need to implement C++ client for Ignite 3.x. Let's adopt C++14 for a new client and consider adopting some library repository for it -- This message was sent by Atlassian Jira (v8.20.10#820010)
