[jira] [Assigned] (IGNITE-19955) Fix create zone on restart rewrites existing data nodes because of trigger key inconsistnecy
[
https://issues.apache.org/jira/browse/IGNITE-19955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mirza Aliev reassigned IGNITE-19955:
Assignee: Mirza Aliev
> Fix create zone on restart rewrites existing data nodes because of trigger
> key inconsistnecy
>
>
> Key: IGNITE-19955
> URL: https://issues.apache.org/jira/browse/IGNITE-19955
> Project: Ignite
> Issue Type: Bug
>Reporter: Mirza Aliev
>Assignee: Mirza Aliev
>Priority: Major
> Labels: ignite-3
>
> Currently we have the logic for initialisation of newly created zone that it
> writes keys
> {noformat}
> zoneDataNodesKey(zoneId), zoneScaleUpChangeTriggerKey(zoneId),
> zoneScaleDownChangeTriggerKey(zoneId), zonesChangeTriggerKey(zoneId)
> {noformat}
> to metastorage, and condition is
> {noformat}
> static CompoundCondition triggerKeyConditionForZonesChanges(long
> revision, int zoneId) {
> return or(
> notExists(zonesChangeTriggerKey(zoneId)),
>
> value(zonesChangeTriggerKey(zoneId)).lt(ByteUtils.longToBytes(revision))
> );
> {noformat}
> Recovery process implies that the create zone event will be processed again,
> but with the higher revision, so data nodes will be rewritten.
> We need to handle this situation, so data nodes will be consistent after
> restart.
> Possible solution is to change condition to
> {noformat}
> static SimpleCondition triggerKeyConditionForZonesCreation(long revision,
> int zoneId) {
> return notExists(zonesChangeTriggerKey(zoneId));
> }
> static SimpleCondition triggerKeyConditionForZonesDelete(int zoneId) {
> return exists(zonesChangeTriggerKey(zoneId));
> }
> {noformat}
>
> so we could not rely on revision and check only existence of the key, when we
> create or remove zone. The problem in this solution is that reordering of the
> create and remove on some node could lead to not consistent state for zones
> key in metastorage
> *UPD*:
> This problem will be resolves once we implement
> https://issues.apache.org/jira/browse/IGNITE-20561
> In this ticket we need to unmute all tickets that were muted by this ticket
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-21007) Enhance CompletableFutureExceptionMatcher API
[
https://issues.apache.org/jira/browse/IGNITE-21007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792697#comment-17792697
]
Kirill Tkalenko commented on IGNITE-21007:
--
Looks good.
> Enhance CompletableFutureExceptionMatcher API
> -
>
> Key: IGNITE-21007
> URL: https://issues.apache.org/jira/browse/IGNITE-21007
> Project: Ignite
> Issue Type: Improvement
>Reporter: Aleksandr Polovtcev
>Assignee: Aleksandr Polovtcev
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> It would be nice if {{CompletableFutureExceptionMatcher}} accepted a
> {{Matcher}} for inspecting the error messages of exceptions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (IGNITE-20673) Sql. Independent multi statement queries need to be executed in parallel.
[
https://issues.apache.org/jira/browse/IGNITE-20673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Orlov reassigned IGNITE-20673:
-
Assignee: Konstantin Orlov
> Sql. Independent multi statement queries need to be executed in parallel.
> -
>
> Key: IGNITE-20673
> URL: https://issues.apache.org/jira/browse/IGNITE-20673
> Project: Ignite
> Issue Type: Improvement
> Components: sql
>Affects Versions: 3.0.0-beta1
>Reporter: Evgeny Stanilovsky
>Assignee: Konstantin Orlov
>Priority: Major
> Labels: ignite-3
>
> Independent or (other words) unrelated queries need to be executed in
> parallel.
> Multi statement request can partially or fully consist of unrelated queries,
> i.e. queries which execution is not depends of previous query execution.
> Simple example:
>
> {noformat}
> --- start multistatement ---
> CREATE TABLE t1
> CREATE TABLE t2
> SELECT * FROM t3
> --- end multistatement ---
> {noformat}
> more complex case:
> {noformat}
> --- start multistatement ---
> CREATE TABLE t1 | paralleled
> CREATE TABLE t2 | paralleled
> SELECT * FROM t2| not paralleled, waiting for previous execution
> CREATE TABLE t3 | paralleled
> --- end multistatement ---
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-20605) Restore scaleUp/scaleDown timers
[ https://issues.apache.org/jira/browse/IGNITE-20605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792665#comment-17792665 ] Vladislav Pyatkov commented on IGNITE-20605: Merged 3e8d1aa383c1434f55c4d97c3a2539c264b5c859 > Restore scaleUp/scaleDown timers > > > Key: IGNITE-20605 > URL: https://issues.apache.org/jira/browse/IGNITE-20605 > Project: Ignite > Issue Type: Bug >Reporter: Mirza Aliev >Assignee: Mirza Aliev >Priority: Major > Labels: ignite-3 > Time Spent: 40m > Remaining Estimate: 0h > > h3. *Motivation* > We need to restore timers that were scheduled before node restart. > h3. *Definition of done* > Timers are rescheduled after restart > h3. *Implementation notes* > It is valid to just schedule local timers according to scaleUp/ScaleDown > timers values from the Catalog, and as a revision take maxScUpFromMap or > maxScDownFromMap from topologyAugmentationMap, where maxScUpFromMap and > maxScDownFromMap are max revision from topologyAugmentationMap of the entry, > which was associated with addition and removal of nodes respectively -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21003) Switch to the 'always use current schema' approach
[ https://issues.apache.org/jira/browse/IGNITE-21003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Puchkovskiy updated IGNITE-21003: --- Description: This issue is to think about the problem and design it first, not about implementing it right away. Currently, for an RW transaction, we take schema at the beginning of the transaction and run the whole transaction on that schema (indices are an exception, but this is not visible to the end user). The transaction still notices most schema changes (if a change happens before a read/write in the transaction, the transaction gets aborted), but it does not notice a table created after the transaction had started. IGNITE-20107 addressed this issue, but it was decided that we need to design this in more depth. An alternative is to always use the latest schema on each operation (still having schema validation). This might have some downsides/bring difficulties: # Same query might return data with different schemas # It's not clear against which schema to validate the current schema at execution of each operation (probably, still against the initial one, defined as Max(txStartTs, tableCreationTs)) # A query would have to use the same query timestamp on each node executing its fragments, so it would have to propagate this timestamp The schema synchronization design was created starting with an assumption that the schema is taken for the start of a transaction, so the design should be revised carefully when switching to the proposed one. h2. Proposed changes It seems that this can be achieved with the following: # Always execute a KV/SQL operation/query using the current schema (obtained using the schema sync procedure for now()) # An operation/query executed distributively (like SQL queries that produce a few fragments executed on different nodes) must pass that query timestamp to each of the nodes participating in its execution; they must use this timestamp to get the 'current' schema # In each read/write operation processed in a PartitionReplicaListener, instead of failing the operation if the current schema is different from the initial transaction schema, do the (already implemented) forward compatibility check (so a few white-listed change types, like adding a nullable column, will be allowed). This is optional, but the 'fail any read/write if the table schema is changed in any way' rule was introduced to disallow a user see any effects of a DDL in the middle of the transaction (except for its abortion); if we allow each query see new schema, this strict rule kinda makes no sense anymore. (On commit, we still do forward compatibility check) # To do the commit/read/write forward compatibility check, take the initial table schema not at transaciton start, but at the moment when the transaction had first enlisted the table. For this, we might need a mechanism to pass the 'tableId->enlistTs' map with each transactional operation/query (back and forth), analogously to how it's done to maintain maxObservableTimestamp. # Probably we should make column rename forward incompatible as the user will now have to switch to the new name immediately. was: This issue is to think about the problem and design it first, not about implementing it right away. Currently, for an RW transaction, we take schema at the beginning of the transaction and run the whole transaction on that schema (indices are an exception, but this is not visible to the end user). The transaction still notices most schema changes (if a change happens before a read/write in the transaction, the transaction gets aborted), but it does not notice a table created after the transaction had started. IGNITE-20107 addressed this issue, but it was decided that we need to design this in more depth. An alternative is to always use the latest schema on each operation (still having schema validation). This might have some downsides/bring difficulties: # Same query might return data with different schemas # It's not clear against which schema to validate the current schema at execution of each operation (probably, still against the initial one, defined as Max(txStartTs, tableCreationTs)) # A query would have to use the same query timestamp on each node executing its fragments, so it would have to propagate this timestamp The schema synchronization design was created starting with an assumption that the schema is taken for the start of a transaction, so the design should be revised carefully when switching to the proposed one. > Switch to the 'always use current schema' approach > -- > > Key: IGNITE-21003 > URL: https://issues.apache.org/jira/browse/IGNITE-21003 > Project: Ignite > Issue Type: Improvement >Reporter: Roman Puchkovskiy >Priority: Major >
[jira] [Updated] (IGNITE-21008) Update Ignite dependency: Netty
[ https://issues.apache.org/jira/browse/IGNITE-21008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksandr Nikolaev updated IGNITE-21008: Labels: ise (was: ) > Update Ignite dependency: Netty > --- > > Key: IGNITE-21008 > URL: https://issues.apache.org/jira/browse/IGNITE-21008 > Project: Ignite > Issue Type: Improvement >Reporter: Aleksandr Nikolaev >Assignee: Aleksandr Nikolaev >Priority: Major > Labels: ise > Fix For: 2.17 > > > Update Ignite dependency: Netty 4.1.89.Final to 4.1.101.Final -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-21008) Update Ignite dependency: Netty
Aleksandr Nikolaev created IGNITE-21008: --- Summary: Update Ignite dependency: Netty Key: IGNITE-21008 URL: https://issues.apache.org/jira/browse/IGNITE-21008 Project: Ignite Issue Type: Improvement Reporter: Aleksandr Nikolaev Assignee: Aleksandr Nikolaev Fix For: 2.17 Update Ignite dependency: Netty 4.1.89.Final to 4.1.101.Final -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21007) Enhance CompletableFutureExceptionMatcher API
[
https://issues.apache.org/jira/browse/IGNITE-21007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Polovtcev updated IGNITE-21007:
-
Description: It would be nice if {{CompletableFutureExceptionMatcher}}
accepted a {{Matcher}} for inspecting the error messages of exceptions.
> Enhance CompletableFutureExceptionMatcher API
> -
>
> Key: IGNITE-21007
> URL: https://issues.apache.org/jira/browse/IGNITE-21007
> Project: Ignite
> Issue Type: Improvement
>Reporter: Aleksandr Polovtcev
>Assignee: Aleksandr Polovtcev
>Priority: Major
> Labels: ignite-3
>
> It would be nice if {{CompletableFutureExceptionMatcher}} accepted a
> {{Matcher}} for inspecting the error messages of exceptions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-21007) Enhance CompletableFutureExceptionMatcher API
[
https://issues.apache.org/jira/browse/IGNITE-21007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Polovtcev updated IGNITE-21007:
-
Fix Version/s: 3.0.0-beta2
> Enhance CompletableFutureExceptionMatcher API
> -
>
> Key: IGNITE-21007
> URL: https://issues.apache.org/jira/browse/IGNITE-21007
> Project: Ignite
> Issue Type: Improvement
>Reporter: Aleksandr Polovtcev
>Assignee: Aleksandr Polovtcev
>Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
>
> It would be nice if {{CompletableFutureExceptionMatcher}} accepted a
> {{Matcher}} for inspecting the error messages of exceptions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IGNITE-21007) Enhance CompletableFutureExceptionMatcher API
Aleksandr Polovtcev created IGNITE-21007: Summary: Enhance CompletableFutureExceptionMatcher API Key: IGNITE-21007 URL: https://issues.apache.org/jira/browse/IGNITE-21007 Project: Ignite Issue Type: Improvement Reporter: Aleksandr Polovtcev Assignee: Aleksandr Polovtcev -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-21006) Introduction of auxiliary class CompletableFutures
[ https://issues.apache.org/jira/browse/IGNITE-21006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirill Tkalenko updated IGNITE-21006: - Reviewer: Aleksandr Polovtcev > Introduction of auxiliary class CompletableFutures > -- > > Key: IGNITE-21006 > URL: https://issues.apache.org/jira/browse/IGNITE-21006 > Project: Ignite > Issue Type: Improvement >Reporter: Kirill Tkalenko >Assignee: Kirill Tkalenko >Priority: Major > Labels: ignite-3 > Fix For: 3.0.0-beta2 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > To reduce memory consumption when using > *java.util.concurrent.CompletableFuture*, a helper class > *org.apache.ignite.internal.util.CompletableFutures* with helper constants > and methods will be added. > For example, in the project there are many places where > *completedFuture(null)* is used, which can be easily replaced with the > constant *CompletableFutures#NULL_COMPLETED_FUTURE*, since any subsequent > execution from it will be performed immediately and not asynchronously. -- This message was sent by Atlassian Jira (v8.20.10#820010)
