[jira] [Updated] (IGNITE-17274) Startup of large numbers of servers slowed by linear lookup in IgniteServiceProcessor
[ https://issues.apache.org/jira/browse/IGNITE-17274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Tupitsyn updated IGNITE-17274: Fix Version/s: 2.14 > Startup of large numbers of servers slowed by linear lookup in > IgniteServiceProcessor > - > > Key: IGNITE-17274 > URL: https://issues.apache.org/jira/browse/IGNITE-17274 > Project: Ignite > Issue Type: Improvement > Components: managed services >Affects Versions: 2.13 >Reporter: Arthur Naseef >Assignee: Arthur Naseef >Priority: Major > Fix For: 2.14 > > Time Spent: 20m > Remaining Estimate: 0h > > Using a small POC, spinning up many servers is slow. In addition, the > startup time appears to be exponential. > Using timing measurements, found a linear lookup inside the > IgniteServiceProcessor that is taking most of the time. > Replacing that linear lookup with a Map lookup, and maintaining the map, > significantly speeds up the process, and startup time is now linear with the > number of services started. > Note this was tested with 20K and 50K services on a 1-node ignite cluster. > Timings against the stock 2.13.0 code come in at 30s for 20K and 250s for 50K > services. Modifying the linear lookup to use a Map, the timing come in at 8s > for 20K and 14s for 50K services. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17274) Startup of large numbers of servers slowed by linear lookup in IgniteServiceProcessor
[ https://issues.apache.org/jira/browse/IGNITE-17274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Tupitsyn updated IGNITE-17274: Ignite Flags: Release Notes Required (was: Docs Required,Release Notes Required) > Startup of large numbers of servers slowed by linear lookup in > IgniteServiceProcessor > - > > Key: IGNITE-17274 > URL: https://issues.apache.org/jira/browse/IGNITE-17274 > Project: Ignite > Issue Type: Improvement > Components: managed services >Affects Versions: 2.13 >Reporter: Arthur Naseef >Assignee: Arthur Naseef >Priority: Major > Fix For: 2.14 > > Time Spent: 20m > Remaining Estimate: 0h > > Using a small POC, spinning up many servers is slow. In addition, the > startup time appears to be exponential. > Using timing measurements, found a linear lookup inside the > IgniteServiceProcessor that is taking most of the time. > Replacing that linear lookup with a Map lookup, and maintaining the map, > significantly speeds up the process, and startup time is now linear with the > number of services started. > Note this was tested with 20K and 50K services on a 1-node ignite cluster. > Timings against the stock 2.13.0 code come in at 30s for 20K and 250s for 50K > services. Modifying the linear lookup to use a Map, the timing come in at 8s > for 20K and 14s for 50K services. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17274) Startup of large numbers of servers slowed by linear lookup in IgniteServiceProcessor
[ https://issues.apache.org/jira/browse/IGNITE-17274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Tupitsyn updated IGNITE-17274: Component/s: managed services > Startup of large numbers of servers slowed by linear lookup in > IgniteServiceProcessor > - > > Key: IGNITE-17274 > URL: https://issues.apache.org/jira/browse/IGNITE-17274 > Project: Ignite > Issue Type: Improvement > Components: managed services >Affects Versions: 2.13 >Reporter: Arthur Naseef >Assignee: Arthur Naseef >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Using a small POC, spinning up many servers is slow. In addition, the > startup time appears to be exponential. > Using timing measurements, found a linear lookup inside the > IgniteServiceProcessor that is taking most of the time. > Replacing that linear lookup with a Map lookup, and maintaining the map, > significantly speeds up the process, and startup time is now linear with the > number of services started. > Note this was tested with 20K and 50K services on a 1-node ignite cluster. > Timings against the stock 2.13.0 code come in at 30s for 20K and 250s for 50K > services. Modifying the linear lookup to use a Map, the timing come in at 8s > for 20K and 14s for 50K services. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (IGNITE-17274) Startup of large numbers of servers slowed by linear lookup in IgniteServiceProcessor
[ https://issues.apache.org/jira/browse/IGNITE-17274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Tupitsyn reassigned IGNITE-17274: --- Assignee: Arthur Naseef > Startup of large numbers of servers slowed by linear lookup in > IgniteServiceProcessor > - > > Key: IGNITE-17274 > URL: https://issues.apache.org/jira/browse/IGNITE-17274 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.13 >Reporter: Arthur Naseef >Assignee: Arthur Naseef >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Using a small POC, spinning up many servers is slow. In addition, the > startup time appears to be exponential. > Using timing measurements, found a linear lookup inside the > IgniteServiceProcessor that is taking most of the time. > Replacing that linear lookup with a Map lookup, and maintaining the map, > significantly speeds up the process, and startup time is now linear with the > number of services started. > Note this was tested with 20K and 50K services on a 1-node ignite cluster. > Timings against the stock 2.13.0 code come in at 30s for 20K and 250s for 50K > services. Modifying the linear lookup to use a Map, the timing come in at 8s > for 20K and 14s for 50K services. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17274) Startup of large numbers of servers slowed by linear lookup in IgniteServiceProcessor
[ https://issues.apache.org/jira/browse/IGNITE-17274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17560616#comment-17560616 ] Arthur Naseef commented on IGNITE-17274: The patch adds a new Map to track services by name, in addition to the original map that tracks the services by service-id. Changes to the original map (get/put/...) updated to also update the new map. Lookup by name now uses the map instead of a linear search of the original map's value set. > Startup of large numbers of servers slowed by linear lookup in > IgniteServiceProcessor > - > > Key: IGNITE-17274 > URL: https://issues.apache.org/jira/browse/IGNITE-17274 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.13 >Reporter: Arthur Naseef >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Using a small POC, spinning up many servers is slow. In addition, the > startup time appears to be exponential. > Using timing measurements, found a linear lookup inside the > IgniteServiceProcessor that is taking most of the time. > Replacing that linear lookup with a Map lookup, and maintaining the map, > significantly speeds up the process, and startup time is now linear with the > number of services started. > Note this was tested with 20K and 50K services on a 1-node ignite cluster. > Timings against the stock 2.13.0 code come in at 30s for 20K and 250s for 50K > services. Modifying the linear lookup to use a Map, the timing come in at 8s > for 20K and 14s for 50K services. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17274) Startup of large numbers of servers slowed by linear lookup in IgniteServiceProcessor
[ https://issues.apache.org/jira/browse/IGNITE-17274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17560611#comment-17560611 ] Arthur Naseef commented on IGNITE-17274: Pull request: [https://github.com/apache/ignite/pull/10123] > Startup of large numbers of servers slowed by linear lookup in > IgniteServiceProcessor > - > > Key: IGNITE-17274 > URL: https://issues.apache.org/jira/browse/IGNITE-17274 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.13 >Reporter: Arthur Naseef >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Using a small POC, spinning up many servers is slow. In addition, the > startup time appears to be exponential. > Using timing measurements, found a linear lookup inside the > IgniteServiceProcessor that is taking most of the time. > Replacing that linear lookup with a Map lookup, and maintaining the map, > significantly speeds up the process, and startup time is now linear with the > number of services started. > Note this was tested with 20K and 50K services on a 1-node ignite cluster. > Timings against the stock 2.13.0 code come in at 30s for 20K and 250s for 50K > services. Modifying the linear lookup to use a Map, the timing come in at 8s > for 20K and 14s for 50K services. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (IGNITE-17274) Startup of large numbers of servers slowed by linear lookup in IgniteServiceProcessor
[ https://issues.apache.org/jira/browse/IGNITE-17274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17560608#comment-17560608 ] Arthur Naseef commented on IGNITE-17274: POC code used to test: [https://github.com/opennms-forge/distributed-scheduling-poc/] Performance Notes: [https://github.com/opennms-forge/distributed-scheduling-poc/wiki/Ignite-Startup-Performance] Pull request is imminent. > Startup of large numbers of servers slowed by linear lookup in > IgniteServiceProcessor > - > > Key: IGNITE-17274 > URL: https://issues.apache.org/jira/browse/IGNITE-17274 > Project: Ignite > Issue Type: Improvement >Affects Versions: 2.13 >Reporter: Arthur Naseef >Priority: Major > > Using a small POC, spinning up many servers is slow. In addition, the > startup time appears to be exponential. > Using timing measurements, found a linear lookup inside the > IgniteServiceProcessor that is taking most of the time. > Replacing that linear lookup with a Map lookup, and maintaining the map, > significantly speeds up the process, and startup time is now linear with the > number of services started. > Note this was tested with 20K and 50K services on a 1-node ignite cluster. > Timings against the stock 2.13.0 code come in at 30s for 20K and 250s for 50K > services. Modifying the linear lookup to use a Map, the timing come in at 8s > for 20K and 14s for 50K services. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IGNITE-17274) Startup of large numbers of servers slowed by linear lookup in IgniteServiceProcessor
Arthur Naseef created IGNITE-17274: -- Summary: Startup of large numbers of servers slowed by linear lookup in IgniteServiceProcessor Key: IGNITE-17274 URL: https://issues.apache.org/jira/browse/IGNITE-17274 Project: Ignite Issue Type: Improvement Affects Versions: 2.13 Reporter: Arthur Naseef Using a small POC, spinning up many servers is slow. In addition, the startup time appears to be exponential. Using timing measurements, found a linear lookup inside the IgniteServiceProcessor that is taking most of the time. Replacing that linear lookup with a Map lookup, and maintaining the map, significantly speeds up the process, and startup time is now linear with the number of services started. Note this was tested with 20K and 50K services on a 1-node ignite cluster. Timings against the stock 2.13.0 code come in at 30s for 20K and 250s for 50K services. Modifying the linear lookup to use a Map, the timing come in at 8s for 20K and 14s for 50K services. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17212) Sql. Add support for DEFAULT operator
[ https://issues.apache.org/jira/browse/IGNITE-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Evgeny Stanilovsky updated IGNITE-17212: Description: We need to support DEFAULT operator. This is technically a port of IGNITE-16018 One of start points : SchemaDescriptorConverter#convertDefault was:We need to support DEFAULT operator. This is technically a port of IGNITE-16018 > Sql. Add support for DEFAULT operator > - > > Key: IGNITE-17212 > URL: https://issues.apache.org/jira/browse/IGNITE-17212 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Konstantin Orlov >Assignee: Konstantin Orlov >Priority: Major > Labels: ignite-3 > > We need to support DEFAULT operator. This is technically a port of > IGNITE-16018 > One of start points : SchemaDescriptorConverter#convertDefault -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (IGNITE-17273) Stream returned by Files.list must be closed
[
https://issues.apache.org/jira/browse/IGNITE-17273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Polovtcev updated IGNITE-17273:
-
Reviewer: Kirill Tkalenko
> Stream returned by Files.list must be closed
>
>
> Key: IGNITE-17273
> URL: https://issues.apache.org/jira/browse/IGNITE-17273
> Project: Ignite
> Issue Type: Bug
>Reporter: Aleksandr Polovtcev
>Assignee: Aleksandr Polovtcev
>Priority: Minor
> Labels: ignite-3
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Current Ignite code (especially {{CheckpointMarkersStorage}}) sometimes
> doesn't close the stream returned by the Files.list method.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (IGNITE-17251) IgniteWalConverter is unable to show WAL of encrypted caches
[
https://issues.apache.org/jira/browse/IGNITE-17251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17560581#comment-17560581
]
Ignite TC Bot commented on IGNITE-17251:
{panel:title=Branch: [pull/10118/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/10118/head] Base: [master] : New Tests
(1)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}Dev Utils{color} [[tests
1|https://ci.ignite.apache.org/viewLog.html?buildId=6654245]]
* {color:#013220}DevUtilsTestSuite:
IgniteEncryptedWalConverterTest.testIgniteWalConverter - PASSED{color}
{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=6652629buildTypeId=IgniteTests24Java8_RunAll]
> IgniteWalConverter is unable to show WAL of encrypted caches
>
>
> Key: IGNITE-17251
> URL: https://issues.apache.org/jira/browse/IGNITE-17251
> Project: Ignite
> Issue Type: Bug
>Reporter: Aleksandr Polovtcev
>Assignee: Aleksandr Polovtcev
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{RecordDataV1Serializer}} contains a bug when parsing WALs of encrypted
> caches. This leads to {{StandaloneWalRecordsIterator}} incorrectly
> interpreting encrypted WAL records thus failing to parse the whole WAL if any
> of these records are encountered.
> The easiest way to reproduce the issue is to populate a cache and print its
> WAL with and without encryption enabled. WAL output with encryption enabled
> will be significantly shorter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (IGNITE-17198) Minimal implementation of pure in-memory storage (with in-memory RAFT)
Title: Message Title Roman Puchkovskiy updated an issue Ignite / IGNITE-17198 Minimal implementation of pure in-memory storage (with in-memory RAFT) Change By: Roman Puchkovskiy Summary: Prototype Minimal implementation of pure in-memory storage (with in-memory RAFT) Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (IGNITE-17273) Stream returned by Files.list must be closed
Title: Message Title Aleksandr Polovtcev created an issue Ignite / IGNITE-17273 Stream returned by Files.list must be closed Issue Type: Bug Assignee: Aleksandr Polovtcev Created: 29/Jun/22 15:46 Labels: ignite-3 Priority: Minor Reporter: Aleksandr Polovtcev Current Ignite code (especially CheckpointMarkersStorage) sometimes doesn't close the stream returned by the Files.list method. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17203) SQL API: Implement scale and precision metadata
Title: Message Title Andrey Mashenkov updated an issue Ignite / IGNITE-17203 SQL API: Implement scale and precision metadata Change By: Andrey Mashenkov Fix Version/s: 3.0.0-alpha6 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17272) Logical recovery works incorrectly for encrypted caches
Title: Message Title Aleksandr Polovtcev updated an issue Ignite / IGNITE-17272 Logical recovery works incorrectly for encrypted caches Change By: Aleksandr Polovtcev Ignite Flags: Docs Required,Release Notes Required Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (IGNITE-17272) Logical recovery works incorrectly for encrypted caches
Title: Message Title Aleksandr Polovtcev created an issue Ignite / IGNITE-17272 Logical recovery works incorrectly for encrypted caches Issue Type: Bug Assignee: Aleksandr Polovtcev Created: 29/Jun/22 14:05 Priority: Major Reporter: Aleksandr Polovtcev When encryption is enabled for a particular cache, its WAL records get encrypted and wrapped in an EncryptedRecord. This encrypted record type is considered a PHYSICAL record, which leads to such records being omitted during logical recovery regardless of the fact that it can contain logical records. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17271) Sort out sqlogic tests on Ignite 3
Title: Message Title
Taras Ledkov updated an issue
Ignite / IGNITE-17271
Sort out sqlogic tests on Ignite 3
Change By:
Taras Ledkov
All the tests from the lists below are muted by rename {{}} -> {{_ignored}}.If the muted test already contains muted part (file {{_ignored}} already exists) the file is renamed to {{_ignored_old}}. Tests to investigate:*Implement: {{STRING_AGG}}*{code}sql/aggregate/aggregates/test_aggregate_types.testsql/aggregate/aggregates/test_distinct_string_agg.testsql/aggregate/aggregates/test_perfect_ht.testsql/aggregate/aggregates/test_string_agg.testsql/aggregate/aggregates/test_string_agg_big.testsql/aggregate/aggregates/test_string_agg_many_groups.test{code}*Metadata issue*{code}Caused by: java.lang.AssertionError: Unexpected type of result: NULL at org.apache.ignite.internal.sql.engine.util.TypeUtils.columnType(TypeUtils.java:343) at org.apache.ignite.internal.sql.engine.prepare.PrepareServiceImpl.lambda$resultSetMetadata$5(PrepareServiceImpl.java:271) at org.apache.ignite.internal.sql.engine.prepare.LazyResultSetMetadata.columns(LazyResultSetMetadata.java:43) at org.apache.ignite.internal.sql.script.SqlScriptRunner.lambda$sql$2(SqlScriptRunner.java:173) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195){code}{code}sql/aggregate/aggregates/test_aggregate_types_scalar.testsql/aggregate/aggregates/test_avg.testsql/aggregate/aggregates/test_scalar_aggr.testsql/function/generic/test_nvl.testsql/function/numeric/test_truncate.testsql/function/string/test_caseconvert.testsql/function/string/test_initcap.testsql/function/string/test_left.testsql/function/string/test_repeat.testsql/function/string/test_replace.testsql/function/string/test_reverse.testsql/function/string/test_right.testsql/function/string/test_substring.testsql/function/string/test_trim.testsql/types/null/test_null.test{code}*Support INTERVAL TYPE*{code}sql/function/interval/test_extract.testsql/types/interval/test_interval_ops.test{code}*Unsorted / Not expected results*{code}sql/function/timestamp/test_extract.testsql/function/timestamp/test_extract_ms.testsql/function/timestamp/test_timestampadd.testsql/insert/test_insert_type.testsql/order/test_order_same_value.test_slowsql/types/blob/test_blob.testsql/types/blob/test_blob_function.testsql/types/blob/test_blob_operator.testsql/types/blob/test_blob_string.testsql/types/collections/array.testsql/types/collections/array.testsql/types/collections/array.testsql/types/collections/array.testsql/types/decimal/cast_from_decimal.testsql/types/decimal/cast_to_decimal.testsql/types/interval/interval_constants.testsql/types/interval/test_interval_addition.testsql/types/null/test_is_null.test{code}
[jira] [Updated] (IGNITE-17271) Sort out sqlogic tests on Ignite 3
Title: Message Title
Taras Ledkov updated an issue
Ignite / IGNITE-17271
Sort out sqlogic tests on Ignite 3
Change By:
Taras Ledkov
Tests to investigate:*Implement: {{STRING_AGG}}*{code}sql/aggregate/aggregates/test_aggregate_types.testsql/aggregate/aggregates/test_distinct_string_agg.testsql/aggregate/aggregates/test_perfect_ht.testsql/aggregate/aggregates/test_string_agg.testsql/aggregate/aggregates/test_string_agg_big.testsql/aggregate/aggregates/test_string_agg_many_groups.test{code}*Metadata issue*{code}Caused by: java.lang.AssertionError: Unexpected type of result: NULL at org.apache.ignite.internal.sql.engine.util.TypeUtils.columnType(TypeUtils.java:343) at org.apache.ignite.internal.sql.engine.prepare.PrepareServiceImpl.lambda$resultSetMetadata$5(PrepareServiceImpl.java:271) at org.apache.ignite.internal.sql.engine.prepare.LazyResultSetMetadata.columns(LazyResultSetMetadata.java:43) at org.apache.ignite.internal.sql.script.SqlScriptRunner.lambda$sql$2(SqlScriptRunner.java:173) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195){code}{code}sql/aggregate/aggregates/test_aggregate_types_scalar.testsql/aggregate/aggregates/test_avg.testsql/aggregate/aggregates/test_scalar_aggr.testsql/function/generic/test_nvl.testsql/function/numeric/test_truncate.testsql/function/string/test_caseconvert.testsql/function/string/test_initcap.testsql/function/string/test_left.testsql/function/string/test_repeat.testsql/function/string/test_replace.testsql/function/string/test_reverse.testsql/function/string/test_right.testsql/function/string/test_substring.testsql/function/string/test_trim.test sql/types/null/test_null.test {code}*Support INTERVAL TYPE*{code}sql/function/interval/test_extract.test sql/types/interval/test_interval_ops.test {code}*Unsorted / Not expected results*{code}sql/function/timestamp/test_extract.testsql/function/timestamp/test_extract_ms.testsql/function/timestamp/test_timestampadd.testsql/insert/test_insert_type.testsql/order/test_order_same_value.test_slowsql/types/blob/test_blob.testsql/types/blob/test_blob_function.testsql/types/blob/test_blob_operator.testsql/types/blob/test_blob_string.testsql/types/collections/array.testsql/types/collections/array.testsql/types/collections/array.testsql/types/collections/array.testsql/types/decimal/cast_from_decimal.testsql/types/decimal/cast_to_decimal.test sql/types/interval/interval_constants.test sql/types/interval/test_interval_addition.testsql/types/null/test_is_null.test {code}
[jira] [Updated] (IGNITE-17271) Sort out sqlogic tests on Ignite 3
Title: Message Title
Taras Ledkov updated an issue
Ignite / IGNITE-17271
Sort out sqlogic tests on Ignite 3
Change By:
Taras Ledkov
Tests to investigate:*Implement: {{STRING_AGG}}*{code}sql/aggregate/aggregates/test_aggregate_types.testsql/aggregate/aggregates/test_distinct_string_agg.testsql/aggregate/aggregates/test_perfect_ht.testsql/aggregate/aggregates/test_string_agg.testsql/aggregate/aggregates/test_string_agg_big.testsql/aggregate/aggregates/test_string_agg_many_groups.test{code}*Metadata issue*{code}Caused by: java.lang.AssertionError: Unexpected type of result: NULL at org.apache.ignite.internal.sql.engine.util.TypeUtils.columnType(TypeUtils.java:343) at org.apache.ignite.internal.sql.engine.prepare.PrepareServiceImpl.lambda$resultSetMetadata$5(PrepareServiceImpl.java:271) at org.apache.ignite.internal.sql.engine.prepare.LazyResultSetMetadata.columns(LazyResultSetMetadata.java:43) at org.apache.ignite.internal.sql.script.SqlScriptRunner.lambda$sql$2(SqlScriptRunner.java:173) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195){code}{code}sql/aggregate/aggregates/test_aggregate_types_scalar.testsql/aggregate/aggregates/test_avg.testsql/aggregate/aggregates/test_scalar_aggr.testsql/function/generic/test_nvl.testsql/function/numeric/test_truncate.testsql/function/string/test_caseconvert.testsql/function/string/test_initcap.testsql/function/string/test_left.testsql/function/string/test_repeat.testsql/function/string/test_replace.testsql/function/string/test_reverse.testsql/function/string/test_right.testsql/function/string/test_substring.testsql/function/string/test_trim.test{code} *Support INTERVAL TYPE*{code}sql/function/interval/test_extract.test{code}*Unsorted / Not expected results*{code}sql/function/timestamp/test_extract.testsql/function/timestamp/test_extract_ms.testsql/function/timestamp/test_timestampadd.testsql/insert/test_insert_type.testsql/order/test_order_same_value.test_slowsql/types/blob/test_blob.testsql/types/blob/test_blob_function.testsql/types/blob/test_blob_operator.testsql/types/blob/test_blob_string.testsql/types/collections/array.testsql/types/collections/array.testsql/types/collections/array.testsql/types/collections/array.testsql/types/decimal/cast_from_decimal.testsql/types/decimal/cast_to_decimal.test{code}
[jira] [Updated] (IGNITE-17251) IgniteWalConverter is unable to show WAL of encrypted caches
Title: Message Title Roman Puchkovskiy updated an issue Ignite / IGNITE-17251 IgniteWalConverter is unable to show WAL of encrypted caches Change By: Roman Puchkovskiy Reviewer: Roman Puchkovskiy Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17271) Sort out sqlogic tests on Ignite 3
Title: Message Title
Taras Ledkov updated an issue
Ignite / IGNITE-17271
Sort out sqlogic tests on Ignite 3
Change By:
Taras Ledkov
Tests to investigate:*Implement: {{STRING_AGG}}*{code}sql/aggregate/aggregates/test_aggregate_types.testsql/aggregate/aggregates/test_distinct_string_agg.testsql/aggregate/aggregates/test_perfect_ht.testsql/aggregate/aggregates/test_string_agg.testsql/aggregate/aggregates/test_string_agg_big.testsql/aggregate/aggregates/test_string_agg_many_groups.test{code} *Metadata issue*{code} Caused by: java.lang.AssertionError: Unexpected type of result: NULL at org.apache.ignite.internal. sql .engine.util.TypeUtils.columnType(TypeUtils.java:343) at org.apache.ignite.internal.sql.engine.prepare.PrepareServiceImpl.lambda$resultSetMetadata$5(PrepareServiceImpl.java:271) at org.apache.ignite.internal.sql.engine.prepare.LazyResultSetMetadata.columns(LazyResultSetMetadata.java:43) at org.apache.ignite.internal.sql.script.SqlScriptRunner.lambda$sql$2(SqlScriptRunner.java:173) at java.base / java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195){code}{code}sql/ aggregate/aggregates/test_aggregate_types_scalar.testsql/aggregate/aggregates/test_avg.testsql/aggregate/aggregates/test_scalar_aggr.testsql/function/generic/test_nvl.testsql/function/numeric/test_truncate.testsql/function/string/test_caseconvert.testsql/function/string/test_initcap.testsql/function/string/test_left.testsql/function/string/test_repeat.testsql/function/string/test_replace.testsql/function/string/test_reverse.testsql/function/string/test_right.testsql/function/string/test_substring.testsql/function/string/test_trim.test{code}
Add Comment
[jira] [Updated] (IGNITE-17268) Port CREATE TABLE AS SELECT to Ignite 3
Title: Message Title
Taras Ledkov updated an issue
Ignite / IGNITE-17268
Port CREATE TABLE AS SELECT to Ignite 3
Change By:
Taras Ledkov
Implement CREATE TABLE AS SELECT for Ignite 3.Blocks tests:{code}sql/aggregate/aggregates/test_group_by_many_groups.test_slowsql/delete/test_large_delete.testsql/filter/test_filter_clause.testsrc/sqlLogicTest/sql/filter/test_transitive_filters.testsrc/sqlLogicTest/sql/filter/test_zonemap.testsql/join/full_outer/test_full_outer_join_many_matches.testsql/order/test_order_large.test sql/types/decimal/decimal_aggregates.test {code}
Add Comment
This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17271) Sort out sqlogic tests on Ignite 3
Title: Message Title
Taras Ledkov updated an issue
Ignite / IGNITE-17271
Sort out sqlogic tests on Ignite 3
Change By:
Taras Ledkov
Tests to investigate:*Implement: {{STRING_AGG}}*{code}sql/aggregate/aggregates/test_aggregate_types.testsql/aggregate/aggregates/test_distinct_string_agg.testsql/aggregate/aggregates/test_perfect_ht.testsql/aggregate/aggregates/test_string_agg.testsql/aggregate/aggregates/test_string_agg_big.testsql/aggregate/aggregates/test_string_agg_many_groups.test{code} *Metadata issue*{code}sql/aggregate/aggregates/test_aggregate_types_scalar.testsql/aggregate/aggregates/test_avg.testsql/aggregate/aggregates/test_scalar_aggr.testsql/function/generic/test_nvl.testsql/function/numeric/test_truncate.testsql/function/string/test_caseconvert.testsql/function/string/test_initcap.testsql/function/string/test_left.testsql/function/string/test_repeat.testsql/function/string/test_replace.testsql/function/string/test_reverse.testsql/function/string/test_right.testsql/function/string/test_substring.testsql/function/string/test_trim.test{code}
Add Comment
This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17271) Sort out sqlogic tests on Ignite 3
Title: Message Title
Taras Ledkov updated an issue
Ignite / IGNITE-17271
Sort out sqlogic tests on Ignite 3
Change By:
Taras Ledkov
Tests to investigate: * Implement: {{STRING_AGG}} * {code}sql/aggregate/aggregates/test_aggregate_types.testsql/aggregate/aggregates/test_distinct_string_agg.testsql/aggregate/aggregates/test_perfect_ht.testsql/aggregate/aggregates/test_string_agg.testsql/aggregate/aggregates/test_string_agg_big.testsql/aggregate/aggregates/test_string_agg_many_groups.test{code}
Add Comment
This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17271) Sort out sqlogic tests on Ignite 3
Title: Message Title
Taras Ledkov updated an issue
Ignite / IGNITE-17271
Sort out sqlogic tests on Ignite 3
Change By:
Taras Ledkov
Tests to investigate: Implement: {{STRING_AGG}}{code}sql/aggregate/aggregates/test_aggregate_types.testsql/aggregate/aggregates/test_distinct_string_agg.testsql/aggregate/aggregates/test_perfect_ht.testsql/aggregate/aggregates/test_string_agg.testsql/aggregate/aggregates/test_string_agg_big.testsql/aggregate/aggregates/test_string_agg_many_groups.test{code}
Add Comment
This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Assigned] (IGNITE-17203) SQL API: Implement scale and precision metadata
Title: Message Title Andrey Mashenkov assigned an issue to Andrey Mashenkov Ignite / IGNITE-17203 SQL API: Implement scale and precision metadata Change By: Andrey Mashenkov Assignee: Andrey Mashenkov Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17197) Change the default storage engine (for tables)
Title: Message Title Roman Puchkovskiy updated an issue Ignite / IGNITE-17197 Change the default storage engine (for tables) Change By: Roman Puchkovskiy Epic Link: IGNITE- 16851 16923 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17271) Sort out sqlogic tests on Ignite 3
Title: Message Title Taras Ledkov updated an issue Ignite / IGNITE-17271 Sort out sqlogic tests on Ignite 3 Change By: Taras Ledkov Labels: ignite-3 tech-debt Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (IGNITE-17271) Sort out sqlogic tests on Ignite 3
Title: Message Title Taras Ledkov created an issue Ignite / IGNITE-17271 Sort out sqlogic tests on Ignite 3 Issue Type: Task Assignee: Unassigned Components: sql Created: 29/Jun/22 13:11 Priority: Major Reporter: Taras Ledkov Tests to investigate: Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (IGNITE-17002) Indexes rebuild in Maintenance Mode
Title: Message Title Semyon Danilov commented on IGNITE-17002 Re: Indexes rebuild in Maintenance Mode The new command's syntax is as follows: --cache schedule_indexes_rebuild --node-id nodeId --cache-names cacheName[index1,...indexN],cacheName2,cacheName3[index1] --group-names groupName1,groupName2,...groupNameN Schedules rebuild of the indexes for specified caches via the Maintenance Mode. Schedules rebuild of specified caches and cache-groups Parameters: --node-id - (Optional) Specify node for indexes rebuild. If not specified, schedules rebuild on all nodes. --cache-names - Comma-separated list of cache names with optionally specified indexes. If indexes are not specified then all indexes of the cache will be scheduled for the rebuild operation. Can be used simultaneously with cache group names. --group-names - Comma-separated list of cache group names for which indexes should be scheduled for the rebuild. Can be used simultaneously with cache names. Thanks for the review, merged to the master branch Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (IGNITE-17002) Indexes rebuild in Maintenance Mode
Title: Message Title Ignite TC Bot commented on IGNITE-17002 Re: Indexes rebuild in Maintenance Mode Branch: [pull/10042/head] Base: [master] : No blockers found! Branch: [pull/10042/head] Base: [master] : New Tests (13) Control Utility [tests 12 IgniteControlUtilityTestSuite: CommandHandlerParsingTest.testScheduleIndexRebuildArgs - PASSED IgniteControlUtilityTestSuite: CommandHandlerParsingTest.testScheduleIndexRebuildWrongArgs - PASSED IgniteControlUtilityTestSuite: GridCommandHandlerScheduleIndexRebuildTest.testConsecutiveCommandInvocations - PASSED IgniteControlUtilityTestSuite: GridCommandHandlerScheduleIndexRebuildTest.testErrors - PASSED IgniteControlUtilityTestSuite: GridCommandHandlerScheduleIndexRebuildTest.testCorruptedIndexRebuildCacheWithGroup - PASSED IgniteControlUtilityTestSuite: GridCommandHandlerScheduleIndexRebuildTest.testSpecificIndexes - PASSED IgniteControlUtilityTestSuite: GridCommandHandlerScheduleIndexRebuildTest.testCorruptedIndexRebuildCacheOnAllNodes - PASSED IgniteControlUtilityTestSuite: GridCommandHandlerScheduleIndexRebuildTest.testCacheGroupParameter - PASSED IgniteControlUtilityTestSuite: GridCommandHandlerScheduleIndexRebuildTest.testCorruptedIndexRebuildCacheWithGroupOnAllNodes - PASSED IgniteControlUtilityTestSuite: GridCommandHandlerScheduleIndexRebuildTest.testRebuild - PASSED IgniteControlUtilityTestSuite: GridCommandHandlerScheduleIndexRebuildTest.testCacheGroupParameterWithCacheNames - PASSED ... and 1 new tests PDS (Indexing) [tests 1 IgnitePdsWithIndexingTestSuite: MaintenanceRebuildIndexUtilsSelfTest.testConstructFromMap - PASSED TeamCity --> Run :: All Results
[jira] [Updated] (IGNITE-16913) Provide affective way to write BinaryRow into byte buffer
Title: Message Title Ivan Bessonov updated an issue Ignite / IGNITE-16913 Provide affective way to write BinaryRow into byte buffer Change By: Ivan Bessonov Epic Link: IGNITE-16923 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17267) Preparing a checkpoint for a DeltaFilePageStore
Title: Message Title Kirill Tkalenko updated an issue Ignite / IGNITE-17267 Preparing a checkpoint for a DeltaFilePageStore Change By: Kirill Tkalenko To implement the *DeltaFilePageStore*, we need to prepare a checkpoint for their creation.What should be done:# Sort dirty pages (*FullPageId*) by *groupId*, *partitionId*, *pageIdx*;# In several threads, write dirty pages in batches for each partition and know the number of these pages.What can we look at:* *org.apache.ignite.internal.pagememory.persistence.checkpoint.CheckpointWorkflow#beginCheckpoint** *org.apache.ignite.internal.pagememory.persistence.checkpoint.CheckpointWorkflow#splitAndSortCheckpointPagesIfNeeded** *org.apache.ignite.internal.pagememory.persistence.checkpoint.Checkpointer#writePages*Some implementation notes:* We can replace *CheckpointWorkflow#parallelSortThreshold* with a constant at 40k, according to the article from 1 [paper|https://www . researchgate.net/publication/331742843_Threshold_Analysis_and_Comparison_of_Sequential_and_Parallel_Divide_and_Conquer_Sorting_Algorithms]. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17267) Preparing a checkpoint for a DeltaFilePageStore
Title: Message Title Kirill Tkalenko updated an issue Ignite / IGNITE-17267 Preparing a checkpoint for a DeltaFilePageStore Change By: Kirill Tkalenko To implement the *DeltaFilePageStore*, we need to prepare a checkpoint for their creation.What should be done:# Sort dirty pages (*FullPageId*) by *groupId*, *partitionId*, *pageIdx*;# In several threads, write dirty pages in batches for each partition and know the number of these pages.What can we look at:* *org.apache.ignite.internal.pagememory.persistence.checkpoint.CheckpointWorkflow#beginCheckpoint** *org.apache.ignite.internal.pagememory.persistence.checkpoint.CheckpointWorkflow#splitAndSortCheckpointPagesIfNeeded** *org.apache.ignite.internal.pagememory.persistence.checkpoint.Checkpointer#writePages* Some implementation notes:* We can replace *CheckpointWorkflow#parallelSortThreshold* with a constant at 40k, according to the article from 1. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17270) Fix issues with Kubernetes docs
Title: Message Title Igor Gusev updated an issue Ignite / IGNITE-17270 Fix issues with Kubernetes docs Change By: Igor Gusev Environment: 1) Check this section of the AWS EKS page: [https://ignite.apache.org/docs/latest/installation/kubernetes/amazon-eks-deployment#partition-awareness]The Partition Awareness is a top-level section and the "Connecting to REST API" is placed under it. Which is wrong. The Partition Awareness must be located as a sub-section of the "Connecting with Thin Clients" section that is above. If it's impossible to have another level under this tree then rename:"Partition Awareness" -> "Connecting with Thin Clients and Partition Awareness" 2) ThinClientKubernetesAddressFinder is mentioned here:[https://ignite.apache.org/docs/latest/thin-clients/java-thin-client#partition-awareness]Add a link to tutorials so users can see how to apply the doc. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Assigned] (IGNITE-17270) Fix issues with Kubernetes docs
Title: Message Title Igor Gusev assigned an issue to Igor Gusev Ignite / IGNITE-17270 Fix issues with Kubernetes docs Change By: Igor Gusev Assignee: Igor Gusev Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17270) Fix issues with Kubernetes docs
Title: Message Title Igor Gusev updated an issue Ignite / IGNITE-17270 Fix issues with Kubernetes docs Change By: Igor Gusev 1) Check this section of the AWS EKS page: [https://ignite.apache.org/docs/latest/installation/kubernetes/amazon-eks-deployment#partition-awareness]The Partition Awareness is a top-level section and the "Connecting to REST API" is placed under it. Which is wrong. The Partition Awareness must be located as a sub-section of the "Connecting with Thin Clients" section that is above. If it's impossible to have another level under this tree then rename:"Partition Awareness" -> "Connecting with Thin Clients and Partition Awareness" 2) ThinClientKubernetesAddressFinder is mentioned here:[https://ignite.apache.org/docs/latest/thin-clients/java-thin-client#partition-awareness]Add a link to tutorials so users can see how to apply the doc. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (IGNITE-17270) Fix issues with Kubernetes docs
Title: Message Title Igor Gusev created an issue Ignite / IGNITE-17270 Fix issues with Kubernetes docs Issue Type: Task Assignee: Unassigned Components: documentation Created: 29/Jun/22 11:53 Environment: 1) Check this section of the AWS EKS page: https://ignite.apache.org/docs/latest/installation/kubernetes/amazon-eks-deployment#partition-awareness The Partition Awareness is a top-level section and the "Connecting to REST API" is placed under it. Which is wrong. The Partition Awareness must be located as a sub-section of the "Connecting with Thin Clients" section that is above. If it's impossible to have another level under this tree then rename: "Partition Awareness" -> "Connecting with Thin Clients and Partition Awareness" 2) ThinClientKubernetesAddressFinder is mentioned here: https://ignite.apache.org/docs/latest/thin-clients/java-thin-client#partition-awareness Add a link to tutorials so users can see how to apply the doc. Priority: Major Reporter: Igor Gusev Add Comment
[jira] [Created] (IGNITE-17269) Java service interceptor
Title: Message Title Pavel Pereslegin created an issue Ignite / IGNITE-17269 Java service interceptor Issue Type: Sub-task Assignee: Pavel Pereslegin Components: managed services Created: 29/Jun/22 11:14 Labels: iep-79 ise Priority: Major Reporter: Pavel Pereslegin Implement interceptor for java service. Add Comment
[jira] [Updated] (IGNITE-16655) Volatile RAFT log for pure in-memory storages
Title: Message Title Ivan Bessonov updated an issue Ignite / IGNITE-16655 Volatile RAFT log for pure in-memory storages Change By: Ivan Bessonov h3. Original issue descriptionFor in-memory storage Raft logging can be optimized as we don't need to have it active when topology is stable.Each write can directly go to in-memory storage at much lower cost than synchronizing it with disk so it is possible to avoid writing Raft log.As nodes don't have any state and always join cluster clean we always need to transfer full snapshot during rebalancing - no need to keep long Raft log for historical rebalancing purposes.So we need to implement API for Raft component enabling configuration of Raft logging process.h3. More detailed descriptionApparently, we can't completely ignore writing to log. There are several situations where it needs to be collected: * During a regular workload, each node needs to have a small portion of log in case if it becomes a leader. There might be a number of "slow" nodes outside of "quorum" that require older data to be re-sent to them. Log entry can be truncated only when all nodes reply with "ack" or fail, otherwise log entry should be preserved. * During a clean node join - it will need to apply part of the log that wasn't included in the full-rebalance snapshot. So, everything, starting with snapshots applied index, will have to be preserved.It feels like the second option is just a special case of the first one - we can't truncate log until we receive all acks. And we can't receive an ack from the joining node until it finishes its rebalancing procedure.So, it all comes to the aggressive log truncation to make it short.Preserved log can be quite big in reality, there must be a disk offloading operation available.The easiest way to achieve it is to write into a RocksDB instance with WAL disabled. It'll store everything in memory until the flush, and even then the amount of flushed data will be small on stable topology. Absence of WAL is not an issue, the entire rocks instance can be dropped on restart, since it's supposed to be volatile.To avoid even the smallest flush, we can use additional volatile structure, like ring buffer or concurrent map, to store part of the log, and transfer records into RocksDB only on structure overflow. This sounds more compilcated and makes memory management more difficult. But, we should take it into consideration anyways. * Potentially, we could use a volatile page memory region for this purpose, since it already has a good control over the amount of memory used. But, memory overflow should be carefully processed, usually it's treated as an error and might even cause node failure.
[jira] [Updated] (IGNITE-16655) Volatile RAFT log for pure in-memory storages
Title: Message Title Ivan Bessonov updated an issue Ignite / IGNITE-16655 Volatile RAFT log for pure in-memory storages Change By: Ivan Bessonov h3. Original issue descriptionFor in-memory storage Raft logging can be optimized as we don't need to have it active when topology is stable.Each write can directly go to in-memory storage at much lower cost than synchronizing it with disk so it is possible to avoid writing Raft log.As nodes don't have any state and always join cluster clean we always need to transfer full snapshot during rebalancing - no need to keep long Raft log for historical rebalancing purposes.So we need to implement API for Raft component enabling configuration of Raft logging process.h3. More detailed description Apparently, we can't completely ignore writing to log. There are several situations where it needs to be collected: * During a regular workload, each node needs to have a small portion of log in case if it becomes a leader. There might be a number of "slow" nodes outside of "quorum" that require older data to be re-sent to them. Log entry can be truncated only when all nodes reply with "ack" or fail, otherwise log entry should be preserved. * During a clean node join - it will need to apply part of the log that wasn't included in the full-rebalance snapshot. So, everything, starting with snapshots applied index, will have to be preserved.It feels like the second option is just a special case of the first one - we can't truncate log until we receive all acks. And we can't receive an ack from the joining node until it finishes its rebalancing procedure.So, it all comes to the aggressive log truncation to make it short.Preserved log can be quite big in reality, there must be a disk offloading operation available.The easiest way to achieve it is to write into a RocksDB instance with WAL disabled. It'll store everything in memory until the flush, and even then the amount of flushed data will be small on stable topology. Absence of WAL is not an issue, the entire rocks instance can be dropped on restart, since it's supposed to be volatile.To avoid even the smallest flush, we can use additional volatile structure, like ring buffer or concurrent map, to store part of the log, and transfer records into RocksDB only on structure overflow. This sounds more compilcated and makes memory management more difficult. But, we should take it into consideration anyways.
[jira] [Updated] (IGNITE-17268) Port CREATE TABLE AS SELECT to Ignite 3
Title: Message Title Taras Ledkov updated an issue Ignite / IGNITE-17268 Port CREATE TABLE AS SELECT to Ignite 3 Change By: Taras Ledkov Labels: ignite-3 tech-debt Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17268) Port CREATE TABLE AS SELECT to Ignite 3
Title: Message Title
Taras Ledkov updated an issue
Ignite / IGNITE-17268
Port CREATE TABLE AS SELECT to Ignite 3
Change By:
Taras Ledkov
Implement CREATE TABLE AS SELECT for Ignite 3.Blocks tests:{code}sql/aggregate/aggregates/test_group_by_many_groups.test_slowsql/delete/test_large_delete.testsql/filter/test_filter_clause.testsrc/sqlLogicTest/sql/filter/test_transitive_filters.testsrc/sqlLogicTest/sql/filter/test_zonemap.testsql/join/full_outer/test_full_outer_join_many_matches.test sql/order/test_order_large.test {code}
Add Comment
This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17268) Port CREATE TABLE AS SELECT to Ignite 3
Title: Message Title
Taras Ledkov updated an issue
Ignite / IGNITE-17268
Port CREATE TABLE AS SELECT to Ignite 3
Change By:
Taras Ledkov
Implement CREATE TABLE AS SELECT for Ignite 3.Blocks tests:{code}sql/aggregate/aggregates/test_group_by_many_groups.test_slowsql/delete/test_large_delete.testsql/filter/test_filter_clause.test src/sqlLogicTest/sql/filter/test_transitive_filters.testsrc/sqlLogicTest/sql/filter/test_zonemap.testsql/join/full_outer/test_full_outer_join_many_matches.test {code}
Add Comment
This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17268) Port CREATE TABLE AS SELECT to Ignite 3
Title: Message Title
Taras Ledkov updated an issue
Ignite / IGNITE-17268
Port CREATE TABLE AS SELECT to Ignite 3
Change By:
Taras Ledkov
Implement CREATE TABLE AS SELECT for Ignite 3.Blocks tests:{code}sql/aggregate/aggregates/test_group_by_many_groups.test_slow sql/delete/test_large_delete.test {code}
Add Comment
This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17268) Port CREATE TABLE AS SELECT to Ignite 3
Title: Message Title
Taras Ledkov updated an issue
Ignite / IGNITE-17268
Port CREATE TABLE AS SELECT to Ignite 3
Change By:
Taras Ledkov
Implement CREATE TABLE AS SELECT for Ignite 3.Blocks tests:{code}sql/aggregate/aggregates/test_group_by_many_groups.test_slowsql/delete/test_large_delete.test sql/filter/test_filter_clause.test {code}
Add Comment
This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (IGNITE-17268) Port CREATE TABLE AS SELECT to Ignite 3
Title: Message Title Taras Ledkov created an issue Ignite / IGNITE-17268 Port CREATE TABLE AS SELECT to Ignite 3 Issue Type: Bug Assignee: Unassigned Components: sql Created: 29/Jun/22 10:21 Priority: Major Reporter: Taras Ledkov Implement CREATE TABLE AS SELECT for Ignite 3. Blocks tests: sql/aggregate/aggregates/test_group_by_many_groups.test_slow Add Comment
[jira] [Assigned] (IGNITE-17186) Port script runner and tests to Ignite 3
Title: Message Title Taras Ledkov assigned an issue to Taras Ledkov Ignite / IGNITE-17186 Port script runner and tests to Ignite 3 Change By: Taras Ledkov Assignee: Taras Ledkov Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (IGNITE-17267) Preparing a checkpoint for a DeltaFilePageStore
Title: Message Title Kirill Tkalenko created an issue Ignite / IGNITE-17267 Preparing a checkpoint for a DeltaFilePageStore Issue Type: Improvement Assignee: Kirill Tkalenko Created: 29/Jun/22 09:38 Fix Versions: 3.0.0-alpha6 Labels: ignite-3 Priority: Major Reporter: Kirill Tkalenko To implement the DeltaFilePageStore, we need to prepare a checkpoint for their creation. What should be done: Sort dirty pages (FullPageId) by groupId, partitionId, pageIdx; In several threads, write dirty pages in batches for each partition and know the number of these pages. What can we look at: org.apache.ignite.internal.pagememory.persistence.checkpoint.CheckpointWorkflow#beginCheckpoint org.apache.ignite.internal.pagememory.persistence.checkpoint.CheckpointWorkflow#splitAndSortCheckpointPagesIfNeeded org.apache.ignite.internal.pagememory.persistence.checkpoint.Checkpointer#writePages
[jira] [Assigned] (IGNITE-17240) Provide an ability to configure logging backend through IgniteClient.Builder
Title: Message Title Konstantin Orlov assigned an issue to Konstantin Orlov Ignite / IGNITE-17240 Provide an ability to configure logging backend through IgniteClient.Builder Change By: Konstantin Orlov Assignee: Konstantin Orlov Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Assigned] (IGNITE-17245) AbstractContinuousQuery.setIncludeExpired does not return this
Title: Message Title Stephen Darlington assigned an issue to Stephen Darlington Ignite / IGNITE-17245 AbstractContinuousQuery.setIncludeExpired does not return this Change By: Stephen Darlington Assignee: Stephen Darlington Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-16892) Update lock manager in order to soupport S, X and I locks
Title: Message Title Alexander Lapin updated an issue Ignite / IGNITE-16892 Update lock manager in order to soupport S, X and I locks Change By: Alexander Lapin Summary: Update lock manager in order to soupport S, X and I locks . Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-14986) Re-work error handling in meta storage component in accordance with error groups
Title: Message Title Vyacheslav Koptilin updated an issue Ignite / IGNITE-14986 Re-work error handling in meta storage component in accordance with error groups Change By: Vyacheslav Koptilin Summary: Re-work error handling in metastorage meta storage component in accordance with error scopes and prefixes groups Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-14931) Define common error scopes and prefix
Title: Message Title Vyacheslav Koptilin updated an issue Ignite / IGNITE-14931 Define common error scopes and prefix Change By: Vyacheslav Koptilin *UPD Error groups and error codesFirst proposed abstraction is a concept of error groups. It is similar to what was called an ErrorScope in [Devlist Discussion|https : * Related //lists.apache.org/thread/l20pp2ws06zt86b33jo851ph1t56hqd0].The main idea is that all errors in Ignite will be grouped. The way to TableManager identify an error type is to have a pair - (group; code), where code represents an integer number, unique within a group.For example, (TABLE, 1) and (SQL, 1) are both valid errors despite the apparent collision.Each group defines a collection of errors that belong to a single component/module/vendor. For example, RAFT, TABLE or SQL. It is convenient to have an integer code for groups as well, so that users won’t have to compare strings in their code. Hence each group must be identified by a unique name and code. See class ErrorGroup for the reference. Uniqueness must be guaranteed by a newGroup method, which is a single point to create new error groups.So, each error type could be represented in two different ways - as a human-readable string and as an integer number. First is used exclusively in text - logs and error messages. Second - exclusively in code:Name should be formatted like this: IGN-XXX-nnn, where XXX is a group name and nnn is an unique error code within a group. Additional IGN prefix will simplify googling the error.Number code should be calculated using following code fragment:(groupCode <<< 16) | (0x & errorCode).This restricts all group codes and individual error codes to 16 bits only , which is still more than enough . So, numeric error code includes both group code and an internal unique code. These codes should be stored in constants and be documented. Add Comment
[jira] [Created] (IGNITE-17266) No exit codes from cli
Title: Message Title Alexander Belyak created an issue Ignite / IGNITE-17266 No exit codes from cli Issue Type: Bug Affects Versions: 3.0.0-alpha5 Assignee: Unassigned Components: control.sh Created: 29/Jun/22 07:55 Priority: Major Reporter: Alexander Belyak I expect that ./ignite util returns some meaningful exit codes in case of error: $ ./ignite node stop n2 Can't find node with namen2 $ echo $? 0 or $ ./ignite node stopp n2 Unmatched arguments from index 1: 'stopp', 'n2' Did you mean: stop or list or start? $ echo $? 0 Just like the other command-line utils: $ mkdir mkdir: missing operand Try 'mkdir --help' for more information. $ echo $? 1
[jira] [Updated] (IGNITE-17266) No exit codes from cli
Title: Message Title Alexander Belyak updated an issue Ignite / IGNITE-17266 No exit codes from cli Change By: Alexander Belyak Labels: ignite-3-cli-tool Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (IGNITE-17234) "version" and "probe" REST commands should not require authentication
Title: Message Title Ignite TC Bot commented on IGNITE-17234 Re: "version" and "probe" REST commands should not require authentication Branch: [pull/10114/head] Base: [master] : No blockers found! Branch: [pull/10114/head] Base: [master] : No new tests found! TeamCity --> Run :: All Results Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Commented] (IGNITE-17111) Remove the ability to set the lazy flag in SqlFieldsQuery
Title: Message Title Luchnikov Alexander commented on IGNITE-17111 Re: Remove the ability to set the lazy flag in SqlFieldsQuery Evgeny Stanilovsky If in LAZY mode it is possible to set the page size equal to Long.MAX_VALUE, then are tests a mandatory requirement? Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (IGNITE-17265) Support RAFT snapshot streaming for PageMemory storage
Title: Message Title Roman Puchkovskiy created an issue Ignite / IGNITE-17265 Support RAFT snapshot streaming for PageMemory storage Issue Type: Improvement Assignee: Unassigned Created: 29/Jun/22 06:54 Fix Versions: 3.0.0-alpha6 Labels: ignite-3 Priority: Major Reporter: Roman Puchkovskiy IGNITE-17254 API needs to be implemented for RocksDB storage Add Comment
[jira] [Created] (IGNITE-17264) Support RAFT snapshot streaming for RocksDB storage
Title: Message Title Roman Puchkovskiy created an issue Ignite / IGNITE-17264 Support RAFT snapshot streaming for RocksDB storage Issue Type: Improvement Assignee: Unassigned Created: 29/Jun/22 06:53 Fix Versions: 3.0.0-alpha6 Labels: ignite-3 Priority: Major Reporter: Roman Puchkovskiy IGNITE-17254 API needs to be implemented for RocksDB storage Add Comment
[jira] [Updated] (IGNITE-17263) Implement leader to replica safe time propagation
Title: Message Title Alexander Lapin updated an issue Ignite / IGNITE-17263 Implement leader to replica safe time propagation Change By: Alexander Lapin Summary: Implement leader to replica safe time propagation . Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (IGNITE-17263) Implement leader to replica safe time propagation.
Title: Message Title Alexander Lapin created an issue Ignite / IGNITE-17263 Implement leader to replica safe time propagation. Issue Type: Improvement Assignee: Unassigned Created: 29/Jun/22 06:52 Labels: ignite-3 transaction3_ro Priority: Major Reporter: Alexander Lapin Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (IGNITE-17262) Implement RAFT snapshot streamer
Title: Message Title Roman Puchkovskiy created an issue Ignite / IGNITE-17262 Implement RAFT snapshot streamer Issue Type: Improvement Assignee: Unassigned Created: 29/Jun/22 06:49 Fix Versions: 3.0.0-alpha6 Labels: ignite-3 Priority: Major Reporter: Roman Puchkovskiy Infrastructure for this should be implemented in IGNITE-17253 . In this ticket, streaming using this infrastructure (along with changes to write commands/their handling) should be implemented. See IGNITE-17083 Add Comment
[jira] [Created] (IGNITE-17261) Implement write intent resolution logic for RO reads
Title: Message Title Alexander Lapin created an issue Ignite / IGNITE-17261 Implement write intent resolution logic for RO reads Issue Type: Improvement Assignee: Unassigned Created: 29/Jun/22 06:47 Labels: ignite-3 transaction3_ro Priority: Major Reporter: Alexander Lapin Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17253) Support for snapshot streaming in RAFT state machine
Title: Message Title Roman Puchkovskiy updated an issue Ignite / IGNITE-17253 Support for snapshot streaming in RAFT state machine Change By: Roman Puchkovskiy In JRaft, snapshotting works through files: first, a snapshot is written to a file, later it is sent to follower(s).This requires additional FS space, and it is a bit strange for in-memory cases.We need to switch to streaming way of doing snapshots: when the leader needs to bootstrap a follower, it opens a 'channel' to the follower, starts producing a snapshot and streaming it to the follower. The follower applies the snapshot in the same streaming way. This ticket is about modifying JRaft internal infrastructure to switch to snapshot streaming. See IGNITE-17083 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17260) Enrich IgniteTransactions and Transaction interfaces with RO related methods
Title: Message Title
Alexander Lapin updated an issue
Ignite / IGNITE-17260
Enrich IgniteTransactions and Transaction interfaces with RO related methods
Change By:
Alexander Lapin
It's required to explicitly register an intent of starting readOnly transaction, so that overloaded versions {code:java}Transaction begin( boolean readOnly long timestamp ){code} and{code:java}CompletableFuture beginAsync( boolean readOnly long timestamp ){code}should be added to IgniteTransactions together with corresponding implementation.Besides that Transaction interface should be extended with{code:java}boolean isReadOnly();{code} method. and {code:java}long timestamp();{code} methods.
Add Comment
This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17259) Populate ReplicaListener with handlers for RO requests
Title: Message Title Alexander Lapin updated an issue Ignite / IGNITE-17259 Populate ReplicaListener with handlers for RO requests Change By: Alexander Lapin For general context please check IGNITE-17252. ReplicaListener itself together with RW related handlers will be introduced in IGNITE-17258. Within given ticket handlers for processing RO requests should be introduced to map ro-timestamp aware requests to direct storage reads. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17259) Populate ReplicaListener with handlers for RO requests
Title: Message Title Alexander Lapin updated an issue Ignite / IGNITE-17259 Populate ReplicaListener with handlers for RO requests Change By: Alexander Lapin For general context please check IGNITE-17252. ReplicaListener itself together with RW related handlers will be introduced in IGNITE-17258. Within given ticket handlers for processing RO requests should be introduced Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17260) Enrich IgniteTransactions and Transaction interfaces with RO related methods
Title: Message Title
Alexander Lapin updated an issue
Ignite / IGNITE-17260
Enrich IgniteTransactions and Transaction interfaces with RO related methods
Change By:
Alexander Lapin
It's required to explicitly register an intent of starting readOnly transaction, so that overloaded versions {code:java}Transaction begin(boolean readOnly){code} and{code:java}CompletableFuture beginAsync(boolean readOnly){code}should be added to IgniteTransactions together with corresponding implementation.Besides that Transaction interface should be extended with{code:java}boolean isReadOnly();{code}method.
Add Comment
This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (IGNITE-17260) Enrich IgniteTransactions and Transaction interfaces with RO related methods
Title: Message Title Alexander Lapin created an issue Ignite / IGNITE-17260 Enrich IgniteTransactions and Transaction interfaces with RO related methods Issue Type: Improvement Assignee: Unassigned Created: 29/Jun/22 06:27 Labels: ignite-3 transaction3_ro Priority: Major Reporter: Alexander Lapin Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Created] (IGNITE-17259) Populate ReplicaListener with handlers for RO requests
Title: Message Title Alexander Lapin created an issue Ignite / IGNITE-17259 Populate ReplicaListener with handlers for RO requests Issue Type: Improvement Assignee: Unassigned Created: 29/Jun/22 06:23 Labels: ignite-3 transaction3_ro Priority: Major Reporter: Alexander Lapin Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17222) Need to propagate HLC with transaction protocol events
Title: Message Title Alexander Lapin updated an issue Ignite / IGNITE-17222 Need to propagate HLC with transaction protocol events Change By: Alexander Lapin Labels: ignite-3 transaction3_ro Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17214) Implement HLC
Title: Message Title Alexander Lapin updated an issue Ignite / IGNITE-17214 Implement HLC Change By: Alexander Lapin Labels: ignite-3 transaction3_ro Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-17221) Need to propagate HLC with RAFT events
Title: Message Title Alexander Lapin updated an issue Ignite / IGNITE-17221 Need to propagate HLC with RAFT events Change By: Alexander Lapin Labels: ignite-3 transaction3_ro Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-16894) Adjust already implemented RW tx protocol in order to match new one
Title: Message Title Alexander Lapin updated an issue Ignite / IGNITE-16894 Adjust already implemented RW tx protocol in order to match new one Change By: Alexander Lapin * Tx commit timestamp generation * Partition enlistment * Lock acquiring/release * Intent resolution * Commit/Abort * Cleanuph3. Upd:Resolved, because logic mentioned above will be completely implemented within other transactions3_rw tickets , mainly in https://issues . apache.org/jira/browse/IGNITE-17258 and Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-16894) Adjust already implemented RW tx protocol in order to match new one
Title: Message Title Alexander Lapin updated an issue Ignite / IGNITE-16894 Adjust already implemented RW tx protocol in order to match new one Change By: Alexander Lapin * Tx commit timestamp generation * Partition enlistment * Lock acquiring/release * Intent resolution * Commit/Abort * Cleanuph3. Upd:Resolved, because logic mentioned above will be completely implemented within other transactions3_rw tickets, mainly in https://issues.apache.org/jira/browse/IGNITE-17258 and https://issues.apache.org/jira/browse/IGNITE-16882 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-16894) Adjust already implemented RW tx protocol in order to match new one
Title: Message Title Alexander Lapin updated an issue Ignite / IGNITE-16894 Adjust already implemented RW tx protocol in order to match new one Change By: Alexander Lapin * Tx commit timestamp generation * Partition enlistment * Lock acquiring/release * Intent resolution * Commit/Abort * Cleanup Tx h3 . Phase1 Upd:Resolved, because logic mentioned above will be completely implemented within other transactions3_rw tickets. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (IGNITE-15931) Implement storage for tx states
Title: Message Title Alexander Lapin updated an issue Ignite / IGNITE-15931 Implement storage for tx states Change By: Alexander Lapin Currently tx states [1] are stored in memory and Occurred that ParitionStorage will be lost on node failure removed soon , but they shouldn so it ' t.Need s required to implement explicit tx state persistent storage. Generally speaking it'll store states in the Storage (similar map of txId to PartitionStorage transaction meta, that will include txnState(committed/aborted ) , list of enlisted partition ids and commit timestamp . [1] org.apache.ignite.internal.tx.impl.TxManagerImpl#statesTx.Phase1 Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
