[jira] [Commented] (IGNITE-10516) Storage is corrupted after CREATE INDEX IF NOT EXISTS on different tables
[
https://issues.apache.org/jira/browse/IGNITE-10516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261377#comment-17261377
]
Stanislav Lukyanov commented on IGNITE-10516:
-
Checked the reproducer - still fails on today's master.
> Storage is corrupted after CREATE INDEX IF NOT EXISTS on different tables
> -
>
> Key: IGNITE-10516
> URL: https://issues.apache.org/jira/browse/IGNITE-10516
> Project: Ignite
> Issue Type: Bug
>Reporter: Stanislav Lukyanov
>Priority: Major
>
> Given two tables in the same schema, we can't create an index with the same
> name for both tables. In other words, the following code leads to an error -
> which is good.
> {code}
> CREATE INDEX IDX on T1 (COL);
> CREATE INDEX IDX on T2 (COL);
> {code}
> If used with `IF NOT EXISTS`, the queries pass. It might be OK or not - one
> needs to look into SQL spec to check if the second operation should be a
> no-op (because IDX exists) or fail (because IDX exists for a different table,
> so the caller is probably doing something wrong)
> {code}
> CREATE INDEX IDX on T1 (COL);
> CREATE INDEX IF NOT EXISTS IDX on T2 (COL);
> {code}
> However, if persistence is enabled, the node will fail to restart complaining
> about duplicate index names.
> {code}
> class org.apache.ignite.IgniteCheckedException: Duplicate index name
> [cache=SQL_PUBLIC_T2, schemaName=PUBLIC, idxName=IDX, existingTable=T,
> table=T2]
> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1183)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2040)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1732)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
> at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:656)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:959)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:900)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:888)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:854)
> at
> org.apache.ignite.IndexWithSameNameTest.test(IndexWithSameNameTest.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.access$001(GridAbstractTest.java:150)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$6.evaluate(GridAbstractTest.java:2104)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2119)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.IgniteException: Duplicate index name
> [cache=SQL_PUBLIC_T2, schemaName=PUBLIC, idxName=IDX, existingTable=T,
> table=T2]
> at
> org.apache.ignite.internal.processors.query.GridQueryProcessor.registerCache0(GridQueryProcessor.java:1650)
> at
> org.apache.ignite.internal.processors.query.GridQueryProcessor.onCacheStart0(GridQueryProcessor.java:803)
> at
> org.apache.ignite.internal.processors.query.GridQueryProcessor.onCacheStart(GridQueryProcessor.java:866)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCacheInRecoveryMode(GridCacheProcessor.java:2595)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor.access$1400(GridCacheProcessor.java:204)
> at
> org.apache.ignite.internal.processors.cache.GridCacheProcessor$CacheRecoveryLifecycle.afterBinaryMemoryRestore(GridCacheProcessor.java:5481)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreBinaryMemory(GridCacheDatabaseSharedManager.java:947)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.startMemoryRestore(GridCacheDatabaseSharedManager.java:1922)
> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1050)
> ... 18 more
> {code}
> It looks like the second index (on T2) is partially created after all.
> Need to either block index creation by `CREATE INDEX IF NOT EXISTS`
> completely, or just fail that query when the table names don't match (if SQL
> spec allows it).
--
This message was sent by Atlassian Jira
[jira] [Resolved] (IGNITE-9964) SQL: query by affinity key fails when a table is created from a custom template
[ https://issues.apache.org/jira/browse/IGNITE-9964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Lukyanov resolved IGNITE-9964. Resolution: Duplicate > SQL: query by affinity key fails when a table is created from a custom > template > --- > > Key: IGNITE-9964 > URL: https://issues.apache.org/jira/browse/IGNITE-9964 > Project: Ignite > Issue Type: Bug >Reporter: Stanislav Lukyanov >Priority: Major > > SELECT by affinity key doesn't work on a table that was created with a custom > cache template if entries were added through JCache (using withKeepBinary()). > If the first row was added via INSERT and after that another row was added > via withKeepBinary(), SELECT by affinity key works correctly. > If the first row was added via withKeepBinary() and after that another row > was added via INSERT, SELECT by affinity key returns nothing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-9964) SQL: query by affinity key fails when a table is created from a custom template
[ https://issues.apache.org/jira/browse/IGNITE-9964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261372#comment-17261372 ] Stanislav Lukyanov commented on IGNITE-9964: Cannot reproduce on master. Closing as duplicate of IGNITE-5795 > SQL: query by affinity key fails when a table is created from a custom > template > --- > > Key: IGNITE-9964 > URL: https://issues.apache.org/jira/browse/IGNITE-9964 > Project: Ignite > Issue Type: Bug >Reporter: Stanislav Lukyanov >Priority: Major > > SELECT by affinity key doesn't work on a table that was created with a custom > cache template if entries were added through JCache (using withKeepBinary()). > If the first row was added via INSERT and after that another row was added > via withKeepBinary(), SELECT by affinity key works correctly. > If the first row was added via withKeepBinary() and after that another row > was added via INSERT, SELECT by affinity key returns nothing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-8732) SQL: REPLICATED cache cannot be left-joined to PARTITIONED
[
https://issues.apache.org/jira/browse/IGNITE-8732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stanislav Lukyanov updated IGNITE-8732:
---
Description:
*Steps to reproduce*
# Run
{{org.apache.ignite.sqltests.ReplicatedSqlTest#testLeftJoinReplicatedPartitioned}}
# Observe that we have 2x results on 2-node cluster
*Root Cause*
{{left LEFT JOIN right ON cond}} operation assumes full scan of of a left
expression. Currently we perform this scan on every node and then simply merge
results on reducer. Two nodes, two scans of {{REPLICATED}} cache, 2x results.
*Potential Solutions*
We may consider several solutions. Deeper analysis is required to understand
which is the right one.
# Perform deduplication on reducer - this most prospective and general
technique, described in more details below
# Treat {{REPLICATED}} cache as {{PARTITIONED}}. Essentially, we just need to
pass proper backup filter. But what if {{REPLICATED}} cache spans more nodes
than {{PARTITIONED}}? We cannot rely on primary/backup in this case
# Implement additional execution phase as follows:
{code:java}
SELECT left.cols, right.cols FROM left INNER JOIN right ON cond;
// Get "inner join" part
UNION
UNICAST SELECT left.cols, [NULL].cols FROM left WHERE left.id NOT IN ([ids from
the first phase]) // Get "outer join" part
{code}
*Reducer Deduplication*
The idea is to get all data locally and then perform final deduplication. This
may incur high network overhead, because of lot of duplicated left parts would
be transferred. However, this could be optimized greatly with the following
techniques applied one after another
# Semi-jions: {{left}} is {{joined}} on mapper node, but instead of sending
{{(left, right)}} relation, we send {{(left) + (right)}}
# In case {{left}} part is known to be idempotent (i.e. it produces the same
result set on all nodes), only one node will send {{(left) + (right)}}, other
nodes will send {{(right)}} only
# Merge {{left}} results with if needed (i.e. if idempotence-related opto was
not applicable)
# Join {{left}} and {{right}} parts on reducer
*UPDATE*
After a few attempts at the implementation, the solution of treating REPLICATED
cache as PARTITIONED looks the most practical. The solution works in a limited
case:
* REPLICATED and PARTITIONED both have the same affinity function, number of
partitions, node filter
** Note that REPLICATED has a different number of partitions by default
* The JOIN is done on an affinity column of both caches
** Note that users often don’t create affinity keys for REPLICATED caches today
* distributedJoins=false (distributed joins aren’t supported for now)
was:
*Steps to reproduce*
# Run
{{org.apache.ignite.sqltests.ReplicatedSqlTest#testLeftJoinReplicatedPartitioned}}
# Observe that we have 2x results on 2-node cluster
*Root Cause*
{{left LEFT JOIN right ON cond}} operation assumes full scan of of a left
expression. Currently we perform this scan on every node and then simply merge
results on reducer. Two nodes, two scans of {{REPLICATED}} cache, 2x results.
*Potential Solutions*
We may consider several solutions. Deeper analysis is required to understand
which is the right one.
# Perform deduplication on reducer - this most prospective and general
technique, described in more details below
# Treat {{REPLICATED}} cache as {{PARTITIONED}}. Essentially, we just need to
pass proper backup filter. But what if {{REPLICATED}} cache spans more nodes
than {{PARTITIONED}}? We cannot rely on primary/backup in this case
# Implement additional execution phase as follows:
{code}
SELECT left.cols, right.cols FROM left INNER JOIN right ON cond;
// Get "inner join" part
UNION
UNICAST SELECT left.cols, [NULL].cols FROM left WHERE left.id NOT IN ([ids from
the first phase]) // Get "outer join" part
{code}
*Reducer Deduplication*
The idea is to get all data locally and then perform final deduplication. This
may incur high network overhead, because of lot of duplicated left parts would
be transferred. However, this could be optimized greatly with the following
techniques applied one after another
# Semi-jions: {{left}} is {{joined}} on mapper node, but instead of sending
{{(left, right)}} relation, we send {{(left) + (right)}}
# In case {{left}} part is known to be idempotent (i.e. it produces the same
result set on all nodes), only one node will send {{(left) + (right)}}, other
nodes will send {{(right)}} only
# Merge {{left}} results with if needed (i.e. if idempotence-related opto was
not applicable)
# Join {{left}} and {{right}} parts on reducer
> SQL: REPLICATED cache cannot be left-joined to PARTITIONED
> --
>
> Key: IGNITE-8732
> URL: https://issues.apache.org/jira/browse/IGNITE-8732
> Project: Ignite
>
[jira] [Commented] (IGNITE-8732) SQL: REPLICATED cache cannot be left-joined to PARTITIONED
[
https://issues.apache.org/jira/browse/IGNITE-8732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261318#comment-17261318
]
Stanislav Lukyanov commented on IGNITE-8732:
[~Vodrážka] I have implemented a partial solution based on approach #2 (treat
replicated cache as partitioned). Will be proceeding with PR and merge now.
> SQL: REPLICATED cache cannot be left-joined to PARTITIONED
> --
>
> Key: IGNITE-8732
> URL: https://issues.apache.org/jira/browse/IGNITE-8732
> Project: Ignite
> Issue Type: Improvement
> Components: sql
>Affects Versions: 2.5
>Reporter: Vladimir Ozerov
>Assignee: Stanislav Lukyanov
>Priority: Major
> Labels: sql-engine
>
> *Steps to reproduce*
> # Run
> {{org.apache.ignite.sqltests.ReplicatedSqlTest#testLeftJoinReplicatedPartitioned}}
> # Observe that we have 2x results on 2-node cluster
> *Root Cause*
> {{left LEFT JOIN right ON cond}} operation assumes full scan of of a left
> expression. Currently we perform this scan on every node and then simply
> merge results on reducer. Two nodes, two scans of {{REPLICATED}} cache, 2x
> results.
> *Potential Solutions*
> We may consider several solutions. Deeper analysis is required to understand
> which is the right one.
> # Perform deduplication on reducer - this most prospective and general
> technique, described in more details below
> # Treat {{REPLICATED}} cache as {{PARTITIONED}}. Essentially, we just need to
> pass proper backup filter. But what if {{REPLICATED}} cache spans more nodes
> than {{PARTITIONED}}? We cannot rely on primary/backup in this case
> # Implement additional execution phase as follows:
> {code}
> SELECT left.cols, right.cols FROM left INNER JOIN right ON cond;
> // Get "inner join" part
> UNION
> UNICAST SELECT left.cols, [NULL].cols FROM left WHERE left.id NOT IN ([ids
> from the first phase]) // Get "outer join" part
> {code}
> *Reducer Deduplication*
> The idea is to get all data locally and then perform final deduplication.
> This may incur high network overhead, because of lot of duplicated left parts
> would be transferred. However, this could be optimized greatly with the
> following techniques applied one after another
> # Semi-jions: {{left}} is {{joined}} on mapper node, but instead of sending
> {{(left, right)}} relation, we send {{(left) + (right)}}
> # In case {{left}} part is known to be idempotent (i.e. it produces the same
> result set on all nodes), only one node will send {{(left) + (right)}}, other
> nodes will send {{(right)}} only
> # Merge {{left}} results with if needed (i.e. if idempotence-related opto was
> not applicable)
> # Join {{left}} and {{right}} parts on reducer
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-13364) Improve index inline defaults
[ https://issues.apache.org/jira/browse/IGNITE-13364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261316#comment-17261316 ] Stanislav Lukyanov commented on IGNITE-13364: - The patch LGTM! I've started a devlist discussion on this since this has a compatibility impact and probably should be discussed before merging. https://lists.apache.org/thread.html/r22ca5271f590816517d4afbc7df633c1c3f39d01657241902e710595%40%3Cdev.ignite.apache.org%3E > Improve index inline defaults > -- > > Key: IGNITE-13364 > URL: https://issues.apache.org/jira/browse/IGNITE-13364 > Project: Ignite > Issue Type: Improvement > Components: sql >Reporter: Evgeniy Rudenko >Assignee: Evgeniy Rudenko >Priority: Major > Fix For: 2.10 > > Time Spent: 0.5h > Remaining Estimate: 0h > > We need to improve how inline size is calculated by default for > variable-length types. > Currently if a varlength type is encountered inline size just defaults to 10, > which is almost always not enough. > A more sensible behavior would be the following: > 1. Add a fixed default to the inline size calculation for every > variable-length type. For example, if the default inlined size for a string > is 10 then an index like (INT, VARCHAR, VARCHAR, INT) should have inline size > default as 5 + 10 + 10 + 5 = 30 (5 for each int, 10 for each string). > 2. Add special support for VARCHAR_FIXED - if a VARCHAR has known length then > that length is used for inline size calculation -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (IGNITE-13963) Excessive classloading of PlatformDotNetSessionLockResult on thin client connection
[
https://issues.apache.org/jira/browse/IGNITE-13963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Kurbanov updated IGNITE-13963:
Affects Version/s: 2.9
> Excessive classloading of PlatformDotNetSessionLockResult on thin client
> connection
> ---
>
> Key: IGNITE-13963
> URL: https://issues.apache.org/jira/browse/IGNITE-13963
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.9
>Reporter: Anton Kurbanov
>Assignee: Anton Kurbanov
>Priority: Major
>
> Empty constructor must be added to
> org.apache.ignite.internal.processors.platform.websession.PlatformDotNetSessionLockResult.
> Multiple thin clients connections are causing this class to generate
> constructors:
> {code:java}
> [Loaded *sun.reflect.GeneratedSerializationConstructorAccessor99* from
> __JVM_DefineClass__]
> {code}
> This is caused by U.forceEmptyConstructor(cls) call from
> org.apache.ignite.internal.binary.BinaryClassDescriptor#constructor.
> With lots of clients it is possible to create high load on metaspace which
> results in lots of metaspace GC.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (IGNITE-13963) Excessive classloading of PlatformDotNetSessionLockResult on thin client connection
[
https://issues.apache.org/jira/browse/IGNITE-13963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Kurbanov updated IGNITE-13963:
Ignite Flags: Release Notes Required (was: Docs Required,Release Notes
Required)
> Excessive classloading of PlatformDotNetSessionLockResult on thin client
> connection
> ---
>
> Key: IGNITE-13963
> URL: https://issues.apache.org/jira/browse/IGNITE-13963
> Project: Ignite
> Issue Type: Bug
>Reporter: Anton Kurbanov
>Assignee: Anton Kurbanov
>Priority: Major
>
> Empty constructor must be added to
> org.apache.ignite.internal.processors.platform.websession.PlatformDotNetSessionLockResult.
> Multiple thin clients connections are causing this class to generate
> constructors:
> {code:java}
> [Loaded *sun.reflect.GeneratedSerializationConstructorAccessor99* from
> __JVM_DefineClass__]
> {code}
> This is caused by U.forceEmptyConstructor(cls) call from
> org.apache.ignite.internal.binary.BinaryClassDescriptor#constructor.
> With lots of clients it is possible to create high load on metaspace which
> results in lots of metaspace GC.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IGNITE-13963) Excessive classloading of PlatformDotNetSessionLockResult on thin client connection
Anton Kurbanov created IGNITE-13963:
---
Summary: Excessive classloading of PlatformDotNetSessionLockResult
on thin client connection
Key: IGNITE-13963
URL: https://issues.apache.org/jira/browse/IGNITE-13963
Project: Ignite
Issue Type: Bug
Reporter: Anton Kurbanov
Assignee: Anton Kurbanov
Empty constructor must be added to
org.apache.ignite.internal.processors.platform.websession.PlatformDotNetSessionLockResult.
Multiple thin clients connections are causing this class to generate
constructors:
{code:java}
[Loaded *sun.reflect.GeneratedSerializationConstructorAccessor99* from
__JVM_DefineClass__]
{code}
This is caused by U.forceEmptyConstructor(cls) call from
org.apache.ignite.internal.binary.BinaryClassDescriptor#constructor.
With lots of clients it is possible to create high load on metaspace which
results in lots of metaspace GC.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (IGNITE-11110) UnsupportedOperationException: null when stopping grid
[
https://issues.apache.org/jira/browse/IGNITE-0?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Kurbanov updated IGNITE-0:
Release Note: Fixed NPE on stopping grid node with enabled collision SPI.
> UnsupportedOperationException: null when stopping grid
> --
>
> Key: IGNITE-0
> URL: https://issues.apache.org/jira/browse/IGNITE-0
> Project: Ignite
> Issue Type: Bug
>Affects Versions: 2.7
>Reporter: Jens Borgland
>Assignee: Anton Kurbanov
>Priority: Major
> Fix For: 2.10
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> After upgrading to 2.7 we've started getting these errors when stopping grids:
> {noformat}
> java.lang.UnsupportedOperationException: null
> at
> org.jsr166.ConcurrentLinkedHashMap.clear(ConcurrentLinkedHashMap.java:1551)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
> org.apache.ignite.internal.processors.job.GridJobProcessor.stop(GridJobProcessor.java:264)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
> org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:2356)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:2228)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2612)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2575)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:379)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at org.apache.ignite.Ignition.stop(Ignition.java:225)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
> org.apache.ignite.internal.IgniteKernal.close(IgniteKernal.java:3568)
> ~[ignite-core-2.7.0.jar:2.7.0]
> {noformat}
> At first glance it looks likely that it was introduced with commit
> [d04d764|https://github.com/apache/ignite/commit/d04d76440ce86873de7aebc8c03d39484cd1e3e5]
> (the {{GridJobProcessor}} still calls {{clear()}} on maps).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
