[jira] [Created] (IGNITE-14507) Un-deprecate IGNITE_BINARY_SORT_OBJECT_FIELDS

[Stanislav Lukyanov (Jira)]

Stanislav Lukyanov created IGNITE-14507:
---

 Summary: Un-deprecate IGNITE_BINARY_SORT_OBJECT_FIELDS
 Key: IGNITE-14507
 URL: https://issues.apache.org/jira/browse/IGNITE-14507
 Project: Ignite
  Issue Type: Improvement
  Components: binary
Reporter: Stanislav Lukyanov


IGNITE_BINARY_SORT_OBJECT_FIELDS should not be deprecated - on the contrary, 
every new environment better have this option. We need to advertise it more, so 
we need to remove the deprecation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13985) C++: Continuous Queries should fail to register if remote filter is in C++ and there are non-C++ affinity nodes

[Stanislav Lukyanov (Jira)]

Stanislav Lukyanov created IGNITE-13985:
---

 Summary: C++: Continuous Queries should fail to register if remote 
filter is in C++ and there are non-C++ affinity nodes
 Key: IGNITE-13985
 URL: https://issues.apache.org/jira/browse/IGNITE-13985
 Project: Ignite
  Issue Type: Improvement
Reporter: Stanislav Lukyanov


See https://issues.apache.org/jira/browse/IGNITE-6800.

Currently, if there is a pure Java node in the cluster and a CQ is being 
registered from C++, the C++ remote filter will just be ignored on the pure 
Java node leading to unexpected CQ notifications.

Need to add a validation to prevent incorrect results.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13975) Validate that tables beind joined have the same partition distribution

[Stanislav Lukyanov (Jira)]

Stanislav Lukyanov created IGNITE-13975:
---

 Summary: Validate that tables beind joined have the same partition 
distribution
 Key: IGNITE-13975
 URL: https://issues.apache.org/jira/browse/IGNITE-13975
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Stanislav Lukyanov


To be able to join two PARTITIONED tables one needs to perform the JOIN on an 
affinity key, and also to make sure that the tables have the same partition 
distribution - otherwise, the result will be incorrect.

Validation of whether the affinity key is being used for JOIN is to be done in 
https://issues.apache.org/jira/browse/IGNITE-13019.

The goal of this ticket is to add validation of the partition distribution.

Whenever a JOIN of PARTITIONED tables is being done, the SQL engine needs to 
ensure that
 * The tables use the same affinity function
 * The tables have the same number of partitions
 * The tables have the same node filters

Same is for when REPLICATED tables are used and 
`treatReplicatedAsPartitioned=true` (see 
https://issues.apache.org/jira/browse/IGNITE-8732).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13777) idle_verify should report real size of the partitions

[Stanislav Lukyanov (Jira)]

Stanislav Lukyanov created IGNITE-13777:
---

 Summary: idle_verify should report real size of the partitions
 Key: IGNITE-13777
 URL: https://issues.apache.org/jira/browse/IGNITE-13777
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


Currently, idle_verify checks the content of partitions (through hash) and the 
partition size that is stored in the partition meta. It will be better if 
idle_verify also counts the entries inside the partition and returns both the 
size from the meta AND the real size.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13341) OOME is ignored if thrown in a client connector thread

[Stanislav Lukyanov (Jira)]

Stanislav Lukyanov created IGNITE-13341:
---

 Summary: OOME is ignored if thrown in a client connector thread
 Key: IGNITE-13341
 URL: https://issues.apache.org/jira/browse/IGNITE-13341
 Project: Ignite
  Issue Type: Bug
  Components: thin client
Reporter: Stanislav Lukyanov


If a thin client operation causes OOME then the thread will die and an error 
will be logged, but that's all. The clients are left hanging and waiting for 
reply until timeout.

Correct behavior in this case would be to trigger Failure Handler.

Example of the error:
{code}
Exception in thread "client-connector-#69%xxx%" java.lang.OutOfMemoryError: 
Java heap space
Aug 08, 2020 7:08:43 AM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: GridWorker 
[name=message-received-notify, igniteInstanceName=xxx, finished=false, 
heartbeatTs=1596859722580, hashCode=1209004925, interrupted=false, 
runner=client-connector-#66%xxx%]
java.lang.OutOfMemoryError: Java heap space
at 
org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk.reallocate(BinaryMemoryAllocatorChunk.java:68)
at 
org.apache.ignite.internal.binary.streams.BinaryHeapOutputStream.ensureCapacity(BinaryHeapOutputStream.java:64)
at 
org.apache.ignite.internal.binary.streams.BinaryAbstractOutputStream.unsafeEnsure(BinaryAbstractOutputStream.java:261)
at 
org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteBinaryObject(BinaryWriterExImpl.java:970)
at 
org.apache.ignite.internal.binary.BinaryClassDescriptor.write(BinaryClassDescriptor.java:756)
at 
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:231)
at 
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:164)
at 
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:151)
at 
org.apache.ignite.internal.binary.BinaryWriterExImpl.doWriteObject(BinaryWriterExImpl.java:523)
at 
org.apache.ignite.internal.binary.BinaryWriterExImpl.writeObject(BinaryWriterExImpl.java:1502)
at 
org.apache.ignite.internal.processors.platform.client.ClientObjectResponse.encode(ClientObjectResponse.java:43)
at 
org.apache.ignite.internal.processors.platform.client.ClientMessageParser.encode(ClientMessageParser.java:459)
at 
org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:203)
at 
org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:49)
at 
org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:278)
at 
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:108)
at 
org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter$3.body(GridNioAsyncNotifyFilter.java:135)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
at 
org.apache.ignite.internal.util.worker.GridWorkerPool$1.run(GridWorkerPool.java:69)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13340) copyOnRead=false doesn't work for byte array values

[Stanislav Lukyanov (Jira)]

Stanislav Lukyanov created IGNITE-13340:
---

 Summary: copyOnRead=false doesn't work for byte array values
 Key: IGNITE-13340
 URL: https://issues.apache.org/jira/browse/IGNITE-13340
 Project: Ignite
  Issue Type: Bug
  Components: cache
Reporter: Stanislav Lukyanov


If near cache is used and copyOnRead=false is set then the Java object should 
only be created on the first read, and all subsequent reads must use the same 
copy.

However, when byte array value is used (e.g. `put(42, new byte[100]`) then the 
copying is done on every read. It seems that the reason is that byte array 
values have their own implementation of `CacheObject` - 
`CacheObjectByteArrayImpl`. That implementation doesn't use 
`CacheObjectValueContext::storeValue` which controls whether copying needs to 
be done; CacheObjectImpl uses it.

Need to correctly implement copyOnRead=false for  caches.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-11977) Data streamer pool MXBean is registered as ThreadPoolMXBean instead of StripedExecutorMXBean

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-11977:
---

 Summary: Data streamer pool MXBean is registered as 
ThreadPoolMXBean instead of StripedExecutorMXBean
 Key: IGNITE-11977
 URL: https://issues.apache.org/jira/browse/IGNITE-11977
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


Data streamer pool is registered with a ThreadPoolMXBean while it is actually a 
StripedExecutor and can use a StripedExecutorMXBean.

Need to change the registration in the IgniteKernal code. It should be 
registered the same way as the striped executor pool.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (IGNITE-11976) @SpringResource is silently ignored if no Spring context is provided

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-11976:
---

 Summary: @SpringResource is silently ignored if no Spring context 
is provided
 Key: IGNITE-11976
 URL: https://issues.apache.org/jira/browse/IGNITE-11976
 Project: Ignite
  Issue Type: Improvement
  Components: spring
Affects Versions: 2.7
Reporter: Stanislav Lukyanov


@SpringResource annotation is silently ignored and the annotated field is null 
if Spring context.

For @SpringResource to work the node needs to be started with 
IgniteSpring::start instead of Ignition::start, but the user may not know that.

Need to add a warning for this.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (IGNITE-11968) FailureProcessor reports tcp-comm-worker as blocked when resolving communication errors in ZK

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-11968:
---

 Summary: FailureProcessor reports tcp-comm-worker as blocked when 
resolving communication errors in ZK
 Key: IGNITE-11968
 URL: https://issues.apache.org/jira/browse/IGNITE-11968
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 2.7
Reporter: Stanislav Lukyanov


When multiple clients fail (stopped) at the same time and ZK discovery is used 
FailureProcessor treats tcp-comm-worker as blocked with the following trace:
{code}
Thread [name="tcp-comm-worker-#1", id=122, state=WAITING, blockCnt=1465, 
waitCnt=4557]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178)
at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
at 
o.a.i.spi.discovery.zk.internal.ZookeeperDiscoveryImpl.resolveCommunicationError(ZookeeperDiscoveryImpl.java:359)
at 
o.a.i.spi.discovery.zk.ZookeeperDiscoverySpi.resolveCommunicationFailure(ZookeeperDiscoverySpi.java:259)
at 
o.a.i.i.managers.discovery.GridDiscoveryManager.resolveCommunicationError(GridDiscoveryManager.java:2549)
at 
o.a.i.i.managers.GridManagerAdapter$1.resolveCommunicationFailure(GridManagerAdapter.java:626)
at 
o.a.i.spi.communication.tcp.TcpCommunicationSpi.processClientCreationError(TcpCommunicationSpi.java:3530)
at 
o.a.i.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3483)
at 
o.a.i.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2987)
at 
o.a.i.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2870)
at 
o.a.i.spi.communication.tcp.TcpCommunicationSpi.access$6000(TcpCommunicationSpi.java:271)
at 
o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.processDisconnect(TcpCommunicationSpi.java:4489)
at 
o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4294)
at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120)
at 
o.a.i.spi.communication.tcp.TcpCommunicationSpi$5.body(TcpCommunicationSpi.java:2237)
at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11894) Add fetchSize to JDBC cache stores

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-11894:
---

 Summary: Add fetchSize to JDBC cache stores
 Key: IGNITE-11894
 URL: https://issues.apache.org/jira/browse/IGNITE-11894
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Reporter: Stanislav Lukyanov


JDBC's PreparedStatement accepts a fetchSize parameter which defines how many 
rows will be loaded from the DB at a time. Currently the only way to change 
that is by specifying it in a customer implementation of the 
JdbcDialect::fetchSize method (and even then it seems not be be used in some 
cases).

Would be good to have a fetchSize property in all of JDBC-based cache stores.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11836) keyFields must be empty if keyType is an SQL type

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-11836:
---

 Summary: keyFields must be empty if keyType is an SQL type
 Key: IGNITE-11836
 URL: https://issues.apache.org/jira/browse/IGNITE-11836
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Stanislav Lukyanov


Currently if QueryEntity.keyType is an SQL type (e.g. String or Integer) then 
keyFields property is simply ignored. This creates room for a misconfiguration.

Let's add a check that keyFields is null or empty if keyType is an SQL type.

Probably the same should be done for valueType and valueFields.

Also, fields property should be handled somehow. An obvious case is that fields 
should be null or empty when both key and value are SQL types. There is 
probably more invariants to enforce though.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11829) Distribute joins fail if number of tables > 7

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-11829:
---

 Summary: Distribute joins fail if number of tables > 7
 Key: IGNITE-11829
 URL: https://issues.apache.org/jira/browse/IGNITE-11829
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Affects Versions: 2.7
Reporter: Stanislav Lukyanov


Distributed joins fail with ArrayIndexOutOfBounds when the total number of 
tables is > 7.
Example:
{code}
try (Ignite ignite = 
Ignition.start("examples/config/example-ignite.xml");) {
IgniteCache cache = ignite.createCache("foo");

cache.query(new SqlFieldsQuery("CREATE TABLE Person(ID INTEGER 
PRIMARY KEY, NAME VARCHAR(100));"));
cache.query(new SqlFieldsQuery("INSERT INTO Person(ID, NAME) VALUES 
(1, 'Ed'), (2, 'Ann'), (3, 'Emma');"));

cache.query(new SqlFieldsQuery("SELECT *\n" +
"FROM PERSON P1\n" +
"JOIN PERSON P2 ON P1.ID = P2.ID\n" +
"JOIN PERSON P3 ON P1.ID = P3.ID\n" +
"JOIN PERSON P4 ON P1.ID = P4.ID\n" +
"JOIN PERSON P5 ON P1.ID = P5.ID\n" +
"JOIN PERSON P6 ON P1.ID = P6.ID\n" +
"JOIN PERSON P7 ON P1.ID = P7.ID\n" +
"JOIN PERSON P8 ON P1.ID = 
P8.ID").setDistributedJoins(true).setEnforceJoinOrder(false));

}
{code}
throws
{code}
Exception in thread "main" javax.cache.CacheException: General error: 
"java.lang.ArrayIndexOutOfBoundsException" [5-197]
at 
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:832)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:765)
at 
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.query(GatewayProtectedCacheProxy.java:403)
at 
org.apache.ignite.examples.ExampleNodeStartup.main(ExampleNodeStartup.java:60)
Caused by: class 
org.apache.ignite.internal.processors.query.IgniteSQLException: General error: 
"java.lang.ArrayIndexOutOfBoundsException" [5-197]
at 
org.apache.ignite.internal.processors.query.h2.QueryParser.parseH2(QueryParser.java:454)
at 
org.apache.ignite.internal.processors.query.h2.QueryParser.parse0(QueryParser.java:156)
at 
org.apache.ignite.internal.processors.query.h2.QueryParser.parse(QueryParser.java:121)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.querySqlFields(IgniteH2Indexing.java:1191)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor$3.applyx(GridQueryProcessor.java:2261)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor$3.applyx(GridQueryProcessor.java:2257)
at 
org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:53)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:2767)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.lambda$querySqlFields$1(GridQueryProcessor.java:2277)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuerySafe(GridQueryProcessor.java:2297)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:2250)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:2177)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:817)
... 3 more
Caused by: org.h2.jdbc.JdbcSQLException: General error: 
"java.lang.ArrayIndexOutOfBoundsException" [5-197]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:357)
at org.h2.message.DbException.get(DbException.java:168)
at org.h2.message.DbException.convert(DbException.java:307)
at org.h2.message.DbException.toSQLException(DbException.java:280)
at org.h2.message.TraceObject.logAndConvert(TraceObject.java:357)
at org.h2.jdbc.JdbcConnection.prepareStatement(JdbcConnection.java:308)
at 
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.prepare(GridSqlQuerySplitter.java:1770)
at 
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.split0(GridSqlQuerySplitter.java:299)
at 
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.split(GridSqlQuerySplitter.java:219)
at 
org.apache.ignite.internal.processors.query.h2.QueryParser.parseH2(QueryParser.java:415)
... 15 more
Caused by: java.lang.ArrayIndexOutOfBoundsException
at java.lang.System.arraycopy(Native Method)
at 
org.apache.ignite.internal.processors.query.h2.opt.join.CollocationModel.childFilters(CollocationModel.java:240)
at 

[jira] [Created] (IGNITE-11533) Improve memory usage reporting in the logs

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-11533:
---

 Summary: Improve memory usage reporting in the logs
 Key: IGNITE-11533
 URL: https://issues.apache.org/jira/browse/IGNITE-11533
 Project: Ignite
  Issue Type: Improvement
Reporter: Stanislav Lukyanov


Currently node's memory usage is reported in the log like this
{code}
^-- PageMemory [pages=66407]
^-- Heap [used=303MB, free=91,6%, comm=849MB]
^-- Off-heap [used=260MB, free=62,73%, comm=580MB]
^--   sysMemPlc region [used=0MB, free=99,21%, comm=40MB]
^--   TxLog region [used=0MB, free=100%, comm=40MB]
^--   Default_Region region [used=260MB, free=47,97%, comm=500MB]
{code}
And it also used to report "Non heap" metrics which were removed in IGNITE-9305.

This has several issues
1. "PageMemory" line alone doesn't help to understand the Ignite off-heap 
memory usage - page size is needed to calculated the used size, and maxSize of 
the data regions are needed to calculate the amount of the memory left.
2. "Non heap" could actually be useful. The line used to show "free=-1%" in 
most setups. This is because metaspace pool of the JVM is not limited by 
default which makes the total size unlimited. In general, it would be good to 
see distribution among the non-heap pools (metaspace, direct buffers) to be 
able to tune JVM settings accordingly.

Let's do the following:
- Don't print "PageMemory" line - it's cryptic and adds little value to the 
"Off-heap" section
- Bring "Non heap" back in some way
-- Don't print "free=-1%" - either print "N/A" or don't print "free" at all
-- Add a suggestion to limit metaspace (same as we do for direct memory) - 
which will enable correct "free" calculation
-- Look into possibility of showing distinct metrics for direct memory and 
metaspace, so that the user can tell which to tune



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11471) JDBC: Thin driver should start with a random server to connect

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-11471:
---

 Summary: JDBC: Thin driver should start with a random server to 
connect
 Key: IGNITE-11471
 URL: https://issues.apache.org/jira/browse/IGNITE-11471
 Project: Ignite
  Issue Type: Improvement
  Components: jdbc
Reporter: Stanislav Lukyanov


Thin JDBC driver uses a list of server addresses to connect to. Connections 
created in a single JVM will use round-robin to choose the server to connect, 
yielding a uniform distribution. However, the round-robin always starts from 
the first server in the list. It means that multiple JVMs with the same JDBC 
connection URL and one connection each will all connect to the same server.

Need to choose the first server randomly to have a better distribution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11365) Partition sizes differ after streaming with allowOverwrite=false with concurrent PME

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-11365:
---

 Summary: Partition sizes differ after streaming with 
allowOverwrite=false with concurrent PME
 Key: IGNITE-11365
 URL: https://issues.apache.org/jira/browse/IGNITE-11365
 Project: Ignite
  Issue Type: Improvement
Reporter: Stanislav Lukyanov
 Attachments: DataStreamerRebalanceMissingDocTest.java

Scenario
- Start 3 nodes
- Run N threads, each thread starts a replicated cache and then streams data to 
that cache with allowOverwrite=false

During the streaming there are "Partition validation failed" warnings.
After the streaming in all threads ends, some caches have incorrect size and 
local SQL queries return less data.

Seems that PME can break an ongoing streaming with allowOverwrite=false.
allowOverwrite=true is a workaround.

Reproducer is attached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11355) Mention CacheConfiguration.sqlIndexMaxInlineSize and QueryGroupIndex.inlineSize in inline size suggestions

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-11355:
---

 Summary: Mention CacheConfiguration.sqlIndexMaxInlineSize and 
QueryGroupIndex.inlineSize in inline size suggestions
 Key: IGNITE-11355
 URL: https://issues.apache.org/jira/browse/IGNITE-11355
 Project: Ignite
  Issue Type: Improvement
Reporter: Stanislav Lukyanov


Currently the suggestion for changing inline size of the PK and AK indexes 
reads like this
{code}
recommendation = "set system property "
+ IgniteSystemProperties.IGNITE_MAX_INDEX_PAYLOAD_SIZE + " 
with recommended size " +
"(be aware it will be used by default for all indexes 
without explicit inline size)";
{code}
However, there is also property CacheConfiguration.sqlIndexMaxInlineSize that 
can be used. In fact, in most cases it is even better to use the cache 
configuration property instead of the system property because 1) it is local, 
not global 2) it is a part of the static configuration and doesn't depend of 
the correct environment setup

The suggestion for other indexes reads 
{code}
recommendation = "use INLINE_SIZE option for CREATE INDEX 
command, " +
"QuerySqlField.inlineSize for annotated classes, or 
QueryIndex.inlineSize for explicit " +
"QueryEntity configuration";
{code}
It should also mention QueryGroupIndex.inlineSize for group indexes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11312) JDBC: Thin driver doesn't reports incorrect property names

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-11312:
---

 Summary: JDBC: Thin driver doesn't reports incorrect property names
 Key: IGNITE-11312
 URL: https://issues.apache.org/jira/browse/IGNITE-11312
 Project: Ignite
  Issue Type: Improvement
  Components: jdbc
Reporter: Stanislav Lukyanov


JDBC driver reports the properties it supports via getPropertyInfo method. It 
currently reports the property names as simple strings, like 
"enforceJoinOrder". However, when the properties are processed on connect they 
are looked up with prefix "ignite.jdbc", e.g. "ignite.jdbc.enforceJoinOrder".

Because of this UI tools like DBeaver can't properly pass the properties to 
Ignite. For example, when "enforceJoinOrder" is set to true in "Connection 
settings" -> "Driver properties" menu of DBeaver it has no effect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11230) Fix Javadoc for DataStorageConfiguration

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-11230:
---

 Summary: Fix Javadoc for DataStorageConfiguration
 Key: IGNITE-11230
 URL: https://issues.apache.org/jira/browse/IGNITE-11230
 Project: Ignite
  Issue Type: Improvement
  Components: documentation
Reporter: Stanislav Lukyanov


Need to polish DataStorageConfiguration Javadoc:
- add links to set* methods from get* counterparts
- add links to the default values for all properties



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11146) SQL: QueryIndex and DDL generate indexes with different default names

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-11146:
---

 Summary: SQL: QueryIndex and DDL generate indexes with different 
default names
 Key: IGNITE-11146
 URL: https://issues.apache.org/jira/browse/IGNITE-11146
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Stanislav Lukyanov


`QueryIndex` with a name set and `CREATE INDEX` without a specified name create 
indexes with different names, even if all parameters are the same - `CREATE 
INDEX` produces a lower-case name, and `QueryIndex` produces an upper-case one.

For example
{code}
new QueryIndex("a")
{code}
in a table "Foo" creates an index with name
{code}
FOO_A_ASC_IDX
{code}
while the query
{code}
CREATE INDEX ON PUBLIC.FOO (a)
{code}
creates an index
{code}
foo_a_asc_idx
{code}

It would be better if DDL generated an upper-case name since that's the default 
in SQL.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11141) Add Java method to create a cache from a template

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-11141:
---

 Summary: Add Java method to create a cache from a template
 Key: IGNITE-11141
 URL: https://issues.apache.org/jira/browse/IGNITE-11141
 Project: Ignite
  Issue Type: Task
Reporter: Stanislav Lukyanov


Currently cache templates can only be used via REST or SQL: 
https://apacheignite.readme.io/docs/cache-template.
It seems reasonable to add the same functionality to Java.

The most flexible way would probably be to add a method like 
`cacheConfiguration(String)` that would return a configuration of a template 
with the given name. Then to create a cache from a template one would write 
code like this
{code}
CacheConfiguration cfg = ignite.cacheConfiguration("myTemplate");
cfg.setName("myCacheFromTemplate");
ignite.createCache(cfg);
{code}

As a bonus, the `cacheConfiguration(String)` method may work with regular 
caches too. For them it would be a shortcut for 
`ignite.cache("myCache").getConfiguration()`.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10983) Check that persistenceEnabled is consistent on all nodes

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10983:
---

 Summary: Check that persistenceEnabled is consistent on all nodes
 Key: IGNITE-10983
 URL: https://issues.apache.org/jira/browse/IGNITE-10983
 Project: Ignite
  Issue Type: Task
Reporter: Stanislav Lukyanov


Currently it is possible to have a cluster where the same data region is 
persistent on some nodes and not persistent on others. This use case doesn't 
have enough testing, so it's better to deny it for now by adding a check for 
that and not allowing a node with a different persistenceEnabled value to join 
the cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10952) Document SET STREAMING command options

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10952:
---

 Summary: Document SET STREAMING command options
 Key: IGNITE-10952
 URL: https://issues.apache.org/jira/browse/IGNITE-10952
 Project: Ignite
  Issue Type: Task
  Components: documentation
Reporter: Stanislav Lukyanov


SET STREAMING command (https://apacheignite-sql.readme.io/docs/set) has a 
number of supported options related to the underlying data streamer 
configuration. However, they are not documented.

Need to add documentation (BNFs and examples) for all of the options.

The list of options can be found in the SqlSetStreamingCommand source code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10922) Add a way to override any configuration property via a system property

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10922:
---

 Summary: Add a way to override any configuration property via a 
system property
 Key: IGNITE-10922
 URL: https://issues.apache.org/jira/browse/IGNITE-10922
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


>From time to time we introduce system properties as a shorthand way to set a 
>specific configuration parameter, e.g. IGNITE-10921.

It would be nice to have a standard way to override any property like that.
E.g. instead of 
-DIGNITE_CONSISTENT_ID=foo 
we would use something like
-Dorg.apache.ignite.IgniteConfiguration.consistentId=foo

This mechanism should work for any configuration sections. It seems that it is 
rather hard to implement for complex types, but perhaps we could start by doing 
this for simple properties (strings, numbers, etc).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10921) Add IGNITE_CONSISTENT_ID system property

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10921:
---

 Summary: Add IGNITE_CONSISTENT_ID system property
 Key: IGNITE-10921
 URL: https://issues.apache.org/jira/browse/IGNITE-10921
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


IGNITE-10900 explains the benefits of setting consistent ID explicitly and 
suggests adding a warning it isn't. However, currently changing consistentId 
for each node is rather cumbersome if the nodes share the configuration.

It would be good to add a new system property as a shorthand way to set 
consistent ID.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10900) Print a warning if native persistence is used without an explicit consistent ID

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10900:
---

 Summary: Print a warning if native persistence is used without an 
explicit consistent ID
 Key: IGNITE-10900
 URL: https://issues.apache.org/jira/browse/IGNITE-10900
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


Experience shows that when Native Persistence is enabled, it is better to 
explicitly set ConsistentIDs than use the autogenerated ones.

First, it simplifies managing the baseline topology. It is much easier to 
manage it via control.sh when the nodes have stable and meaningful names.

Second, it helps to avoid certain shoot-yourself-in-the-foot issues. E.g. if 
one loses all the data of a baseline node, when that node is restarted it 
doesn't have its old autogenerated consistent ID - so it is not a part of the 
baseline anymore. This may be unexpected and confusing.

Finally, having explicit consistent IDs improves the general stability of the 
setup - one knows what the the set of nodes, where they run and what they're 
called.

All in all, it seems beneficial to urge users to explicitly configure 
consistent IDs. We can do this by introducing a warning that is printed every 
time a new consistent ID is automatically generated. It should also be printed 
when a node doesn't have an explicit consistent ID and picks up one from an 
existing peristence folder.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10887) Docs: Align performance tips for .Net with Java

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10887:
---

 Summary: Docs: Align performance tips for .Net with Java
 Key: IGNITE-10887
 URL: https://issues.apache.org/jira/browse/IGNITE-10887
 Project: Ignite
  Issue Type: Bug
  Components: documentation
Reporter: Stanislav Lukyanov


It seems that the .Net Performnace Tips page 
(https://apacheignite-net.readme.io/docs/performance-tips) is outdated/not 
aligned with what we have in Java.

I think the best way would be to remove most (or all) info from the .Net part, 
and only leave a link to the main page, since the performance tips should be 
roughly the same for all platforms.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10856) cassandra-store seems to be broken by incorrect guava version

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10856:
---

 Summary: cassandra-store seems to be broken by incorrect guava 
version
 Key: IGNITE-10856
 URL: https://issues.apache.org/jira/browse/IGNITE-10856
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


IGNITE-9131 upgrade guava from 18 to 25.
However, cassandra-driver-core:3.0 dependency of the Ignite's cassandra-store 
requires guava 16-19.
Need to fix the dependency conflict there.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10808) Discovery message queue may build up with TcpDiscoveryMetricsUpdateMessage

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10808:
---

 Summary: Discovery message queue may build up with 
TcpDiscoveryMetricsUpdateMessage
 Key: IGNITE-10808
 URL: https://issues.apache.org/jira/browse/IGNITE-10808
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov
 Attachments: IgniteMetricsOverflowTest.java

A node receives a new metrics update message every `metricsUpdateFrequency` 
milliseconds, and the message will be put at the top of the queue (because it 
is a high priority message).
If processing one message takes more than `metricsUpdateFrequency` then 
multiple `TcpDiscoveryMetricsUpdateMessage` will be in the queue. A long enough 
delay (e.g. caused by a network glitch or GC) may lead to the queue building up 
tens of metrics update messages which are essentially useless to be processed. 
Finally, if processing a message on average takes a little more than 
`metricsUpdateFrequency` (even for a relatively short period of time, say, for 
a minute due to network issues) then the message worker will end up processing 
only the metrics updates and the cluster will essentially hang.

Reproducer is attached. In the test, the queue first builds up and then very 
slowly being teared down, causing "Failed to wait for PME" messages.

Need to change ServerImpl's SocketReader not to put another metrics update 
message to the top of the queue if it already has one (or replace the one on 
the top with new one).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10563) Allow manual fsync for WAL

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10563:
---

 Summary: Allow manual fsync for WAL
 Key: IGNITE-10563
 URL: https://issues.apache.org/jira/browse/IGNITE-10563
 Project: Ignite
  Issue Type: Improvement
Reporter: Stanislav Lukyanov


When walMode is set to LOG_ONLY or BACKGROUND there is a gap between successful 
return of cache write operations and actual of the data to the persistent 
memory. This gap is, while usually low, generally can be of any length and 
depends on low-level system parameters (e.g. sysctl memory and IO settings on 
Linux).

On the other hand, there are situations when a user may want to ensure that at 
certain points all writes have been propagated to the disk.

For example, say a large batch of data is being inserted into Ignite from an 
upstream system. After finishing the batch we want to ensure that all of the 
data is secure, and after that we remove the batch from the upstream storage. 
In other words, we want
{code}
map = upstream.getData();
cache.putAll(map);
cache.fsync(); // <---
upstream.removeAll(map);
{code}
Currently there is no easy way to ensure that certain write (or all writes 
until a point in time) has been flushed to a device. We can only rely on 
settings like WAL sync interval, checkpoint timeout, etc.

It would be nice to have a way to manually call fsync() for WAL to have a 
strong guarantee that all previous updates have been fully written on disk.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10516) Storage is corrupted after CREATE INDEX IF NOT EXISTS on different tables

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10516:
---

 Summary: Storage is corrupted after CREATE INDEX IF NOT EXISTS on 
different tables
 Key: IGNITE-10516
 URL: https://issues.apache.org/jira/browse/IGNITE-10516
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


Given two tables in the same schema, we can't create an index with the same 
name for both tables. In other words, the following code leads to an error - 
which is good
{code}
CREATE INDEX IDX on T1 (COL);
CREATE INDEX IDX on T2 (COL);
{code}

If used with `IF NOT EXISTS`, the queries pass. It might be OK or not - one 
needs to look into SQL spec to check if the second operation should be a no-op 
(because IDX exists) or fail (because IDX exists for a different table, so the 
caller is probably doing something wrong)
{code}
CREATE INDEX IDX on T1 (COL);
CREATE INDEX IF NOT EXISTS IDX on T2 (COL);
{code}

However, if persistence is enabled, the node will fail to restart complaining 
about duplicate index names.
{code}
class org.apache.ignite.IgniteCheckedException: Duplicate index name 
[cache=SQL_PUBLIC_T2, schemaName=PUBLIC, idxName=IDX, existingTable=T, table=T2]
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1183)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2040)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1732)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:656)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:959)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:900)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:888)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:854)
at 
org.apache.ignite.IndexWithSameNameTest.test(IndexWithSameNameTest.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$001(GridAbstractTest.java:150)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$6.evaluate(GridAbstractTest.java:2104)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2119)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteException: Duplicate index name 
[cache=SQL_PUBLIC_T2, schemaName=PUBLIC, idxName=IDX, existingTable=T, table=T2]
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.registerCache0(GridQueryProcessor.java:1650)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.onCacheStart0(GridQueryProcessor.java:803)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.onCacheStart(GridQueryProcessor.java:866)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCacheInRecoveryMode(GridCacheProcessor.java:2595)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.access$1400(GridCacheProcessor.java:204)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor$CacheRecoveryLifecycle.afterBinaryMemoryRestore(GridCacheProcessor.java:5481)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreBinaryMemory(GridCacheDatabaseSharedManager.java:947)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.startMemoryRestore(GridCacheDatabaseSharedManager.java:1922)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1050)
... 18 more
{code}
It looks like the second index (on T2) is partially created after all.

Need to either block index creation by `CREATE INDEX IF NOT EXISTS` completely, 
or just fail that query when the table names don't match (if SQL spec allows 
it).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10506) Emphasize the need to close the query cursors in docs

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10506:
---

 Summary: Emphasize the need to close the query cursors in docs
 Key: IGNITE-10506
 URL: https://issues.apache.org/jira/browse/IGNITE-10506
 Project: Ignite
  Issue Type: Bug
  Components: documentation
Reporter: Stanislav Lukyanov


Currently the need to close query cursors is mentioned only in two places:
- Queries docs on readme.io: 
https://apacheignite.readme.io/docs/cache-queries#section-querycursor
- QueryCursor::close javadoc 
https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/query/QueryCursor.html#close--

A failure to close a cursor may lead to severe resource leaks. It seems 
reasonable to better emphasize and explain how to approach cursors closing. 
Most importantly, we need to have a mention of that in the Java API secton of 
SQL readme.io docs (https://apacheignite-sql.readme.io/docs/java-sql-api).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10225) Striped pool starvation check reports false-positive warnings if a job starts right before the check

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10225:
---

 Summary: Striped pool starvation check reports false-positive 
warnings if a job starts right before the check
 Key: IGNITE-10225
 URL: https://issues.apache.org/jira/browse/IGNITE-10225
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


StripedExecutor::checkStarvation checks if two conditions are met: a stripe is 
executing a job and the stripe's job counter hasn't changed since the last 
check.
These conditions will be met if no jobs have been executed for a long time and 
a job (even a very short one) has just started.

This check isn't supposed to be exact and allowed to produce false-positives. 
However, the check is supposed to detect jobs running for more than starvation 
check interval (IGNITE_STARVATION_CHECK_INTERVAL), but in this case it may 
detect even very short ones.

To fix this we may require a stripe to have no progress in two consecutive 
checks - then we will have much less chance for a false-positive, but still 
detect hanging or very long jobs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10223) Affinity methods should return List instead of Collection

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10223:
---

 Summary: Affinity methods should return List instead of Collection
 Key: IGNITE-10223
 URL: https://issues.apache.org/jira/browse/IGNITE-10223
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


Methods of class Affinity should return List instead of Collection when order 
of nodes matters.

The following methods are supposed to always return a collection with primary 
node preceding backups:
Affinity::mapPartitionToPrimaryAndBackups
Affinity::mapKeyToPrimaryAndBackups

The return type for both methods is Collection. The Javadocs say: 
"Note that primary node is always first in the returned collection".
In general, the notion of "order" and "first element" is not defined for a 
Collection (we could play with iterators, but its awkward from both code and 
logic perspective). It makes more sense for this methods to return List instead.

The task is to introduce List equivalents of these methods. It would also be 
nice to check if other APIs have similar issues.

The tricky part is compatibility.
Changing return type from Collection to List is source compatible (meaning the 
old client code will compile with the new method), but not binary compatible 
(meaning the already compiled client code will break because the signature has 
changed).
A compatible solution would be to
- deprecate the existing methods
- add methods with different names, like mapPartitionToPrimaryAndBackups*List*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10152) Allow warming up data region before a node has joined

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10152:
---

 Summary: Allow warming up data region before a node has joined
 Key: IGNITE-10152
 URL: https://issues.apache.org/jira/browse/IGNITE-10152
 Project: Ignite
  Issue Type: New Feature
Reporter: Stanislav Lukyanov


After a node with existing PDS joins the cluster its performance will be low 
for a while - until it loads all data into RAM. This is a feature of Ignite - 
cold restart - that allows nodes to start as fast as possible.
However, it would be good to also allow "hot restart" - a way to preload as 
much data into RAM as needed before the node has joined. Then the node will not 
slow the cluster down during its warm-up.

There are several recent improvements with similar goals:
- IGNITE-8873 added methods to preload data of a specific partition
- IGNITE-7196 and IGNITE-9420 moved data recovery to before node joins the 
cluster

Combining the above approaches we could have allow a user to trigger 
synchronous preloading right after the data recovery but before discovery 
starts up.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10150) DisappearedCacheWasNotFoundMessageSelfTest fails with NPE

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10150:
---

 Summary: DisappearedCacheWasNotFoundMessageSelfTest fails with NPE
 Key: IGNITE-10150
 URL: https://issues.apache.org/jira/browse/IGNITE-10150
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


https://ci.ignite.apache.org/viewLog.html?buildId=2226328=buildResultsDiv=IgniteTests24Java8_Queries2

{code}
java.lang.NullPointerException
at 
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.nodesForPartitions(GridReduceQueryExecutor.java:1702)
at 
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:690)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$8.iterator(IgniteH2Indexing.java:1801)
at 
org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator(QueryCursorImpl.java:95)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$9.iterator(IgniteH2Indexing.java:1869)
at 
org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator(QueryCursorImpl.java:95)
at 
org.apache.ignite.internal.processors.cache.QueryCursorImpl.getAll(QueryCursorImpl.java:114)
at 
org.apache.ignite.internal.processors.query.h2.twostep.DisappearedCacheWasNotFoundMessageSelfTest.testDisappearedCacheWasNotFoundMessage(DisappearedCacheWasNotFoundMessageSelfTest.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2206)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:144)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:2121)
at java.lang.Thread.run(Thread.java:748)
{code]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10067) Tracking progress of index building

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10067:
---

 Summary: Tracking progress of index building
 Key: IGNITE-10067
 URL: https://issues.apache.org/jira/browse/IGNITE-10067
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


When a new index is being created or indexes are being rebuilt in case a node 
is restarted without index.bin, the performance of queries is low until the 
index is ready. It would be good to track the process of indexing to have an 
approximate completion time.

Currently we only have "started" and "finished" messages in the logs for each 
index.
It would be good to also know
- All indexes to be built
- Number (+ percentage?) of processed entries (or pages, if it's hard to update 
progress for each entry)
- Number (+ percentage?) of entries left 

It seems reasonable to start collecting this data and just print it in the logs 
for now. `metricsLogFrequency` could be reused as the frequency of logging. 
After community has some experience with using this, this metrics could be 
exposed via Java API/JMX/etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10059) Local scan query against LOST partition fails

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10059:
---

 Summary: Local scan query against LOST partition fails
 Key: IGNITE-10059
 URL: https://issues.apache.org/jira/browse/IGNITE-10059
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


Local scan queries executed against a LOST partition always fail.
Stack trace:
{code}
javax.cache.CacheException: No queryable nodes for partition 0 [forced local 
query=GridCacheQueryAdapter [type=SCAN, clsName=null, clause=null, filter=null, 
transform=null, part=0, incMeta=false, metrics=GridCacheQueryMetricsAdapter 
[minTime=9223372036854775807, maxTime=0, sumTime=0, avgTime=0.0, execs=0, 
completed=0, fails=0], pageSize=1024, timeout=0, incBackups=false, 
forceLocal=true, dedup=false, 
prj=org.apache.ignite.internal.cluster.ClusterGroupAdapter@3a806080, 
keepBinary=false, subjId=null, taskHash=0, mvccSnapshot=null]]

at 
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:697)
at 
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.query(GatewayProtectedCacheProxy.java:376)
at 
org.apache.ignite.internal.processors.cache.distributed.IgniteCachePartitionLossPolicySelfTest.checkQueryPasses(IgniteCachePartitionLossPolicySelfTest.java:767)
at 
org.apache.ignite.internal.processors.cache.distributed.IgniteCachePartitionLossPolicySelfTest.validateQuery0(IgniteCachePartitionLossPolicySelfTest.java:720)
at 
org.apache.ignite.internal.processors.cache.distributed.IgniteCachePartitionLossPolicySelfTest.validateQuery(IgniteCachePartitionLossPolicySelfTest.java:691)
at 
org.apache.ignite.internal.processors.cache.distributed.IgniteCachePartitionLossPolicySelfTest.checkLostPartition(IgniteCachePartitionLossPolicySelfTest.java:497)
at 
org.apache.ignite.internal.processors.cache.distributed.IgniteCachePartitionLossPolicySelfTest.testReadOnlyAll(IgniteCachePartitionLossPolicySelfTest.java:185)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2209)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:144)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:2124)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteCheckedException: No queryable nodes 
for partition 0 [forced local query=GridCacheQueryAdapter [type=SCAN, 
clsName=null, clause=null, filter=null, transform=null, part=0, incMeta=false, 
metrics=GridCacheQueryMetricsAdapter [minTime=9223372036854775807, maxTime=0, 
sumTime=0, avgTime=0.0, execs=0, completed=0, fails=0], pageSize=1024, 
timeout=0, incBackups=false, forceLocal=true, dedup=false, 
prj=org.apache.ignite.internal.cluster.ClusterGroupAdapter@3a806080, 
keepBinary=false, subjId=null, taskHash=0, mvccSnapshot=null]]
at 
org.apache.ignite.internal.processors.cache.query.GridCacheQueryAdapter.executeScanQuery(GridCacheQueryAdapter.java:541)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl$1.applyx(IgniteCacheProxyImpl.java:410)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl$1.applyx(IgniteCacheProxyImpl.java:408)
at 
org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:36)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:2713)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:407)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:689)
... 15 more
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10058) resetLostPartitions() leaves an additional copy of a partition in the cluster

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10058:
---

 Summary: resetLostPartitions() leaves an additional copy of a 
partition in the cluster
 Key: IGNITE-10058
 URL: https://issues.apache.org/jira/browse/IGNITE-10058
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


If there are several copies of a LOST partition, resetLostPartitions() will 
leave all of them in the cluster as OWNING.

Scenario:
1) Start 4 nodes, a cache with backups=0 and READ_WRITE_SAFE, fill the cache
2) Stop one node - some partitions are recreated on the remaining nodes as LOST
3) Start one node - the LOST partitions are being rebalanced to the new node 
from the existing ones
4) Wait for rebalance to complete
5) Call resetLostPartitions()
After that the partitions that were LOST become OWNING on all nodes that had 
them. Eviction of these partitions doesn't start.

Need to correctly evict additional copies of LOST partitions either after 
rebalance on step 4 or after resetLostPartitions() call on step 5.
Current resetLostPartitions() implementation does call checkEvictions(), but 
the ready affinity assignment contains several nodes per partition for some 
reason.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10057) SQL queries hang during rebalance if there are LOST partitions

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10057:
---

 Summary: SQL queries hang during rebalance if there are LOST 
partitions
 Key: IGNITE-10057
 URL: https://issues.apache.org/jira/browse/IGNITE-10057
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Stanislav Lukyanov


When there are both LOST and MOVING partitions in the cluster, SQL queries will 
hang until rebalance has completed and there aren't any MOVING partitions. 
During that time the reducer node keeps printing
{code}
[2018-10-30 10:26:06,787][INFO 
][test-runner-#1%cache.IndexingCachePartitionLossPolicySelfTest%][GridReduceQueryExecutor]
 Failed to calculate nodes for SQL query (partition has no owners, but 
corresponding cache group has data nodes) [qryId=13, cacheIds=[1254100233], 
cacheName=partitioned, cacheId=1254100233, part=0, cacheGroupId=1254100233]
[2018-10-30 10:26:06,798][INFO 
][test-runner-#1%cache.IndexingCachePartitionLossPolicySelfTest%][GridReduceQueryExecutor]
 Failed to calculate nodes for SQL query (partition has no owners, but 
corresponding cache group has data nodes) [qryId=14, cacheIds=[1254100233], 
cacheName=partitioned, cacheId=1254100233, part=0, cacheGroupId=1254100233]
[2018-10-30 10:26:06,818][INFO 
][test-runner-#1%cache.IndexingCachePartitionLossPolicySelfTest%][GridReduceQueryExecutor]
 Failed to calculate nodes for SQL query (partition has no owners, but 
corresponding cache group has data nodes) [qryId=15, cacheIds=[1254100233], 
cacheName=partitioned, cacheId=1254100233, part=0, cacheGroupId=1254100233]
[2018-10-30 10:26:06,849][INFO 
][test-runner-#1%cache.IndexingCachePartitionLossPolicySelfTest%][GridReduceQueryExecutor]
 Failed to calculate nodes for SQL query (partition has no owners, but 
corresponding cache group has data nodes) [qryId=16, cacheIds=[1254100233], 
cacheName=partitioned, cacheId=1254100233, part=0, cacheGroupId=1254100233]
[2018-10-30 10:26:06,889][INFO 
][test-runner-#1%cache.IndexingCachePartitionLossPolicySelfTest%][GridReduceQueryExecutor]
 Failed to calculate nodes for SQL query (partition has no owners, but 
corresponding cache group has data nodes) [qryId=17, cacheIds=[1254100233], 
cacheName=partitioned, cacheId=1254100233, part=0, cacheGroupId=1254100233]
[2018-10-30 10:26:06,940][INFO 
][test-runner-#1%cache.IndexingCachePartitionLossPolicySelfTest%][GridReduceQueryExecutor]
 Failed to calculate nodes for SQL query (partition has no owners, but 
corresponding cache group has data nodes) [qryId=18, cacheIds=[1254100233], 
cacheName=partitioned, cacheId=1254100233, part=0, cacheGroupId=1254100233]
[2018-10-30 10:26:07,001][INFO 
][test-runner-#1%cache.IndexingCachePartitionLossPolicySelfTest%][GridReduceQueryExecutor]
 Failed to calculate nodes for SQL query (partition has no owners, but 
corresponding cache group has data nodes) [qryId=19, cacheIds=[1254100233], 
cacheName=partitioned, cacheId=1254100233, part=0, cacheGroupId=1254100233]
[2018-10-30 10:26:07,072][INFO 
][test-runner-#1%cache.IndexingCachePartitionLossPolicySelfTest%][GridReduceQueryExecutor]
 Failed to calculate nodes for SQL query (partition has no owners, but 
corresponding cache group has data nodes) [qryId=20, cacheIds=[1254100233], 
cacheName=partitioned, cacheId=1254100233, part=0, cacheGroupId=1254100233]
[2018-10-30 10:26:07,152][INFO 
][test-runner-#1%cache.IndexingCachePartitionLossPolicySelfTest%][GridReduceQueryExecutor]
 Failed to calculate nodes for SQL query (partition has no owners, but 
corresponding cache group has data nodes) [qryId=21, cacheIds=[1254100233], 
cacheName=partitioned, cacheId=1254100233, part=0, cacheGroupId=1254100233]
[2018-10-30 10:26:07,243][INFO 
][test-runner-#1%cache.IndexingCachePartitionLossPolicySelfTest%][GridReduceQueryExecutor]
 Failed to calculate nodes for SQL query (partition has no owners, but 
corresponding cache group has data nodes) [qryId=22, cacheIds=[1254100233], 
cacheName=partitioned, cacheId=1254100233, part=0, cacheGroupId=1254100233]
[2018-10-30 10:26:07,343][INFO 
][test-runner-#1%cache.IndexingCachePartitionLossPolicySelfTest%][GridReduceQueryExecutor]
 Failed to calculate nodes for SQL query (partition has no owners, but 
corresponding cache group has data nodes) [qryId=23, cacheIds=[1254100233], 
cacheName=partitioned, cacheId=1254100233, part=0, cacheGroupId=1254100233]
[2018-10-30 10:26:07,454][INFO 
][test-runner-#1%cache.IndexingCachePartitionLossPolicySelfTest%][GridReduceQueryExecutor]
 Failed to calculate nodes for SQL query (partition has no owners, but 
corresponding cache group has data nodes) [qryId=24, cacheIds=[1254100233], 
cacheName=partitioned, cacheId=1254100233, part=0, cacheGroupId=1254100233]
[2018-10-30 10:26:07,575][INFO 
][test-runner-#1%cache.IndexingCachePartitionLossPolicySelfTest%][GridReduceQueryExecutor]
 Failed to calculate nodes for SQL query (partition has no owners, but 

[jira] [Created] (IGNITE-10044) LOST partition is marked as OWNING after the owner rejoins with existing persistent data

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10044:
---

 Summary: LOST partition is marked as OWNING after the owner 
rejoins with existing persistent data
 Key: IGNITE-10044
 URL: https://issues.apache.org/jira/browse/IGNITE-10044
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


When persistence is enabled LOST partition may become OWNING without a call to 
resetLostPartitions.

If a partition is marked as LOST, and a node with that partition in persistence 
joins, the partition becomes OWNING. IgniteCache::lostPartitions doesn't return 
that partition anymore.

Apparently, it only affects the lost partitions list, while needsRecovery flag 
stays set until resetLostPartitions is called.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10043) Lost partitions list is reset when only one server is alive in the cluster

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10043:
---

 Summary: Lost partitions list is reset when only one server is 
alive in the cluster
 Key: IGNITE-10043
 URL: https://issues.apache.org/jira/browse/IGNITE-10043
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


When server nodes are killed one by one IgniteCache.lostPartitions() returns 
non-empty list up until there is just one node left. When only one node is 
alive, the lostPartitions() returns an empty collection even though the 
partition loss events has been fired and the partitions are missing.

This only affects cluster with persistence enabled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-10041) Partition loss policies work incorrectly with BLT

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-10041:
---

 Summary: Partition loss policies work incorrectly with BLT
 Key: IGNITE-10041
 URL: https://issues.apache.org/jira/browse/IGNITE-10041
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


Current partition loss handling (initially designed before persistence and BLT 
were introduced) aligns poorly with BLT semantics. The general problem is that 
partition loss handling assumes that rebalancing always happens on topology 
change - which is not the case with BLT.

The features with semantics conflicting with BLT are:
- PartitionLossPolicy.READ_ONLY_ALL and PartitionLossPolicy.READ_WRITE_ALL
READ_*_ALL policies assume that there is "some" partition created in the 
cluster which we could read or write, even though it is marked as LOST. With 
BLT the LOST partition only exists on an offline node, so there is no place to 
route the requests to.

- PartitionLossPolicy.IGNORE
Somewhat similar to the READ_*_ALL, but IGNORE also expects that the lost 
partition is immediately recreated on another node - which is impossible 
without changing affinity, i.e. without resetting BLT.

- IgniteCache::withPartitionRecover
withPartitionRecover is supposed to ignore partition loss policies and allow 
all operations, but when a partition is lost in a BLT cluster it's actually 
unavailable, there is straightforward way to ignore it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9964) SQL: query by affinity key fails when a table is created from a custom template

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-9964:
--

 Summary: SQL: query by affinity key fails when a table is created 
from a custom template
 Key: IGNITE-9964
 URL: https://issues.apache.org/jira/browse/IGNITE-9964
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


SELECT by affinity key doesn't work on a table that was created with a custom 
cache template if entries were added through JCache (using withKeepBinary()).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9902) ScanQuery doesn't take lost partitions into account when persistence is enabled

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-9902:
--

 Summary: ScanQuery doesn't take lost partitions into account when 
persistence is enabled
 Key: IGNITE-9902
 URL: https://issues.apache.org/jira/browse/IGNITE-9902
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


Same as IGNITE-9841, but about scan queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9900) Upgrade annotations.jar to a new version

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-9900:
--

 Summary: Upgrade annotations.jar to a new version
 Key: IGNITE-9900
 URL: https://issues.apache.org/jira/browse/IGNITE-9900
 Project: Ignite
  Issue Type: Improvement
Reporter: Stanislav Lukyanov
Assignee: Stanislav Lukyanov


OWASP Dependency Check reports that annotations.jar of version 13.0 is affected 
by vulnerability CVE-2017-8316,
while it obviously isn't (the CVE is about XXE attack, and annotations.jar is, 
well, annotations).

The issue is that NVD database only says that "intellij_idea" is affected, and 
OWASP doesn't know better than to map annotations.jar to the intellij_idea 
product.

While the problem could be (and perhaps will be, see 
https://youtrack.jetbrains.com/issue/IDEA-200601) sorted out on the level of 
OWASP and NVD, the easiest way to workaround this is to upgrade annotations.jar 
to a new version (currently 16.0.3). It will help Ignite users to avoid 
annoying false-positive warnings from OWASP.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9841) SQL take lost partitions into account when persistence is enabled

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-9841:
--

 Summary: SQL take lost partitions into account when persistence is 
enabled
 Key: IGNITE-9841
 URL: https://issues.apache.org/jira/browse/IGNITE-9841
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Stanislav Lukyanov
 Attachments: IgniteSqlQueryWithLostPartitionsTest.java

IGNITE-8927 changed SQL queries to honor lost partitions. However, it doesn't 
work if persistence is enabled. E.g. if there are lost partitions then `select 
* from T` fails for in-memory caches (good) and silently ignores the lost 
partitions for persistent caches (bad).

See attached reproducer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9825) Need test for cache writes on unstable topology

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-9825:
--

 Summary: Need test for cache writes on unstable topology
 Key: IGNITE-9825
 URL: https://issues.apache.org/jira/browse/IGNITE-9825
 Project: Ignite
  Issue Type: Test
Reporter: Stanislav Lukyanov
Assignee: Stanislav Lukyanov


It appears that ignite-core test suites have no tests for a simple case - cache 
writes on unstable topology.

Need to add a simple test covering this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9651) Binary objects comparison may fail when the same reference occurs more than once

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-9651:
--

 Summary: Binary objects comparison may fail when the same 
reference occurs more than once
 Key: IGNITE-9651
 URL: https://issues.apache.org/jira/browse/IGNITE-9651
 Project: Ignite
  Issue Type: Bug
  Components: binary
Reporter: Stanislav Lukyanov
 Attachments: RepeatingFieldSqlTest.java, RepeatingFieldTest.java

If a composite object contains multiple occurrences of the same POJO that POJO 
will be properly serialized only the first time, and all other references will 
refer to that first object. Because of that the same object may have different 
views in binary format when serialized on its own and when serialized as a part 
of another object.

Having multiple binary views of the same object leads to several issues with 
binary objects comparison. Examples in pseudocode are below, JUnit tests are 
attached


Example 1 (RepeatingFieldTest.java)
{code}
class Key { int i, Pojo pojo1, Pojo pojo2 }
class Pojo { int i }

cache.put(new Key(42, new Pojo(42), new Pojo(42)), 42);

// new Pojo is created for both fields - works
assertEquals(INT, cache.get(new Key(42, new Pojo(42), new Pojo(42;

// same Pojo is used for both fields - fails
// this is because the second field is serialized as a reference to the first 
one, and binary representation differs
Pojo obj = new Pojo(42);
assertEquals(42, cache.get(new Key(42, obj, obj)));
{code}


Example 2 (RepeatingFieldSqlTest.java)

{code}
class Pojo { int i }
class Key { int i, Pojo pojo }
class Value { Pojo pojo, Key key }

Pojo obj = new Pojo(INT);
Key key = new Key(INT, obj);
Value val = new Value(key, obj);
cache.put(key, val);

// supposed to return 1 row, but returns 0
SqlFieldsQuery qry = new SqlFieldsQuery("select * from Value where _key = key");

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9526) Allow explicitly setting index inline size for Java objects

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-9526:
--

 Summary: Allow explicitly setting index inline size for Java 
objects
 Key: IGNITE-9526
 URL: https://issues.apache.org/jira/browse/IGNITE-9526
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Stanislav Lukyanov
Assignee: Stanislav Lukyanov


Currently Ignite never inlines indexed objects when they are POJOs. It makes 
sense, as there is currently no way of inlining a POJO other than by inlining 
the binary object, and the headers of the binary objects (i.e. their first 24 
bytes) are usually the same for the same indexed field.

However, this approach has downsides.
Silently ignoring the specified inline size might be surprising for a user 
willing to trade (a lot of) memory for speed and setting a large inline size 
for that purpose.
On the other hand, inlining headers and just several first bytes of POJOs in 
binary format has proven to be beneficial. It is especially beneficial in case 
page replacements are happening, as this greatly reduces the number of data 
pages required to traverse an index tree.

The proposal is to allow POJO inlining in case the inline size is given for the 
specific POJO field. So, unlike strings and byte arrays, POJOs will not use the 
max inline size by default - as in most cases inlining POJOs isn't going to 
help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9331) Add thin client tests for custom TLS protocols

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-9331:
--

 Summary: Add thin client tests for custom TLS protocols
 Key: IGNITE-9331
 URL: https://issues.apache.org/jira/browse/IGNITE-9331
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


IGNITE-6167 added an ability to specify a custom TLS protocol for Ignite 
connections. It includes tests for regular Ignite node and Java Thin client.

Need to add tests for other thin clients: JDBC, ODBC, .Net, Node.js, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9315) Eviction meta from near cache may appear in DHT entries

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-9315:
--

 Summary: Eviction meta from near cache may appear in DHT entries
 Key: IGNITE-9315
 URL: https://issues.apache.org/jira/browse/IGNITE-9315
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov
Assignee: Stanislav Lukyanov


There are cases when eviction metadata from near cache's eviction policy may be 
added to a DHT cache entry, leading to unpredictable behavior.

The bug is caused by coding mistakes when a DHT entry is being updated, but 
near cache entry is being touched.
E.g. in GridDhtPartitionDemander::preloadEntry:
{code}
GridCacheContext cctx = grp.sharedGroup() ? ctx.cacheContext(entry.cacheId()) : 
grp.singleCacheContext();

cached = cctx.dhtCache().entryEx(entry.key());
//cctx = cached.context();

if (log.isDebugEnabled())
log.debug("Rebalancing key [key=" + entry.key() + ", part=" + p + ", node=" 
+ from.id() + ']');

if (preloadPred == null || preloadPred.apply(entry)) {
if (cached.initialValue(
entry.value(),
entry.version(),
entry.ttl(),
entry.expireTime(),
true,
topVer,
cctx.isDrEnabled() ? DR_PRELOAD : DR_NONE,
false
)) {
cctx.evicts().touch(cached, topVer); // Start tracking.
{code}

Here `cached` is always DHT entry, but `cctx` can be near.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9247) CPP Thin: implement GetAll

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-9247:
--

 Summary: CPP Thin: implement GetAll
 Key: IGNITE-9247
 URL: https://issues.apache.org/jira/browse/IGNITE-9247
 Project: Ignite
  Issue Type: New Feature
  Components: thin client
Reporter: Stanislav Lukyanov


Need to implement GetAll in C++ Thin client.

Currently, there is no way to extract values from cache via C++ Thin client 
without knowing the keys beforehand. GetAll would be the easiest way to do that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9182) Propagate lazy flag to synthetic SELECT used by first step of DELETE

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-9182:
--

 Summary: Propagate lazy flag to synthetic SELECT used by first 
step of DELETE
 Key: IGNITE-9182
 URL: https://issues.apache.org/jira/browse/IGNITE-9182
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Stanislav Lukyanov


When DELETE is executed it first executed SELECT with the same WHERE, and then 
deletes all results via an invoke(). If a large number of entries is to be 
deleted, it may lead to OOM.

The task is to allow to perform the SELECT with the parent query's LAZY flag.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8952) Move CacheStore::loadCache to a separate interface

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-8952:
--

 Summary: Move CacheStore::loadCache to a separate interface
 Key: IGNITE-8952
 URL: https://issues.apache.org/jira/browse/IGNITE-8952
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


3rd party persistence has three way of interacting with the backing storage:
- JCache's CacheLoader methods for read-through
- JCache's CacheWriter methods for write-through
- Ignite's CacheStore::loadCache for bulk cache preloading

Usually, one just uses a CacheStore implementation and provides a no-op 
implementation for the methods that are not needed (e.g. no-op implementation 
for write() if write-through isn't used). However, this adds certain redundancy.

In particular, if a user only wants cache preloading (loadCache()) but no 
read-through or write-through, they will need to provide a bunch of these no-op 
methods, which can seem messy.

Instead, it would be nice to move loadCache() method to a separate interface, 
say CachePreloader. This way we'll have 4 interfaces:
- JCache's CacheLoader for read-through
- JCache's CacheWriter for write-through
- Ignite's CachePreloader for loadCache()
- Ignite's CacheStore for all three together

One can either choose any combination of loader, writer and preloader, OR set 
the store.

The task is to
- extract CacheStore::loadCache into a new interface CachePreloader 
- add CacheConfiguration::setCachePreloader
- make sure that setCachePreloader works alone or in combination with loader 
and writer, and doesn't work with store



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8776) Eviction policy MBeans are never registered if evictionPolicyFactory is used

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-8776:
--

 Summary: Eviction policy MBeans are never registered if 
evictionPolicyFactory is used
 Key: IGNITE-8776
 URL: https://issues.apache.org/jira/browse/IGNITE-8776
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.5
Reporter: Stanislav Lukyanov


Eviction policy MBeans, such as LruEvictionPolicyMBean, are never registered if 
evictionPolicyFactory is set instead of evictionPolicy (the latter is 
deprecated).

This happens because GridCacheProcessor::registerMbean attempts to find either 
an *MBean interface or IgniteMBeanAware interface on the passed object. It 
works for LruEvictionPolicy but not for LruEvictionPolicyFactory (which doesn't 
implement these interfaces).

The code needs to be adjusted to handle factories correctly.
New tests are needed to make sure that all standard beans are registered 
(IgniteKernalMbeansTest does that for kernal mbeans - need the same for cache 
beans).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8774) Daemon moves cluster to compatibility mode when joins

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-8774:
--

 Summary: Daemon moves cluster to compatibility mode when joins
 Key: IGNITE-8774
 URL: https://issues.apache.org/jira/browse/IGNITE-8774
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


When a daemon node joins the cluster seems to switch to compatibility mode 
(allowing nodes without baseline support). It prevents baseline nodes from 
being restarted.

Example:
{code}
Ignite ignite1 = 
IgnitionEx.start("examples/config/persistentstore/example-persistent-store.xml",
 "srv1");
Ignite ignite2 = 
IgnitionEx.start("examples/config/persistentstore/example-persistent-store.xml",
 "srv2");
ignite2.cluster().active(true);

IgnitionEx.setClientMode(true);
IgnitionEx.setDaemon(true);
Ignite daemon = 
IgnitionEx.start("examples/config/persistentstore/example-persistent-store.xml",
 "daemon");
IgnitionEx.setClientMode(false);
IgnitionEx.setDaemon(false);

ignite2.close();
IgnitionEx.start("examples/config/persistentstore/example-persistent-store.xml",
 "srv2");
{code}

The attempt to restart ignite2 throws an exception:
{code}
[2018-06-11 18:45:25,766][ERROR][tcp-disco-msg-worker-#39%srv2%][root] Critical 
system error detected. Will be handled accordingly to configured handler 
[hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, 
failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class 
o.a.i.IgniteException: Node with BaselineTopology cannot join mixed cluster 
running in compatibility mode]]
class org.apache.ignite.IgniteException: Node with BaselineTopology cannot join 
mixed cluster running in compatibility mode
at 
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.onGridDataReceived(GridClusterStateProcessor.java:714)
at 
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$5.onExchange(GridDiscoveryManager.java:883)
at 
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.onExchange(TcpDiscoverySpi.java:1939)
at 
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processNodeAddedMessage(ServerImpl.java:4354)
at 
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2744)
at 
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2536)
at 
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6775)
at 
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2621)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8726) Highlight that memory metrics are local for a node in the docs

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-8726:
--

 Summary: Highlight that memory metrics are local for a node in the 
docs
 Key: IGNITE-8726
 URL: https://issues.apache.org/jira/browse/IGNITE-8726
 Project: Ignite
  Issue Type: Improvement
  Components: documentation
Reporter: Stanislav Lukyanov


Memory Metrics (DataRegionMetrics and DataStorageMetrics) in Ignite are local 
for each node. However, this is not highlighted in the documentation enough. 
The code snippets suggest to just call `ignite.dataRegionMetrics()` which seems 
to be a bit at odds with the general use case of Ignite servers being started 
via ignite.sh.

It would be good to have an easily noticeable warning that the metrics will 
only return data for the local node (and that, for example, on client they 
would typically always print 0).

Also, would be nice to include a couple of practical approaches other than JMX 
to collect metrics. E.g. a snippet of client code getting metrics from all 
servers:
{code}
Collection metricsFromNodes = ignite.compute().broadcast(() -> {
Ignite ignite = Ignition.localIgnite();

StringBuilder sb = new StringBuilder();

sb.append("Node: " + ignite.name());

for (DataRegionMetrics metrics : ignite.dataRegionMetrics()) {
// append metrics to the builder 
}

return sb.toString();
});

for (String metricsString : metricsFromNodes)
System.out.println(metricsString);
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8627) Verify local partition map against global

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-8627:
--

 Summary: Verify local partition map against global
 Key: IGNITE-8627
 URL: https://issues.apache.org/jira/browse/IGNITE-8627
 Project: Ignite
  Issue Type: Improvement
Reporter: Stanislav Lukyanov
Assignee: Alexey Goncharuk


IGNITE-7467 introduced verification of partition update counters and sizes 
between replicas.

Similarly, it's possible to verify that local copy of the partition map doesn't 
differ from the map on the other nodes, detecting issues that may lead to 
inconsistent reads or data loss.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8611) Binary marshaller documentation should cover how data classes can or can't be changed

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-8611:
--

 Summary: Binary marshaller documentation should cover how data 
classes can or can't be changed
 Key: IGNITE-8611
 URL: https://issues.apache.org/jira/browse/IGNITE-8611
 Project: Ignite
  Issue Type: Improvement
  Components: documentation
Reporter: Stanislav Lukyanov


Binary marshaller docs (https://apacheignite.readme.io/docs/binary-marshaller) 
give an idea that a class structure may be changed (fields can be added or 
removed) and the marshaller will handle such change.

However, not all changes are supported.
One corner case is when an enum value is stored in the cache: if the order of 
the enum constants is changed, or if a new constant is added at the start or at 
the middle of the constants list, it will lead to an error. This is because the 
enums are stored as ordinals (integers), and the ordinals of an enum depend on 
the order of values in the code.

The task is to update the documentation with the description of data class 
changes that are incompatible from binary marshallers point of view. At least 
the enum case should be covered. If more cases are discovered, they should be 
documented as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8550) CacheAbstractJdbcStore expects merge to always return 1 but MySQL may also return 2 or 0

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-8550:
--

 Summary: CacheAbstractJdbcStore expects merge to always return 1 
but MySQL may also return 2 or 0
 Key: IGNITE-8550
 URL: https://issues.apache.org/jira/browse/IGNITE-8550
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Stanislav Lukyanov


CacheAbstractJdbcStore.write attempts to execute a merge update if it is 
available, and expects the merge to always return 1 (as the number of updated 
entries is always 1).

However, MySQL's `INSERT ... ON DUPLICATE KEY UPDATE` 
(https://dev.mysql.com/doc/refman/8.0/en/insert-on-duplicate.html) may return 0 
or 2, depending on what was updated:
{quote}With ON DUPLICATE KEY UPDATE, the affected-rows value per row is 1 if 
the row is inserted as a new row, 2 if an existing row is updated, and 0 if an 
existing row is set to its current values.{quote}
Because of that, CacheAbstractJdbcStore may report a false warning.

Need to consider either removing the warning or special-case the MySQL dialect 
to allow to return values other than 1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8486) Update ConcurrentLinkedDeque in Ignite's master repository to the latest JDK version

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-8486:
--

 Summary: Update ConcurrentLinkedDeque in Ignite's master 
repository to the latest JDK version
 Key: IGNITE-8486
 URL: https://issues.apache.org/jira/browse/IGNITE-8486
 Project: Ignite
  Issue Type: Improvement
Reporter: Stanislav Lukyanov
Assignee: Stanislav Lukyanov


Ignite still uses copies of several JSR 166 (j.u.concurrent) classes in it's 
sources. Those copies are now outdated compared to the latest versions used in 
JDK.
In particular, `ConcurrentLinkedDeque` has received a couple of correctness 
fixes recently (https://bugs.openjdk.java.net/browse/JDK-8188900, 
https://bugs.openjdk.java.net/browse/JDK-8189387). It would be good to have 
them in Ignite as well to protect ourselves from possible issues.

The task is to update Ignite's `ConcurrentLinkedDeque8` to the latest version 
of `ConcurrentLinkedDeque`, although keeping compatibility with earlier Java 
version (e.g. JDK's version now uses Java 9's VarHandles which we can't use 
yet).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8470) NPE when starting LOCAL cache on a client with no data regions

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-8470:
--

 Summary: NPE when starting LOCAL cache on a client with no data 
regions
 Key: IGNITE-8470
 URL: https://issues.apache.org/jira/browse/IGNITE-8470
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov


If a LOCAL cache is started on a client and the client has no data regions 
configured then an NPE is thrown with no error message:

{code}
class org.apache.ignite.IgniteCheckedException: null
at 
org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7284)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:259)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:232)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:159)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:151)
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.onKernalStart(GridCachePartitionExchangeManager.java:632)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStart(GridCacheProcessor.java:829)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1043)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1973)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1716)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1144)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:642)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:849)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:798)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapIndexScanTest.beforeTestsStarted(IgniteCacheOffheapIndexScanTest.java:78)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.setUp(GridAbstractTest.java:599)
at 
org.apache.ignite.testframework.junits.common.GridCommonAbstractTest.setUp(GridCommonAbstractTest.java:485)
at junit.framework.TestCase.runBare(TestCase.java:139)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at 
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at 
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.lang.NullPointerException
at 
org.apache.ignite.internal.processors.cache.CacheGroupContext.(CacheGroupContext.java:207)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCacheGroup(GridCacheProcessor.java:1983)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1881)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCachesOnLocalJoin(GridCacheProcessor.java:1773)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initCachesOnLocalJoin(GridDhtPartitionsExchangeFuture.java:784)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:666)
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2344)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
{code}

If assertions are enabled (-ea), an AssertionError is thrown instead:
{code}
java.lang.AssertionError
at 
org.apache.ignite.internal.processors.cache.CacheGroupContext.(CacheGroupContext.java:185)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.startCacheGroup(GridCacheProcessor.java:1983)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1881)
at 

[jira] [Created] (IGNITE-8210) Rebalancing sometimes is not triggered when a node is added to the baseline topology

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-8210:
--

 Summary: Rebalancing sometimes is not triggered when a node is 
added to the baseline topology
 Key: IGNITE-8210
 URL: https://issues.apache.org/jira/browse/IGNITE-8210
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Reporter: Stanislav Lukyanov


When a cluster already has baseline topology set and activated adding a new 
node to the baseline should lead to rebalancing. However, in some cases the new 
node will not receive any data when joined and added to the baseline; if a 
cluster is later deactivated and activated again, the rebalancing happens 
properly and partitions are assigned and transferred to the new node.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8134) Services can't be deployed on servers outside of baseline topology

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-8134:
--

 Summary: Services can't be deployed on servers outside of baseline 
topology
 Key: IGNITE-8134
 URL: https://issues.apache.org/jira/browse/IGNITE-8134
 Project: Ignite
  Issue Type: Bug
  Components: managed services, persistence
Reporter: Stanislav Lukyanov
Assignee: Stanislav Lukyanov


If a node is not a part of the baseline topology, the services will never be 
deployed on it. In particular, if that node calls a synchronous deploy* method, 
the method will hang.
After the node is added to the baseline, all previously initiated deployments 
succeed (and deploy* methods return).

It seems that the issue is with the continuous query started by the 
GridServiceProcessor on the ignite-sys-cache.

Example:
=
public class BltServicesBug {
public static void main(String[] args) {
// start one node
IgniteConfiguration cfg1 = new IgniteConfiguration()
.setIgniteInstanceName("node1")
.setDataStorageConfiguration(
new DataStorageConfiguration()
.setDefaultDataRegionConfiguration(
new DataRegionConfiguration()
.setPersistenceEnabled(true)
)
);
try (Ignite ignite1 = Ignition.start(cfg1)) {

// activate and set baseline topology
ignite1.cluster().active(true);

// start another node
IgniteConfiguration cfg2 = new IgniteConfiguration(cfg1)
.setIgniteInstanceName("node2");
try (Ignite ignite2 = Ignition.start(cfg2)) {
// try to deploy a service;
// this call hangs until the second node is added to the BLT 
(e.g. externally via control.sh)
ignite2.services().deployNodeSingleton("myService", new 
MyServiceImpl());

System.out.println("> Deployed");
}
}

}

private static class MyServiceImpl implements Service {
@Override public void cancel(ServiceContext ctx) { }
@Override public void init(ServiceContext ctx) { }
@Override public void execute(ServiceContext ctx) { }
}
}
=



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8133) Baseline topology documentation improvement

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-8133:
--

 Summary: Baseline topology documentation improvement
 Key: IGNITE-8133
 URL: https://issues.apache.org/jira/browse/IGNITE-8133
 Project: Ignite
  Issue Type: Bug
  Components: documentation
Affects Versions: 2.4
Reporter: Stanislav Lukyanov
Assignee: Stanislav Lukyanov


Baseline topology concept was added to Ignite in 2.4 by IEP-4. This changed 
Ignite cluster behavior when persistence is enabled (first of all, activation 
and rebalancing timings).
It seems that the current documentation may be confusing.

For example, the sentence
{quote}Note that the baseline topology is not set when the cluster is started 
for the first time; that's the only time when a manual intervention is 
needed.{quote}
may lead one to thinking that baseline topology is not used by default and 
needs to be enabled only if one wants to use it.

Also, the documentation describes the tools and commands that are used to 
manage the baseline topology and activation, but doesn't give guidelines on 
which nodes should be in the topology, when should it be changed, etc.

The documentation should be enhanced to
- give clear understanding that baseline topology always needs to be considered 
as a part of the cluster architecture when persistence is enabled;
- provide overview of the behavioral changes compared to AI 2.3 (use a 
note/warning block for that to separate it from the main text?);
- provide basic guidelines and suggestions of how one can start a new cluster 
and manage it (when to activate/deactivate, when to change baseline topology, 
what happens and what needs to be done when a node fails or joins, how to use 
consistentId)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8023) Processors are incorrectly initialized if during a node's reconnection cluster changes its state

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-8023:
--

 Summary: Processors are incorrectly initialized if during a node's 
reconnection cluster changes its state
 Key: IGNITE-8023
 URL: https://issues.apache.org/jira/browse/IGNITE-8023
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.4, 2.3, 2.5
Reporter: Stanislav Lukyanov
Assignee: Stanislav Lukyanov


If a node reconnects during the cluster activation process then some of the 
GridProcessor instances of that node will be incorrectly initialized. While 
GridClusterStateProcessor will correctly report the active cluster state, other 
processors that are sensitive to the cluster state, e.g. GridServiceProcessor, 
will be not initialized.

 

This is a problem similar to IGNITE-7753, except that the timing (and the code 
that should handle this) is different.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7908) NULLS LAST works incorrectly for PARTITIONED caches

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-7908:
--

 Summary: NULLS LAST works incorrectly for PARTITIONED caches
 Key: IGNITE-7908
 URL: https://issues.apache.org/jira/browse/IGNITE-7908
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Stanislav Lukyanov


When a `SELECT  ORDER BY ... NULLS LAST` query is executed on a partitioned 
cache, the null values at the end of the result are interleaved with non-null 
values. 
It seems that for each node participating in the query all null values that are 
stored on that node are returned immediately after the last non-null value. As 
a result, non-null values are correctly ordered and null values appear in a few 
"portions" (the number of "portions" is equal to the number of nodes) among the 
last values.

Example:
 
{code:java}
public class NullsLastBug {
public static void main(String[] args) throws Exception {
try (
Ignite srv1 = 
IgnitionEx.start("examples/config/example-ignite.xml", "server-1");
Ignite srv2 = 
IgnitionEx.start("examples/config/example-ignite.xml", "server-2");
Ignite srv3 = 
IgnitionEx.start("examples/config/example-ignite.xml", "server-3")
) {
Ignition.setClientMode(true);
try (Ignite client = 
IgnitionEx.start("examples/config/example-ignite.xml", "client")) {
IgniteCache cache = client.getOrCreateCache(
new CacheConfiguration("cache")
.setCacheMode(CacheMode.PARTITIONED)
.setIndexedTypes(Long.class, Box.class)
);

for (long i = 0; i < 30; i++) {
Long num = (i % 2 != 0) ? i : null;
cache.put(i, new Box(num));
}

SqlFieldsQuery selectAll = new SqlFieldsQuery("SELECT num FROM 
Box ORDER BY num NULLS LAST");
for (List list : cache.query(selectAll))
System.out.println(list);
}
}
}
}

class Box implements Serializable {
@QuerySqlField(index = true) private Long num;

Box(Long num) {
this.num = num;
}
}
{code}
 

Output:
 
{code}
 [1]
 [3]
 [5]
 [7]
 [9]
 [11]
 [13]
 [15]
 [17]
 [19]
 [21]
 [23]
 [null]
 [null]
 [25]
 [27]
 [null]
 [null]
 [null]
 [null]
 [null]
 [null]
 [null]
 [null]
 [29]
 [null]
 [null]
 [null]
 [null]
 [null]
{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7904) ComputeTaskFuture.get() throws incorrect exception if ComputeTask.result()

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-7904:
--

 Summary: ComputeTaskFuture.get() throws incorrect exception if 
ComputeTask.result()
 Key: IGNITE-7904
 URL: https://issues.apache.org/jira/browse/IGNITE-7904
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov
Assignee: Stanislav Lukyanov


ComputeTask.result() javadoc says: "Throws: IgniteException - If handling a job 
result caused an error effectively rejecting a failover. This exception will be 
thrown out of ComputeTaskFuture.get() method."

However, GridFutureAdapter calls IgniteUtils.cast(Throwable) on the exception 
before throwing it from get(), and the latter method trims the stack trace to 
the first occurence of an IgniteCheckedException. Because of that, get() throws 
not the IgniteException thrown from the ComputeTask.result() but one of its 
causes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7771) Names of Ignite JMX beans should not be quoted unless required

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-7771:
--

 Summary: Names of Ignite JMX beans should not be quoted unless 
required
 Key: IGNITE-7771
 URL: https://issues.apache.org/jira/browse/IGNITE-7771
 Project: Ignite
  Issue Type: Bug
Reporter: Stanislav Lukyanov
Assignee: Stanislav Lukyanov


Names of Ignite JMX beans and bean groups are currently quoted if they contain 
non-alphanumeric characters. This leads to names with spaces, e.g. Thread 
Pools, appearing as "Thread Pools". Moreover, Thread Pools and "Thread Pools" 
are recognized by JMX as distinct names, so code accessing that MBean needs to 
take that into account.

It would be better not to quote the bean and group names unless they contain 
specific control characters. That way users won't need to quote names in their 
JMX clients.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7753) Processors are incorrectly initialized if a node joins during cluster activation

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-7753:
--

 Summary: Processors are incorrectly initialized if a node joins 
during cluster activation
 Key: IGNITE-7753
 URL: https://issues.apache.org/jira/browse/IGNITE-7753
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.3, 2.4, 2.5
Reporter: Stanislav Lukyanov
Assignee: Stanislav Lukyanov


If a node joins during the cluster activation process (while the related 
exchange operation is in progress), then some of the GridProcessor instances of 
that node will be incorrectly initialized. While GridClusterStateProcessor will 
correctly report the active cluster state, other processors that are sensitive 
to the cluster state, e.g. GridServiceProcessor, will be not initialized.

A reproducer is below. 
===
Ignite server = 
IgnitionEx.start("examples/config/persistentstore/example-persistent-store.xml",
 "server");

CyclicBarrier barrier = new CyclicBarrier(2);
Thread activationThread = new Thread(() -> {
try {
barrier.await();
server.active(true);
}
catch (Exception e) {
e.printStackTrace(); // TODO implement.
}
});
activationThread.start();
barrier.await();

IgnitionEx.setClientMode(true);
Ignite client = 
IgnitionEx.start("examples/config/persistentstore/example-persistent-store.xml",
 "client");

activationThread.join();

client.services().deployClusterSingleton("myClusterSingleton", new 
SimpleMapServiceImpl<>());
===

Here a single server node is started, then simultaneously a client node is 
being started and the cluster is being activated, then client attempts to 
deploy a service. As the result, the thread calling the deploy method hangs 
forever with a stack trace like this:
===
"main@1" prio=5 tid=0x1 nid=NA waiting
  java.lang.Thread.State: WAITING
  at sun.misc.Unsafe.park(Unsafe.java:-1)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
  at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
  at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
  at 
org.apache.ignite.internal.util.IgniteUtils.awaitQuiet(IgniteUtils.java:7505)
  at 
org.apache.ignite.internal.processors.service.GridServiceProcessor.serviceCache(GridServiceProcessor.java:290)
  at 
org.apache.ignite.internal.processors.service.GridServiceProcessor.writeServiceToCache(GridServiceProcessor.java:728)
  at 
org.apache.ignite.internal.processors.service.GridServiceProcessor.deployAll(GridServiceProcessor.java:634)
  at 
org.apache.ignite.internal.processors.service.GridServiceProcessor.deployAll(GridServiceProcessor.java:600)
  at 
org.apache.ignite.internal.processors.service.GridServiceProcessor.deployMultiple(GridServiceProcessor.java:488)
  at 
org.apache.ignite.internal.processors.service.GridServiceProcessor.deployClusterSingleton(GridServiceProcessor.java:469)
  at 
org.apache.ignite.internal.IgniteServicesImpl.deployClusterSingleton(IgniteServicesImpl.java:120)
===

The behavior depends on the timings - the client has to join in the middle of 
the activation's exchange process. Putting Thread.sleep(4000) into 
GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest seems to work on a 
development laptop.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7687) SQL SELECT doesn't update TTL for Touched/AccessedExpiryPolicy

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-7687:
--

 Summary: SQL SELECT doesn't update TTL for 
Touched/AccessedExpiryPolicy
 Key: IGNITE-7687
 URL: https://issues.apache.org/jira/browse/IGNITE-7687
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.5
Reporter: Stanislav Lukyanov


SQL SELECT queries don't update TTLs when TouchedExpiryPolicy or 
AccessedExpiryPolicy is used (unlike IgniteCache::get which does update the 
TTLs).

Example (modified SqlDmlExample):

CacheConfiguration orgCacheCfg = new 
CacheConfiguration(ORG_CACHE)
.setIndexedTypes(Long.class, Organization.class)
.setExpiryPolicyFactory(TouchedExpiryPolicy.factoryOf(new 
Duration(TimeUnit.SECONDS, 10)));

IgniteCache orgCache = 
ignite.getOrCreateCache(orgCacheCfg);

SqlFieldsQuery qry = new SqlFieldsQuery("insert into Organization (_key, 
id, name) values (?, ?, ?)");
orgCache.query(qry.setArgs(1L, 1L, "ASF"));
orgCache.query(qry.setArgs(2L, 2L, "Eclipse"));

SqlFieldsQuery qry1 = new SqlFieldsQuery("select id, name from Organization 
as o");
for (int i = 0; ;i++) {
List res = orgCache.query(qry1).getAll();
print("i = " + i);
for (Object next : res)
System.out.println(">>> " + next);
U.sleep(5000);
}

Output:
>>> i = 0
>>> [1, ASF]
>>> [2, Eclipse]

>>> i = 1
>>> [1, ASF]
>>> [2, Eclipse]

>>> i = 2

>>> i = 3
...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7611) Document logger configuration options

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-7611:
--

 Summary: Document logger configuration options
 Key: IGNITE-7611
 URL: https://issues.apache.org/jira/browse/IGNITE-7611
 Project: Ignite
  Issue Type: Bug
  Components: documentation
Reporter: Stanislav Lukyanov
Assignee: Stanislav Lukyanov


Existing documentation on readme.io 
(https://apacheignite.readme.io/docs/logging) gives details on how to enable 
each of the supported logging frameworks, and some info on the default 
configuration (e.g. IGNITE_QUIET).

The documentation should also cover some other features of Ignite logging, such 
as recently added features of automatic reconfiguration of log4j 1.x and 2.x 
(IGNITE-6946) and DEV_ONLY marker in logging messages (IGNITE-7284), as well as 
other features like automatic metrics logging (MetricsLogFrequency) and 
performance suggestions on start (IGNITE_PERFORMANCE_SUGGESTIONS_DISABLED).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7508) GridKernalContextImpl::isDaemon creates contention on system properties access

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-7508:
--

 Summary: GridKernalContextImpl::isDaemon creates contention on 
system properties access
 Key: IGNITE-7508
 URL: https://issues.apache.org/jira/browse/IGNITE-7508
 Project: Ignite
  Issue Type: Bug
  Components: general
Reporter: Stanislav Lukyanov


GridKernalContextImpl::isDaemon reads system property IGNITE_DAEMON on every 
call, leading to contention on the system properties lock. The lock is shown as 
contended in the Java Mission Control analysis of a JFR recording of the 
IgnitePutGetBenchmark.

The fix would be to cache IGNITE_DAEMON value (e.g. in IgniteUtils) since it 
isn't supposed to be changed during the JVM's lifetime anyway.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7398) InstanceNotFoundException appears in the log when a cache is closed and MBeans are disabled

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-7398:
--

 Summary: InstanceNotFoundException appears in the log when a cache 
is closed and MBeans are disabled
 Key: IGNITE-7398
 URL: https://issues.apache.org/jira/browse/IGNITE-7398
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.4
Reporter: Stanislav Lukyanov
Priority: Minor


If IGNITE_MBEANS_DISABLED is set to true, whenever a cache is started the 
following error is logged:
javax.management.InstanceNotFoundException: 
org.apache:clsLdr=18b4aac2,igniteInstanceName=mbeans.GridMBeanDisableSelfTest0,group=Cache
 groups,name=MyCache
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
at 
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.cleanup(GridCacheProcessor.java:606)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.stopCacheGroup(GridCacheProcessor.java:2264)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.stopCacheGroup(GridCacheProcessor.java:2253)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.stopCaches(GridCacheProcessor.java:907)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.stop(GridCacheProcessor.java:872)
at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:2316)
at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:2182)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2512)
at 
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2475)
at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:362)
at org.apache.ignite.Ignition.stop(Ignition.java:224)
at org.apache.ignite.internal.IgniteKernal.close(IgniteKernal.java:3509)
at 
org.apache.ignite.util.mbeans.GridMBeanDisableSelfTest.testCorrectMBeanInfo(GridMBeanDisableSelfTest.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2001)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:133)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1916)
at java.lang.Thread.run(Thread.java:748)

That's because GridCacheProcessor::cleanup unconditionally attempts to 
unregister cache's MBean, not checking whether or not the cache actually has an 
MBean registered for it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7334) MBeans fail to be created for caches with special symbols in names

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-7334:
--

 Summary: MBeans fail to be created for caches with special symbols 
in names
 Key: IGNITE-7334
 URL: https://issues.apache.org/jira/browse/IGNITE-7334
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 2.4
Reporter: Stanislav Lukyanov
Assignee: Stanislav Lukyanov


If a cache contains a certain symbols, e.g. an asterisk, its creation will fail 
due to an exception during MBean creation. This is because of insufficient 
escaping in the shared MBean registration code in IgniteUtils.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)