Re:Deprecated GridSslContextFactory removal

[Taras Ledkov]

+1 for remove

Re:[DISCUSSION] Removal of ignitevisorcmd

[Taras Ledkov]

Hi,

+1 for remove Visor related code.
Unfortunately we have to migrate `control-utility` to use IgniteClient (thin 
client) before drop GridClient. I guess we will do it in the future.

Also, the old thin JDBC based on the GridClient (classes placed at the 
org.apache.ignite.internal.jdbc.*) must be removed.

[ANNOUNCE] Apache Ignite 2.14.0 Released

[Taras Ledkov]

The Apache Ignite Community is pleased to announce the release of
Apache Ignite 2.14.0.

Apache Ignite® is an in-memory computing platform for transactional,
analytical, and streaming workloads delivering in-memory speeds at
a petabyte scale.
https://ignite.apache.org

For the full list of changes, you can refer to the RELEASE_NOTES list
which is trying to catalogue the most significant improvements for
this version of the platform.
https://ignite.apache.org/releases/2.14.0/release_notes.html

Download the latest Ignite version from here:
https://ignite.apache.org/download.cgi

Please let us know if you encounter any problems:
https://ignite.apache.org/community/resources.html#ask

--
With best regards,
Taras Ledkov

[RESULT] [VOTE] Release Apache Ignite 2.14.0 RC0

[Taras Ledkov]

Dear community,

The release of Apache Ignite 2.14.0 RC0 has been accepted.

Vote result: The vote PASSES with six+1 votes (5 bindings), 
one 0 vote, 
and no -1 votes.

+1 votes:
- Alex Plehanov (binding)
- Pavel Tupitsyn (binding)
- Ivan Daschinsky (binding)
- Maxim Muzafarov (binding)
- Dmitriy Pavlov (binding)
- mwiesenberg 

0 votes:
- Ilya Kasnacheev

The new release will be announced soon.

--
With best regards,
Taras Ledkov

[VOTE] Release Apache Ignite 2.14.0 RC0

[Taras Ledkov]

Dear Community,
 
The release candidate is ready.

I have uploaded release candidate to
https://dist.apache.org/repos/dist/dev/ignite/2.14.0-rc0/
https://dist.apache.org/repos/dist/dev/ignite/packages_2.14.0-rc0/
  
The following staging can be used for testing:
https://repository.apache.org/content/repositories/orgapacheignite-1552/
  
Tag name is 2.14.0-rc0:
https://gitbox.apache.org/repos/asf?p=ignite.git;a=tag;h=refs/tags/2.14.0-rc0
  
2.14.0 most important changes:
* Added new experimental, Apache Calcite based SQL engine
* Added support of IGNITE_TO_STRING_INCLUDE_SENSITIVE option by Сonsistency 
check command
* CPP Thin: Implemented asynchronous network events handling
* CPP thin: Implemented continuous queries
* Deprecated IgniteServices#service(String) and IgniteServices#services(String).
* Implemented CDC for in-memory caches
* Implemented NUMA aware allocator for Ignite durable memory
* Implemented Read Repair strategies
* Implemented array component type in binary object
* Removed the legacy service grid implementation  

RELEASE NOTES:
https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=RELEASE_NOTES.txt;hb=ignite-2.14
  
Complete list of resolved issues: 
https://issues.apache.org/jira/issues/?jql=(project%20%3D%20'Ignite'%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20('2.14'))%20AND%20(component%20is%20EMPTY%20OR%20component%20not%20in%20(documentation))%20and%20status%20in%20('CLOSED'%2C%20'RESOLVED')%20AND%20resolution%20in(Fixed%2C%20Done%2C%20Implemented%2C%20Delivered)%20ORDER%20BY%20priority
  

DEVNOTES 
https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob_plain;f=DEVNOTES.txt;hb=ignite-2.14
  
The vote is formal, see voting guidelines 
https://www.apache.org/foundation/voting.html
  
+1 - to accept Apache Ignite 2.14.0-rc0
0 - don't care either way
-1 - DO NOT accept Apache Ignite Ignite 2.14.0-rc0 (explain why)
  
See notes on how to verify release here 
https://www.apache.org/info/verification.html
and 
https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification
  
This vote will be open for at least 3 days till Mon Oct 3, 2022, 20:00 UTC. 
https://www.timeanddate.com/countdown/vote?iso=20221003T20=0=VOTE+on+the+Apache+Ignite+Release+2.14.0+RC0=sanserif

--
With best regards,
Taras Ledkov

Re:Apache Ignite 2.14 RELEASE [Time, Scope, Manager]

[Taras Ledkov]

Hi, Ivan.

My apologies.
A blocking performance issue has been fixed [1]. And the release candidate is 
verified and ready to be voted on. What do you think about the 2.14.1 release, 
or can the fix wait until 2.15?
I'm going to start VOTE just now.

[1]. https://issues.apache.org/jira/browse/IGNITE-17764

Re:Apache Ignite 2.14 RELEASE [Time, Scope, Manager]

[Taras Ledkov]

Dear Ignite Community!

Release 2.14 has been delayed due to performance issues (compare with 2.13) for 
IgniteSqlQueryBenchmark in in-memory mode. We're looking.

Any suggestions and support are welcome.

Re:Apache Ignite 2.14 RELEASE [Time, Scope, Manager]

[Taras Ledkov]

Dear Ignite Community!

I'm announcing a code freeze for the 2.14 release [1]. Only blockers
are accepted to be included to the scope.

[1]. https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.14

Re:Apache Ignite 2.14 RELEASE [Time, Scope, Manager]

[Taras Ledkov]

Dear Ignite Community!

The planning release date was updated [1]:

Code Freeze: September 7, 2022
Voting Date: September 21, 2022
Release Date: September 28, 2022

I delayed code freeze for two weeks because there is bunch of important patches 
for the release that require a bit of time to finalize (see unresolved issues 
for the release).

[1]. https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.14

Re:Contributor access rights

[Taras Ledkov]

Hello!

I have added you to JIRA and Wiki as contributor.

Please make sure to read
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute

Re:Apache Ignite 2.14 RELEASE [Time, Scope, Manager]

[Taras Ledkov]

Hi,

1. Petr, could you grant me TC permissions to run release builds please?
2. Pavel, please cherry pick your patch for IGNITE-17559 [1] to the release 
branch ignite-2.14 [2]. Or we could move the issue to the next release?

[1]. https://issues.apache.org/jira/browse/IGNITE-17559
[2]. https://github.com/apache/ignite/tree/ignite-2.14

Re:Apache Ignite 2.14 RELEASE [Time, Scope, Manager]

[Taras Ledkov]

Dear Ignite Community!

I have created the 2.14 release page. [1]

I have cut the release branch: 'ignite-2.14'. Please, cherry-pick new issues if 
needed.

The 2.14 scope is frozen. Unresolved not-blocking issues will be moved on the 
code freeze stage. The planning release date was updated:

Scope Freeze: August 18, 2022
Code Freeze: August 25, 2022
Voting Date: September 8, 2022
Release Date: September 15, 2022

[1]: https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.14

PS. Fix typos at the prev message.

 --
 With best regards,
 Taras Ledkov

Re:Apache Ignite 2.14 RELEASE [Time, Scope, Manager]

[Taras Ledkov]

Dear Ignite Community!

I have created the 2.13 release page. [1]

I have cut the release branch: 'ignite-2.13'. Please, cherry-pick new issues if 
needed.

The 2.14 scope is frozen. Unresolved not-blocking issues will be moved on the 
code freeze stage. The planning release date was updated:

Scope Freeze: August 18, 2022
Code Freeze: August 25, 2022
Voting Date: September 8, 2022
Release Date: September 15, 2022

[1]: https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.14

 --
 With best regards,
 Taras Ledkov

Apache Ignite 2.14 RELEASE [Time, Scope, Manager]

[Taras Ledkov]

Dear Ignite Community!

I suggest starting Apache Ignite 2.14 release activities.

We've accumulated a hundred resolved [1] issues with new
features and bug fixes which are waiting for their release date. For
example,
- bunch of Calcite based SQL engine fixes and improvements;
- thin clients improvements;
- fixes at the core: update counters, binary meta, etc.

I want to propose myself to be the release manager of the planning release.

I propose the following timeline:

Scope Freeze: August 17, 2022
Code Freeze: August 24, 2022
Voting Date: September 7, 2022
Release Date: September 14, 2022

[1]. 
https://issues.apache.org/jira/browse/IGNITE-16958?jql=(project%20%3D%20%27Ignite%27%20AND%20fixVersion%20is%20not%20empty%20AND%20fixVersion%20in%20(%272.14%27))%20AND%20(component%20is%20EMPTY%20OR%20component%20not%20in%20(documentation))%20and%20status%20in%20(%27CLOSED%27%2C%20%27RESOLVED%27)%20ORDER%20BY%20priority

--
With best regards,
Taras Ledkov

Re: [VOTE] Add micronaut dependency to Ignite 3 (reopened)

[Taras Ledkov]

+1

On Tue, May 31, 2022 at 12:03 PM Aleksandr Pakhomov 
wrote:

> Alexander,
>
> I am not sure that now we can define all
> security standards that will be implemented
> by Ignite 3. But I can mention the most popular:
> OAuth2/OpenId, LDAP.
>
> I’ve mentioned them in IEP but I believe that
> the definite set of security protocols has to
> be discussed in another IEP.
>
> > On 30 May 2022, at 12:13, Alexander Polovtcev 
> wrote:
> >
> > Aleksandr,
> > Thanks for the update, but can you please explain what security
> > capabilities are planned to be used? Maybe it should be included in the
> IEP
> > as well.
> >
> > On Sun, May 29, 2022 at 11:03 PM Aleksandr Pakhomov  >
> > wrote:
> >
> >> Dear community,
> >>
> >> We’ve had a productive discussion about
> >> a micronaut dependency [1]. As a result,
> >> we’ve came to the conclusion that the
> >> security support in micronaut is a good
> >> reason to use this library. I am reopening the
> >> vote for adding the micronaut dependency to
> >> the rest module. Also, micronaut-serde is
> >> dropped from the list because Ignite 3
> >> already uses jackson as a serialisation lib.
> >>
> >> List of dependencies:
> >> io.micronaut:micronaut-inject:jar:3.4.1
> >> io.micronaut:micronaut-core:jar:3.4.1
> >> io.micronaut:micronaut-http-server-netty:jar:3.4.1
> >> io.micronaut:micronaut-http-server:jar:3.4.1
> >> io.micronaut:micronaut-router:jar:3.4.1
> >> io.micronaut:micronaut-http-netty:jar:3.4.1
> >> io.micronaut:micronaut-buffer-netty:jar:3.4.1
> >> io.micronaut:micronaut-runtime:jar:3.4.1
> >> io.micronaut:micronaut-http:jar:3.4.1
> >> io.micronaut:micronaut-aop:jar:3.4.1
> >> io.micronaut:micronaut-context:jar:3.4.1
> >> io.micronaut:micronaut-core-reactive:jar:3.4.1
> >>
> >> More information about motivation and
> >> implementation details could be found
> >> in IEP-87 [2].
> >>
> >> The vote is formal, see voting guidelines [3].
> >>
> >> +1 - to accept additional dependencies to be included to Java code
> >> Guidelines [4]
> >> 0 - don’t care either way
> >> -1 - DO NOT accept (explain why)
> >>
> >> This vote will be open for at leas 3 days till Wed June 1, 2022,
> >> 23:00 Moscow TZ [5].
> >>
> >>
> >> [1] https://lists.apache.org/thread/0nq6wx8t9r036mrjsk6n592gwnvvqbhs <
> >> https://lists.apache.org/thread/0nq6wx8t9r036mrjsk6n592gwnvvqbhs <
> https://lists.apache.org/thread/0nq6wx8t9r036mrjsk6n592gwnvvqbhs>>
> >> [2]
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-87%3A+Open+API+support+for+REST
> <
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-87%3A+Open+API+support+for+REST
> >
> >> <
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-87:+Open+API+support+for+REST
> <
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-87:+Open+API+support+for+REST
> >>
> >>
> >> [3] https://www.apache.org/foundation/voting.html <
> https://www.apache.org/foundation/voting.html> <
> >> https://www.apache.org/foundation/voting.html <
> https://www.apache.org/foundation/voting.html>>
> >> [4]
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Java+Code+Style+Guide#JavaCodeStyleGuide-2Using3rdpartylibraries
> <
> https://cwiki.apache.org/confluence/display/IGNITE/Java+Code+Style+Guide#JavaCodeStyleGuide-2Using3rdpartylibraries
> >
> >> <
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/Java+Code+Style+Guide#JavaCodeStyleGuide-2Using3rdpartylibraries
> <
> https://cwiki.apache.org/confluence/display/IGNITE/Java+Code+Style+Guide#JavaCodeStyleGuide-2Using3rdpartylibraries
> >
> >>>
> >> [5]
> >>
> https://www.timeanddate.com/countdown/generic?iso=20220601T23=166=%5BVOTE%5D+Add+micronaut+dependency+to+Ignite+3+%28reopened%29=cursive
> <
> https://www.timeanddate.com/countdown/generic?iso=20220601T23=166=%5BVOTE%5D+Add+micronaut+dependency+to+Ignite+3+%28reopened%29=cursive
> >
> >> <
> >>
> https://www.timeanddate.com/countdown/generic?iso=20220601T23=166=[VOTE]+Add+micronaut+dependency+to+Ignite+3+(reopened)=cursive
> <
> https://www.timeanddate.com/countdown/generic?iso=20220601T23=166=[VOTE]+Add+micronaut+dependency+to+Ignite+3+(reopened)=cursive
> >>
> >>
> >
> >
> >
> > --
> > With regards,
> > Aleksandr Polovtcev
>
>

[DISCUSSION] Validation key type and schema on a data insertion [IGNITE-16098]

[Taras Ledkov]

Hi, Igniters.

There was know issue described at the [IGNITE- 16098] case 1.
These issue is the cause of not symmetric cache API and SQL API:
we cannot create the same table by SQL and cache API (with se identical
internal structures).
In more detail: it is impossible now to create a sorted PK index with
unwrapped fields through cach eAPI.

I think it's more convenient when SQL and the cache API have equal
capabilities.

Also I found the second scenario that can break indexes.

Please take a look at the issue [1] description, and let me know what you
think.

[1]. https://issues.apache.org/jira/browse/IGNITE-16098

Add ability to specify INLINE_SIZE for PK sorted index and key's schema validation [IGNITE-11402]

[Taras Ledkov]

Hi Igniters,

There is an issue to add ability to specify INLINE_SIZE for sorted index on
the primary key and index on affinite fields [1].
Please take a look at the last comment (proposal).

I duplicate the proposal here.

*Proposal:*
add to the public QueryEntity properties:
- Integer primaryKeyInlineSize - use wrapper object to compatibility (null
value - default behavior);
- Integer affinityFieldInlineSize - use wrapper object to compatibility
(null value - default behavior);

The approach has a contradiction with cache API behavior.
Now there is no ability to specify PK sorted index with unwrapped PK fields
by cache API + QueryEntity. This functionality is available only from SQL
command 'CREATE TABLE' (see H2TableDescriptor#extractKeyColumns)

So,
- by SQL command we cannot create 'wrapped' PK sorted index for composite
PK;
- by cache API we cannot create 'unwrapped' PK sorted index for composite
PK.

I propose to add the public property:
Boolean QueryEntity#unwrapPrimaryKeyFieldsForSortedIndex
to make SQL & cache API symmetrical.

*There is a pitfall here*
User may specify the key class like below:

class MyKey{
  @QuerySqlField
  int id0;

  @QuerySqlField
  int id1;

  int hiddenId;
}

In this case two objects MyKey(0, 0, 0) & MyKey(0,0,1) are different and
may be put into cache as a two different keys.
But they are similar for SQL PK.
But this scenario now may be produced by SQL command CREATE TABLE + cache
API.

I propose to add key schema validation to the
QueryTypeDescriptorImpl#validateKeyAndValue.
[1]. https://issues.apache.org/jira/browse/IGNITE-11402

Re: IEP-61 Transaction API desing for Ignite 3

[Taras Ledkov]

Hi colleagues,

2Pavel:
> RecordView txView = view.withTransaction();
Can we use the syntax (see below) to attach the table / operation to the
started transaction?
RecordView  txPersonView =
person.recordView().withTransaction(txView.transaction());


On Mon, Nov 29, 2021 at 1:34 PM Pavel Tupitsyn  wrote:

> Alexei,
>
> I agree that runInTransaction is confusing and error-prone.
>
> But we already have view.withTransaction(), which seems to be the most
> boilerplate-free approach.
> The example above will look like this:
>
> public void testMixedPutGet() throws TransactionException {
> RecordView view = accounts.recordView();
>
> view.upsert(makeValue(1, BALANCE_1));
>
> RecordView txView = view.withTransaction();
>
> txView.getAsync(makeKey(1)).thenCompose(r ->
> txView.upsertAsync(makeValue(1, r.doubleValue("balance") + DELTA),
> tx)).thenCompose(txView.transaction().commitAsync()).join();
>
> assertEquals(BALANCE_1 + DELTA,
> view.get(makeKey(1)).doubleValue("balance"));
> }
>
> Is there any problem with this?
>
> On Mon, Nov 29, 2021 at 10:45 AM Alexei Scherbakov <
> alexey.scherbak...@gmail.com> wrote:
>
> > Folks,
> >
> > Recently I've pushed transactions support phase 1 for Ignite 3, see [1].
> > Feel free to give feedback.
> > Current implementation attempts to automatically enlist a table into
> > transaction if it's started using [2] or [3] by using thread local
> context,
> > similar to Ignite 2 approach, to reduce the amount of boilerplate code.
> > But it turns out such an approach still has unacceptable drawbacks from a
> > user experience point of view.
> >
> > Consider the example [4]:
> >
> > public void testMixedPutGet() throws TransactionException {
> > accounts.recordView().upsert(makeValue(1, BALANCE_1));
> >
> > igniteTransactions.runInTransaction(tx -> {
> > var txAcc = accounts.recordView().withTransaction(tx);
> >
> > txAcc.getAsync(makeKey(1)).thenCompose(r ->
> > txAcc.upsertAsync(makeValue(1, r.doubleValue("balance") +
> DELTA))).join();
> > });
> >
> > assertEquals(BALANCE_1 + DELTA,
> > accounts.recordView().get(makeKey(1)).doubleValue("balance"));
> > }
> >
> > Here we *have to* to manually enlist a table if it's used in async chain
> > call, because the caller thread will be different and the chained
> operation
> > will be executed in separate tx.
> > This works similarly in Ignite 2 and is very confusing.
> >
> > To avoid this, I propose to add an explicit Transaction argument to each
> > table API method. Null value means to start the implicit transaction
> > (autocommit mode). For example:
> >
> > /**
> >  * Asynchronously inserts a record into the table if it doesn't exist
> > or replaces the existed one.
> >  *
> >  * @param rec A record to insert into the table. The record cannot be
> > {@code null}.
> >  * @param tx The transaction or {@code null} to auto commit.
> >  * @return Future representing pending completion of the operation.
> >  */
> > @NotNull CompletableFuture upsertAsync(@NotNull R rec,
> @Nullable
> > Transaction tx);
> >
> > The example [4] turns to
> >
> > public void testMixedPutGet() throws TransactionException {
> > RecordView view = accounts.recordView();
> >
> > view.upsert(makeValue(1, BALANCE_1));
> >
> > igniteTransactions.runInTransaction(tx -> {
> > view.getAsync(makeKey(1), tx).thenCompose(r ->
> > view.upsertAsync(makeValue(1, r.doubleValue("balance") + DELTA),
> > tx)).join();
> > });
> >
> > assertEquals(BALANCE_1 + DELTA,
> > view.get(makeKey(1)).doubleValue("balance"));
> > }
> >
> > Share your thoughts.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-15085
> > [2]
> >
> org.apache.ignite.tx.IgniteTransactions#runInTransaction(java.util.function.Consumer)
> > [3]
> >
> org.apache.ignite.tx.IgniteTransactions#runInTransaction(java.util.function.Function)
> > [4] org.apache.ignite.internal.table.TxAbstractTest#testMixedPutGet
> >
> > ср, 14 июл. 2021 г. в 14:12, Alexei Scherbakov <
> > alexey.scherbak...@gmail.com
> > >:
> >
> > > Andrey,
> > >
> > > 1) "As a user, I'd expect runInTransaction(closure) will create Tx for
> > me,
> > > commit Tx after a successful closure call, and rollback Tx in case of
> > > error."
> > > - I'm ok with this behavior, and will alter javadoc.
> > >
> > > 2) "Transaction tx = beginTx()" - there is no such method "beginTx" in
> > the
> > > proposed API, and I'm not intending to add it.
> > > For the synchronous case I suggest to use "runInTransaction", which
> > > eliminates the need in AutoClosable.
> > >
> > >
> > >
> > > ср, 14 июл. 2021 г. в 13:21, Ivan Daschinsky :
> > >
> > >> > yes, it is stated in the javadoc in the PR.
> > >> Ah, I see.
> > >>
> > >> ср, 14 июл. 2021 г. в 12:16, Alexei Scherbakov <
> > >> alexey.scherbak...@gmail.com
> > >> >:
> > >>
> > >> > Ivan,
> > >> >
> > >> > And what if I have already 

Re: [VOTE] Create separate Jira project and Confluence space for Ignite 3

[Taras Ledkov]

+1

On 05.10.2021 2:52, Valentin Kulichenko wrote:

Hello Community,

As discussed in [1], I would like to propose the creation of a separate
Jira project and Confluence space for Ignite 3.

Ignite 2 and Ignite 3 are developed in parallel in separate repos, so we
need a clear separation in other tools as well - this will help to
streamline the development process. Please refer to the discussion for more
details.

[1]
https://lists.apache.org/thread.html/rdcad3fc64b9f3a848c93089baae2bee1124a97869a94f4a04dd80fdf%40%3Cdev.ignite.apache.org%3E

Voting options:

- +1 - Agree with the suggestion
- 0 - Don't care much about the suggestion
- -1 - Disagree with the suggestion

This is a majority vote.

Voting ends in 72 hours, at 5pm PDT on October 7:
https://www.timeanddate.com/counters/fullscreen.html?mode=a=20211007T17=2021=10=7=17=0=0=224

-Val


--
Taras Ledkov
Mail-To: tled...@gridgain.com

Re: [DISCUSS] IEP-71 Public API for secondary index search

[Taras Ledkov]

Hi,

My proposal:
1. Don't search index by criteria, specify the index name always 
(preferred).


OR

2. Search index by criteria without check the order of criteriones.
Use the Set of criterions instead of the ordered collection.
In the strange case when the both index exist (a, b) and (b, a) - use 
the any index

when index name isn't specified.

On 26.08.2021 16:49, Maksim Timonin wrote:

There are some thoughts about strict field order:
1. Index (A, B) is not equivalent to index (B, A). Some queries may have
different performance on such indexes, and users have to specify the right
index. What if both indexes exist?
2. We should avoid cases when a user uses in query only field B for index
(A, B). We have to force the user to specify range for (A) too, or
explicitly set it (null, null). Otherwise it looks like a mistake.




On Thu, Aug 26, 2021 at 4:39 PM Ivan Daschinsky  wrote:


1. I suppose, that the next step is to implement the api for manually
creating index. I think that user wants to create index that will speed up
his criteria base queries, so he or she will use the same criteria to
define the index. So no problem at all
2. We should print warning or throws exception if there is not any index
that match specific criteria.

BTW, Mongo DB doesn't make user to write index name in query. It just
works.

чт, 26 авг. 2021 г., 15:52 Taras Ledkov :


Hi,


It is an usability nightmare to make user write index name in all

cases.

I don't see any difference between specifying the index name and
specifying the index fields in the right order.
Do you see?

Let's there is the index:
idx_A_B ON TBL (A, B)

Is it OK that the query like below doesn't math the index 'idx_A_B'?
new IndexQuery<>(..)
  .setCriteria(lt("b", 1), lt("a", 2));

On 26.08.2021 15:23, Ivan Daschinsky wrote:

I am against to make user write index name. It is quite simple and
straightforward algorithm to match index to field names, so it is

strange

to compare it to sql engine optimizer.

It is an usability nightmare to make user write index name in all

cases.

чт, 26 авг. 2021 г., 14:42 Maksim Timonin :


Hi, Igniters!

There is a discussion about how to specify an index to query with an
IndexQuery [1]. Currently my PR provides 2 ways to specify index:
1. With a table and index name;
2. With a table and list of index fields (without index name). In this

case

IndexQueryProcessor tries to find an index that matches table and

index

fields in strict order (order of fields in criteria has to match the

order

of fields in index).

Discussion is whether is the second approach valid?

Pros:
1. Currently index name is an optional field for QueryIndex and
QuerySqlField. Then users can create an index with a table and list of
fields. Then, we should provide an opportunity to define an index for
querying the same way as we do for creating.
2. It's required to know the index name to query it (in case the index

was

created without an explicit name). Users can find it and then use it

as

a

constant in code, but I see some troubles there:
2.1. Get index name by querying the system view INDEXES. Note, that

system

views are marked as an experimental feature [2].
2.2. There is a workaround to know an index name with EXPLAIN clause

for

sql query that uses the required index (but it depends on SQL

optimizer).

2.3. Users can use the index name builder, but it is in the
internal package
(org.apache.ignite.internal.processors.query.QueryUtils#indexName).

Then it

can be changed from version to version without warning, and then the

user

can't rely on it in code.
3. Name of the Primary Key index (_key_PK) is predefined and hardcoded

in

Ignite. Users can't set it while creating it, and the name of PK index

is

hardcoded in the internal package too (QueryUtils.PRIMARY_KEY_INDEX).

Cons:
1. It's declared that IndexQuery avoids some SQL steps (like planning,
optimizer) in favor of speed. It looks like that looking for an index

by

list of fields is the work of an optimizer.
2. It can be not obvious that the order of fields in a query has to

match

the order of fields in the related index. We should have it in mind

when

building a query - there should be a check for order of fields before
querying.

  From my side, I think that arguments for enforcing usage of an index

name

for queries are strong enough. But for me it's strange that it's

possible

to create an index without a name, but it's required to use name to

query

it. Also taking in consideration that there is no guaranteed way to

get

an

index name (or I don't know it).

Igniters, what do you think?
[1] https://github.com/apache/ignite/pull/9118#discussion_r642557531
[2]

https://ignite.apache.org/docs/latest/monitoring-metrics/system-views


On Fri, Aug 6, 2021 at 4:04 PM Maksim Timonin <

timonin.ma...@gmail.com>

wrote:


Hi, all!

It's a gentle reminder. There is a PR for the new Index API [1]. It

was

approved by Alex Plekha

Re: [DISCUSS] IEP-71 Public API for secondary index search

[Taras Ledkov]

lock on an index too while

querying.

And I want to discuss the right way. I have in mind the next things:
1. Indexes currently doesn't support transactions, also SQL queries

don't

lock index for queries, so Ignite don't guarantee data consistency;
2. As I understand preparing whole data for SQL queries is required due
to relations between tables. The more complex query and relations we

have,

the much consistency issues we have in result in case of parallel
operations;
3. Querying a single index only (by TextQuery or IndexQuery) doesn't
affect any relations, so we can allow concurrent updates, as it could
affect a query result but it doesn't hurt.

And following these thoughts, it's right to implement lazy iterations
over indexes. What do you think?

Also, there is a second topic to discuss. BPlusTree indexes support

query

parallelism. But CacheQueries don't. There needs to be a change to
infrastructure to support query parallelism, so on this patch [1] I

handle

multiple segments in a single thread. And this works OK, as in the case

of

lazy querying it's very fast to initialize a cursor, so there is not

much

overhead on multiple segments. I ran performance tests and found that in
some cases, IndexQuery beats SqlFieldsQuery even with enabled
queryParallelism (it helps a SqlFieldsQuery much). So the need for
supporting queryParallelism for IndexQuery is required to be tested

well.

As IndexQuery already can help users to speed up some queries I propose

to

check queryParallelism a little bit later. WDYT?

So, those 2 things affect the Apache Ignite release that IndexQuery will
be delivered with. So, please let me know your thoughts.

Any thoughts from the community are welcome too.


[1] https://github.com/apache/ignite/pull/9118
[2] https://github.com/apache/ignite/pull/9086

On Mon, Apr 12, 2021 at 1:52 PM Maksim Timonin 
Andrey,

Thanks! I picked it.

On Mon, Apr 12, 2021 at 1:51 PM Maksim Timonin <

timonin.ma...@gmail.com>

wrote:


Stephen,

I don't see a reason to replace or deprecate IndexingSpi. I'm not
sure how smbd uses it, but it works now.

On Mon, Apr 12, 2021 at 1:42 PM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:


Is this a replacement for IndexingSpi? Put bluntly, do we deprecate
(and remove) it?

Or do you see them as complimentary?


On 12 Apr 2021, at 11:29, Maksim Timonin 

wrote:

Hi Stephen!

Please have a look at the QueryProcessing paragraph [1]. I've

described

why IndexingSpi doesn't fit us well.

[1]


https://cwiki.apache.org/confluence/display/IGNITE/IEP-71+Public+API+for+secondary+index+search#IEP71PublicAPIforsecondaryindexsearch-2)QueryProcessing

On Mon, Apr 12, 2021 at 1:24 PM Stephen Darlington <
stephen.darling...@gridgain.com> wrote:


How does this fit with the current IndexingSpi? Superficially they

appear

to do very similar things?

Regards,
Stephen


On 6 Apr 2021, at 14:13, Maksim Timonin 
wrote:

Hi, Igniters!

I'd like to propose a new feature - opportunity to query and

create

indexes

from public API.

It will help in some cases, where:
1. SQL is not applicable by design of user application;
2. Where IndexScan is preferable than ScanQuery for performance

reasons;

3. Functional indexes are required.

Also it'll be great to have a transactional support for such

queries,

like

the "select for update" query provides. But I don't dig there

much. It

will

be a next step if this API will be implemented.

I've prepared an IEP-71 for that [1] with more details. Please

share your

thoughts.


[1]


https://cwiki.apache.org/confluence/display/IGNITE/IEP-71+Public+API+for+secondary+index+search







--
Taras Ledkov
Mail-To: tled...@gridgain.com

Re: IGNITE-14812: Statistics

[Taras Ledkov]

Hi,

> Taras, can you, please, describe the features that was implemented in 
this merge?

> How users supposed to use them?
> Do we have plans to document?

Sure. Alexander Belyak will describe and file ticket to documentation.

On 23.06.2021 9:27, Nikolay Izhikov wrote:

Hello, Taras.

Thanks for feedback.


AFAIK and as long as I can remember Ignite project try to minimize external 
dependencies usage and adds new external dependency only when there is no other 
way out.

Does it mean we have to incapsulate every external library we want to use?

Taras, can you, please, describe the features that was implemented in this 
merge?
How users supposed to use them?
Do we have plans to document?



23 июня 2021 г., в 09:21, Taras Ledkov  написал(а):

Hi,

We have discussed BCrypt include/add dependency here [1].
AFAIK and as long as I can remember Ignite project try to minimize external 
dependencies usage
and adds new external dependency only when there is no other way out.

[1]. 
http://apache-ignite-developers.2346864.n4.nabble.com/Username-password-authentication-for-thin-clients-tp26058p26954.html

On 23.06.2021 3:08, Valentin Kulichenko wrote:

Dmitry,

As the PMC chair, would you mind contacting legal regarding the matter?
This is not the only example of such code (e.g. [1]), so we should look
into this asap.

[1]
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/mindrot/BCrypt.java

As for this particular commit, can HLL be added as a dependency instead? Is
there any particular reason to include the source code? @Sasha Belyak
 , can you please chime in?

-Val

On Tue, Jun 22, 2021 at 8:10 AM Dmitry Pavlov  wrote:


Hi Nikolai,

thank you for noticing. I guess it's not about license, but about
Intellectual Property (IP) ownership.

AFAIK, Apache License 2.0 is here and AL 2.0 is definetely allowed to be
used in the codebase for an Apache project (
https://www.apache.org/legal/resolved.html)

But licenese and IP owner are 2 sligthly different things. E.g at the end
of any website you can find:
Copyright © 2021 The Apache Software Foundation, Licensed under the Apache
License, Version 2.0.

Incubated projects are mandated to transfer IP to the ASF. And I'm not
aware of any exceptions.

In this PR there are 5 classes which licenses with AL 2.0, but IP owner is
3rd party company.

I'm a bit concerned about having such code in the project. I'd rather
reverted it until we have approval from experts at mailing list:
legal-disc...@apache.org

Sincerely,
Dmitriy Pavlov

On 2021/06/22 14:56:49, Nikolay Izhikov  wrote:

Hello, Igniters.

Recently huge commit was merged [1].

Taras, Alexander, can you, please, explain what is purpose of the commit?
What feature it implemented?

Looked inside the ticket and found no explanation.
Description is "Add statistics collection and usage.»

Do we have plans to document this new feature?

Also, I see very strange license in added files [2]

```
  * Copyright 2013 Aggregate Knowledge, Inc.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
```

Is it OK to have those copyright inside ASF codebase?
Is it some kind of external dependency we adopted as part of the code?
Why do we need it?

[1]

https://github.com/apache/ignite/commit/503a98495433e1d0cf84f8be8c1e2adc57034fbb

[2]

https://github.com/apache/ignite/blob/master/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/stat/hll/serialization/IHLLMetadata.java


--
Taras Ledkov
Mail-To: tled...@gridgain.com


--
Taras Ledkov
Mail-To: tled...@gridgain.com

Re: IGNITE-14812: Statistics

[Taras Ledkov]

Hi,

We have discussed BCrypt include/add dependency here [1].
AFAIK and as long as I can remember Ignite project try to minimize 
external dependencies usage

and adds new external dependency only when there is no other way out.

[1]. 
http://apache-ignite-developers.2346864.n4.nabble.com/Username-password-authentication-for-thin-clients-tp26058p26954.html


On 23.06.2021 3:08, Valentin Kulichenko wrote:

Dmitry,

As the PMC chair, would you mind contacting legal regarding the matter?
This is not the only example of such code (e.g. [1]), so we should look
into this asap.

[1]
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/mindrot/BCrypt.java

As for this particular commit, can HLL be added as a dependency instead? Is
there any particular reason to include the source code? @Sasha Belyak
 , can you please chime in?

-Val

On Tue, Jun 22, 2021 at 8:10 AM Dmitry Pavlov  wrote:


Hi Nikolai,

thank you for noticing. I guess it's not about license, but about
Intellectual Property (IP) ownership.

AFAIK, Apache License 2.0 is here and AL 2.0 is definetely allowed to be
used in the codebase for an Apache project (
https://www.apache.org/legal/resolved.html)

But licenese and IP owner are 2 sligthly different things. E.g at the end
of any website you can find:
Copyright © 2021 The Apache Software Foundation, Licensed under the Apache
License, Version 2.0.

Incubated projects are mandated to transfer IP to the ASF. And I'm not
aware of any exceptions.

In this PR there are 5 classes which licenses with AL 2.0, but IP owner is
3rd party company.

I'm a bit concerned about having such code in the project. I'd rather
reverted it until we have approval from experts at mailing list:
legal-disc...@apache.org

Sincerely,
Dmitriy Pavlov

On 2021/06/22 14:56:49, Nikolay Izhikov  wrote:

Hello, Igniters.

Recently huge commit was merged [1].

Taras, Alexander, can you, please, explain what is purpose of the commit?
What feature it implemented?

Looked inside the ticket and found no explanation.
Description is "Add statistics collection and usage.»

Do we have plans to document this new feature?

Also, I see very strange license in added files [2]

```
  * Copyright 2013 Aggregate Knowledge, Inc.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
```

Is it OK to have those copyright inside ASF codebase?
Is it some kind of external dependency we adopted as part of the code?
Why do we need it?

[1]

https://github.com/apache/ignite/commit/503a98495433e1d0cf84f8be8c1e2adc57034fbb

[2]

https://github.com/apache/ignite/blob/master/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/stat/hll/serialization/IHLLMetadata.java


--
Taras Ledkov
Mail-To: tled...@gridgain.com

Re: Questions regarding sql windows functions using apache ignite: is sql windows function supported in ignite?

[Taras Ledkov]

Hi,

Current H2-based SQL engine doesn't support windows functions.
Now we are working on new SQL engine. [1]

Now windows functions isn't high priority feature.
Welcome to contribution! Please take a look at the dev-branch [2].

[1]. 
https://cwiki.apache.org/confluence/display/IGNITE/IEP-37%3A+New+query+execution+engine

[2]. https://github.com/apache/ignite/tree/sql-calcite

On 05.05.2021 14:34, Xi Wang wrote:

Hi,

My name is Xi Wang and I am a developer from Clobotics. For now we are testing 
the performance of ignite, and the way we are doing is to insert some tables 
and then write sql queries to check the running time.

I am trying to add some windows functions into my query, something like 
over(partition by something order by something). I use it very often on SQL 
Server and Oracle, but I never succeeded when using ignite. I saw some posts 
from ignite community saying that windows function was not supported, and the 
post was from 2018.

So could you please tell me whether windows function is currently compatible to 
Ignite or not for now? If the answer is yes, could you send me some examples? 
At least I couldn’t found anything on google.

I would really appreciate your help.

Bests,
Xi

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10



--
Taras Ledkov
Mail-To: tled...@gridgain.com

[jira] [Created] (IGNITE-14623) Calcite. Sort out test scripts at: sql/aggregate/aggregates/*

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14623:
-

 Summary: Calcite. Sort out test scripts at: 
sql/aggregate/aggregates/*
 Key: IGNITE-14623
 URL: https://issues.apache.org/jira/browse/IGNITE-14623
 Project: Ignite
  Issue Type: Task
Reporter: Taras Ledkov
Assignee: Taras Ledkov


Calcite. Sort out test scripts at: {{sql/aggregate/aggregates/*}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14610) BinaryBuilderReader doesn't supports reference (HANDLE) to collection

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14610:
-

 Summary: BinaryBuilderReader doesn't supports reference (HANDLE) 
to collection
 Key: IGNITE-14610
 URL: https://issues.apache.org/jira/browse/IGNITE-14610
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.10
 Environment: Steps to reproduce:
1. Create object with two objects collection fields, e.g. List
2. Assign the reference of the one field to the second.
3. Serialize object to BinaryObject
4. Create BinaryObjectBuiler from the object.
5. Add/change any field except collections.
6. Try to build new BinaryObject form the builer

Exception is thrown:
{code:java}
class org.apache.ignite.binary.BinaryObjectException: Unsupported protocol 
version: 2
at 
org.apache.ignite.internal.binary.BinaryUtils.checkProtocolVersion(BinaryUtils.java:795)
at 
org.apache.ignite.internal.binary.builder.BinaryObjectBuilderImpl.(BinaryObjectBuilderImpl.java:139)
at 
org.apache.ignite.internal.binary.builder.BinaryBuilderReader.parseValue(BinaryBuilderReader.java:522)
at 
org.apache.ignite.internal.binary.builder.BinaryObjectBuilderImpl.serializeTo(BinaryObjectBuilderImpl.java:286)
at 
org.apache.ignite.internal.binary.builder.BinaryBuilderSerializer.writeValue(BinaryBuilderSerializer.java:100)
at 
org.apache.ignite.internal.binary.builder.BinaryBuilderSerializer.writeValue(BinaryBuilderSerializer.java:53)
at 
org.apache.ignite.internal.binary.builder.BinaryObjectBuilderImpl.serializeTo(BinaryObjectBuilderImpl.java:293)
at 
org.apache.ignite.internal.binary.builder.BinaryBuilderSerializer.writeValue(BinaryBuilderSerializer.java:100)
at 
org.apache.ignite.internal.binary.builder.BinaryBuilderSerializer.writeValue(BinaryBuilderSerializer.java:53)
at 
org.apache.ignite.internal.binary.builder.BinaryObjectBuilderImpl.serializeTo(BinaryObjectBuilderImpl.java:293)
at 
org.apache.ignite.internal.binary.builder.BinaryObjectBuilderImpl.build(BinaryObjectBuilderImpl.java:188)
{code}

Root cause:
BinaryBuilderReader#parseValue doesn’t support handle to collection. Only 
handles to object are expected.

Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.11






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14548) BinaryThreadLocalContext must be cleaned when client is closed

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14548:
-

 Summary: BinaryThreadLocalContext must be cleaned when client is 
closed
 Key: IGNITE-14548
 URL: https://issues.apache.org/jira/browse/IGNITE-14548
 Project: Ignite
  Issue Type: Bug
  Components: binary
Affects Versions: 2.10
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.11


ThreadLocal {{BinaryThreadLocalContext#CTX}} must be cleaned when thin client 
and JDBC thin client are closed.

Fail case: [Stack 
overflow|https://stackoverflow.com/questions/67086723/unable-to-remove-org-apache-ignite-internal-binary-binarythreadlocal-error-in-we?noredirect=1#comment118615654_67086723]

Previous discussion: IGNITE-967



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14546) Calcite engine. MIN/MAX aggregates doesn't support string types

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14546:
-

 Summary: Calcite engine. MIN/MAX aggregates doesn't support string 
types
 Key: IGNITE-14546
 URL: https://issues.apache.org/jira/browse/IGNITE-14546
 Project: Ignite
  Issue Type: Bug
Reporter: Taras Ledkov


MIN/MAX aggregates doesn't support string types.
e.g. 
{{CREATE TABLE TEST(val VARCHAR) }}
{{INSERT INTO TEST VALUES ('A'), ('B'), ('C') }}
{{ SELECT MIN(val), MAX(val) FROM TEST}}

Test: {{src/test/sql/aggregate/aggregates/test_aggr_string.test}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14545) Calcite engine. Unicode literal not supported

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14545:
-

 Summary: Calcite engine. Unicode literal not supported
 Key: IGNITE-14545
 URL: https://issues.apache.org/jira/browse/IGNITE-14545
 Project: Ignite
  Issue Type: Bug
Reporter: Taras Ledkov


Unicode literal not supported.
e.g. {{SELECT }}

Tests:
{{aggregate/aggregates/test_aggr_string.test}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14544) Calcite engine. Implement DISTING aggregates

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14544:
-

 Summary: Calcite engine. Implement DISTING aggregates
 Key: IGNITE-14544
 URL: https://issues.apache.org/jira/browse/IGNITE-14544
 Project: Ignite
  Issue Type: Bug
Reporter: Taras Ledkov


Now, DISTINCT aggregates not implemented.
(e.g. {{SELECT COUNT (DISTINCT lastName) FROM person}})

Tests:
{{aggregate/aggregates/test_count.test_ignored}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14543) Calcite engine. COUNT agregate on scalar values fails

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14543:
-

 Summary: Calcite engine. COUNT agregate on scalar values fails
 Key: IGNITE-14543
 URL: https://issues.apache.org/jira/browse/IGNITE-14543
 Project: Ignite
  Issue Type: Bug
Reporter: Taras Ledkov


Failed queries:
{{SELECT COUNT(*)}}
{{SELECT COUNT(NULL)}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14542) Calcite engine. Need to support TableFunctions / SYSTEM_RANGE dynamic table

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14542:
-

 Summary: Calcite engine. Need to support TableFunctions / 
SYSTEM_RANGE dynamic table
 Key: IGNITE-14542
 URL: https://issues.apache.org/jira/browse/IGNITE-14542
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Taras Ledkov


A lot of cases require dynamic range data source.
Tests:
{{aggregate/aggregates/test_perfect_ht.test_ignored}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14541) Calcite engine. HAVING fails on scalar

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14541:
-

 Summary: Calcite engine. HAVING fails on scalar 
 Key: IGNITE-14541
 URL: https://issues.apache.org/jira/browse/IGNITE-14541
 Project: Ignite
  Issue Type: Bug
Reporter: Taras Ledkov


Query to reproduce:
{{SELECT 42 HAVING 42 > 20}}
Test: {{aggregate/having/test_scalar_having.test_ignore}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14540) Calcite engine. Support correlated subquery in having

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14540:
-

 Summary:  Calcite engine. Support correlated subquery in having
 Key: IGNITE-14540
 URL: https://issues.apache.org/jira/browse/IGNITE-14540
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Taras Ledkov


Correlated subquery in having isn't supported now.
Test: {{aggregate/having/test_corel_subquery_in_having.test_ignored}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14539) Calcite engine. Support DISTINCT ON expression

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14539:
-

 Summary: Calcite engine. Support DISTINCT ON expression
 Key: IGNITE-14539
 URL: https://issues.apache.org/jira/browse/IGNITE-14539
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Taras Ledkov


Now DISTINCT ON expression isn't supported.
e.g.:
{{SELECT DISTINCT ON (lastName) firstName, lastName FROM Person WHERE ... }}
Test: {{aggregate/distinct/test_distinct_on.test_ignored}}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14537) Nested arbitrary queries hangs on optimizing

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14537:
-

 Summary: Nested arbitrary queries hangs on optimizing 
 Key: IGNITE-14537
 URL: https://issues.apache.org/jira/browse/IGNITE-14537
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Taras Ledkov


The query with nested arbitrary subqueries aren't planned.
Test: {{subquery/test_neumann.test_ignore}}

See more: [Unnesting Arbitrary 
Queries|https://subs.emis.de/LNI/Proceedings/Proceedings241/383.pdf]




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14535) Caclite SQL engine capabilities

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14535:
-

 Summary: Caclite SQL engine capabilities
 Key: IGNITE-14535
 URL: https://issues.apache.org/jira/browse/IGNITE-14535
 Project: Ignite
  Issue Type: Improvement
Reporter: Taras Ledkov


Umbrella ticket to track issues related to SQL logical tests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14531) Calcite improvements. Support DISTINCT for aggregates call at the execution

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14531:
-

 Summary: Calcite improvements. Support DISTINCT for aggregates 
call at the execution
 Key: IGNITE-14531
 URL: https://issues.apache.org/jira/browse/IGNITE-14531
 Project: Ignite
  Issue Type: Improvement
Reporter: Taras Ledkov


Now {{Accumulators#accumulatorFactory}} and all implementation of the 
{{Accumulator}} doesn't support DISTINCT (see {{AggregateCall#isDistinct}}). 
Must be implemented.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14508) Calcite integration: introduce SQL logical test runner

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14508:
-

 Summary: Calcite integration: introduce SQL logical test runner
 Key: IGNITE-14508
 URL: https://issues.apache.org/jira/browse/IGNITE-14508
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov


New SQL engine functions must be tested a lot.
We have to create suite / toolkit to run test SQL script that describes test 
statements.  expected behavior and results.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Query API for Ignite 3.0

[Taras Ledkov]

Hi, Ivan.

My comments are inline.

In general about JDBC native API:
I see no reasons why a user must use different query APIs on the client 
and on the server node (e.g. inside services or jobs).

It is complicate and not user-friendly.



1.1. Thin JDBC client will be really thin: provide network communication

layer and transparently map to native API.
But we have already this in AI 2.x

Not complete.
JDBC thin is a implementation of the JDBC specification. But it uses 
Query API `query(SqlFieldsQuery)`.
I see no reason to introduce new internal query API for 3.0 and wrap it 
into JDBC again.

May be we try to use JDBC as native query API?
This way doesn't prohibit to use any internal extension if need.


1.2. Thick JDBC client implementation will be trivial: start client node

and open JDBC connection on the started node.
We have already this in AI 2.x and IMHO this functionality is weird and
need to be removed. No one wants to start new node from DBeaver :)


I guess it will be removed on Ignite 2.x because implementation is weird 
(due to legacy).




1.3 JDBC provides sufficient functionality to implement ODBC (need to

investigate: may be thin protocol may be extended to unify JDBC and ODBC).
We already have this in AI 2.x (JDBC and ODBC is mostly unified)

It is absolutely wrong.
ODBC, JDBC and thin client use the same port and unified head of the 
handshake message.

The protocols are different a lot.


On 06.04.2021 15:40, Ivan Daschinsky wrote:

Hi Taras!

1.1. Thin JDBC client will be really thin: provide network communication

layer and transparently map to native API.
But we have already this in AI 2.x

1.2. Thick JDBC client implementation will be trivial: start client node

and open JDBC connection on the started node.
We have already this in AI 2.x and IMHO this functionality is weird and
need to be removed. No one wants to start new node from DBeaver :)

1.3 JDBC provides sufficient functionality to implement ODBC (need to

investigate: may be thin protocol may be extended to unify JDBC and ODBC).
We already have this in AI 2.x (JDBC and ODBC is mostly unified)


вт, 6 апр. 2021 г. в 15:05, Taras Ledkov :


Hi,

Ignite 3.0 requires a rethinking of the query API.
We have 'cache.query()' and 'SqlFieldQuery' abstractions at the Ignite
2.x native API and several JDBC implementation for clients.

I propose to think about new query/SQL API for the Ignite 3.0.

My vision is something like this:
Ignite will support two query APIs: standard JDBC (on native) and one of
reactive DB API.

1. Native JDBC API, e.g.:
  Connection conn = node.sql().connection(props);

JDBC is the industrial standard of the DB access and we have to support
one.
Also:
1.1. Thin JDBC client will be really thin: provide network communication
layer and transparently map to native API.
1.2. Thick JDBC client implementation will be trivial: start client node
and open JDBC connection on the started node.
1.3. JDBC provides sufficient functionality to implement ODBC (need to
investigate: may be thin protocol may be extended to unify JDBC and ODBC).

2. About reactive DB API.
I don't know of any industrial standard API for DB reactive access now.
There are several candidates:
2.1. R2DBC look like the popular and alive. See [1];
2.2. ADBA (java.sql2 [2]) looks like not alive. Not sure;
2.3. Custom async/reactive API.
e.g. oracle DB use this way [3].

Igniters, WDYT?

[1]. https://github.com/r2dbc/r2dbc-spi
[2].
http://cr.openjdk.java.net/~lancea/apidoc/java/sql2/package-summary.html
[3].

https://docs.oracle.com/en/database/oracle/oracle-database/21/jjdbc/jdbc-reactive-extensions.html#GUID-1C40C43B-3823-4848-8B5A-D2F97A82F79B

--
Taras Ledkov
Mail-To: tled...@gridgain.com



--
Taras Ledkov
Mail-To: tled...@gridgain.com

[DISCUSS] Query API for Ignite 3.0

[Taras Ledkov]

Hi,

Ignite 3.0 requires a rethinking of the query API.
We have 'cache.query()' and 'SqlFieldQuery' abstractions at the Ignite 
2.x native API and several JDBC implementation for clients.


I propose to think about new query/SQL API for the Ignite 3.0.

My vision is something like this:
Ignite will support two query APIs: standard JDBC (on native) and one of 
reactive DB API.


1. Native JDBC API, e.g.:
    Connection conn = node.sql().connection(props);

JDBC is the industrial standard of the DB access and we have to support one.
Also:
1.1. Thin JDBC client will be really thin: provide network communication 
layer and transparently map to native API.
1.2. Thick JDBC client implementation will be trivial: start client node 
and open JDBC connection on the started node.
1.3. JDBC provides sufficient functionality to implement ODBC (need to 
investigate: may be thin protocol may be extended to unify JDBC and ODBC).


2. About reactive DB API.
I don't know of any industrial standard API for DB reactive access now.
There are several candidates:
2.1. R2DBC look like the popular and alive. See [1];
2.2. ADBA (java.sql2 [2]) looks like not alive. Not sure;
2.3. Custom async/reactive API.
e.g. oracle DB use this way [3].

Igniters, WDYT?

[1]. https://github.com/r2dbc/r2dbc-spi
[2]. 
http://cr.openjdk.java.net/~lancea/apidoc/java/sql2/package-summary.html
[3]. 
https://docs.oracle.com/en/database/oracle/oracle-database/21/jjdbc/jdbc-reactive-extensions.html#GUID-1C40C43B-3823-4848-8B5A-D2F97A82F79B


--
Taras Ledkov
Mail-To: tled...@gridgain.com

[DISCUSSION] Common approach to print sensitive information

[Taras Ledkov]

Hi,

I work on ticket IGNITE-14441 [1] to hide sensitive information at the 
log messages produced by SQL.

There are negative comments for the patch.

I guess we have to produce view to work with sensitive information and 
make rules to define sensitive information.


See on the usage of the GridToStringBuilder#includeSensitive. Class 
names and  field names now are considered sensitive.
My train of thought is this: SQL query and query plan contain table name 
(similar to class name) and field name.

So, the query and plan are completely sensitive.

Lets define sensitive info and work with it for Ignite.

Someone proposes introduce one more Ignite property for print SQL 
sensitive info.

I think this leads to complication.

Introduce levels of the sensitivity make sense but all similar 
information must be handled with the same rules.


Igniters, WDYT?

[1]. https://issues.apache.org/jira/browse/IGNITE-14441

--
Taras Ledkov
Mail-To: tled...@gridgain.com

[jira] [Created] (IGNITE-14471) JDBCv2: query cursors leak when node to execute queries is specified

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14471:
-

 Summary: JDBCv2: query cursors leak when node to execute queries 
is specified
 Key: IGNITE-14471
 URL: https://issues.apache.org/jira/browse/IGNITE-14471
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov


Query cursors leak when node to execute queries is specified.
In this case the query tasks are executed on the remote node instead of client 
node launched by the JDBC v2 client.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14461) Track down those who initiated a query

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14461:
-

 Summary: Track down those who initiated a query
 Key: IGNITE-14461
 URL: https://issues.apache.org/jira/browse/IGNITE-14461
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.11


We need to know and track who initiated a query. It could be useful for an IT 
guy to be able to find an application and ask the developers to change the 
code. It could be as part of {{LOCAL_SQL_RUNNING_QUERIES}} system view or may 
be better to have separate view which could be joined with running queries view.

For first glance it should be :

# ip, port, host name, user for thin clients.
# for thick client - nodeId, clientId, ip, host name, 
# task name for compute




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14451) PK becomes corrupted after node restart

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14451:
-

 Summary: PK becomes corrupted after node restart
 Key: IGNITE-14451
 URL: https://issues.apache.org/jira/browse/IGNITE-14451
 Project: Ignite
  Issue Type: Bug
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.11


PK index tree becomes corrupted after node restart in case it was created 
through SQL API and it's fields order differs from the one in field list. For 
example:
CREATE TABLE t (id1 INTEGER, id2 INTEGER, val VARCHAR, CONSTRAINT PK PRIMARY 
KEY (id2, id1)) - this won't survive restart.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14441) Query plan printed for long running query must use IGNITE_TO_STRING_INCLUDE_SENSITIVE policy

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14441:
-

 Summary: Query plan printed for long running query must use 
IGNITE_TO_STRING_INCLUDE_SENSITIVE policy
 Key: IGNITE-14441
 URL: https://issues.apache.org/jira/browse/IGNITE-14441
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.11


Query plan printed for long running query must use 
IGNITE_TO_STRING_INCLUDE_SENSITIVE policy.
Now query parameters may be printed in the query plan.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14274) Calcite integration: refactoring aggregates converter rules: single and map/reduce

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14274:
-

 Summary: Calcite integration: refactoring aggregates converter 
rules: single and map/reduce
 Key: IGNITE-14274
 URL: https://issues.apache.org/jira/browse/IGNITE-14274
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov


We have to use converter rules to create single & map+reduce aggregates instead 
of creates map/reduce aggregates on handle traits at the single aggregates.
Now:
SortAggregateConverterRule
HashAggregateConverterRule

Must be:
SortSingleAggregateConverterRule
SortMapReduceAggregateConverterRule
HashSingleAggregateConverterRule
HashMapReduceAggregateConverterRule




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-14008) SQL tracing: add tag sql.query.id

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-14008:
-

 Summary: SQL tracing: add tag sql.query.id
 Key: IGNITE-14008
 URL: https://issues.apache.org/jira/browse/IGNITE-14008
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Affects Versions: 2.9.1
Reporter: Taras Ledkov
Assignee: Taras Ledkov


It's hard to find a trace fro specified query.
I think we have to add new tag: {{sql.query.id}} for spans:
- {{sql.command.query.execute}}
- {{sql.dml.query.execute}}
- {{sql.cursor.open}}

In this case user can find a query bu nodeId & queryId.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13919) Calcite integration. Single group aggregates

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-13919:
-

 Summary: Calcite integration. Single group aggregates
 Key: IGNITE-13919
 URL: https://issues.apache.org/jira/browse/IGNITE-13919
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Taras Ledkov


Investigate the optimization for single group aggregates,
e.g.: {{SELECT  FROM }}

Now the hash based aggregates are used for this case. 
We can obtain a little performance improvement.
Also the special cases (e.g.: {{SELECT COUNT() FROM }}) may 
be optimized to use table/cache size.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13904) Don't use rows buffers by redce index for simple one-way reducer

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-13904:
-

 Summary: Don't use rows buffers by redce index for simple one-way 
reducer
 Key: IGNITE-13904
 URL: https://issues.apache.org/jira/browse/IGNITE-13904
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.9
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.10


Now the reducer's index buffer {{AbstractReducer#fetched}} is used even for 
simple plain query. It is the cause of increased memory usage for plain query.

Proposed fix:
- don't use index buffer for simple reducer (for plain query).




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13849) Calcite Integration. Add unit tests for all IgniteRel nodes serialization / deserialization

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-13849:
-

 Summary: Calcite Integration. Add unit tests for all IgniteRel 
nodes serialization / deserialization
 Key: IGNITE-13849
 URL: https://issues.apache.org/jira/browse/IGNITE-13849
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov


We have to check serialization for each node (child classes of the IgniteRel).




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13744) Use Table spool for IgniteNestedLoopJoin

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-13744:
-

 Summary: Use Table spool for IgniteNestedLoopJoin
 Key: IGNITE-13744
 URL: https://issues.apache.org/jira/browse/IGNITE-13744
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov


Now {{NestedLoopJoinNode}} uses internal buffer to save all rows of the right 
input.
We have to do refactoring {{IgniteNestedLoopJoin}} to use rewind of the right 
input and use TableSpool for not rewindable inputs.

This refactoring separates implementation the join logic from materialization 
of the right input if it is needed. In the future we can use disk offload for 
TableSpool etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13677) Calcite integration. TableSpool

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-13677:
-

 Summary: Calcite integration. TableSpool
 Key: IGNITE-13677
 URL: https://issues.apache.org/jira/browse/IGNITE-13677
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov


The first step of the IGNITE-13545: adds TableSpool.

We have to add new trait: {{CorretlationTrait}} to check stream correlation.
Without check this trait Filter with correlated variables may be places under 
Exchange.
So, {{IgniteCorrelatedNestedLoopJoin}} requires correlated stream from the 
right input and {{Exchange}} may produce only uncorrelated stream.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: JAR-based code deployment

[Taras Ledkov]

Hi,

Great idea, but some points are not clear for me:
- Redeploy and versioning. Does it mean that several versions of the 
code will be prohibited?
- Will there be any additional requirements for the content of the JAR? 
Metadata, security etc.

If yes, then maybe we should talk about Ignte ARchive (IAR) likes WAR.

On 25.09.2020 4:42, Valentin Kulichenko wrote:

Igniters,

During the meetup dedicated to Ignite 3.0 [1], one of the items I've
mentioned was JAR based code deployment, as an alternative to peer class
loading and other mechanisms for code deployment.

As promised, I've started drafting the IEP for this change [2]. Please let
me know your thoughts.

[1]
https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0#ApacheIgnite3.0-Meetups
[2]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-59%3A+JAR-based+code+deployment

-Val


--
Taras Ledkov
Mail-To: tled...@gridgain.com

Re: Command line interface to manage distributed properties

[Taras Ledkov]

Hi,

Lets finalize our decision.

1. All distributed properties are published. - It's clear.

2. Do we add the description for the property (add method to the 
interface DistributedProperty)?

If yes: is the description is persisted or hard-coded?

3. Permission:
- add one permission to modify all properties (the value of any property 
may be read without permission)

- separate permission for each property.

I'm OK with both cases. It's obvious that one permission for all 
properties is easier to implement.


On 08.09.2020 17:19, Nikolay Izhikov wrote:

Hello, Anton.


1) Publish distributed property

I propose to use SystemView API to accomplish this.
Actually, there is PR for it, already [1]

Distributed property will be available via SQL - «SELECT * FROM 
SYS.DISTRIBUTED_METASTORAGE»
Or via JMX in the corresponding bean.


2) Permission for distributed properties

Do we really need separate permission for each property?
Would it be enough to have one permission «DISTRIBUTED_PROPERTY_WRITE» or 
similar?
WIth it we allow to the user SET operation for any property.

[1] https://github.com/apache/ignite/pull/8225/files


8 сент. 2020 г., в 16:53, Anton Kalashnikov  написал(а):

Hi everyone,

I think I agree with Nikolay that we should make available all our property not 
only some of them. Also maybe it makes sense to split this task into two tasks 
in the following way: Publish all distributed property through public 
interfaces(control.sh, jmx, etc.), Giving possibility to add permission to some 
properties.

In my opinion, we need the following changes(it is just a high overview of my 
ideas):

1) Publish distributed property
- DistributedConfigurationProcessor new methods: propertyList(): 
List, get(propertyName): DistributedProperty
- DistributedProperty new methods: stringView(), 
propagateFromString(valueAsString) - it's discussable, I still don't sure it is 
great place for such methods. Perhaps we should have a some converter inside of 
control.sh or whatever.

Usage:
List allClusterProperties = 
distributedConfigurationProcessor.list();
allClusterProperties.forEach( prop -> System.out.printl(prop.stringView()) );

DistributedProperty baselineAutoAdjustEnabled = 
distributedConfigurationProcessor.get("baselineAutoAdjustEnabled");
baselineAutoAdjustEnabled.propagateFromString("true");//baselineAutoAdjustEnabled.propagate(true)

Open questions:
- How to update complex objects(any object which is not primitive)?
- Does it ok to convert the object to string inside the DistributedProperty?

2) Permission for distributed properties
- New class for permission - unfortunately, it's broking all 
hierarchy(DistributedLongProperty, DistributedBooleanProperty etc.) but maybe 
it is not a big problem
class PermissibleDistributedProperty extends 
SimpleDistributedProperty> {
PermittedDistributedProperty(key, realValue, readPermission, 
writePermission) {
   super(key, new InnerPermissionWrapper(realValue, readPermission, 
writePermission);
 }
}

- I don't know a lot about ignite security so I don't sure where we should 
check the permission in that case - it can be a new special processor or just 
inside a job

One more idea - instead of creating the PermittedDistributedProperty we can store 
some mapping  separately(it can be 
static mapping or it can be stored in some new DistributedProperty). But in this 
case, it is possible to lost permission after property renaming.

--
Best regards,
Anton Kalashnikov



05.09.2020, 10:09, "Nikolay Izhikov" :

Hello, Taras.

One more thing:


  --property list - prints list of the available properties with description, 
e.g.:

We have a convenient API to show Ignite internal objects - System Views [1]

Any system view available via SQL and JMX.
It seems we should have METASTORAGE view instead of this option.

P.S. Should we add some CMD interface for system views?

[1] https://apacheignite.readme.io/docs/system-views


  3 сент. 2020 г., в 10:37, Nikolay Izhikov  написал(а):

  Hello, Taras.


  I guess some properties (may be future properties) shouldn't be published 
through generic cmd line interface.

  With marker interface user have to wait for a new release to fix not 
published property.
  New release is a very long way for fixing one tiny configuration value.

  Also, we shouldn’t hide anything from the administrator.

  I’m sure that hiding any internals from our users is always a bad idea and 
hides some issue in the codebase.
  Let’s do it in Apache Way? :) - «Not restriction but common sense»

  We can have some kind of `IgniteSystemProperty` with default read-only list 
and description to it -
  «User, you edit this properties fully on your own. We can’t predict results 
of such kind of edits»
  So user can fix this list manually.

  WDYT?


  3 сент. 2020 г., в 10:21, Taras Ledkov  написал(а):

  Hi,


  Why do we want to restrict property management somehow?

  I guess some properties (may be future propert

[jira] [Created] (IGNITE-13403) Update JDBC metadata to match actual capabilities

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-13403:
-

 Summary: Update JDBC metadata to match actual capabilities
 Key: IGNITE-13403
 URL: https://issues.apache.org/jira/browse/IGNITE-13403
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.8.1
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.10


In our JDBC metadata some capabilities are reported not to be supported, while 
in fact they are. E.g. I see
{code}
/** {@inheritDoc} */
@Override public boolean supportsAlterTableWithAddColumn() throws 
SQLException {
return false;
}

/** {@inheritDoc} */
@Override public boolean supportsAlterTableWithDropColumn() throws 
SQLException {
return false;
}
{code}

while in fact `ALTER TABLE ADD/DROP COLUMN` are supported.
Need to go through the list of capabilities and correct them.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Command line interface to manage distributed properties

[Taras Ledkov]

Hi,

>  Why do we want to restrict property management somehow?
I guess some properties (may be future properties) shouldn't be 
published through generic cmd line interface.
May be its require separate more complex cmd line commands, some 
properties may have dependencies and require complex management not only 
set/get.
In this case we can use distributed property without publish one via 
simpel cmd line interface.


On 03.09.2020 10:05, Nikolay Izhikov wrote:

Hello, Taras.

It a shame we don’t have a well-written guide for the development of the Ignite 
management interfaces at the moment.
For now, we have dozen of some management APIs - java, JMX, SQL, control.sh, 
visorcmd.sh, REST

I think we should support 3 manage interfaces for each new command:

* CMD
* JMX
* SQL

You can take as an example implementation of the `KILL` command [1]


If we create the instance of this class the property can be managed by command 
line.

Why do we want to restrict property management somehow?

This operation should be done by the administrator who knows what he or she 
does.
I think we should provide a way to change any property value without any 
restriction for admin.
So our users don’t have to wait «one more release» with only change ‘implements 
SimpleDistributedPublicProperty’ for some property.

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=145724615


2 сент. 2020 г., в 23:11, Taras Ledkov  написал(а):

Hi,

Motivation: we have to manage SQL distributed property by command line and 
introduce common approach to manage distributed properties.
Issue: IGNITE-13186 (see [1])

My proposal is:

Property classes & DistributedConfigurationProcessor changes (see PR [2]):
- introduce PublicProperty interface and implements it at the 
PublicSimpleProperty;
- SimpleDistributedPublicProperty. If we create the instance of this class the 
property can be managed by command line.

Command line interface:
--property list - prints list of the available properties with description, 
e.g.:
 sql.disabledFunctions : Disabled SQL functions
 sql.defaultQueryTimeout : Default query timeout
--property get --name  - prints the property value
--property set --name  --val  - change the property 
value.

Possible we have to add the command:
--property reset --name  - reset property to default value.

Please your comments.
Please pay your attention to concept & design of the publishing a property by 
'PublicProperty' and set of the new commands.

[1]. https://issues.apache.org/jira/browse/IGNITE-13186
[2]. https://github.com/apache/ignite/pull/8208

--
Taras Ledkov
Mail-To: tled...@gridgain.com


--
Taras Ledkov
Mail-To: tled...@gridgain.com

Command line interface to manage distributed properties

[Taras Ledkov]

Hi,

Motivation: we have to manage SQL distributed property by command line 
and introduce common approach to manage distributed properties.

Issue: IGNITE-13186 (see [1])

My proposal is:

Property classes & DistributedConfigurationProcessor changes (see PR [2]):
- introduce PublicProperty interface and implements it at the 
PublicSimpleProperty;
- SimpleDistributedPublicProperty. If we create the instance of this 
class the property can be managed by command line.


Command line interface:
--property list - prints list of the available properties with 
description, e.g.:

    sql.disabledFunctions : Disabled SQL functions
    sql.defaultQueryTimeout : Default query timeout
--property get --name  - prints the property value
--property set --name  --val  - change the 
property value.


Possible we have to add the command:
--property reset --name  - reset property to default value.

Please your comments.
Please pay your attention to concept & design of the publishing a 
property by 'PublicProperty' and set of the new commands.


[1]. https://issues.apache.org/jira/browse/IGNITE-13186
[2]. https://github.com/apache/ignite/pull/8208

--
Taras Ledkov
Mail-To: tled...@gridgain.com

Re: [MTCGA]: new failures in builds [5552028] needs to be handled

[Taras Ledkov]

Hi,

Thanks a lot.
I've already fixed it.

On 19.08.2020 14:42, Ivan Pavlukhin wrote:

A missing link for a previous message https://github.com/apache/ignite/pull/8167

2020-08-19 14:42 GMT+03:00, Ivan Pavlukhin :

Travis gives a green light for a quick fix.

2020-08-19 14:08 GMT+03:00, Ivan Pavlukhin :

Taras,

Could you please take a look into this? It seems that checkstyle fails
build due to a malicious extra space here
https://github.com/apache/ignite/commit/6ff4ac0636da0af89d0f56a417030948b7b5dbee#diff-f025eca4994493a28c47db77d09afdf5R1558

2020-08-19 13:59 GMT+03:00, dpavlov.ta...@gmail.com
:

Hi Igniters,

  I've detected some new issue on TeamCity to be handled. You are more
than
welcomed to help.

  If your changes can lead to this failure(s): We're grateful that you
were
a
volunteer to make the contribution to this project, but things change
and
you may no longer be able to finalize your contribution.
  Could you respond to this email and indicate if you wish to continue
and
fix test failures or step down and some committer may revert you commit.

  *New Critical Failure in master ~Build Apache Ignite~
https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_BuildApacheIgnite?branch=%3Cdefault%3E
  Changes may lead to failure were done by
 - tledkov 
https://ci.ignite.apache.org/viewModification.html?modId=905946

 - Here's a reminder of what contributors were agreed to do
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
 - Should you have any questions please contact dev@ignite.apache.org

Best Regards,
Apache Ignite TeamCity Bot
https://github.com/apache/ignite-teamcity-bot
Notification generated at 13:59:20 19-08-2020



--

Best regards,
Ivan Pavlukhin



--

Best regards,
Ivan Pavlukhin




--
Taras Ledkov
Mail-To: tled...@gridgain.com

[jira] [Created] (IGNITE-13339) Sql query fails with NullPointerException caused by calling CAST in order by clause

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-13339:
-

 Summary: Sql query fails with NullPointerException caused by 
calling CAST in order by clause
 Key: IGNITE-13339
 URL: https://issues.apache.org/jira/browse/IGNITE-13339
 Project: Ignite
  Issue Type: Test
  Components: sql
Affects Versions: 2.8.1
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.9


Sql statements to reproduce:

CREATE TABLE Person(ID INTEGER PRIMARY KEY, NAME VARCHAR(100));
INSERT INTO Person(ID, NAME) VALUES (1, 'Ed'), (2, 'Ann'), (3, 'Emma');
select name,id from Person order by CAST(id as long)

Result:

{code:java}
javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: null
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuerySafe(GridQueryProcessor.java:2572)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:2505)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:2463)
at 
org.apache.ignite.internal.visor.query.VisorQueryUtils.lambda$scheduleQueryStart$0(VisorQueryUtils.java:409)
at 
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7157)
at 
org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:826)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteCheckedException: null
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:3050)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.lambda$querySqlFields$1(GridQueryProcessor.java:2531)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuerySafe(GridQueryProcessor.java:2569)
... 9 more
Caused by: java.lang.NullPointerException
at 
org.apache.ignite.internal.processors.query.h2.sql.GridSqlFunction.getSQL(GridSqlFunction.java:124)
at 
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuery.getSortLimitSQL(GridSqlQuery.java:171)
at 
org.apache.ignite.internal.processors.query.h2.sql.GridSqlSelect.getSQL(GridSqlSelect.java:182)
at 
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.splitQuery(GridSqlQuerySplitter.java:371)
at 
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.split0(GridSqlQuerySplitter.java:280)
at 
org.apache.ignite.internal.processors.query.h2.sql.GridSqlQuerySplitter.split(GridSqlQuerySplitter.java:212)
at 
org.apache.ignite.internal.processors.query.h2.QueryParser.parseH2(QueryParser.java:501)
at 
org.apache.ignite.internal.processors.query.h2.QueryParser.parse0(QueryParser.java:173)
at 
org.apache.ignite.internal.processors.query.h2.QueryParser.parse(QueryParser.java:132)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.querySqlFields(IgniteH2Indexing.java:1115)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:2515)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:2511)
at 
org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:35)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:3027)
... 11 more
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13243) Add tests for commands to manage metadata

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-13243:
-

 Summary: Add tests for commands to manage metadata
 Key: IGNITE-13243
 URL: https://issues.apache.org/jira/browse/IGNITE-13243
 Project: Ignite
  Issue Type: Test
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov


Adds tests for commands to manage metadata (see GG-29247)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13200) SQL create index on invalid data type

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-13200:
-

 Summary: SQL create index on invalid data type
 Key: IGNITE-13200
 URL: https://issues.apache.org/jira/browse/IGNITE-13200
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.8.1
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.9


*Reproduce*
- Create cache with value class
{code}
private static class Value {
@QuerySqlField
int val_int;
java.util.Date val_date;
}
{code}
- alter table with command
{{ALTER TABLE TEST ADD COLUMN (VAL_DATE DATE)}}
- try to create index with command
{{CREATE INDEX TEST_VAL_DATE_IDX ON TEST(VAL_DATE)}}

{{CorruptedTreeException}} is thrown, the node is stopped.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13198) Calcite integration. Rework error / cancle login on execution

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-13198:
-

 Summary: Calcite integration. Rework error / cancle login on 
execution
 Key: IGNITE-13198
 URL: https://issues.apache.org/jira/browse/IGNITE-13198
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov


Change error & cancel logic on query execution.
*Error*
- propagated from source to downstream nodes.
- Always propagated to the RootNode;
- RootNode receive an error and cancel all execution tree.

*Cancel*
- propagated from downstream to source;
- any node may cancel its execution sub-tree;
- (technical details) to prevent race on Inbox creation Outbox resend Cancel 
message to Inbox.





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13186) Commands for control.sh to manage properties at the distributed metastore

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-13186:
-

 Summary: Commands for control.sh to manage properties at the 
distributed metastore
 Key: IGNITE-13186
 URL: https://issues.apache.org/jira/browse/IGNITE-13186
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Affects Versions: 2.8.1
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.9


Now we store some properties (e.g. all properties of the 
{{DistributedSqlConfiguration}}) at the distributed metastore.

Now there is not way to change this properties for user.

*Proposed fix:*
adds command for {{CommandHandler}} to manage properties stored at the 
distibuted metastore. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13183) Query timeout redesign

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-13183:
-

 Summary: Query timeout redesign
 Key: IGNITE-13183
 URL: https://issues.apache.org/jira/browse/IGNITE-13183
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Affects Versions: 2.8.1
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.9


*Motivation:*
Now the query timeout is set up for each node separately by the node 
configuration. This property isn't propagated for the all nodes of cluster. 
Also user cannot change / disable query timeout without restart all nodes of 
the cluster.

*Proposal fix:*
- Adds the default query timeout property to the 
{{DistributedSqlConfiguration}}. Use distributed metastore to store and manage 
the property.
- Deprecates the {{SqlConfiguration#defaultQueryTimeout}} property and use it 
to set up initial value of new property 
{{DistributedSqlConfiguration#defaultQueryTimeout}}
- Adds info about explicit query timeout to {{GridH2QueryRequest}} (boolean 
flag {{explicitTimeout=false}} by default). This is necessary so that the 
default timeout may be used for queries from old nodes.
- When query timeout is set to zero by old node ({{explicitTimeout=false}}) we 
assume this is the default value and use default timeout for this queries.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13179) Calcite integration. Fix traits at the IgniteLimit

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-13179:
-

 Summary: Calcite integration. Fix traits at the IgniteLimit
 Key: IGNITE-13179
 URL: https://issues.apache.org/jira/browse/IGNITE-13179
 Project: Ignite
  Issue Type: Improvement
Reporter: Taras Ledkov
Assignee: Taras Ledkov


Current implementation of {{IgniteLimit}} doesn't  support distribution trait 
propagation.
Limit must require singleton distribution.

Now we cannot reproduce invalid behavior (see IGNITE-12819) and IgniteLimit 
isn't placed under the Exchange node.

Also Limit must be propogated (copied) under Exchange to reduce unnecessary 
fetches.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Introduce the ability for a user to manage binary types [IGNITE-13154]

[Taras Ledkov]

Hi,

I change the error message [1].
The suggestion at the message contains info about requirement to remove 
all data with old schema,
then remove metadata by control.sh or manually by delete files and 
restart grid.



[1]. 
https://github.com/apache/ignite/pull/7936/files#diff-cb1981ab1a44f213b4cadafd80c611c5R1047


On 19.06.2020 1:55, Denis Magda wrote:

Glad to see it coming! That was a pain in the neck.

Could we also add a hint to the exception message when a type mismatch
happens? Basically, the exception should not only report details about the
mismatch but also suggest how to solve it. The text can mention these new
control.sh commands and advise to remove the old metadata.

-
Denis


On Thu, Jun 18, 2020 at 7:42 AM Taras Ledkov  wrote:


Hi,

I propose to introduce ability for a user to manage cluster binary types.
This functionality is useful when user needs to change schema without
cluster restart.

Motivation:
Rid users of manual operations in the folder 'binary_meta' and cluster
restart when they need change schema.

The proposed implementation contains restrictions (see [1]).
So I guess we add  internal API and command for public command line tool
(control.sh)
with mode 'experimental' mode (the property
IGNITE_ENABLE_EXPERIMENTAL_COMMAND must be set).

Please take a look at the ticket / patch.

Any comments / suggestions?

[1]. https://issues.apache.org/jira/browse/IGNITE-13154

--
Taras Ledkov
Mail-To: tled...@gridgain.com



--
Taras Ledkov
Mail-To: tled...@gridgain.com

Introduce the ability for a user to manage binary types [IGNITE-13154]

[Taras Ledkov]

Hi,

I propose to introduce ability for a user to manage cluster binary types.
This functionality is useful when user needs to change schema without 
cluster restart.


Motivation:
Rid users of manual operations in the folder 'binary_meta' and cluster 
restart when they need change schema.


The proposed implementation contains restrictions (see [1]).
So I guess we add  internal API and command for public command line tool 
(control.sh)
with mode 'experimental' mode (the property 
IGNITE_ENABLE_EXPERIMENTAL_COMMAND must be set).


Please take a look at the ticket / patch.

Any comments / suggestions?

[1]. https://issues.apache.org/jira/browse/IGNITE-13154

--
Taras Ledkov
Mail-To: tled...@gridgain.com

[jira] [Created] (IGNITE-13154) Introduce aility to manage manage binary types by the users

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-13154:
-

 Summary: Introduce aility to manage manage binary types by the 
users
 Key: IGNITE-13154
 URL: https://issues.apache.org/jira/browse/IGNITE-13154
 Project: Ignite
  Issue Type: Improvement
  Components: binary
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.9


We need a way to change schema (including unsupported changes, such as column 
type change) without cluster restart. This is for the case when all data 
associated with the binary type has been removed, so removing the old schema is 
safe.

Now users must stop the cluster and remove the folder with binary metadata 
manually.

The proposed way is to introduce internal API to manage binary types and public 
command line interface (via control.sh). That way one can remove a cache from 
the cluster, then unregister corresponding binary types, then launch a new 
version of an application that would register the new schema and reload the 
data.

*The current implementation has restrictions:*
- all cluster nodes must support remove type feature.
- the cluster must not contains data with type to remove.
- operation of the update type must not be launched on the cluster for the type 
to remove (operation examples: put into cache, BinaryObjectBuilder#build).
- client nodes process metadata operation asynchronously so type may be removed 
at the client after any delay after the remove type operation is completed.
- if the node that contains the old version of the type joins to the cluster 
where type was removed the type is propagated to cluster metadata (because 
metadata tombstone not supported).
- if the node that contains the old version of the type cannot join to the 
cluster where type was removed and then updated to the new version (because 
metadata versioned tombstone not supported).

So, user scenarios looks like:
# Be sure that all server nodes supports remove type feature.
# Remove caches contains the data with type to remove.
# Stops the client node with older version.
# Stops all operation with type to remove (don't create binary objects, don't 
run compute jobs with type to remove).
# Remove the type on the stable topology (and production destination topolog).
# Waits any delay (depends on the cluster size and clients count)
# Produce operations with new version of the type.

*Proposed command line interface*
New commands (all commands are _experimental_ ):
- {{--meta list}} prints info about all available binary types:
{{typeId=, typeName=, fields=, schemas=, 
isEnum=}}
- {{\-\-meta details (\-\- typeId | \-\-typeName )}} prints detailed 
info info about specified type. The type may be specified by type name or type 
ID.
output example:
{code}
typeId=0x1FBFBC0C (532659212)
typeName=TypeName1
Fields:
  name=fld3, type=long[], fieldId=0x2FFF95 (3145621)
  name=fld2, type=double, fieldId=0x2FFF94 (3145620)
  name=fld1, type=Date, fieldId=0x2FFF93 (3145619)
  name=fld0, type=int, fieldId=0x2FFF92 (3145618)
Schemas:
  schemaId=0x6C5CC179 (1818018169), fields=[fld0]
  schemaId=0x70E46431 (1894016049), fields=[fld0, fld1, fld2, fld3]
{code}
- {{\-\-meta remove (\-\- typeId | \-\-typeName ) [\-\-out 
]}} removes metadata for specified type form cluster and saves the 
removed metadata to the specified file. If the file name isn't specified the 
output file name is: {{.bin}}
The command requires confirmation.
*N.B.*: The all session of thin clients (ODBC, JDBC, thin client) are closed 
(to cleanup local cache of the binary metadata).
- {{\-\-meta update \-\-in ]}} update cluster metadata from 
specified file (file name is required)
The command requires confirmation.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-13068) SQL: use cache.localSize instead of index scan to calculate row count statistic

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-13068:
-

 Summary: SQL: use cache.localSize instead of index scan to 
calculate row count statistic 
 Key: IGNITE-13068
 URL: https://issues.apache.org/jira/browse/IGNITE-13068
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.9


The scan of index is slow on big data set on setup where disc i/o requires warm 
up.
This is the reason that restarting the node can take a long time.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Apache Ignite 2.8.1 RELEASE [Time, Scope, Manager]

[Taras Ledkov]

 changed probably.

IGNITE-12735 - Metric exporter implementation could lead to

NullPointerException from gauge which invoke communication

IGNITE-12568 - MessageFactory implementations refactoring

Ivan,
Personally, I’m against any refactoring improvements in bug fix

release.

So, I propose to exclude IGNITE-12756 from 2.8.1

Andrey, what do you think as a committer of this improvements?


21 апр. 2020 г., в 10:44, Alex Plehanov 


написал(а):

Nikolay,

I've cherry-picked IGNITE-12734 to 2.8.1 branch.

вт, 21 апр. 2020 г. в 10:02, Ivan Bessonov 

:

Hello, Igniters,

considering commit "683f22e64f IGNITE-12756 TcpCommunication 
SPI

metrics

improvement" - it depends
on https://issues.apache.org/jira/browse/IGNITE-12735 and
https://issues.apache.org/jira/browse/IGNITE-12568,
both were targeted to 2.9, but this has to be changed probably.

There might be other issues that I didn't find, we should

probably ask

Andrey Gura about it, he is the author of
those changes.

So, release scope should be expanded a little bit, are you ok

with it?
пн, 20 апр. 2020 г. в 19:24, Nikolay Izhikov 

:

Hello, Igniters.

I perform cherry-pick for most commits targeted for 2.8.1

release.

TC bot results -

https://mtcga.gridgain.com/pr.html?serverId=apache=IgniteTests24Java8_RunAll=pull%2F7690%2Fhead=Latest=pull%2F7102%2Fhead 


I need help with cherry picking the following commits:

- 4e6cd2ce04 IGNITE-12759 Getting a SecurityContext from
GridSecurityProcessor - Fixes #7523.
- 60ebc23f99 IGNITE-12848 fix H2Connection leaks on INSERT

(#7649)

- 0b185b192f IGNITE-12800  SQL: local queries cursors must be

closed

or

full read to unlock the GridH2Table. (#7551)
- bcaae8deef IGNITE-12734 Fixed scan query over evicted

partition -

Fixes

#7494.
- 683f22e64f IGNITE-12756 TcpCommunication SPI metrics

improvement

Denis Garus,
Taras Ledkov,
Alexey Plekhanov,
Ivan Bessonov,

Can you help me with above commits and cherry-pick your

contribution

in

ignite-2.8.1 branch?

CHERRY PICKED:

+ d0c155fe43 IGNITE-12772 Fixed JmxMetricExporterSpi uses log

method

which

must not be used in production code (#7604)
+ 00cb1ad7a3 IGNITE-12764: Thin JDBC streaming fails

BatchUpdateException

if function is used (#7615)
+ b8167296b1 IGNITE-12805 Fixed NullPointerException when 
memory

restore

is in progress. Fixes #7562
+ f57509e8e6 IGNITE-12828 Fixes NPE during CQ registration 
with

failed

deployment. (#7620)
+ 826aad8890 IGNITE-12726 Long keys support for metastorage.

(#7606)
+ 2b1d2b4dec IGNITE-12859: Support of .Net service call 
with the

Timestamp

and Guid params. (#7618)
+ 6f3515686f IGNITE-12769: histogramNames cache in

MetricRegistryMBean

removed. (#7549)
+ 795617fc94 IGNITE-12774 Handle "Too many open files" 
exception

-

Fixes

#7516.
+ 3928bb3a13 IGNITE-12799 Fixed wrong SpEL expression.
+ ef4f67e351 IGNITE-12743 Java thin client: Fixed thread

shutdown on
client close when partition awareness is enabled - Fixes 
#7522.

+ e389bb8f55 IGNITE-12728 Collect time statistics on

cache#putAllAsync -

Fixes #7483.
+ 90951c6b2e IGNITE-12711 Fixed tests memory usage. - Fixes

#7469.

+ f32b44deb1 IGNITE-12590: Fix (remove) check KEY at the MERGE

command

(#7321)
+ 59917f0731 IGNITE-12624 Java thin client: typeId 
generation for

system

types fixed - Fixes #7416.
+ cc6f4d7814 IGNITE-12671 Ignoring single messages during 
PME can

prevent

late affinity switch. - Fixes #7425.
+ 02ac292662 IGNITE-11798 Fix memory leak on unstable topology

caused by

partition reservation (#7251)
+ 8ed8576544 IGNITE-12665: SQL: Fix potential race on 
MapResult

close.
+ 465cc444d0 IGNITE-12582 Add Spring EL support in Spring 
Data. -

Fixes

#7411.
+ 100101ccce IGNITE-12605: Reset initial update counter value

before

clearing a partition (#7341)
+ e17887bfbf IGNITE-12654 Some of rentingFutures in
GridDhtPartitionTopologyImpl may accumulate a huge number of

eviction

callbacks - Fixes #7399.
+ bdbe6a79d0 IGNITE-12651 Non-comparable keys for eviction 
policy

cause

failure handle and node shutdown - Fixes #7397.
+ 56a515db6d IGNITE-12631 Incorrect rewriting wal record 
type in

marshalled mode during iteration - Fixes #7371.
+ 14dd160f90 IGNITE-12621 Node leave may cause

NullPointerException

during

IO message processing if security is enabled - Fixes #7366.
+ 67ac1d5d68 IGNITE-12636 Full rebalance instead of a 
historical

one

-

Fixes #7379.
+ 4433485d74 IGNTIE-12468 Java thin client: deserialization of

arrays,

collections and maps fixed - Fixes #7320.
+ 0dfd98388e IGNITE-12618 Affinity cache for version of last

server

event

can be wiped from history - Fixes #7359.
+ e2c597fff1 IGNITE-12013 NullPointerException is thrown by
ExchangeLatchManager during cache creation - Fixes #7335.
+ cebc76f1f5d7d077093ddaecbabcd8db2106c60a - IGNITE-12545

Introduce

listener interface for components to react to partition map

exchange

events
+ a9278eedf7 IGNITE-11797 Fixed partition consistency 
issues for

atomic

and mixed tx-at

Re: Apache Ignite 2.8.1 RELEASE [Time, Scope, Manager]

[Taras Ledkov]

Hi,

Nikolay, i've created PR [1] that contains the SQL-related tickets to 
port into 2.8.1:
- IGNITE-12790 Introduce distributed SQL configuration and ability to 
disable SQL functions.
- IGNITE-12887 Fix handle type mismatch exception on compare values 
while traversing index tree.

- IGNITE-12848 fix H2Connection leaks on INSERT
- IGNITE-12800  SQL: local queries cursors must be closed or full read 
to unlock the GridH2Table.

- IGNITE-12067 User SQL query execution metrics
- IGNITE-12732 Fix for KILL QUERY command hanging.
- IGNITE-12609: SQL: GridReduceQueryExecutor refactoring.
- IGNITE-12444: SQL: Query reduce can fail with NPE on retry.

Wait to TC results and visa from TC bot.

[1]. https://github.com/apache/ignite/pull/7703

On 21.04.2020 12:52, Nikolay Izhikov wrote:

Igniters.

Can someone from Grid Gain help me with setup TC bot to monitor ignite-2.8.1 
branch?


21 апр. 2020 г., в 11:11, Nikolay Izhikov  написал(а):


I've cherry-picked IGNITE-12734 to 2.8.1 branch.


Thanks!


considering commit "683f22e64f IGNITE-12756 TcpCommunication SPI metrics

improvement" - it depends
on https://issues.apache.org/jira/browse/IGNITE-12735 and
https://issues.apache.org/jira/browse/IGNITE-12568,
both were targeted to 2.9, but this has to be changed probably.

IGNITE-12735 - Metric exporter implementation could lead to 
NullPointerException from gauge which invoke communication
IGNITE-12568 - MessageFactory implementations refactoring

Ivan,
Personally, I’m against any refactoring improvements in bug fix release.
So, I propose to exclude IGNITE-12756 from 2.8.1

Andrey, what do you think as a committer of this improvements?



21 апр. 2020 г., в 10:44, Alex Plehanov  написал(а):

Nikolay,

I've cherry-picked IGNITE-12734 to 2.8.1 branch.

вт, 21 апр. 2020 г. в 10:02, Ivan Bessonov :


Hello, Igniters,

considering commit "683f22e64f IGNITE-12756 TcpCommunication SPI metrics
improvement" - it depends
on https://issues.apache.org/jira/browse/IGNITE-12735 and
https://issues.apache.org/jira/browse/IGNITE-12568,
both were targeted to 2.9, but this has to be changed probably.

There might be other issues that I didn't find, we should probably ask
Andrey Gura about it, he is the author of
those changes.

So, release scope should be expanded a little bit, are you ok with it?

пн, 20 апр. 2020 г. в 19:24, Nikolay Izhikov :


Hello, Igniters.

I perform cherry-pick for most commits targeted for 2.8.1 release.

TC bot results -


https://mtcga.gridgain.com/pr.html?serverId=apache=IgniteTests24Java8_RunAll=pull%2F7690%2Fhead=Latest=pull%2F7102%2Fhead

I need help with cherry picking the following commits:

- 4e6cd2ce04 IGNITE-12759 Getting a SecurityContext from
GridSecurityProcessor - Fixes #7523.
- 60ebc23f99 IGNITE-12848 fix H2Connection leaks on INSERT (#7649)
- 0b185b192f IGNITE-12800  SQL: local queries cursors must be closed or
full read to unlock the GridH2Table. (#7551)
- bcaae8deef IGNITE-12734 Fixed scan query over evicted partition - Fixes
#7494.
- 683f22e64f IGNITE-12756 TcpCommunication SPI metrics improvement

Denis Garus,
Taras Ledkov,
Alexey Plekhanov,
Ivan Bessonov,

Can you help me with above commits and cherry-pick your contribution in
ignite-2.8.1 branch?

CHERRY PICKED:

+ d0c155fe43 IGNITE-12772 Fixed JmxMetricExporterSpi uses log method

which

must not be used in production code (#7604)
+ 00cb1ad7a3 IGNITE-12764: Thin JDBC streaming fails BatchUpdateException
if function is used (#7615)
+ b8167296b1 IGNITE-12805 Fixed NullPointerException when memory restore
is in progress. Fixes #7562
+ f57509e8e6 IGNITE-12828 Fixes NPE during CQ registration with failed
deployment. (#7620)
+ 826aad8890 IGNITE-12726 Long keys support for metastorage. (#7606)
+ 2b1d2b4dec IGNITE-12859: Support of .Net service call with the

Timestamp

and Guid params. (#7618)
+ 6f3515686f IGNITE-12769: histogramNames cache in MetricRegistryMBean
removed. (#7549)
+ 795617fc94 IGNITE-12774 Handle "Too many open files" exception - Fixes
#7516.
+ 3928bb3a13 IGNITE-12799 Fixed wrong SpEL expression.
+ ef4f67e351 IGNITE-12743 Java thin client: Fixed thread shutdown on
client close when partition awareness is enabled - Fixes #7522.
+ e389bb8f55 IGNITE-12728 Collect time statistics on cache#putAllAsync -
Fixes #7483.
+ 90951c6b2e IGNITE-12711 Fixed tests memory usage. - Fixes #7469.
+ f32b44deb1 IGNITE-12590: Fix (remove) check KEY at the MERGE command
(#7321)
+ 59917f0731 IGNITE-12624 Java thin client: typeId generation for system
types fixed - Fixes #7416.
+ cc6f4d7814 IGNITE-12671 Ignoring single messages during PME can prevent
late affinity switch. - Fixes #7425.
+ 02ac292662 IGNITE-11798 Fix memory leak on unstable topology caused by
partition reservation (#7251)
+ 8ed8576544 IGNITE-12665: SQL: Fix potential race on MapResult close.
+ 465cc444d0 IGNITE-12582 Add Spring EL support in Spring Data. - Fixes
#7411.
+ 100101ccce IGNITE-12605: Reset initial update counter value before
clearing a partiti

[jira] [Created] (IGNITE-12917) SQL: print warning log message when query's result is big

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-12917:
-

 Summary: SQL: print warning log message when query's result is big
 Key: IGNITE-12917
 URL: https://issues.apache.org/jira/browse/IGNITE-12917
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.9


We have to track queries with big result set.
Print warning log message when query's result is bigger than threshold 
specified by JMX.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12903) Fix ML + SQL examples

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-12903:
-

 Summary: Fix ML + SQL examples
 Key: IGNITE-12903
 URL: https://issues.apache.org/jira/browse/IGNITE-12903
 Project: Ignite
  Issue Type: Task
  Components: examples
Reporter: Taras Ledkov
Assignee: Taras Ledkov


The examples
{{DecisionTreeClassificationTrainerSQLInferenceExample}}
{{DecisionTreeClassificationTrainerSQLTableExample}}
are used CSVREAD function to initial load data into cluster.

Must be changed because this function is disabled by default



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12887) Node stops on type mismatch error between index type and type of value from searched row

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-12887:
-

 Summary: Node stops on type mismatch error between index type and 
type of value from searched row
 Key: IGNITE-12887
 URL: https://issues.apache.org/jira/browse/IGNITE-12887
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.8
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.8.1


Steps to reproduce:
1. Create table with value fields types: (INT, OTHER) and create indexes for 
the fields:
{code}
CREATE TABLE TEST (ID INT PRIMARY KEY, val_int INT, VAL_OBJ OTHER)
CREATE INDEX TEST_VAL_INT ON TEST(VAL_INT)
CREATE INDEX TEST_VAL_OBJ ON TEST(VAL_OBJ)
{code}

2. Add any data to the table:
{code}
INSERT INTO TEST VALUES (0, 0, ?)
{code}

3. Any of the query below crushes and node stops:
SELECT * FROM TEST WHERE VAL_OBJ < CURRENT_TIMESTAMP()
SELECT * FROM TEST WHERE VAL_INT < CURRENT_TIMESTAMP()


*Root cause*: all runtime exception inside {{Index.find}} is converted to 
{{CorruptedTreeException}} and stops the node,



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12886) Introduce separate SQL configuration

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-12886:
-

 Summary: Introduce separate SQL configuration
 Key: IGNITE-12886
 URL: https://issues.apache.org/jira/browse/IGNITE-12886
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.9


A lot of SQL-related configuration parameter are placed at the root of the 
{{IgniteConfiguration}}.
It would be better to move them to a separate configuration class, e.g. 
{{SqlConfiguration}}.

Thread on [Ignite developers 
list|http://apache-ignite-developers.2346864.n4.nabble.com/Introduce-separate-SQL-configuration-td46636.html]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12870) Fix test JdbcThinTransactionsLeaksMvccTest after connection manager refactoring

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-12870:
-

 Summary: Fix test JdbcThinTransactionsLeaksMvccTest after 
connection manager refactoring
 Key: IGNITE-12870
 URL: https://issues.apache.org/jira/browse/IGNITE-12870
 Project: Ignite
  Issue Type: Test
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.9


The test {{JdbcThinTransactionsLeaksMvccTest#testLeaks}} is broken by the patch 
IGNITE-12804. The method  {{detachedConnectionCount}} must be change to check 
used connection after connection manager refactoring.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12848) SQL: H2Connection leaks on INSERT

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-12848:
-

 Summary: SQL: H2Connection leaks on INSERT
 Key: IGNITE-12848
 URL: https://issues.apache.org/jira/browse/IGNITE-12848
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.8
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.8.1


H2 connection leaks on INSERT query when one row is inserted and not constant 
values are used:
e.g.:
{{INSERT INTO T VALUES(1, CURRENT_TIMESTAM());}}

*Root cause:*
Query cursor isn't closed at the IgniteH2Indexing#executeUpdateNonTransactional 
after {{DmlUtils#processSelectResult}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Introduce separate SQL configuration

[Taras Ledkov]

Hi,

> It would be nice to provide a list of properties that can be changed 
at runtime


Ok, I catch the your and Pavel's opinion.
My be add javadoc for each changed parameter instead of list?

e.g. for longQueryWarningTimeout
See {@link 
org.apache.ignite.internal.mxbean.SqlQueryMXBean#setLongQueryWarningTimeout} 


to change this parameter in runtime.

But we cannot reference internal packages in public API.

On 23.03.2020 17:51, Вячеслав Коптилин wrote:

Hello Taras,

I think this is the right way.
I agree with Pavel's comment. It would be nice to provide a list of
properties that can be changed at runtime and remove the word “Initial”
from the class name.

Thanks,
S.

пн, 23 мар. 2020 г. в 16:54, Pavel Tupitsyn :


Hi Taras,

I support the idea, it brings some order to the IgniteConfiguration.

However, I have strong objections to `Initial` prefix:

many parameters may be changed in runtime

This applies to many other config properties, but we don't name them
`Initial`

Let's just make it SqlConfiguration.

On Mon, Mar 23, 2020 at 1:04 PM Taras Ledkov  wrote:


Hi, Igniters.

I propose to introduce separate configuration class for SQL

configuration.

e.g. SqlInitailConfiguration.

Now we have several SQL parameters:
- sqlSchemas;
- defaultQueryTimeout;
- longQueryWarningTimeout;
- sqlQueryHistorySize;

are placed at the IgniteConfiguration.
I propose to deprecate the SQL properties in the IgniteConfiguration and
redirect the calls to child configuration object of
SqlInitailConfiguration.

We are going to add several additional parameters in the nearest future.
I guess separate configuration is better than many plain SQL parameters
at the root IgniteConfiguration.

I insist on the configuration be called 'Initial' because many
parameters may be changed in runtime
(e.g. longQueryWarningTimeout by JMX, set of schemas by creating caches
etc).

I've created PR [1] that contains proposal changes. The patch is not
final clean.
I think PR may be useful to demonstrate the proposal.

Please let me know your opinion.

[1]. https://github.com/apache/ignite/pull/7559


--
Taras Ledkov
Mail-To: tled...@gridgain.com



--
Taras Ledkov
Mail-To: tled...@gridgain.com

Introduce separate SQL configuration

[Taras Ledkov]

Hi, Igniters.

I propose to introduce separate configuration class for SQL configuration.
e.g. SqlInitailConfiguration.

Now we have several SQL parameters:
- sqlSchemas;
- defaultQueryTimeout;
- longQueryWarningTimeout;
- sqlQueryHistorySize;

are placed at the IgniteConfiguration.
I propose to deprecate the SQL properties in the IgniteConfiguration and 
redirect the calls to child configuration object of SqlInitailConfiguration.


We are going to add several additional parameters in the nearest future.
I guess separate configuration is better than many plain SQL parameters 
at the root IgniteConfiguration.


I insist on the configuration be called 'Initial' because many 
parameters may be changed in runtime
(e.g. longQueryWarningTimeout by JMX, set of schemas by creating caches 
etc).


I've created PR [1] that contains proposal changes. The patch is not 
final clean.

I think PR may be useful to demonstrate the proposal.

Please let me know your opinion.

[1]. https://github.com/apache/ignite/pull/7559


--
Taras Ledkov
Mail-To: tled...@gridgain.com

Re: Apache Ignite 2.8.1 RELEASE [Time, Scope, Manager]

[Taras Ledkov]

Hi,

I propose to add the issue [1] related to SQL query execution to this scope.

We had omitted this case and Ignite 2.8 contains serious SQL issue: 
cursor of a local query is not thread-safe.
It is root cause of several SQL issue, e.g. JDBC thin client cannot 
execute query  from replicated cache,

PME may hang after execute such queries from JDBC thin, etc.

[1]. https://issues.apache.org/jira/browse/IGNITE-12800

On 19.03.2020 17:52, Denis Magda wrote:

Igniters,

As long as 2.8.1 is inevitable and we already keep adding critical issues
to the working queue, let's settle on the release time frames and decide
who will be a release manager. This is the time proposed by Maxim and,
personally, I concur with such a schedule:

- Scope Freeze: April 15, 2020
- Code Freeze: April 22, 2020
- Voting Date: April 27, 2020
- Release Date: May 1, 2020

Do we agree on this time? Is there anybody who ready to drive the release
as a release manager?

-
Denis


On Thu, Mar 19, 2020 at 5:50 AM Sergey Antonov 
wrote:


Folks,

I'd like to add ticket IGNITE-12774 Transaction hangs after too many open
files NIO exception [1] to ignite-2.8.1 scope.

[1] https://issues.apache.org/jira/browse/IGNITE-12774

ср, 18 мар. 2020 г. в 16:53, Maxim Muzafarov :


Folks,

Can we add ignite-2.8.1 [2] branch under TC.Bot protection [1]?


[1] https://mtcga.gridgain.com/guard.html
[2] https://github.com/apache/ignite/tree/ignite-2.8.1

On Mon, 16 Mar 2020 at 16:32, Alexey Goncharuk
 wrote:

Folks,

I've walked through all the commits to master since 2.8 branch was cut

and

filtered some tickets that in my opinion are worth including to 2.8.1
release below (note that they are ready end the effort of including

them

to

the release should be low as long as there are no implicit dependencies
between tickets). Please share your opinion on whether we should

include

them to the 2.8.1.

IGNITE-12717 SQL: index creation refactoring
IGNITE-12590 MERGE INTO query is failing on Ignite client node
IGNITE-12671 Update of partition's states can stuck when rebalance
completed during exchange
IGNITE-11798 Memory leak on unstable topology caused by partition
reservation
IGNITE-12665 SQL: Potential race on MapResult close.
IGNITE-12605 Historical (WAL) rebalance can start on a cleared

partition

if

some baseline node leaves the cluster and then joins back.
IGNITE-12654 Some of rentingFutures in GridDhtPartitionTopologyImpl may
accumulate a huge number of eviction callbacks
IGNITE-12631 Incorrect rewriting wal record type in marshalled mode

during

iteration
IGNITE-12621 Node leave may cause NullPointerException during IO

message

processing if security is enabled
IGNITE-12636 Full rebalance instead of a historical one
IGNITE-12618 Affinity cache for version of last server event can be

wiped

from history
IGNITE-12013 NullPointerException is thrown by ExchangeLatchManager

during

cache creation
IGNITE-11797 Fix consistency issues for atomic and mixed tx-atomic

cache

groups.
IGNITE-12557 Destroy of big cache which is not only cache in cache

group

causes IgniteOOME
IGNITE-12567 H2Tree goes into illegal state when non-indexed columns

are

dropped
IGNITE-12569 Can't set serialized enum to a BinaryObject's field
IGNITE-12460 Cluster fails to find the node by consistent ID
IGNITE-12459 Searching checkpoint record in WAL doesn't work with

segment

compaction
IGNITE-12548 Possible tx desync during recovery on near node left.
IGNITE-12546 Prevent partitions owned by other nodes switch their state

to

MOVING due to counter difference on node join.
IGNITE-12551 Partition desync if a partition is evicted then owned

again

and historically rebalanced
IGNITE-12536 Inconsistency between cache data and indexes when cache
operation is interrupted
IGNITE-12403 Throttle page difference output in PageMemoryTracker
IGNITE-12523 Continuously generated thread dumps in failure processor

slow

down the whole system
IGNITE-12489 Error during purges by expiration: Unknown page type


--
BR, Sergey Antonov


--
Taras Ledkov
Mail-To: tled...@gridgain.com

[jira] [Created] (IGNITE-12804) SQL: ConnectionManager refactoring

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-12804:
-

 Summary: SQL: ConnectionManager refactoring
 Key: IGNITE-12804
 URL: https://issues.apache.org/jira/browse/IGNITE-12804
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.8
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.9


As of now ConnectionManager is a mess, really hard to understand. We should 
refactor it.

AlsoThreadLocal connection is a root cause of complicated behavior of local 
queries & map queries.

I guess we have to use simple connection pool. 
For better performance several striped by Thread ID connection pools may be 
used.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12800) SQL: local queries cursors must be closed or full read to unlock the GridH2Table.

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-12800:
-

 Summary: SQL: local queries cursors must be closed or full read to 
unlock the GridH2Table.
 Key: IGNITE-12800
 URL: https://issues.apache.org/jira/browse/IGNITE-12800
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.8
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.9


*Root cause:* local queries cursors must be closed or full read to unlock the 
GridH2Table.

*Proposal fix:*
- modify {{H2ResultSetIterator}} to use "paged mode": iterator reads N records 
into internal buffer and unlock the tables (similar to {{MapQueryResult}}; 
later we have to refactor these classes. They must use one code base to fetch 
data and lok/unlock tables)
- modify the state logic of the {{QueryCursorImpl}} for lazy mode. Now the real 
query cancellation isn't called when result set is gathered. It is not valid 
for lazy mode.
- add wrapper for iterator at the {{RegisteredQueryCursor}} because the state 
of query isn't tracked when results are read via Iterator at the client code.
- fix tests that doesn't close query cursor for local queries.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12740) Supports feature flags on index meta pages

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-12740:
-

 Summary: Supports feature flags on index meta pages
 Key: IGNITE-12740
 URL: https://issues.apache.org/jira/browse/IGNITE-12740
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.9


Some changes on indexing e.g. inl;ine java objects, unwrap PK columns and 
several planned features (change inline pojo format, inline DECIMAL fields etc) 
break backward compatibility.

We have to add metadata about index layout and format to index tree meta-page.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12717) SQL: indexing DB object creation refactoring

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-12717:
-

 Summary: SQL: indexing DB object creation refactoring
 Key: IGNITE-12717
 URL: https://issues.apache.org/jira/browse/IGNITE-12717
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.9


Do refactoring children of the H2 IndexBase to simplify the H2 upgrade.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12590) MERGE INTO query is failing on Ignite client node

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-12590:
-

 Summary: MERGE INTO query is failing on Ignite client node
 Key: IGNITE-12590
 URL: https://issues.apache.org/jira/browse/IGNITE-12590
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.7
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.9


Precondition: server nodes (any amount), webconsole is running.

1. Create the table as following:

CREATE TABLE USERPUBSTATICDATA (BOOK VARCHAR, DESK VARCHAR, TRADERS VARCHAR, 
REGION VARCHAR, LOB VARCHAR, EXCLUDE VARCHAR, TRANSIT VARCHAR, 
MAPBOOKTOTHISBOOK VARCHAR, CONSTRAINT USERPUBSTATICDATA_PK PRIMARY KEY 
(BOOK,DESK)) WITH "template=replicated"

2. Execute merge into query:

MERGE INTO USERPUBSTATICDATA(BOOK, DESK, TRADERS, REGION, LOB, EXCLUDE, 
TRANSIT, MAPBOOKTOTHISBOOK) VALUES('CADOIS', 'FRT TOR', 'Robin Das/Dave 
Carlson', 'Toronto', 'FRT', null, null, 'CADOIS');

Step 2 is successfull on Web Console and called on IgniteCache from the server 
node, but fails when called from the Ignite client node with exception:
{code}
javax.cache.CacheException: Invalid column name in KEYS clause of MERGE - it 
may include only key and/or affinity columns: BOOK
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Feature masks for thin clients

[Taras Ledkov]

Alexei,

After the flags is introduced we can change the flag set instead of 
change protocol version.

Of course, we will need to up the protocol  version for introducing flags.

On 23.01.2020 15:47, Alexei Scherbakov wrote:

Igor Sapego,

I do not understand how feature masks can remove the necessity of having
protocol versioning.
A protocol for one feature can change from release to release.

чт, 23 янв. 2020 г. в 15:36, Igor Sapego :


Hi Igniters,

As we have a lot of different thin clients now, maintained by different
people, the issues with our backward compatibility mechanism becomes
more and more prominent.

Currently, we use protocol versioning as the only approach to provide
backward compatibility. The main issue of this approach is that we can
not skip some change in protocol and implement i.e. protocol of version
1.5 without implementation of 1.4. There are many cases when one may
want to do so: e.g. when feature provided in 1.4 is not relevant for a
specific client, or when protocol version 1.5 contains urgent fix or
feature
which is easy to implement, but its blocked by not-so-urgent and
hard-to-implement feature introduced in 1.4.

So to fix this issue I propose to introduce another backward compatibility
mechanism. The idea is to send "supported features" mask by a client to
a server, which should be answered with the same mask by the server.
The resulting set of enabled features is acquired with a simple logical
"AND"
operation on these two masks.

This change has many other positive effects:
1. It improves readability and also potentially simplifies debugging.
2. It gives users the ability to enable or disable features of thin clients
on both
server and client as they desire.

What are your thoughts guys?

Best Regards,
Igor




--
Taras Ledkov
Mail-To: tled...@gridgain.com

Re: Feature masks for thin clients

[Taras Ledkov]

Hi,

The great idea!

What do you think about thin JDBC and ODBC?
Should we have common feature flags set and client specific feature flag 
set?

Or each client has independent flags set even if they are duplicated?

On 23.01.2020 15:36, Igor Sapego wrote:

Hi Igniters,

As we have a lot of different thin clients now, maintained by different
people, the issues with our backward compatibility mechanism becomes
more and more prominent.

Currently, we use protocol versioning as the only approach to provide
backward compatibility. The main issue of this approach is that we can
not skip some change in protocol and implement i.e. protocol of version
1.5 without implementation of 1.4. There are many cases when one may
want to do so: e.g. when feature provided in 1.4 is not relevant for a
specific client, or when protocol version 1.5 contains urgent fix or feature
which is easy to implement, but its blocked by not-so-urgent and
hard-to-implement feature introduced in 1.4.

So to fix this issue I propose to introduce another backward compatibility
mechanism. The idea is to send "supported features" mask by a client to
a server, which should be answered with the same mask by the server.
The resulting set of enabled features is acquired with a simple logical
"AND"
operation on these two masks.

This change has many other positive effects:
1. It improves readability and also potentially simplifies debugging.
2. It gives users the ability to enable or disable features of thin clients
on both
server and client as they desire.

What are your thoughts guys?

Best Regards,
Igor


--
Taras Ledkov
Mail-To: tled...@gridgain.com

[jira] [Created] (IGNITE-12537) Util DbH2ServerStartup failed

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-12537:
-

 Summary: Util DbH2ServerStartup failed
 Key: IGNITE-12537
 URL: https://issues.apache.org/jira/browse/IGNITE-12537
 Project: Ignite
  Issue Type: Bug
  Components: examples
Affects Versions: 2.7
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.8


{{DbH2ServerStartup}} fails with exception:

{code:java}Exception in thread "main" class org.apache.ignite.IgniteException: 
Failed to start database TCP server
at 
org.apache.ignite.examples.util.DbH2ServerStartup.main(DbH2ServerStartup.java:86)
Caused by: org.h2.jdbc.JdbcSQLNonTransientConnectionException: Database 
"mem:ExampleDb" not found, and IFEXISTS=true, so we cant auto-create it 
[90146-199]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:617)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:427)
at org.h2.message.DbException.get(DbException.java:205)
at org.h2.message.DbException.get(DbException.java:181)
at org.h2.engine.Engine.openSession(Engine.java:67)
at org.h2.engine.Engine.openSession(Engine.java:201)
at org.h2.engine.Engine.createSessionAndValidate(Engine.java:178)
at org.h2.engine.Engine.createSession(Engine.java:161)
at org.h2.server.TcpServerThread.run(TcpServerThread.java:160)
at java.lang.Thread.run(Thread.java:748)

at org.h2.message.DbException.getJdbcSQLException(DbException.java:617)
at org.h2.engine.SessionRemote.done(SessionRemote.java:607)
at org.h2.engine.SessionRemote.initTransfer(SessionRemote.java:143)
at org.h2.engine.SessionRemote.connectServer(SessionRemote.java:431)
at 
org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:317)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:169)
at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:148)
at org.h2.Driver.connect(Driver.java:69)
at 
org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:189)
at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352)
at 
org.h2.jdbcx.JdbcDataSource.getPooledConnection(JdbcDataSource.java:384)
at 
org.h2.jdbcx.JdbcConnectionPool.getConnectionNow(JdbcConnectionPool.java:234)
at 
org.h2.jdbcx.JdbcConnectionPool.getConnection(JdbcConnectionPool.java:199)
at 
org.apache.ignite.examples.util.DbH2ServerStartup.populateDatabase(DbH2ServerStartup.java:56)
at 
org.apache.ignite.examples.util.DbH2ServerStartup.main(DbH2ServerStartup.java:74){code}

This issue blocks all store examples



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12536) Inconsistency between cache data and indexes when cache operation is interrupted

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-12536:
-

 Summary: Inconsistency between cache data and indexes when cache 
operation is interrupted
 Key: IGNITE-12536
 URL: https://issues.apache.org/jira/browse/IGNITE-12536
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.7
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.8


*Root cause:*
Inconsistency between cache and indexes happens when cache operation put/remove 
is interrupted (e.g. thread is interrupted). The cache operation is finished, 
GridH2Table#lock(boolean) }} is interrupted because {{Lock#lockInterruptibly is 
used.

*Possible fix:*
Use not interruptible lock for cache operation and interruptible lock for SQL 
operation.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12484) Fix issues related to client cache stop and SQL metadata retrieval

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-12484:
-

 Summary: Fix issues related to client cache stop and SQL metadata 
retrieval
 Key: IGNITE-12484
 URL: https://issues.apache.org/jira/browse/IGNITE-12484
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.7
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.8




After IGNITE-6173 the problem is appeared. We need to fix it by properly 
handling client cache stop. Main idea is that SQL schema metadata (tables, 
indexes, etc.) should not be removed on client cache.close().

But the implementation should be carefully designed as many factors should be 
taken into account, e.g. cache destroy, persistence, client reconnect and etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-11910) JDBC v2: adds 'schema' to URL parameters, makes 'cache' parameter is optional fpr the most cases

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-11910:
-

 Summary: JDBC v2: adds 'schema' to URL parameters, makes 'cache' 
parameter is optional fpr the most cases
 Key: IGNITE-11910
 URL: https://issues.apache.org/jira/browse/IGNITE-11910
 Project: Ignite
  Issue Type: Bug
  Components: jdbc
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.8


We have to do changes for the JDBC v2:
- makes 'cache' parameter is optional.
- add 'schema' parameter to URL.

When cache and schema parameters are not defined 'PUBLIC' schema is used.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11755) Memory leak H2 connections at the ConnectionManager#detachedConns

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-11755:
-

 Summary: Memory leak H2 connections at the 
ConnectionManager#detachedConns
 Key: IGNITE-11755
 URL: https://issues.apache.org/jira/browse/IGNITE-11755
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.7
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.8


{{ConnectionManager#detachedConns}} leaks on mvcc transnational SELECT.
Reproduce: 
1. CREATE TABLE with enabled MVCC
2. Do SELECTs.
3. Each query is executed at the new JDBC thin connection. A connection is 
closed after query.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11754) Memory leak on the GridCacheTxFinishSync#threadMap

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-11754:
-

 Summary: Memory leak on the GridCacheTxFinishSync#threadMap
 Key: IGNITE-11754
 URL: https://issues.apache.org/jira/browse/IGNITE-11754
 Project: Ignite
  Issue Type: Bug
  Components: general, mvcc
Affects Versions: 2.7
Reporter: Taras Ledkov
 Fix For: 2.8


The {{GridCacheTxFinishSync#threadMap}} is not cleared when tx thread is 
terminated.
So, memory leak happens when transactions are executed inside new start/stopped 
threads.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11726) SQL: don't store default precision and scale in the QueryEntity for CHAR/VARCHAR and DECIMAL types

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-11726:
-

 Summary: SQL: don't store default precision and scale in the 
QueryEntity for CHAR/VARCHAR and DECIMAL types
 Key: IGNITE-11726
 URL: https://issues.apache.org/jira/browse/IGNITE-11726
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Affects Versions: 2.7
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.8


If table is created by SQL query, e.g.:
{{CREATE TABLE test (id INT, val_dec DECIMAL, val_char VARCHAR)}}
and CHAR/VARCHAR  and DECIMAL types are used with default precision (size for 
char/varchar) and scale the default precision and scale mustn't be passed into 
{{QueryEntity}} for created cache.

This is cause of compatibility problems.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11720) Document new SQL system view "COLUMNS"

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-11720:
-

 Summary: Document new SQL system view "COLUMNS"
 Key: IGNITE-11720
 URL: https://issues.apache.org/jira/browse/IGNITE-11720
 Project: Ignite
  Issue Type: Task
  Components: documentation
Affects Versions: 2.7
Reporter: Taras Ledkov
 Fix For: 2.8


See the ticket IGNITE-11434 for list of view's columns.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11694) Add documentation for SqlFieldsQuery.updateBatchSize into thin clients docs

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-11694:
-

 Summary: Add documentation for SqlFieldsQuery.updateBatchSize into 
thin clients docs
 Key: IGNITE-11694
 URL: https://issues.apache.org/jira/browse/IGNITE-11694
 Project: Ignite
  Issue Type: Task
  Components: documentation
Affects Versions: 2.7
Reporter: Taras Ledkov
 Fix For: 2.8


The property {{SqlFieldsQuery.updateBatchSize}} is introduced by the patch 
IGNITE-11499.

ODBC, thin JDBC, thin client documentation should be changed.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11693) Add support SqlFieldsQuery.updateBatchSize for ODBC client.

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-11693:
-

 Summary: Add support SqlFieldsQuery.updateBatchSize for ODBC 
client.
 Key: IGNITE-11693
 URL: https://issues.apache.org/jira/browse/IGNITE-11693
 Project: Ignite
  Issue Type: Task
  Components: odbc
Affects Versions: 2.7
Reporter: Taras Ledkov
 Fix For: 2.8


Add support {{SqlFieldsQuery.updateBatchSize}} for ODBC client.
The property {{SqlFieldsQuery.updateBatchSize}} is introduced by the patch 
IGNITE-11499.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11692) Add support SqlFieldsQuery.updateBatchSize for thin client protocol and java thin client

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-11692:
-

 Summary: Add support SqlFieldsQuery.updateBatchSize for thin 
client protocol and java thin client
 Key: IGNITE-11692
 URL: https://issues.apache.org/jira/browse/IGNITE-11692
 Project: Ignite
  Issue Type: Task
  Components: thin client
Affects Versions: 2.7
Reporter: Taras Ledkov
 Fix For: 2.8


Add support {{SqlFieldsQuery.updateBatchSize}} for thin client protocol and 
java thin client.
The property {{SqlFieldsQuery.updateBatchSize}} is introduced by the patch 
IGNITE-11499.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-11656) The GridCacheMapEntry#lockEntry cannot be interrupted

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-11656:
-

 Summary: The GridCacheMapEntry#lockEntry cannot be interrupted
 Key: IGNITE-11656
 URL: https://issues.apache.org/jira/browse/IGNITE-11656
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Affects Versions: 2.7
Reporter: Taras Ledkov


Now the method {{GridCacheMapEntry#lockEntry}} use {{ReentrantLock#lock}} to 
lock the entry. So, when deadlock happens the locked threads cannot be 
interrupted by {{Thread.interrupt()}}. 

In this case a test and the grid cannot be stoped without kill the process.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)