[jira] [Created] (IGNITE-8204) Tests hang with lazy query flag on

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-8204:
---

 Summary: Tests hang with lazy query flag on
 Key: IGNITE-8204
 URL: https://issues.apache.org/jira/browse/IGNITE-8204
 Project: Ignite
  Issue Type: Bug
  Components: sql
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.5


Affected tests:

{{IgniteCacheClientQueryReplicatedNodeRestartSelfTest.testRestart}}

{{IgniteCacheDistributedPartitionQueryNodeRestartsSelfTest.testJoinQueryUnstableTopology}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8050) Throw a meaningful exception when user issues TX SQL keyword with MVCC turned off

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-8050:
---

 Summary: Throw a meaningful exception when user issues TX SQL 
keyword with MVCC turned off
 Key: IGNITE-8050
 URL: https://issues.apache.org/jira/browse/IGNITE-8050
 Project: Ignite
  Issue Type: Task
Reporter: Alexander Paschenko
 Fix For: 2.5


An exception must be thrown when the user issues TX SQL command (BEGIN, COMMIT, 
ROLLBACK) in absence of MVCC - ingoring these may be confusing and can lead to 
SQL engine behavior to behaving quite differently from what the user expects, 
esp. in terms of data consistency.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8016) Make sure checks in QueryUtils.typeForQueryEntity are sane

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-8016:
---

 Summary: Make sure checks in QueryUtils.typeForQueryEntity are sane
 Key: IGNITE-8016
 URL: https://issues.apache.org/jira/browse/IGNITE-8016
 Project: Ignite
  Issue Type: Task
  Components: cache, sql
Reporter: Alexander Paschenko


If one uses {{AffinityUuid}} (and thus, most likely, any {{AffinityKey}} at 
all) as cache key in {{QueryEntity}} while not having an actual class for value 
type, cache fails to start with error

 

{{Failed to find value class in the node classpath (use default marshaller to 
enable binary objects)}}

 

Not only error message is misleading (as {{BinaryMarshaller}} is always on 
now), the checks themselves must be verified for sanity as ruling them out 
allowed to start cache, put data into it via cache API and perform SQL query 
that worked.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7956) MVCC TX: cache eviction operations for key-value API

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-7956:
---

 Summary: MVCC TX: cache eviction operations for key-value API
 Key: IGNITE-7956
 URL: https://issues.apache.org/jira/browse/IGNITE-7956
 Project: Ignite
  Issue Type: Task
  Components: cache
Reporter: Alexander Paschenko
 Fix For: 2.5


We need to implement MVCC-compatible cache eviction operations for key-value 
API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7955) MVCC TX: cache peek for key-value API

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-7955:
---

 Summary: MVCC TX: cache peek for key-value API
 Key: IGNITE-7955
 URL: https://issues.apache.org/jira/browse/IGNITE-7955
 Project: Ignite
  Issue Type: Task
  Components: cache
Reporter: Alexander Paschenko
 Fix For: 2.5


We need to implement MVCC-compatible cache peek operation for key-value API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7953) MVCC TX: continuous queries

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-7953:
---

 Summary: MVCC TX: continuous queries
 Key: IGNITE-7953
 URL: https://issues.apache.org/jira/browse/IGNITE-7953
 Project: Ignite
  Issue Type: Task
  Components: cache
Reporter: Alexander Paschenko
 Fix For: 2.5


We need to implement MVCC-compatible continuous queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7952) MVCC TX: cache clear routines for key-value API

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-7952:
---

 Summary: MVCC TX: cache clear routines for key-value API
 Key: IGNITE-7952
 URL: https://issues.apache.org/jira/browse/IGNITE-7952
 Project: Ignite
  Issue Type: Task
  Components: cache
Reporter: Alexander Paschenko
 Fix For: 2.5






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7942) Document streaming in thin JDBC driver

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-7942:
---

 Summary: Document streaming in thin JDBC driver
 Key: IGNITE-7942
 URL: https://issues.apache.org/jira/browse/IGNITE-7942
 Project: Ignite
  Issue Type: Task
  Components: documentation, jdbc, sql
Reporter: Alexander Paschenko
 Fix For: 2.5






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7859) SQL streaming support via native API

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-7859:
---

 Summary: SQL streaming support via native API
 Key: IGNITE-7859
 URL: https://issues.apache.org/jira/browse/IGNITE-7859
 Project: Ignite
  Issue Type: Task
Reporter: Alexander Paschenko


In addition to streaming via thin JDBC driver, ability to run same {{SET 
STREAMING}} command should be added to cache API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: TX SQL: SELECT FOR UPDATE implementation

[Alexander Paschenko]

Hello again, and here I go with some more thoughts on SELECT FOR UPDATE.
After having talking to Vlad, we returned again to question about how
SELECT FOR UPDATE should work with JOINs.

It seems that distributed joins currently are better to be left aside
as long as rework of distributed queries and joins is a separate big
issue.
However, probably we still should think about what we could make work
after all with what we have in our hands now. And it seems that some
scenarios are pretty doable - say, when there's an ordinary JOIN of
few tables, we can simply append keys of those tables to list of
selected columns, and that would work in case of collocated data.

* Alas, major vendors (Postgres, MySQL, Oracle) all also honor
subqueries in JOINs - say, like this:

select p.id , c.id from person p inner join (select * from company) c
on p.company_id = c.id where p.id > 3 and p.id < 10 for update;

Obviously, this is a bit corner case as such JOIN is not much
different from an ordinary multi-table one, but it gives overall idea
of what I'm talking about - major databases lock what's been selected
in subquery as well.

That said, aforementioned case probably could also be processed as
suggested above - that is, we take innermost query, append key to list
of its selected columns, so that we can refer to that new column from
outer query, etc, etc. In principle we could process even
multiple-level nesting of queries with such approach.
Also no major vendors, except probably MySQL, support GROUP BY with
SELECT FOR UPDATE, and this simplifies our task a bit. :)

If someone can think of an example of a query that cannot be processed
in the way described above, please write about it.

* Also it looks like no major vendors out of three mentioned above
lock record from tables mentioned in subqueries in SELECT columns or
in WHERE - say, this query

select p.id, (select count(*) from company) from person p

will lock only records in Person table.

Thoughts? Should we attempt to implement JOINs support right away? Or
would it be better to leave it be for (near) future and start with
simple implementation that would ban JOINs altogether?

Regards,
Alex


2018-01-25 10:58 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>:
> Alex,
>
> Thank you for detailed analysis. My 50 cents:
>
> 1) I would not allow GROUP BYs in the first place. These are good
> candidates for further iterations IMO
> 2) Same as p.1
> 3) In final TX SQL solution we woll lock rows, not keys
> (GridCacheMapEntry). Can we simply lock every row returned from the query?
> 4) Same as p.1
> 5) Yes, it should work the same way we would implement it for normal queries
>
> Also I am not quite understand why should we keep results on map node until
> all keys are locked. This increases memory pressure on the server. Instead,
> I would start sending batches to reducer immediately, but do not return the
> very first result to the user until all results are collected. This way
> pressure is moved to the client what increases cluster stability. Will that
> work?
>
> On Wed, Jan 24, 2018 at 6:23 PM, Alexander Paschenko <
> alexander.a.pasche...@gmail.com> wrote:
>
>> Hello Igniters,
>>
>> I'd like to bring up the discussion about implementation details of
>> https://issues.apache.org/jira/browse/IGNITE-6937 about
>> support of SELECT FOR UPDATE statements as a part of overall activity
>> on transactional SQL.
>>
>> That kind of statements allows the user proactively obtain row level
>> locks for the records that they may (or may not) want to
>> update in further statements of the same transaction.
>>
>> Suggested general approach to implementation is as follows:
>>
>> - Perform two-step SELECT as usual (e.g. split, map, reduce stages), BUT
>> - Map nodes start giving away their first result pages for such query
>> only after they have traversed whole their result set
>> and have obtained locks for all keys corresponding to selected rows
>> (e.g. rows matching WHERE clause).
>>
>> I've made some research on how some RDBMSs behave (MySQL, Oracle, Pg)
>> and found that they all also consider SELECT FOR UPDATE
>> finished and give away its result set only after all locks have been
>> acquired, so in general suggested strategy is similar to
>> what other databases do. Alas, everyone concerned is welcome to share
>> their views on how this feature could work in Ignite - as well as on
>> some stuff stated below. These are some caveats that I'd like to
>> discuss in addition to overall approach.
>>
>> First: should we allow GROUP BY clause in SELECT FOR UPDATE? Oracle
>> and Pg do not support it in any way, while MySQL does.
>>
>> Second: should we allow JOIN clause in SELECT FOR UPDATE? All
>&g

[jira] [Created] (IGNITE-7738) Allow 'multiple statements' in thin JDBC streaming mode

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-7738:
---

 Summary: Allow 'multiple statements' in thin JDBC streaming mode
 Key: IGNITE-7738
 URL: https://issues.apache.org/jira/browse/IGNITE-7738
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Alexander Paschenko


We need to update thin JDBC protocol to let user run multiple statements via 
Statement.execute(String) when connection is in streamed mode. Currently in 
streaming mode the server always receives all SQL in batches and its batch 
processing logic does not allow multiple statements altogether. If we're able 
to recognize initial nature of the statement, we'll be able to act 
appropriately, and for this to be possible additional information must be 
passed with each query in the batch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-7664) SQL: throw sane exception on unsupported SQL statements

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-7664:
---

 Summary: SQL: throw sane exception on unsupported SQL statements
 Key: IGNITE-7664
 URL: https://issues.apache.org/jira/browse/IGNITE-7664
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.5


Inspired by this SO issue:

[https://stackoverflow.com/questions/48708238/ignite-database-create-schema-assertionerror]

We should handle unsupported stuff more gracefully both in core code and 
drivers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Ignite SQL: How to drop table without explicit cache name

[Alexander Paschenko]

Guys,

Right, first option is the way to go. Options 2 and 3 are surrogates
that do not free you from concept of cache but rather allow you to
obtain one.

I believe in near future we'll propose public API for cacheless queries.

- Alex

2018-02-01 19:38 GMT+03:00 Valentin Kulichenko :
> Nikolay,
>
> I believe the first option is the way to go. However, I'd also like to hear
> from other community members.
>
> Vladimir, Alex P, can you comment on this? I know we currently don't have
> public API to execute DDL statement without having a cache. However, looks
> like we use private API of GridQueryProcessor to overcome this in JDBC
> drivers and Web Console. We now need a similar solution for data frames
> integration, can you confirm that we're moving in the right direction?
>
> -Val
>
> On Thu, Feb 1, 2018 at 4:04 AM, Nikolay Izhikov  wrote:
>
>> Hello, guys.
>>
>> I working on support of saving Spark DataFrame to Ignite [1].
>>
>> I want to execute "DROP TABLE XXX" query.
>> To execute some SQL query in regulary way one need some cache.
>> That cache has to differs from table cache to execute "DROP TABLE"
>> successfully.
>>
>> I founded 3 different ways to execute SQL query without explicit cache
>> name.
>> Please, tell me which way is right?
>> Do we need all 3 way to make a query?
>>
>> 1. JdbcRequestHandler - [2]
>>
>> ```
>>
>> private final GridKernalContext ctx;
>> ...
>> List> results = 
>> ctx.query().querySqlFields(qry,
>> true, protocolVer.compareTo(VER_2_3_0) < 0);
>>
>> ```
>>
>> 2. GridCacheProcessor - [3]
>>
>> ```
>>
>> /**
>> * Gets public cache that can be used for query execution.
>> * If cache isn't created on current node it will be started.
>> */
>> public IgniteCacheProxy getOrStartPublicCache(boolean start,
>> boolean inclLoc) throws IgniteCheckedException {
>>
>> ```
>>
>> 3. QueryCommandHandler - [4]
>>
>> ```
>>
>> protected static final String DFLT_CACHE_NAME = "default";
>> ...
>> String cacheName = req.cacheName() == null ? DFLT_CACHE_NAME :
>> req.cacheName();
>>
>> ```
>>
>>
>> [1] https://issues.apache.org/jira/browse/IGNITE-7337
>> [2] https://github.com/apache/ignite/blob/master/modules/
>> core/src/main/java/org/apache/ignite/internal/processors/
>> odbc/jdbc/JdbcRequestHandler.java#L310
>> [3] https://github.com/apache/ignite/blob/master/modules/
>> core/src/main/java/org/apache/ignite/internal/processors/
>> cache/GridCacheProcessor.java#L1648
>> [4] https://github.com/apache/ignite/blob/master/modules/
>> core/src/main/java/org/apache/ignite/internal/processors/
>> rest/handlers/query/QueryCommandHandler.java#L318

TX SQL: SELECT FOR UPDATE implementation

[Alexander Paschenko]

Hello Igniters,

I'd like to bring up the discussion about implementation details of
https://issues.apache.org/jira/browse/IGNITE-6937 about
support of SELECT FOR UPDATE statements as a part of overall activity
on transactional SQL.

That kind of statements allows the user proactively obtain row level
locks for the records that they may (or may not) want to
update in further statements of the same transaction.

Suggested general approach to implementation is as follows:

- Perform two-step SELECT as usual (e.g. split, map, reduce stages), BUT
- Map nodes start giving away their first result pages for such query
only after they have traversed whole their result set
and have obtained locks for all keys corresponding to selected rows
(e.g. rows matching WHERE clause).

I've made some research on how some RDBMSs behave (MySQL, Oracle, Pg)
and found that they all also consider SELECT FOR UPDATE
finished and give away its result set only after all locks have been
acquired, so in general suggested strategy is similar to
what other databases do. Alas, everyone concerned is welcome to share
their views on how this feature could work in Ignite - as well as on
some stuff stated below. These are some caveats that I'd like to
discuss in addition to overall approach.

First: should we allow GROUP BY clause in SELECT FOR UPDATE? Oracle
and Pg do not support it in any way, while MySQL does.

Second: should we allow JOIN clause in SELECT FOR UPDATE? All
mentioned major vendors support it, but with Ignite implementation
likely
will not be straightforward - this has to do with pt. 3.

Third: in general case, how do we detect *keys* that we should lock?
Consider following statement: SELECT name FROM person FOR UPDATE.
This query does not mention table key at all. Moreover, if we have
JOIN clause in the query, it is concerned with more than one table
(=cache), thus making it even harder to detect which keys we should
lock. Suggested solution is as follows: on split stage,
append to MAP query key columns for all concerned tables (i.e. SELECT
a.name, b.surname from a join b will become
SELECT a.name, b.surname, a._key, b._key) - with that done, we'll
definitely be able to lock everything we need on MAP nodes.

Fourth: should we allow LIMIT clause in SELECT FOR UPDATE? In general,
RDBMS vendors do not recommend combining LIMIT with
SELECT FOR UPDATE. First, this is because LIMIT results are subject to
ordering, and in this case it's harder to predict what records
will be locked after all. Second, some vendors state that they lock
not just what LIMIT yields, but also *everything before it* - say,
if your SELECT returns 100 records and you apply LIMIT 50,60, then all
first 60 records will be locked even if you don't need first 50.
Moreover, in case of Ignite, any LIMIT will make query need merge
table, all data will have to be copied to reduce node first,
yadda yadda yadda, and we won't be able to actually lock anything
until all data is on reduce node. See pt. 5 for more on this.

Fifth: in fact, above question stands for all cases when we need merge
table: how things should work when we have to pull all data to
reduce node prior to actually filtering anything? Should we aim to
support such cases in SELECT FOR UPDATE?

Regards,
Alex

[jira] [Created] (IGNITE-7300) Allow expressions in SQL INSERTs within transactions

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-7300:
---

 Summary: Allow expressions in SQL INSERTs within transactions
 Key: IGNITE-7300
 URL: https://issues.apache.org/jira/browse/IGNITE-7300
 Project: Ignite
  Issue Type: Bug
 Environment: The problem is related to IGNITE-7267 - the latter honors 
raw rows, but drops support for inserts with expressions which yield local 
subqueries. To fix this, {{UpdatePlan.isLocalSubquery()}} must be honored.
Reporter: Alexander Paschenko
Assignee: Igor Seliverstov
 Fix For: 2.4






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7001) Refactor dynamic indexes test to use SQL API

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-7001:
---

 Summary: Refactor dynamic indexes test to use SQL API
 Key: IGNITE-7001
 URL: https://issues.apache.org/jira/browse/IGNITE-7001
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.4


Tests for dynamic indexes should be refactored like those in IGNITE-6416 to use 
SQL API instead of relying on internal structures.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6996) Smarter handling of id fields in SQL values

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-6996:
---

 Summary: Smarter handling of id fields in SQL values
 Key: IGNITE-6996
 URL: https://issues.apache.org/jira/browse/IGNITE-6996
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Alexander Paschenko


Consider such case:

User wants to have a composite value (many value fields in {{QueryEntity}}) 
with one field associated with value's id (most likely matching cache key too).

Currently in order to insert such an object we will have to do something like
{{INSERT INTO Person(_key, id, name) values(1, 1, 'John')}}

And there's no way to avoid such a redundant repeat of the same value.

Suggested approach: I believe that we should specifically handle the case when 
user specifies {{keyFieldName}} in configuration and specified field is field 
of the value.

In such case, we could just do {{INSERT INTO Person(id, name) values(1, 
'John')}} and derive {{_key}} value from {{id}} column. (And vice versa.)

At a glance, this also will require following tweaks:
- forbid performing SQL {{UPDATE}} on such column ({{id}} in above example);
- on an {{INSERT}}, check that {{_key}} and {{id}} values are the same, if both 
specified.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6950) Update documentation for user facing CREATE TABLE params

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-6950:
---

 Summary: Update documentation for user facing CREATE TABLE params
 Key: IGNITE-6950
 URL: https://issues.apache.org/jira/browse/IGNITE-6950
 Project: Ignite
  Issue Type: Task
  Security Level: Public (Viewable by anyone)
  Components: documentation, sql
Affects Versions: 2.4
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.4


Changes to documentation should be made to reflect additional spelling for some 
{{CREATE TABLE}} params added in IGNITE-6270.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6637) No proper cleanup of statements cache is done on table drop

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-6637:
---

 Summary: No proper cleanup of statements cache is done on table 
drop
 Key: IGNITE-6637
 URL: https://issues.apache.org/jira/browse/IGNITE-6637
 Project: Ignite
  Issue Type: Bug
  Security Level: Public (Viewable by anyone)
  Components: sql
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko


We should cleanup statements cache whenever cache is deregistered - otherwise 
it's possible to retrieve from statements cache a statement that is prepared 
from H2 perspective but may not be executed.
Reproducer attached.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6529) JDBC drivers don't provide correct metadata about columns' nullability

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-6529:
---

 Summary: JDBC drivers don't provide correct metadata about  
columns' nullability
 Key: IGNITE-6529
 URL: https://issues.apache.org/jira/browse/IGNITE-6529
 Project: Ignite
  Issue Type: Bug
  Components: jdbc, sql
Reporter: Alexander Paschenko


For complete implementation of IGNITE-5648, we have to make JDBC drivers return 
correct nullability flag for {{NOT NULL}} columns - currently corresponding 
param of {{QueryField}} is ignored.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6518) Smarter analysis of INSERT and MERGE statements at parsing stage

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-6518:
---

 Summary: Smarter analysis of INSERT and MERGE statements at 
parsing stage
 Key: IGNITE-6518
 URL: https://issues.apache.org/jira/browse/IGNITE-6518
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Alexander Paschenko
 Fix For: 2.3


We could analyze INSERT and MERGE statements to detect that they don't specify 
data for key and/or value to notify users early that their query can't be 
executed within Ignite - prior to building plans and attempting to actually do 
anything.
(Note how we check that CREATE TABLE doesn't declare columns for key - logic 
here would be similar.)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6270) Rename user facing CREATE TABLE param names to split words properly

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-6270:
---

 Summary: Rename user facing CREATE TABLE param names to split 
words properly
 Key: IGNITE-6270
 URL: https://issues.apache.org/jira/browse/IGNITE-6270
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Alexander Paschenko
 Fix For: 2.3


Currently, some {{CREATE TABLE}} params specified inside {{WITH}} clause look 
like this: {{AFFINITYKEY}}, i.e. words are not separated by dashes as they 
should be.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6269) Document CREATE TABLE param for cache write sync mode on readme.io

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-6269:
---

 Summary: Document CREATE TABLE param for cache write sync mode on 
readme.io
 Key: IGNITE-6269
 URL: https://issues.apache.org/jira/browse/IGNITE-6269
 Project: Ignite
  Issue Type: Task
  Components: documentation, sql
Reporter: Alexander Paschenko
 Fix For: 2.3






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Key-value access to caches created with DDL

[Alexander Paschenko]

Sounds good to me.

– Alex

2017-09-02 9:12 GMT+02:00 Denis Magda :
> Igniters,
>
> Consider a table and cache created this way:
>
> CREATE TABLE `city` (
>   `ID` INT(11),
>   `Name` CHAR(35),
>   `CountryCode` CHAR(3),
>   `District` CHAR(20),
>   `Population` INT(11),
>   PRIMARY KEY (`ID`, `CountryCode`)
> ) WITH "template=partitioned, backups=1, affinityKey=CountryCode";
> Now in addition to SQL I want to process the data from this cache using 
> *key-value* and *compute affinity* APIs. To do that I need to build a valid 
> key first (like CityKey(cityId, countryCode) where countryCode is an 
> affinityKey).
>
> Seems it’s not achievable (or at least usable) because the key type name is 
> defined by DDL/DML and a bit scary - 
> SQL_PUBLIC_CITY_3f4e9fbf_3464_4598_8394_1307b86dc4e7_KEY.
>
> Will my case be addressed if we do the following?
>
> - Let a user pass a key type’s name into WITH clause. Assume CityKey was 
> chosen for this example.
>
> - DML operations keep using the same BinaryObjectBuilders for the key 
> generation internally but passing CityKey as a type name instead of the 
> auto-generated one.
>
> - The end user now can construct a valid key using the same 
> BinaryObjectBuilders or CityKey object (making sure hashCode and equals are 
> implemented properly) and, for instance, execute a compute.affinityRun over 
> the City cache.
>
> Please suggest another alternatives if I overlook something.
>
> —
> Denis
>
>

Re: Create own SQL parser

[Alexander Paschenko]

Dmitry,

Ignite mode in H2 is about parsing only, it does not address other
issues pointed out by Vlad. Alas, implementing _parser_ surely won't
take no years, whilst things like smart query optimization, etc,
probably don't have and can't have proper "finished" state - it's
something that you can improve again and again as there's never too
much performance.

My opinion: we should use Ignite mode in H2 to extend classic commands
with keywords of our own fashion while laying foundation for our own
SQL engine by starting work on parser core to provide for processing
of Ignite specific commands. I'll try to come up with some prototype
in near future.

- Alex

2017-08-03 4:43 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>:
> Vladimir, this sounds like a task that would take years to design,
> implement, and polish. Can we just aim to improve the Ignite mode in H2,
> which is much more feasible  in my view?
>
> On Wed, Aug 2, 2017 at 2:59 PM, Vladimir Ozerov <voze...@gridgain.com>
> wrote:
>
>> Alex P.,
>>
>> The very problem with non-SELECT and non-DML commands is that we do not
>> support most of what is supported by H2, and vice versa - H2 doesn't
>> support most things that we need, like cache properties, templates, inline
>> indexes, etc.. Another important thing is that at some point we will add a
>> kind of SQL-based command-line or scripting utility(es) for Ignite [1].
>> Mature products has rich set of commands, which are outside of SQL
>> standard. E.g., we would like to manage transaction settings (concurrency,
>> isolation) on per-session basis, grant and rewoke Ignite-specific roles,
>> gather some metrics from the cluster, etc.. It doesn't make sense to
>> develop it in H2.
>>
>> Actual H2 parsing logic takes about a dozen KLOCs. But parsing core is much
>> smaller and most of overall parser's code relates to SELECT and DML
>> statements, which are mostly not needed for DDL and administrative
>> commands. That said, I think it is perfectly fine to move Ignite-specific
>> commands to Ignite's own parser.
>>
>> Alex K.,
>>
>> Having the whole own SQL engine is very cool thing, as it gives us
>> unlimited capabilities in terms of performance and UX. But this is a very
>> huge thing. H2's core which is used in Ignite is about as big as all
>> existing Ignite's SQL logic in terms of lines of codes. So I would put this
>> question out of scope for now. We should focus on new features, usability
>> and documentation for now, and try getting as much as possible from the
>> given architecture.
>>
>> [1] https://issues.apache.org/jira/browse/IGNITE-5608
>>
>>
>> On Wed, Aug 2, 2017 at 3:38 PM, Alexey Kuznetsov <akuznet...@apache.org>
>> wrote:
>>
>> > From my opinion we could start investing in our own parser and SQL engine
>> > step by step, little by little
>> >  and one day drop H2 at all.
>> >
>> >  Having own parser and engine will give us a freedom to any optimizations
>> > and any syntax we like.
>> >
>> > Also that will be for one dependency less and we could have SQL out of
>> the
>> > box with no third-party dependencies.
>> >
>> >
>> > On Wed, Aug 2, 2017 at 7:25 PM, Alexander Paschenko <
>> > alexander.a.pasche...@gmail.com> wrote:
>> >
>> > > I'd like to point out that we already do have Ignite mode in H2 parser
>> > > (thanks Sergi) and thus have AFFINITY keyword support. Is is suggested
>> > > that we should abandon H2 way at all? Or should we suggest adding to
>> > > H2 only rather minor stuff (like some keywords for existing commands)
>> > > whilst introducing completely new commands for our own parser?
>> > >
>> > > - Alex
>> > >
>> > > 2017-08-02 9:01 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>:
>> > > > No, it will work as follows:
>> > > >
>> > > > Model parse(String sql) {
>> > > > Model res = tryParseWithIgnite(sql); // Parse what we can
>> > > >
>> > > > if (res == null)
>> > > > res = parseWithH2(sql);
>> > > >
>> > > > return res;
>> > > > }
>> > > >
>> > > > We will need a number of custom commands which are not present in H2.
>> > > >
>> > > > On Wed, Aug 2, 2017 at 3:44 AM, Dmitriy Setrakyan <
>> > dsetrak...@apache.org
>> > > >
>> > > > wrote:
>> > > >
>> > 

[jira] [Created] (IGNITE-5837) Fix logic in DynamicIndexAbstractConcurrentSelfTest

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-5837:
---

 Summary: Fix logic in DynamicIndexAbstractConcurrentSelfTest
 Key: IGNITE-5837
 URL: https://issues.apache.org/jira/browse/IGNITE-5837
 Project: Ignite
  Issue Type: Test
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-5703) Add CREATE TABLE param for cache write sync mode

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-5703:
---

 Summary: Add CREATE TABLE param for cache write sync mode
 Key: IGNITE-5703
 URL: https://issues.apache.org/jira/browse/IGNITE-5703
 Project: Ignite
  Issue Type: Task
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.1


Inspired by IGNITE-5702: instead of always enforcing {{FULL_SYNC}} mode, we 
better give the user an opportunity to set it explicitly like we do with other 
params (backups number, atomicity mode, etc.).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-5653) Add to query execution plan debug data for joins

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-5653:
---

 Summary: Add to query execution plan debug data for joins
 Key: IGNITE-5653
 URL: https://issues.apache.org/jira/browse/IGNITE-5653
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Affects Versions: 2.0
 Environment: Plan should output table type (replicated/partitioned) 
and colocation information if possible. If we have this than we can warn (or 
throw exception) if users try to join non colocated tables with local joins.
Reporter: Alexander Paschenko
 Fix For: 2.2






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-5652) Print slow query warnings on client node

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-5652:
---

 Summary: Print slow query warnings on client node
 Key: IGNITE-5652
 URL: https://issues.apache.org/jira/browse/IGNITE-5652
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Affects Versions: 2.0
 Environment: Currently, only worker (MAP) nodes of the query print 
long query execution time warning to their console, for usability it would be 
nice to propagate this to the client node as well.
Reporter: Alexander Paschenko
 Fix For: 2.2






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-5651) Add long query timeout property

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-5651:
---

 Summary: Add long query timeout property
 Key: IGNITE-5651
 URL: https://issues.apache.org/jira/browse/IGNITE-5651
 Project: Ignite
  Issue Type: Task
Affects Versions: 2.0
 Environment: It would be nice to allow users setting long execution 
timeout when sending a query - by doing so, user won't start getting dull long 
execution warnings which may be well expected. Instead, they will set their own 
acceptable timeout and will start seeing warnings only when they want to.
Reporter: Alexander Paschenko
 Fix For: 2.2






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-5650) Add convenient API for getting column values

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-5650:
---

 Summary: Add convenient API for getting column values
 Key: IGNITE-5650
 URL: https://issues.apache.org/jira/browse/IGNITE-5650
 Project: Ignite
  Issue Type: Task
Affects Versions: 2.0
Reporter: Alexander Paschenko
 Fix For: 2.2


It's desirable to have some API for getting column values from query results as 
current API operates only rows (raw Lists).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-5482) Implement basic caching of query results

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-5482:
---

 Summary: Implement basic caching of query results
 Key: IGNITE-5482
 URL: https://issues.apache.org/jira/browse/IGNITE-5482
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.0
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.1


Sergi suggested that we reuse results of the same queries running 
simultaneously - i.e. if a query is being executed with the same arguments and 
flags again and again, there's no need to do actual querying of data, instead 
we can really run query once while other simultaneous runs will wait for those 
results.

This strategy will be implemented on MAP stage of distributed query.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-5449) Add complex DDL+DML test.

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-5449:
---

 Summary: Add complex DDL+DML test.
 Key: IGNITE-5449
 URL: https://issues.apache.org/jira/browse/IGNITE-5449
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.1


We need a test that will test data flow behavior in chain of DML+DDL operations 
as a whole.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Apache Ignite 2.1 scope

[Alexander Paschenko]

IGNITE-5327 Create predefined cache templates for CREATE TABLE command
- minor comments left, ETA is Friday.

IGNITE-5380 Validate cache QueryEntities in discovery thread - in
progress, the meat of code is written, but need to add lots of tests.
ETA is Friday.

IGNITE-5188 Support AFFINITY KEY keyword for CREATE TABLE command - in
progress, made few first small steps, ETA is Friday.

Rest is closed/patch available, ignite-4994 has been moved to 2.2.

- Alex

2017-06-01 19:03 GMT+03:00 Sergey Chugunov :
>1. IGNITE-5386 Inactive mode must be forced on starting up grid with
>persistence is enabled
>It is important improvement to fix critical bug IGNITE-5363.
>Working on it, ETA - tomorrow.
>2. IGNITE-5375 New PersistentStoreMetrics, MemoryMetrics interface
>improvements
>A lot of discussions were on this topic, ticket created only today and
>requires several days to implement.
>
>
>
> On Thu, Jun 1, 2017 at 6:56 PM, Taras Ledkov  wrote:
>
>> Folks,
>>
>> IGNITE-4922 JDBC Driver: renew thin client based solution:
>>
>> On 2.1 the functionality of the new thin client JDBC driver will be
>> between deprecated Ignite thin JDBC and Ignite JDBCv2.
>> 1. The most functions of SQL query (include DML) are implemented and ready
>> for review;
>> 2. The most functions of JDBC metadata are implemented and ready for
>> review;
>> 3. Transactions, batching, streaming, blobs, scrollable / writable cursors
>> will not be supported in 2.1.
>>
>>
>>
>> On 01.06.2017 18:43, Vladimir Ozerov wrote:
>>
>>> Folks,
>>>
>>> We are almost reached proposed feature-complete date (June 2), Could you
>>> please share current status of your major features?
>>>
>>> On Tue, May 16, 2017 at 3:51 AM, Dmitriy Setrakyan >> >
>>> wrote:
>>>
>>> Looks a little tight. Let's hope we can make it.

 On Mon, May 15, 2017 at 1:29 PM, Denis Magda  wrote:

 Well, let me propose the following milestones for 2.1 release then.
>
> Code freeze: June 2nd.
> Final QA and benchmarking: June 5 - June 8
> Voting: ~ June 9
> Release: ~ June 13
>
> Also I heard H2 has to be released once again to support Ignite’s CREATE
> table command. Think that we should talk to H2 folks to make it happen
> in
> June 22nd - June 2nd time frame.
>
> —
> Denis
>
> On May 11, 2017, at 2:26 AM, Pavel Tupitsyn 
>>
> wrote:
>
>> As for .NET, I would propose to concentrate on peer deployment
>>
> (IGNITE-2492)
>
>> and related stuff, like IGNITE-1894 .NET: Delegate support in the API
>>
> via

> extension methods.
>>
>> SQL Dependency does not look important to me, we can reschedule it for
>> later versions.
>>
>> On Thu, May 11, 2017 at 12:01 PM, Dmitriy Setrakyan <
>>
> dsetrak...@apache.org>
>
>> wrote:
>>
>> Vyacheslav, I think it is worth the research, but you should always
>>>
>> keep

> data querying and indexing in mind. For example, I don't see how
>>>
>> by-page

> compression will solve it.
>>>
>>> On Thu, May 11, 2017 at 1:52 AM, Vyacheslav Daradur <
>>>
>> daradu...@gmail.com>
>
>> wrote:
>>>
>>> Dmitriy,

 I'm researching a best way for this future.

 At the moment I found only one way (querying and indexing

>>> compatible),

> this
>>>
 is per-objects-field compression.

 But there is a good proffit only for long strings or fields with

>>> large

> objects.

 Maybe it makes sense just to introduce compression for string fileds.

 I'm researching the new page-memory architecture as applied to

>>> by-page

> compression.

 2017-05-11 11:30 GMT+03:00 Dmitriy Setrakyan >> :
>
>> On Thu, May 11, 2017 at 12:44 AM, Vyacheslav Daradur <
>
 daradu...@gmail.com>

> wrote:
>
> Denis,
>>
>> The described roadmap looks great!
>>
>> Additional, I vote for introducing an ability (OOTB) to store
>>
> objects

> in

> a
>
>> cache in a compressed form.
>> This will allow to store more data at the cost of incriasing of CPU
>> utilization.
>>
>> One of the problems with compression is indexing and querying. How
>
 do

> we
>>>
 index the data if it is compressed?
>
>
>
>> 2017-05-11 4:23 GMT+03:00 Denis Magda :
>>
>> Igniters,
>>>
>>> Let me start a discussion around the scope for 2.1 release.
>>>
>>> In my 

Re: ALTER TABLE command support

[Alexander Paschenko]

Hi guys,

To my knowledge, our binary format is currently resilient to
adding/removing fields of individual objects while having those
objects logically belonging to the same value type - thanks to all
efforts with binary resolvers and stable field sorting. Thus, ALTER
TABLE implemented as suggested (descriptors manipulation) indeed will
not need actual data modification, amirite?

That said, in my view at a first glance the task boils down to following:

1. Make type descriptors and table descriptors mutable in principle
2. Add Ignite ALTER TABLE command (at first approach let's limit
ourselves with columns add/remove)
3. Implement its propagation and distributed execution (CREATE INDEX
style - this mechanism of distributed schema changes has already been
implemented by Vlad)

4. On local stage of distributed execution:

4.1. Issue on local H2 instance combination of DROP TABLE and then
CREATE TABLE with new columns list.
This is a lot like what H2 does when you ask it to ALTER TABLE, but H2
actually also does data copying via SELECT which we don't need to do
as far as I can see due to binary format resiliency I wrote above -
our combination of local CREATE and DROP will modify local H2 metadata
to look the way we ultimately need.

4.2. After we're finished with H2 metadata, we're safe to modify type
and table descriptors.

Of course some amount of additional work will be needed to make sure
that such operations are mutually exclusive with cache gets/puts, or
index creation, or cache creation (also implies manipulations with
type descriptors and may also cause a CREATE TABLE in its own right).
But still approach in general could be as described above.

I also have some ideas about support for constraints which is in fact
is also concerned with descriptors mutation, but I believe this is not
the topic of this conversation.

Thoughts?

– Alex

2017-05-31 23:36 GMT+03:00 Denis Magda :
> Agree. The gradual approach is the way to go for us.
>
> —
> Denis
>
>> On May 31, 2017, at 1:20 PM, Dmitriy Setrakyan  wrote:
>>
>> I think a fully functional ALTER TABLE command may be hard to implement, as
>> it includes changes of columns, types, constraints, etc... We should take a
>> gradual approach here and implement this command by phases.
>>
>> I would propose that in the first phase we simply add the capability to add
>> and remove columns to a table. This way, given that we already support
>> CREATE INDEX command, the new column can be immediately indexed within the
>> cluster and used for query execution.
>>
>> Does this sound like a plan?
>>
>> D.
>>
>> On Wed, May 31, 2017 at 11:52 AM, Denis Magda  wrote:
>>
>>> Sergi, Vovan, Alexander P.,
>>>
>>> It’s great that we added CREATE/DROP index commands support to Ignite. As
>>> the next step, I think we need to move forward and plan to add ALTER TABLE
>>> command to the list.
>>>
>>> The reason we should have this command is simple. If a user adds a new
>>> field to Person class (while the cluster is up and running) then there
>>> should be a way to access the field from SQL and index it later if needed.
>>> Presently, this is not supported.
>>>
>>> What will be our efforts to support this?
>>>
>>> —
>>> Denis
>

[jira] [Created] (IGNITE-5291) SQL: plans cache does not clear on cache stop

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-5291:
---

 Summary: SQL: plans cache does not clear on cache stop
 Key: IGNITE-5291
 URL: https://issues.apache.org/jira/browse/IGNITE-5291
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.0
 Environment: After cache-schema decoupling mutation of 
{{DmlStatementsProcessor#planCache}} appears to be broken (different pieces of 
code contexts operate different schema names), have to do this consistently.
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.1






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-5289) SQL: forbid WITH queries

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-5289:
---

 Summary: SQL: forbid WITH queries
 Key: IGNITE-5289
 URL: https://issues.apache.org/jira/browse/IGNITE-5289
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko


Recursive queries starting with WITH keyword must be explicitly forbidden.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-5116) Handle cache destroy in DmlStatementsProcessor

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-5116:
---

 Summary: Handle cache destroy in DmlStatementsProcessor
 Key: IGNITE-5116
 URL: https://issues.apache.org/jira/browse/IGNITE-5116
 Project: Ignite
  Issue Type: Bug
  Components: SQL
Affects Versions: 1.9
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.1


We have to clear plans cache for Ignite cache that is being destroyed as 
otherwise cache context leaks with obsolete DML execution plans



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-4998) Add more dynamic indexing tests for various cache modes

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4998:
---

 Summary: Add more dynamic indexing tests for various cache modes
 Key: IGNITE-4998
 URL: https://issues.apache.org/jira/browse/IGNITE-4998
 Project: Ignite
  Issue Type: Sub-task
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.0






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-4953) Rework logic of concurrent schema changes

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4953:
---

 Summary: Rework logic of concurrent schema changes
 Key: IGNITE-4953
 URL: https://issues.apache.org/jira/browse/IGNITE-4953
 Project: Ignite
  Issue Type: Bug
  Components: SQL
Affects Versions: 2.0
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.0


H2's prepared statements store references to indexes that were present when the 
statement was parsed and initialized - this means that currently there's no way 
to prevent index usage if it goes down between the moment when the statement is 
created and actually executed. Have to come up with some new locking schema.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: CREATE TABLE SQL command syntax

[Alexander Paschenko]

Sergi,

First, it would be as little as overriding the part responsible for
CREATE TABLE - there's no need to touch anything else as luckily H2
parser is internally structured well enough.

Second, although it is not all-around perfect, I am most confident
that this is far better than dragging into H2 bunch of stuff that they
don't really need just because we need it there or can smug it there.

I think I'll just spend some time in the weekend and come up with a
prototype as otherwise this talk seems to be just a chit-chat.

- Alex

2017-04-12 14:38 GMT+03:00 Sergi Vladykin <sergi.vlady...@gmail.com>:
> So basically in inherited class you are going co copy/paste base class
> methods and tweak them? I don't like this approach.
>
> Sergi
>
> 2017-04-12 14:07 GMT+03:00 Alexander Paschenko <
> alexander.a.pasche...@gmail.com>:
>
>> Sergi,
>>
>> As I've written in my previous post, it would be just inheriting Parser on
>> Ignite side and plugging its instance in SINGLE place. Just making H2's
>> Parser internal methods protected instead of private would let us do the
>> trick.
>>
>> — Alex
>>
>> среда, 12 апреля 2017 г. пользователь Sergi Vladykin написал:
>>
>> > I don't see how you make H2 Parser extendable, you will have to add
>> plugin
>> > call to every *potentially* extendable place in it. In general this does
>> > not work. As H2 guy I would also reject patch like this.
>> >
>> > Sergi
>> >
>> > 2017-04-12 13:10 GMT+03:00 Alexander Paschenko <
>> > alexander.a.pasche...@gmail.com <javascript:;>>:
>> >
>> > > Sergi,
>> > >
>> > > Please have a closer look at what I've written in my first post. I
>> don't
>> > > see why we have to cling to H2 and its parsing modes all the time —
>> after
>> > > all, we're just talking string processing now, aren't we? (Yes, complex
>> > and
>> > > non trivial, but still.)
>> > >
>> > > What's wrong with idea of patching H2 to allow custom parsing? (With
>> the
>> > > parsing itself living in Ignite code, obviously, not in H2.).
>> > >
>> > > What I propose is just to make H2's Parser class extendable and make H2
>> > > aware of its descendants via config params. And that's all with respect
>> > to
>> > > H2, nothing more.
>> > >
>> > > After that, on Ignite side we do all we want with our parser based on
>> > > theirs. It resembles story with custom types — first we make H2
>> > extendable
>> > > in the way we need, then we introduce exact features we need on Ignite
>> > > side.
>> > >
>> > > — Alex
>> > >
>> > > среда, 12 апреля 2017 г. пользователь Sergi Vladykin написал:
>> > >
>> > > > It definitely makes sense to add a separate mode for Ignite in H2.
>> > Though
>> > > > it is wrong to think that it will allow us to add any crazy syntax we
>> > > want
>> > > > (and it is actually a wrong idea imo), only the minor variations of
>> the
>> > > > existing syntax. But this must be enough.
>> > > >
>> > > > I believe we should end up with something like
>> > > >
>> > > > CREATE TABLE person
>> > > > (
>> > > >   id INT PRIMARY KEY,
>> > > >   orgId INT AFFINITY KEY,
>> > > >   name VARCHAR
>> > > > )
>> > > > WITH "cfg:my_config_template.xml"
>> > > >
>> > > > Sergi
>> > > >
>> > > >
>> > > > 2017-04-12 7:54 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org
>> > <javascript:;>
>> > > > <javascript:;>>:
>> > > >
>> > > > > Agree, the updated syntax looks better. One change though: KEY ->
>> > > PRIMARY
>> > > > > KEY.
>> > > > >
>> > > > > Sergi, what do you think?
>> > > > >
>> > > > > D.
>> > > > >
>> > > > > On Tue, Apr 11, 2017 at 9:50 PM, Pavel Tupitsyn <
>> > ptupit...@apache.org <javascript:;>
>> > > > <javascript:;>>
>> > > > > wrote:
>> > > > >
>> > > > > > I think "WITH" syntax is ugly and cumbersome.
>> > > > > >
>> > > > > > We should go with this one:
>> > &g

Re: CREATE TABLE SQL command syntax

[Alexander Paschenko]

Sergi,

As I've written in my previous post, it would be just inheriting Parser on
Ignite side and plugging its instance in SINGLE place. Just making H2's
Parser internal methods protected instead of private would let us do the
trick.

— Alex

среда, 12 апреля 2017 г. пользователь Sergi Vladykin написал:

> I don't see how you make H2 Parser extendable, you will have to add plugin
> call to every *potentially* extendable place in it. In general this does
> not work. As H2 guy I would also reject patch like this.
>
> Sergi
>
> 2017-04-12 13:10 GMT+03:00 Alexander Paschenko <
> alexander.a.pasche...@gmail.com <javascript:;>>:
>
> > Sergi,
> >
> > Please have a closer look at what I've written in my first post. I don't
> > see why we have to cling to H2 and its parsing modes all the time — after
> > all, we're just talking string processing now, aren't we? (Yes, complex
> and
> > non trivial, but still.)
> >
> > What's wrong with idea of patching H2 to allow custom parsing? (With the
> > parsing itself living in Ignite code, obviously, not in H2.).
> >
> > What I propose is just to make H2's Parser class extendable and make H2
> > aware of its descendants via config params. And that's all with respect
> to
> > H2, nothing more.
> >
> > After that, on Ignite side we do all we want with our parser based on
> > theirs. It resembles story with custom types — first we make H2
> extendable
> > in the way we need, then we introduce exact features we need on Ignite
> > side.
> >
> > — Alex
> >
> > среда, 12 апреля 2017 г. пользователь Sergi Vladykin написал:
> >
> > > It definitely makes sense to add a separate mode for Ignite in H2.
> Though
> > > it is wrong to think that it will allow us to add any crazy syntax we
> > want
> > > (and it is actually a wrong idea imo), only the minor variations of the
> > > existing syntax. But this must be enough.
> > >
> > > I believe we should end up with something like
> > >
> > > CREATE TABLE person
> > > (
> > >   id INT PRIMARY KEY,
> > >   orgId INT AFFINITY KEY,
> > >   name VARCHAR
> > > )
> > > WITH "cfg:my_config_template.xml"
> > >
> > > Sergi
> > >
> > >
> > > 2017-04-12 7:54 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org
> <javascript:;>
> > > <javascript:;>>:
> > >
> > > > Agree, the updated syntax looks better. One change though: KEY ->
> > PRIMARY
> > > > KEY.
> > > >
> > > > Sergi, what do you think?
> > > >
> > > > D.
> > > >
> > > > On Tue, Apr 11, 2017 at 9:50 PM, Pavel Tupitsyn <
> ptupit...@apache.org <javascript:;>
> > > <javascript:;>>
> > > > wrote:
> > > >
> > > > > I think "WITH" syntax is ugly and cumbersome.
> > > > >
> > > > > We should go with this one:
> > > > > CREATE TABLE Person (id int AFFINITY KEY, uid uuid KEY, firstName
> > > > > varchar, lastName varchar)
> > > > >
> > > > > All databases (i.e. [1], [2]) work this way, I see no reason to
> > invent
> > > > > something different and confuse the users.
> > > > >
> > > > > [1]
> > > > > https://docs.microsoft.com/en-us/sql/t-sql/statements/create
> > > > > -table-transact-sql#syntax-1
> > > > > [2] https://www.postgresql.org/docs/9.1/static/sql-
> createtable.html
> > > > >
> > > > > On Wed, Apr 12, 2017 at 6:12 AM, Alexander Paschenko <
> > > > > alexander.a.pasche...@gmail.com <javascript:;> <javascript:;>>
> wrote:
> > > > >
> > > > > > Dmitry,
> > > > > >
> > > > > > For H2 it would be something like this - please note all those
> > > quotes,
> > > > > > commas and equality signs that would be mandatory:
> > > > > >
> > > > > > CREATE TABLE Person (id int, uid uuid, firstName varchar,
> lastName
> > > > > > varchar) WITH "keyFields=id,uuid","affinityKey=id"
> > > > > >
> > > > > > With suggested approach, it would be something like
> > > > > >
> > > > > > CREATE TABLE Person (id int AFFINITY KEY, uid uuid KEY, firstName
> > > > > > varchar, lastName varchar)
> > > > > >

Re: CREATE TABLE SQL command syntax

[Alexander Paschenko]

Sergi,

Please have a closer look at what I've written in my first post. I don't
see why we have to cling to H2 and its parsing modes all the time — after
all, we're just talking string processing now, aren't we? (Yes, complex and
non trivial, but still.)

What's wrong with idea of patching H2 to allow custom parsing? (With the
parsing itself living in Ignite code, obviously, not in H2.).

What I propose is just to make H2's Parser class extendable and make H2
aware of its descendants via config params. And that's all with respect to
H2, nothing more.

After that, on Ignite side we do all we want with our parser based on
theirs. It resembles story with custom types — first we make H2 extendable
in the way we need, then we introduce exact features we need on Ignite side.

— Alex

среда, 12 апреля 2017 г. пользователь Sergi Vladykin написал:

> It definitely makes sense to add a separate mode for Ignite in H2. Though
> it is wrong to think that it will allow us to add any crazy syntax we want
> (and it is actually a wrong idea imo), only the minor variations of the
> existing syntax. But this must be enough.
>
> I believe we should end up with something like
>
> CREATE TABLE person
> (
>   id INT PRIMARY KEY,
>   orgId INT AFFINITY KEY,
>   name VARCHAR
> )
> WITH "cfg:my_config_template.xml"
>
> Sergi
>
>
> 2017-04-12 7:54 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org
> <javascript:;>>:
>
> > Agree, the updated syntax looks better. One change though: KEY -> PRIMARY
> > KEY.
> >
> > Sergi, what do you think?
> >
> > D.
> >
> > On Tue, Apr 11, 2017 at 9:50 PM, Pavel Tupitsyn <ptupit...@apache.org
> <javascript:;>>
> > wrote:
> >
> > > I think "WITH" syntax is ugly and cumbersome.
> > >
> > > We should go with this one:
> > > CREATE TABLE Person (id int AFFINITY KEY, uid uuid KEY, firstName
> > > varchar, lastName varchar)
> > >
> > > All databases (i.e. [1], [2]) work this way, I see no reason to invent
> > > something different and confuse the users.
> > >
> > > [1]
> > > https://docs.microsoft.com/en-us/sql/t-sql/statements/create
> > > -table-transact-sql#syntax-1
> > > [2] https://www.postgresql.org/docs/9.1/static/sql-createtable.html
> > >
> > > On Wed, Apr 12, 2017 at 6:12 AM, Alexander Paschenko <
> > > alexander.a.pasche...@gmail.com <javascript:;>> wrote:
> > >
> > > > Dmitry,
> > > >
> > > > For H2 it would be something like this - please note all those
> quotes,
> > > > commas and equality signs that would be mandatory:
> > > >
> > > > CREATE TABLE Person (id int, uid uuid, firstName varchar, lastName
> > > > varchar) WITH "keyFields=id,uuid","affinityKey=id"
> > > >
> > > > With suggested approach, it would be something like
> > > >
> > > > CREATE TABLE Person (id int AFFINITY KEY, uid uuid KEY, firstName
> > > > varchar, lastName varchar)
> > > >
> > > > While this may not look like a drastic improvement in this particular
> > > > case, we someday most likely will want either an all-custom CREATE
> > > > CACHE command, or a whole bunch of new options for CREATE TABLE, if
> we
> > > > decide not to go with CREATE CACHE - I personally think that stuff
> > > > like
> > > >
> > > > CREATE TABLE ... WITH
> > > > "keyFields=id,uuid","affinityKey=id","cacheType=
> > partitioned","atomicity=
> > > > atomic","partitions=3"
> > > >
> > > > which will arise if we continue to try to stuff everything into WITH
> > > > will just bring more ugliness with time, and that's not to mention
> > > > that new CREATE CACHE syntax will be impossible or relatively hard to
> > > > introduce as we will have to approve it with H2 folks, and that's how
> > > > it will be with any new param or command that we want.
> > > >
> > > > Allowing to plug custom parser into H2 (as we do now with table
> > > > engine) will let us introduce any syntax we want and focus on
> > > > usability and not on compromises and workarounds (which WITH keyword
> > > > currently is).
> > > >
> > > > - Alex
> > > >
> > > > 2017-04-12 5:11 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org
> <javascript:;>>:
> > > > > Alexeander,
> > > > >
> > 

Re: CREATE TABLE SQL command syntax

[Alexander Paschenko]

Dmitry,

For H2 it would be something like this - please note all those quotes,
commas and equality signs that would be mandatory:

CREATE TABLE Person (id int, uid uuid, firstName varchar, lastName
varchar) WITH "keyFields=id,uuid","affinityKey=id"

With suggested approach, it would be something like

CREATE TABLE Person (id int AFFINITY KEY, uid uuid KEY, firstName
varchar, lastName varchar)

While this may not look like a drastic improvement in this particular
case, we someday most likely will want either an all-custom CREATE
CACHE command, or a whole bunch of new options for CREATE TABLE, if we
decide not to go with CREATE CACHE - I personally think that stuff
like

CREATE TABLE ... WITH
"keyFields=id,uuid","affinityKey=id","cacheType=partitioned","atomicity=atomic","partitions=3"

which will arise if we continue to try to stuff everything into WITH
will just bring more ugliness with time, and that's not to mention
that new CREATE CACHE syntax will be impossible or relatively hard to
introduce as we will have to approve it with H2 folks, and that's how
it will be with any new param or command that we want.

Allowing to plug custom parser into H2 (as we do now with table
engine) will let us introduce any syntax we want and focus on
usability and not on compromises and workarounds (which WITH keyword
currently is).

- Alex

2017-04-12 5:11 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>:
> Alexeander,
>
> Can you please provide an example of what the CREATE TABLE command would
> look like if we use WITH syntax from H2 vs. what you are proposing?
>
> D.
>
> On Tue, Apr 11, 2017 at 6:35 PM, Alexander Paschenko <
> alexander.a.pasche...@gmail.com> wrote:
>
>> Hello Igniters,
>>
>> Yup, it's THAT time once again as we haven't ultimately settled on
>> anything with the subj. as of yet, but I believe that now with DDL on
>> its way this talk can't be avoided anymore (sorry guys).
>>
>> The last time we talked about Ignite specific stuff we need to have in
>> CREATE TABLE (key fields list, affinity key, am I missing anything?),
>> the simplest approach suggested by Sergi was that we simply use WITH
>> part of H2's CREATE TABLE to pass stuff we need.
>>
>> This could work, but needless to say that such commands would look plain
>> ugly.
>>
>> I think we should go with custom syntax after all, BUT not in a way
>> suggested before by Sergi (propose Apache Ignite mode to H2).
>>
>> Instead, I suggest that we propose to H2 patch that would allow
>> plugging in *custom SQL parser* directly based on theirs (quite
>> elegant one) – I've had a look at their code, and this should not be
>> hard.
>>
>> Work on such a patch making syntax parsing overridable would take a
>> couple days which is not much time AND would give us the opportunity
>> to introduce to Ignite virtually any syntax we wish - both now and in
>> the future. Without worrying about compatibility with H2 ever again,
>> that is.
>>
>> Thoughts? After we agree on this principally and after H2 patch for
>> custom parsing is ready, we can roll our sleeves and focus on syntax
>> itself.
>>
>> - Alex
>>

CREATE TABLE SQL command syntax

[Alexander Paschenko]

Hello Igniters,

Yup, it's THAT time once again as we haven't ultimately settled on
anything with the subj. as of yet, but I believe that now with DDL on
its way this talk can't be avoided anymore (sorry guys).

The last time we talked about Ignite specific stuff we need to have in
CREATE TABLE (key fields list, affinity key, am I missing anything?),
the simplest approach suggested by Sergi was that we simply use WITH
part of H2's CREATE TABLE to pass stuff we need.

This could work, but needless to say that such commands would look plain ugly.

I think we should go with custom syntax after all, BUT not in a way
suggested before by Sergi (propose Apache Ignite mode to H2).

Instead, I suggest that we propose to H2 patch that would allow
plugging in *custom SQL parser* directly based on theirs (quite
elegant one) – I've had a look at their code, and this should not be
hard.

Work on such a patch making syntax parsing overridable would take a
couple days which is not much time AND would give us the opportunity
to introduce to Ignite virtually any syntax we wish - both now and in
the future. Without worrying about compatibility with H2 ever again,
that is.

Thoughts? After we agree on this principally and after H2 patch for
custom parsing is ready, we can roll our sleeves and focus on syntax
itself.

- Alex

[jira] [Created] (IGNITE-4947) Create AI 2.0 TC suites

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4947:
---

 Summary: Create AI 2.0 TC suites
 Key: IGNITE-4947
 URL: https://issues.apache.org/jira/browse/IGNITE-4947
 Project: Ignite
  Issue Type: Task
Affects Versions: 2.0
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.0


Due to OptimizedMarshaller removal from public API per IGNITE-4938, we need 
all-new post-OptimizedMarshaller set of TC suites that will be used by default 
after 2.0 is released.

What has to be done:

- Remove all OptimizedMarshaller specific suites
- Make all 'binary' suites 'standard'



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Apache Ignite 2.0 Release

[Alexander Paschenko]

I've fixed https://issues.apache.org/jira/browse/IGNITE-4354, PR is
https://github.com/apache/ignite/pull/1759.
Pavel, thank you very much for bringing that to my attention.

- Alex

2017-04-07 20:28 GMT+03:00 Sergi Vladykin :
> A bunch of SQL related tickets is delayed until H2 release on the next week.
>
> Sergi
>
> 2017-04-07 19:36 GMT+03:00 Alexey Goncharuk :
>
>> Folks,
>>
>> The status of ignite-3477 is as follows: we've fixed almost all tests and
>> currently waiting for ignite-4535 and ignite-4534 to be merged, which will
>> happen once TC for those branches is completed. This should happen over the
>> weekend.
>>
>> 2017-04-06 19:12 GMT+03:00 Alexey Kuznetsov :
>>
>> >  I've prepared
>> >
>> >  IGNITE-4349  -
>> > Discontinue the schema-import utility
>> >
>> > Anton, could you review my changes related to Ignite *build*?
>> > See: http://reviews.ignite.apache.org/ignite/review/IGNT-CR-163
>> >
>> >
>> > --
>> > Alexey Kuznetsov
>> >
>>

[jira] [Created] (IGNITE-4905) ClassCastException in DML benchmarks

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4905:
---

 Summary: ClassCastException in DML benchmarks
 Key: IGNITE-4905
 URL: https://issues.apache.org/jira/browse/IGNITE-4905
 Project: Ignite
  Issue Type: Bug
  Components: SQL
Affects Versions: 1.9
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.0






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-4903) More SQL side tests for CREATE/DROP INDEX

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4903:
---

 Summary: More SQL side tests for CREATE/DROP INDEX
 Key: IGNITE-4903
 URL: https://issues.apache.org/jira/browse/IGNITE-4903
 Project: Ignite
  Issue Type: Sub-task
  Components: SQL
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.0


IGNITE-4656 needs more tests to check indexes' actual usage and correctness.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Question about OBJ binary representation

[Alexander Paschenko]

Hi Vyacheslav,

1: Yes, exactly.

2: Hash code is written for all BinaryObjects. Starting with Ignite
1.9, hashCode implementations of original classes are never used to
compute hash codes for corresponding binary objects.

- Alex

2017-03-23 12:58 GMT+03:00 Vyacheslav Daradur :
> Following second question.
>
> In which cases it needs to write postWriteHashCode?
>
> 2017-03-23 12:27 GMT+03:00 Vyacheslav Daradur :
>
>> Hello everyone.
>>
>> Please, explain me, in which cases at marshalling after object header (24
>> bytes) it needs to write Class.getName?
>>
>> I understand that it means that this class isn't registered in
>> BinaryContext, and at deserializing we use it for loading Class with
>> ClassLoader.
>>
>> Please explain real scenarios.
>>
>> --
>> Best Regards, Vyacheslav
>>
>
>
>
> --
> Best Regards, Vyacheslav

Re: Documenting Apache Ignite 1.9 Features

[Alexander Paschenko]

I've written new sections about streaming both on that hidden jdbc 1.9
page and in (hidden for now and created by myself) DML 1.9 page -
figured that it might be a good thing to give the user a hint about
having a way to stream data via SQL on DML page as they could probably
miss it if we don't mention it anywhere besides JDBC page.

- Alex

2017-03-03 6:16 GMT+03:00 Denis Magda :
> Vovan, Alex P.,
>
> I've cloned JDBC page for version 1.9 and made it hidden:
> https://apacheignite.readme.io/docs/jdbc-driver-19
>
> Please add new properties added as a part of IGNITE-4169 to the properties
> table there.
>
> I would have done this on my own if I hadn't lost track of what we actually
> did :)
>
> --
> Denis
>
> On Thu, Mar 2, 2017 at 10:45 AM, Denis Magda  wrote:
>
>> Vovan,
>>
>> If Alex can’t finish the doc tomorrow could you write a paragraph or so
>> about
>>
>> Data Streamer Mode DML (*Alexander P.*). Please add a section to
>>> “Distributed DML” page:
>>>https://issues.apache.org/jira/browse/IGNITE-4169
>>>
>>>
>> ?
>>
>> To be honest I lost track what exactly was supported.
>>
>> —
>> Denis
>>
>> On Mar 2, 2017, at 8:14 AM, Igor Sapego  wrote:
>>
>> Denis, I will try to complete my part today.
>>
>> Best Regards,
>> Igor
>>
>> On Thu, Mar 2, 2017 at 7:12 PM, Denis Magda  wrote:
>>
>>> Igor, Alex P.,
>>>
>>> Will you be able to complete your part tomorrow? I’ll highly appreciate
>>> this because Prachi and me need some time to do a final review.
>>>
>>> —
>>> Denis
>>>
>>>
>>> Begin forwarded message:
>>>
>>> *From: *Denis Magda 
>>> *Subject: **Documenting Apache Ignite 1.9 Features*
>>> *Date: *February 28, 2017 at 11:46:03 AM PST
>>> *To: *dev@ignite.apache.org
>>> *Cc: *Vladimir Ozerov , "
>>> alexander.a.pasche...@gmail.com" , Igor
>>> Sapego 
>>> *Reply-To: *dev@ignite.apache.org
>>>
>>> Igniters,
>>>
>>> We’ve already documented most of the features and improvements except the
>>> ones below.
>>>
>>> Vovan, Igor, Alexander P., please spare some time this week preparing the
>>> docs. Once you’re done Prachi and I will do a final review. All the docs
>>> have to be ready by the time AI 1.9 gets released officially.
>>>
>>> 1) Parallel Query Execution (*Vovan*). Please add a section to
>>> “Distributed Queries” page:
>>>https://issues.apache.org/jira/browse/IGNITE-4106
>>>
>>> 2) Data Streamer Mode DML (*Alexander P.*). Please add a section to
>>> “Distributed DML” page:
>>>https://issues.apache.org/jira/browse/IGNITE-4169
>>>
>>> 3) C++ LoadCache (*Igor*). You can use Java cache store and loading
>>> approaches pages as a template:
>>>https://issues.apache.org/jira/browse/IGNITE-4670
>>>
>>> 4) C++ DML and SQL Grid documentation (*Igor*). Please do the same way as
>>> Pavel did for .NET:
>>>https://issues.apache.org/jira/browse/IGNITE-4620
>>>
>>> This page might be useful if you don’t remember how we document a new
>>> feature that is missing in current release:
>>> https://cwiki.apache.org/confluence/display/IGNITE/Documentation
>>>
>>> —
>>> Denis
>>>
>>>
>>>
>>
>>

[jira] [Created] (IGNITE-4742) Flaky IgniteCacheUpdateSqlQuerySelfTest.testTypeConversions

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4742:
---

 Summary: Flaky 
IgniteCacheUpdateSqlQuerySelfTest.testTypeConversions
 Key: IGNITE-4742
 URL: https://issues.apache.org/jira/browse/IGNITE-4742
 Project: Ignite
  Issue Type: Bug
  Components: SQL
Affects Versions: 1.8
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
Priority: Minor
 Fix For: 1.9


This test fails randomly on various query related suites and very seldom 
locally. Seems to be test problem and not of what is tested.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-4732) Invalid ids quoting logic in DML and GridSqlFunction

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4732:
---

 Summary: Invalid ids quoting logic in DML and GridSqlFunction
 Key: IGNITE-4732
 URL: https://issues.apache.org/jira/browse/IGNITE-4732
 Project: Ignite
  Issue Type: Bug
  Components: SQL
Affects Versions: 1.9
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 1.9






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Apache Ignite 1.9

[Alexander Paschenko]

Hi,

There are some questions regarding IGNITE-4169 - probably some design
approval will be required from Dmitry S. However, chances are good
that we'll make it in time.

- Alex

2017-02-16 14:48 GMT+03:00 Anton Vinogradov <avinogra...@gridgain.com>:
> Igniters,
>
> We're planning to have a code freeze tomorrow, Feb 17 19.00 MSK.
> Here's the list of issues we especially expect to be done in 1.9:
>
> IGNITE-4106 SQL: parallelize sql queries over cache local partitions Andrew
> Mashenkov
> IGNITE-4670 CPP: Implement LoadCache method Task Igor Sapego
> IGNITE-3860 Improve distributed SQL support. Sergi Vladykin
> IGNITE-3013 Support sorted merge index for SQL Sergi Vladykin
> IGNITE-4212 Ignite Benchmarking Simplification and Automation Oleg Ostanin
> IGNITE-4169 Data streamer mode for DML Alexander Paschenko
> IGNITE-3710 Upgrade ignite-spark module to Spark 2.0 Evgenii Zhuravlev
> IGNITE-1794 Ignite should support Hibernate 5 Mykola Pereyma
> IGNITE-4523 Allow distributed SQL query execution over explicit set of
> partitions Alexei Scherbakov
>
> Guys, please confirm that you will be able to complete listed tasks till
> Feb 17 19.00 MSK.
>
> On Thu, Feb 16, 2017 at 2:31 PM, Anton Vinogradov <avinogra...@gridgain.com>
> wrote:
>
>> Denis,
>>
>> Evgenii Zhuravlev going to fix this issue. I'll provide him help if
>> necessarry.
>>
>> On Wed, Feb 15, 2017 at 8:03 PM, Denis Magda <dma...@apache.org> wrote:
>>
>>> Anton,
>>>
>>> The implementation is correct and was reviewed by Val long time ago. The
>>> only issue that is left is the compilation in *TeamCity* only.
>>> You fixed it for one scala version and now should simply do the same for
>>> other scala version by adding a new suite to TeamCity and adjusting the pom.
>>>
>>> —
>>> Denis
>>>
>>> > On Feb 15, 2017, at 1:20 AM, Anton Vinogradov <avinogra...@gridgain.com>
>>> wrote:
>>> >
>>> > Denis,
>>> >
>>> > I don't know what is the plan for this issue.
>>> > You asked me to fix compilation and thats what I done :) , but I'm not
>>> sure
>>> > that issue implemented in proper way. I don't like changes I've done,
>>> they
>>> > looks redundant although it works.
>>> > Do we have somebody more familiar with spark inside the community?
>>> > We have to review what we already done and figure out what's left.
>>> >
>>> > On Wed, Feb 15, 2017 at 5:01 AM, Denis Magda <dma...@apache.org> wrote:
>>> >
>>> >> Continuing Spark related topic...
>>> >>
>>> >> This ticket definitely must be included in 1.9:
>>> >> https://issues.apache.org/jira/browse/IGNITE-3710 <
>>> >> https://issues.apache.org/jira/browse/IGNITE-3710>
>>> >>
>>> >> We can’t longer afford torture Ignite Shared RDD users who have to use
>>> an
>>> >> outdated Spark version.
>>> >>
>>> >> Anton, could you finalize it this week? As far as I remember, the only
>>> >> thing that is left is to add a missing test suite to Team City.
>>> >>
>>> >> —
>>> >> Denis
>>> >>
>>> >>> On Feb 14, 2017, at 5:00 PM, Denis Magda <dma...@apache.org> wrote:
>>> >>>
>>> >>> Reviewed and merged the first (!) Ignite Shared RDD examples
>>> contributed
>>> >> by Manish:
>>> >>>
>>> >>> https://issues.apache.org/jira/browse/IGNITE-4526
>>> >>>
>>> >>> Java:
>>> >>> https://github.com/apache/ignite/blob/b461cb47882861356ede58775bd9e2
>>> >> 53dcf26202/examples/src/main/java8/org/apache/ignite/example
>>> s/java8/spark/
>>> >> SharedRDDExample.java
>>> >>>
>>> >>> Scala:
>>> >>> https://github.com/apache/ignite/blob/b461cb47882861356ede58775bd9e2
>>> >> 53dcf26202/examples/src/main/scala/org/apache/ignite/
>>> >> scalar/examples/spark/ScalarSharedRDDExample.scala
>>> >>>
>>> >>> Manish, thanks for making the life of Apache Ignite users easier. No
>>> >> they can refer to your examples while learning Ignite Shared RDD API.
>>> >>>
>>> >>> —
>>> >>> Denis
>>> >>>
>>> >>>> On Feb 14, 2017, at 1:21 AM, Anton Vinogradov <
>>> avinogr

Re: DML data streaming

[Alexander Paschenko]

Folks,

Regarding INSERT semantics in JDBC DML streaming mode - I've left only
INSERTs supports as we'd agreed before.

However, current architecture of streaming related internals does not
give any clear way to intercept key duplicates and inform the user -
say, I can't just throw an exception from stream receiver (which is to
my knowledge the only place where we could filter erroneous keys) as
long as it will lead to whole batch remap and it's clearly not what we
want here.

Printing warning to log from the receiver is of little to no use as it
will happen on data nodes so the end user won't see anything.

What I've introduced for now is optional config param that turns on
allowOverwrite on the streamer used in DML operation.

Does anyone have any thoughts about what could/should be done
regarding informing user about key duplicates in streaming mode? Or
probably we should just let it be as it is now?

Regards,
Alex

2017-02-15 23:42 GMT+03:00 Dmitriy Setrakyan :
> On Wed, Feb 15, 2017 at 4:28 AM, Vladimir Ozerov 
> wrote:
>
>> Ok, let's put aside current fields configuration, I'll create separate
>> thread for it. As far as _KEY and _VAL, proposed change is exactly about
>> mappings:
>>
>> class QueryEntity {
>> ...
>> String keyFieldName;
>> String valFieldName;
>> ...
>> }
>>
>> The key thing is that we will not require users to be aware of our system
>> columns. Normally user should not bother about existence of hidden _KEY and
>> _VAL columns. Instead, we just allow them to optionally reference the whole
>> key and/or val through predefined name.
>>
>>
> Vladimir, how will it work from the DDL perspective. Let's say whenever
> user wants to create a table in Ignite?

Re: DML data streaming

[Alexander Paschenko]

And to avoid further confusion: UPDATE and DELETE are simply
impossible in streaming mode when the key is not completely defined as
long as data streamer operates with key-value pairs and not just
tuples of named values. That's why we can't do DELETE from Person
WHERE id1 = 5 from prev example with streamer - the Key { id1 = 5, id2
= 0 } that would be constructed from such query is just one key and is
handled by streamer as such while semantically that query is not about
ONE key but about ALL keys where id1 = 5.

- Alex

2017-02-10 11:49 GMT+03:00 Alexander Paschenko
<alexander.a.pasche...@gmail.com>:
> Dima,
>>
>> There are several ways to handle it. I would check how other databases
>> handle it, maybe we can borrow something. To the least, we should log such
>> errors in the log for now.
>>
>
> Logging errors would mean introducing some kind of stream receiver to
> do that and thus that would be really the same performance penalty for
> the successful operations. I think we should go with that optional
> flag for semantics after all.
>
>> You don't have to use _key. Primary key is usually a field in the class, so
>> you can use a normal column name. In any case, we should remove any usage
>> of _key before 2.0 is released.
>>
>> Again, if user does not have to specify _key on INSERT, then it is very
>> unclear to me, why user would need to specify _key for UPDATE or DELETE.
>> Something smells here. Can you please provide an example?
>>
>
> UPDATE and DELETE _in streaming mode_ are carried _only_ for "fast"
> optimized cases - i.e. those where _key (and possibly _val) are
> explicitly specified by the user thus allowing us to map UPDATE and
> DELETE directly to cache's replace and remove operations without
> messing with entry processors and doing map-reduce SELECT by given
> criteria.
>
> Say, we have Person { firstName, secondName } with key class Key { id1, id2 }
>
> If I say DELETE from Person WHERE _key = ? and specify arg via JDBC,
> there's no need to do any SELECT - we can just call IgniteCache.remove
> on that key.
>
> But if I say DELETE from Person WHERE id1 = 5 then there's no way to
> avoid MR - we have to find all keys that interest us first by doing
> SELECT as long as we know only partly about what keys the user wants
> to be affected.
>
> It works in the same way for UPDATE. And I hope that it's clear how
> it's different from INSERT - there's no MR by definition (we don't
> allow INSERT FROM SELECT in streaming mode).
>
> AGAIN: this all is said only about streaming mode; non streaming mode
> does those optimizations too, but it also allows complex conditions,
> while streaming mode does not allow them to keep things fast and avoid
> MR.
>
> That's the reason why I suggest that we drop UPDATE and DELETE from
> DML streaming as they mean messing with those soon-hidden columns.
>
> Still we could optimize stuff like DELETE from Person WHERE id1 = 5
> AND id2 = 6 - query involves ALL fields of key AND compares only for
> equality AND has no complex expressions - we can construct key
> unambiguously and still call remove directly.
>
> But to me it does not sound like a really great reason to leave UPDATE
> and DELETE in DML - the users will have to write some specific queries
> to use that while all other stuff will just be declined in that mode.
> And, as I said before, UPDATE and DELETE don't probably perfectly fit
> with primary data streamer use cases - after all, modifying existing
> stuff is not what data streamer is about.
>
> And regarding hiding columns: it's unclear how things will look like
> for caches like <int, int> when we remove _key and _val as long as
> tables for such cases currently have nothing but those two columns.
>
> - Alex
>
>>> 8 февр. 2017 г. 11:33 PM пользователь "Dmitriy Setrakyan" <
>>> dsetrak...@apache.org> написал:
>>>
>>> > Alexander,
>>> >
>>> > Are you suggesting that currently to execute a simple INSERT for 1 row we
>>> > invoke a data streamer on Ignite API? How about an update by a primary
>>> key?
>>> > Why not execute a simple cache put in either case?
>>> >
>>> > I think we had a separate thread where we agreed that the streamer should
>>> > only be turned on if a certain flag on a JDBC connection is set, no?
>>> >
>>> > D.
>>> >
>>> > On Wed, Feb 8, 2017 at 7:00 AM, Alexander Paschenko <
>>> > alexander.a.pasche...@gmail.com> wrote:
>>> >
>>> > > Hello Igniters,
>>> > >
>>> > > I'd like to rais

Re: DML data streaming

[Alexander Paschenko]

Dima,
>
> There are several ways to handle it. I would check how other databases
> handle it, maybe we can borrow something. To the least, we should log such
> errors in the log for now.
>

Logging errors would mean introducing some kind of stream receiver to
do that and thus that would be really the same performance penalty for
the successful operations. I think we should go with that optional
flag for semantics after all.

> You don't have to use _key. Primary key is usually a field in the class, so
> you can use a normal column name. In any case, we should remove any usage
> of _key before 2.0 is released.
>
> Again, if user does not have to specify _key on INSERT, then it is very
> unclear to me, why user would need to specify _key for UPDATE or DELETE.
> Something smells here. Can you please provide an example?
>

UPDATE and DELETE _in streaming mode_ are carried _only_ for "fast"
optimized cases - i.e. those where _key (and possibly _val) are
explicitly specified by the user thus allowing us to map UPDATE and
DELETE directly to cache's replace and remove operations without
messing with entry processors and doing map-reduce SELECT by given
criteria.

Say, we have Person { firstName, secondName } with key class Key { id1, id2 }

If I say DELETE from Person WHERE _key = ? and specify arg via JDBC,
there's no need to do any SELECT - we can just call IgniteCache.remove
on that key.

But if I say DELETE from Person WHERE id1 = 5 then there's no way to
avoid MR - we have to find all keys that interest us first by doing
SELECT as long as we know only partly about what keys the user wants
to be affected.

It works in the same way for UPDATE. And I hope that it's clear how
it's different from INSERT - there's no MR by definition (we don't
allow INSERT FROM SELECT in streaming mode).

AGAIN: this all is said only about streaming mode; non streaming mode
does those optimizations too, but it also allows complex conditions,
while streaming mode does not allow them to keep things fast and avoid
MR.

That's the reason why I suggest that we drop UPDATE and DELETE from
DML streaming as they mean messing with those soon-hidden columns.

Still we could optimize stuff like DELETE from Person WHERE id1 = 5
AND id2 = 6 - query involves ALL fields of key AND compares only for
equality AND has no complex expressions - we can construct key
unambiguously and still call remove directly.

But to me it does not sound like a really great reason to leave UPDATE
and DELETE in DML - the users will have to write some specific queries
to use that while all other stuff will just be declined in that mode.
And, as I said before, UPDATE and DELETE don't probably perfectly fit
with primary data streamer use cases - after all, modifying existing
stuff is not what data streamer is about.

And regarding hiding columns: it's unclear how things will look like
for caches like <int, int> when we remove _key and _val as long as
tables for such cases currently have nothing but those two columns.

- Alex

>> 8 февр. 2017 г. 11:33 PM пользователь "Dmitriy Setrakyan" <
>> dsetrak...@apache.org> написал:
>>
>> > Alexander,
>> >
>> > Are you suggesting that currently to execute a simple INSERT for 1 row we
>> > invoke a data streamer on Ignite API? How about an update by a primary
>> key?
>> > Why not execute a simple cache put in either case?
>> >
>> > I think we had a separate thread where we agreed that the streamer should
>> > only be turned on if a certain flag on a JDBC connection is set, no?
>> >
>> > D.
>> >
>> > On Wed, Feb 8, 2017 at 7:00 AM, Alexander Paschenko <
>> > alexander.a.pasche...@gmail.com> wrote:
>> >
>> > > Hello Igniters,
>> > >
>> > > I'd like to raise few questions regarding data streaming via DML
>> > > statements.
>> > >
>> > > Currently, all types of DML statements are supported (INSERT, UPDATE,
>> > > DELETE, MERGE).
>> > >
>> > > UPDATE and DELETE are supported in streaming mode only when their
>> > > WHERE condition is bounded with _key and/or _val columns, and UPDATE
>> > > works only for _val column directly.
>> > >
>> > > Seeing some activity in direction of hiding _key and _val from the
>> > > user as far as possible, these features seem pointless and should not
>> > > be released, what do you think?
>> > >
>> > > Also INSERT in streaming mode currently does not throw errors on
>> > > duplicate keys and silently ignores such new records (as long as it's
>> > > faster than it would work if we'd introduced receiver that would throw
>> > > exceptions) - this can

Re: DML data streaming

[Alexander Paschenko]

Sergey,

Streaming does not make sense for INSERT FROM SELECT as this pattern does
not match primary use case for streaming (bulk data load to Ignite).

Dima,

No, I suggest that data streamer mode supports full semantic sense of
INSERT (throw an ex if there's a duplicate of PK) optionally and depending
on a flag (that is to be introduced). Currently new records are quietly
ignored on key duplication — it's really just a question of notifying the
user about duplicate keys in streaming mode.

Update by primary key is implemented now, but obviously it involves user
messing with _key column that we're planning to hide from them in near
future.

Streaming is turned on via the flag, just as we've agreed in one of prev
threads. This thread is not about how we turn streaming on but rather about
semantic correctness of INSERT and MERGE in this mode and about whether we
need UPDATE and DELETE in it as they do not essentially load new data into
cache and (_in streaming mode_) make user mess with service columns of _key
and _val.

— Alex
8 февр. 2017 г. 11:33 PM пользователь "Dmitriy Setrakyan" <
dsetrak...@apache.org> написал:

> Alexander,
>
> Are you suggesting that currently to execute a simple INSERT for 1 row we
> invoke a data streamer on Ignite API? How about an update by a primary key?
> Why not execute a simple cache put in either case?
>
> I think we had a separate thread where we agreed that the streamer should
> only be turned on if a certain flag on a JDBC connection is set, no?
>
> D.
>
> On Wed, Feb 8, 2017 at 7:00 AM, Alexander Paschenko <
> alexander.a.pasche...@gmail.com> wrote:
>
> > Hello Igniters,
> >
> > I'd like to raise few questions regarding data streaming via DML
> > statements.
> >
> > Currently, all types of DML statements are supported (INSERT, UPDATE,
> > DELETE, MERGE).
> >
> > UPDATE and DELETE are supported in streaming mode only when their
> > WHERE condition is bounded with _key and/or _val columns, and UPDATE
> > works only for _val column directly.
> >
> > Seeing some activity in direction of hiding _key and _val from the
> > user as far as possible, these features seem pointless and should not
> > be released, what do you think?
> >
> > Also INSERT in streaming mode currently does not throw errors on
> > duplicate keys and silently ignores such new records (as long as it's
> > faster than it would work if we'd introduced receiver that would throw
> > exceptions) - this can be fixed with additional flag that could
> > _optionally_ make INSERT slower but more accurate in semantic.
> >
> > And MERGE in streaming mode currently not totally accurate in
> > semantic, too - on key presence, it will just replace whole value with
> > new one thus potentially making values of some concrete columns/fields
> > lost - this is analogous to
> > https://issues.apache.org/jira/browse/IGNITE-4489, but hardly can be
> > fixed as long as probably it would hit performance and would be
> > unresonably complex to implement.
> >
> > I suggest that we drop all except INSERT and introduce optional flag
> > for its totally correct semantic behavior as described above.
> >
> > - Alex
> >
>

Re: DML data streaming

[Alexander Paschenko]

Also, currently it's possible to run SELECTs on "streamed"
connections, and probably this is odd and should not be released too,
what do you think?

- Alex

2017-02-08 18:00 GMT+03:00 Alexander Paschenko
<alexander.a.pasche...@gmail.com>:
> Hello Igniters,
>
> I'd like to raise few questions regarding data streaming via DML statements.
>
> Currently, all types of DML statements are supported (INSERT, UPDATE,
> DELETE, MERGE).
>
> UPDATE and DELETE are supported in streaming mode only when their
> WHERE condition is bounded with _key and/or _val columns, and UPDATE
> works only for _val column directly.
>
> Seeing some activity in direction of hiding _key and _val from the
> user as far as possible, these features seem pointless and should not
> be released, what do you think?
>
> Also INSERT in streaming mode currently does not throw errors on
> duplicate keys and silently ignores such new records (as long as it's
> faster than it would work if we'd introduced receiver that would throw
> exceptions) - this can be fixed with additional flag that could
> _optionally_ make INSERT slower but more accurate in semantic.
>
> And MERGE in streaming mode currently not totally accurate in
> semantic, too - on key presence, it will just replace whole value with
> new one thus potentially making values of some concrete columns/fields
> lost - this is analogous to
> https://issues.apache.org/jira/browse/IGNITE-4489, but hardly can be
> fixed as long as probably it would hit performance and would be
> unresonably complex to implement.
>
> I suggest that we drop all except INSERT and introduce optional flag
> for its totally correct semantic behavior as described above.
>
> - Alex

DML data streaming

[Alexander Paschenko]

Hello Igniters,

I'd like to raise few questions regarding data streaming via DML statements.

Currently, all types of DML statements are supported (INSERT, UPDATE,
DELETE, MERGE).

UPDATE and DELETE are supported in streaming mode only when their
WHERE condition is bounded with _key and/or _val columns, and UPDATE
works only for _val column directly.

Seeing some activity in direction of hiding _key and _val from the
user as far as possible, these features seem pointless and should not
be released, what do you think?

Also INSERT in streaming mode currently does not throw errors on
duplicate keys and silently ignores such new records (as long as it's
faster than it would work if we'd introduced receiver that would throw
exceptions) - this can be fixed with additional flag that could
_optionally_ make INSERT slower but more accurate in semantic.

And MERGE in streaming mode currently not totally accurate in
semantic, too - on key presence, it will just replace whole value with
new one thus potentially making values of some concrete columns/fields
lost - this is analogous to
https://issues.apache.org/jira/browse/IGNITE-4489, but hardly can be
fixed as long as probably it would hit performance and would be
unresonably complex to implement.

I suggest that we drop all except INSERT and introduce optional flag
for its totally correct semantic behavior as described above.

- Alex

[jira] [Created] (IGNITE-4660) Add DML capabilities to legacy JDBC driver

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4660:
---

 Summary: Add DML capabilities to legacy JDBC driver
 Key: IGNITE-4660
 URL: https://issues.apache.org/jira/browse/IGNITE-4660
 Project: Ignite
  Issue Type: Improvement
  Components: jdbc-driver, SQL
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 2.0


Legacy Ignite JDBC driver lacks DML capabilities, but it turns out that there 
still are plenty of its users who need DML too, so we should de-deprecate it 
and enable updating operations in it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Do not use H2 parser for DDL

[Alexander Paschenko]

Guys,

And what would you say if I suggested that we implement custom grammar
support with ANTLR? It allows you to describe pretty much any grammar
in a declarative way, generates lexer and parser and then allows to
easily process parsed commands by implementing few (generated)
interfaces. Yesterday I gave it a try and it's really simple.
Downsides are use of generated code itself (I'm pretty sure someone is
strongly against it) and relative wordiness of resulting code written
manually. But this approach will no doubt save time and allow any
extensions or changes in syntax in the future w/o worrying about H2 or
anyone 3rd party. Thoughts?

- Alex

2017-01-27 21:56 GMT+03:00 Vladimir Ozerov :
> My point was that we can avoid dependency on 3rd party developers for this
> relatively simple logic.
>
> On Fri, Jan 27, 2017 at 8:07 PM, Dmitriy Setrakyan 
> wrote:
>
>> On Fri, Jan 27, 2017 at 5:45 AM, Sergi Vladykin 
>> wrote:
>>
>> > H2 to some extent supports syntax (and quirks) from other databases. For
>> > example you can start it with MODE=MySQL and it will allow some MySQL
>> > specific syntax to be handled.
>> >
>> > Having said that, IMO the most correct way to handle non-standard syntax
>> is
>> > to introduce H2 MODE=ApacheIgnite and put the needed switches there.
>> >
>> > Though this needs to be negotiated with H2 folks.
>> >
>>
>> Sergi, can you ask the H2 folks about this? I agree, this sounds like the
>> most correct way.
>>

[jira] [Created] (IGNITE-4575) Implement in Ignite wrapper for enums based on H2 user value type

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4575:
---

 Summary: Implement in Ignite wrapper for enums based on H2 user 
value type
 Key: IGNITE-4575
 URL: https://issues.apache.org/jira/browse/IGNITE-4575
 Project: Ignite
  Issue Type: Sub-task
  Components: SQL
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 1.9






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4574) Introduce user value types to H2 w/custom conversion logic

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4574:
---

 Summary: Introduce user value types to H2 w/custom conversion logic
 Key: IGNITE-4574
 URL: https://issues.apache.org/jira/browse/IGNITE-4574
 Project: Ignite
  Issue Type: Sub-task
Reporter: Alexander Paschenko






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4573) Optimize H2 comparisons w/constant

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4573:
---

 Summary: Optimize H2 comparisons w/constant
 Key: IGNITE-4573
 URL: https://issues.apache.org/jira/browse/IGNITE-4573
 Project: Ignite
  Issue Type: Sub-task
  Components: SQL
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko


Currently H2 performs constant conversions for each row - say, for {{WHERE 
person.id = '1'}} the {{'1'}} will be converted to {{int}} for each row which 
is clearly redundant.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4570) Handle CREATE INDEX statements

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4570:
---

 Summary: Handle CREATE INDEX statements
 Key: IGNITE-4570
 URL: https://issues.apache.org/jira/browse/IGNITE-4570
 Project: Ignite
  Issue Type: Sub-task
  Components: SQL
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 1.9


Parse and execute CREATE INDEX boiling down to IgniteCacheEx.createQueryIndex 
introduced by IGNITE-4566



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4569) Create local portion of index w/table locking

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4569:
---

 Summary: Create local portion of index w/table locking
 Key: IGNITE-4569
 URL: https://issues.apache.org/jira/browse/IGNITE-4569
 Project: Ignite
  Issue Type: Sub-task
  Components: cache, SQL
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 1.9


Actual index data modification - initially will render whole SQL table 
inaccessible during initial index buildup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4568) Dynamically create distributed index

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4568:
---

 Summary: Dynamically create distributed index
 Key: IGNITE-4568
 URL: https://issues.apache.org/jira/browse/IGNITE-4568
 Project: Ignite
  Issue Type: Sub-task
  Components: messaging, SQL
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 1.9


Handle index creation messaging using means introduced by IGNITE-4567



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4567) Design and implement means for distributed DDL statements

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4567:
---

 Summary: Design and implement means for distributed DDL statements
 Key: IGNITE-4567
 URL: https://issues.apache.org/jira/browse/IGNITE-4567
 Project: Ignite
  Issue Type: Sub-task
  Components: messaging, SQL
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko


We need some kind of common way to handle stuff like CREATE INDEX/CREATE TABLE 
statements - in other words, run potentially long-running (and possibly cache 
locking) DDL operations on cluster nodes that would allow using new 
tables/indexes only when all nodes have performed requested operations and data 
modifications.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4566) Introduce IgniteCacheEx.createQueryIndex

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4566:
---

 Summary: Introduce IgniteCacheEx.createQueryIndex
 Key: IGNITE-4566
 URL: https://issues.apache.org/jira/browse/IGNITE-4566
 Project: Ignite
  Issue Type: Sub-task
  Components: cache, SQL
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 1.9






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4565) Support CREATE INDEX DDL statements

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4565:
---

 Summary: Support CREATE INDEX DDL statements
 Key: IGNITE-4565
 URL: https://issues.apache.org/jira/browse/IGNITE-4565
 Project: Ignite
  Issue Type: New Feature
  Components: SQL
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 1.9






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4553) Parse CREATE INDEX statements

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4553:
---

 Summary: Parse CREATE INDEX statements
 Key: IGNITE-4553
 URL: https://issues.apache.org/jira/browse/IGNITE-4553
 Project: Ignite
  Issue Type: Task
  Components: SQL
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 1.9






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: DDL implementation details

[Alexander Paschenko]

Sergi,

Regarding CREATE SCHEMA/TABLE parsing:

2017-01-12 18:51 GMT+03:00 Sergi Vladykin :
> Hi,
>
> 1. For now I'm against inventing any custom SQL syntax and implementing
> parsing.
> Currently H2 supports the following syntax:
>
> CREATE TABLE test(...) WITH "myCustomParamString"
>
> This is enough for us to pass the needed parameters.

H2's grammar requires to supply table engine class name as well, you
can't just go with WITH:

http://www.h2database.com/html/grammar.html#create_table

And code of H2's org.h2.command.Parser works exactly like this - you
can't just set default engine in connection string, in order to use
WITH, you have to specify both engine and WITH.

So we have either to force users to specify engine in their queries
(ugly and hardly acceptable), or have to somehow analyze CREATE TABLE
and sneak ENGINE param into the query ourselves.

But this again smells custom parsing, if you ask me.

>
> 2. Agree with AG, we have to separate cache creation from table creation.
> Cache == SQL schema for us. We just have to add the same WITH syntax in H2
> for schema creation like this:
>
> CREATE SCHEMA "MyCacheName" WITH "cacheConfig=myCache.xml"
>

Currently H2 does not allow using WITH with CREATE SCHEMA, proof:
http://www.h2database.com/html/grammar.html#create_schema

Do you suggest that we should somehow introduce this syntax to H2 engine first?

In the light of this, CREATE CACHE might seem legit. Thus, if we
depart from CREATE SCHEMA == create cache and introduce custom syntax
for CREATE CACHE, then we'll end up closer to what Vlad suggest –
schema is a logical object, caches exist on their own right.

Also, to add to Vlad's thoughts on Ignite becoming a database, I would
like to point out that for now we have talked only about schemas,
tables, and caches, but there are also *databases* as organizational
entities. Shouldn't we start thinking about those as well?

- Alex

Re: DDL implementation details

[Alexander Paschenko]

Dima,

2017-01-13 21:20 GMT+03:00 Dmitriy Setrakyan :
> How about dynamic caches? Isn't the metadata already maintained across all
> nodes every time a new cache is created? What am I missing here?

Right, however, this section of the docs

https://apacheignite.readme.io/docs/jcache#section-dynamic-cache

does not clarify Ignite's behavior regarding dynamic caches when a new
node joins the cluster. Will it just pick up dynamic cache
configurations from the peers without being explicitly configured? If
yes, I suggest that we at least mention it in the docs. (Sorry if I'm
missing something but I skimmed through the documentation and have not
found a clear answer to this.) Could someone clarify please?

- Alex

Re: DDL implementation details

[Alexander Paschenko]

Dima,

>> But, as you have justly noted, we have to keep nodes joining the
>> cluster up-to-date about what schemas and tables need to be created in
>> order for those nodes to participate in distributed queries. And,
>> correct me if I'm wrong, but this is relevant even outside of context
>> of 2.0 and page memory and persistent stores, amirite?
>>
>
> This should already be supported in Ignite. Otherwise, how are we able to
> run queries today?
>

Currently SQL structures identity is maintained between nodes via
configuration identity. Nodes participating in distributed queries
must be started w/identical configs, there's no mechanism to propagate
SQL related configuration between peer nodes. This works as long as
SQL structures are created currently just once, at node startup.
However, if we wish to create those structures dynamically, we have to
maintain some sort of that metadata delivery to new nodes.

- Alex

>>
>> 2017-01-13 1:47 GMT+08:00 Sergi Vladykin <sergi.vlady...@gmail.com>:
>> > The xml config was only for example. We can put in this configuration
>> > string cache config parameters directly like this:
>> >
>> > CREATE SCHEMA "MyCacheName" WITH
>> > "cacheMode=REPLICATED;atomicityMode=ATOMIC"
>> >
>> > Sergi
>> >
>> > 2017-01-12 20:21 GMT+03:00 Alexander Paschenko <
>> > alexander.a.pasche...@gmail.com>:
>> >
>> >> Sergi, Alexey G.,
>> >>
>> >> I see your point and am rather inclined to agree that we should let
>> >> current notion of "single schema - multiple tables" live.
>> >>
>> >> Still, if we create schema with cache config file, what's the whole
>> >> point of SQL then if the user anyway has to write XML? This probably
>> >> could be useful to propagate configuration to all cluster nodes tho.
>> >>
>> >> And if we skip CREATE TABLE now, it means that we leave user facing
>> >> the need to write XML configuration, no other options. Is this what we
>> >> want?
>> >>
>> >> Still I must admit that leaving user with his familiar XML stuff looks
>> >> attractive - no messing with bunch of unknown new params, just write
>> >> your XML and go. Also it's portable and allows to re-use
>> >> configurations easily, so undoubtedly is a good approach from some
>> >> point.
>> >>
>> >> - Alex
>> >>
>> >> 2017-01-12 23:51 GMT+08:00 Sergi Vladykin <sergi.vlady...@gmail.com>:
>> >> > Hi,
>> >> >
>> >> > 1. For now I'm against inventing any custom SQL syntax and
>> implementing
>> >> > parsing.
>> >> > Currently H2 supports the following syntax:
>> >> >
>> >> > CREATE TABLE test(...) WITH "myCustomParamString"
>> >> >
>> >> > This is enough for us to pass the needed parameters.
>> >> >
>> >> > 2. Agree with AG, we have to separate cache creation from table
>> creation.
>> >> > Cache == SQL schema for us. We just have to add the same WITH syntax
>> in
>> >> H2
>> >> > for schema creation like this:
>> >> >
>> >> > CREATE SCHEMA "MyCacheName" WITH "cacheConfig=myCache.xml"
>> >> >
>> >> > 3. If we want to create tables then I suggest to put this
>> functionality
>> >> to
>> >> > 2.0+PageMemory right away and think where and how we are going to
>> store
>> >> all
>> >> > the related metadata.This is especially important for persistent
>> >> storages.
>> >> >
>> >> > Sergi
>> >> >
>> >> >
>> >> > 2017-01-12 16:56 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>:
>> >> >
>> >> >> I am afraid in this case user will have to define too much schemes -
>> >> >> boilerplate.
>> >> >> Does it make sense at all to pack multiple tuples into a single cache
>> >> from
>> >> >> user perspective?
>> >> >>
>> >> >> On Thu, Jan 12, 2017 at 4:40 PM, Alexey Goncharuk <
>> >> >> alexey.goncha...@gmail.com> wrote:
>> >> >>
>> >> >> > Alexander,
>> >> >> >
>> >> >> > Will we keep the old option to have multiple tables in one cache?
>> If
>> >

Re: DDL implementation details

[Alexander Paschenko]

BTW, I have also did some H2 guts inspection, and, as I see it now,
it's not impossible to do custom parsing without (an awful lot) of
ugliness. What we would have to do is basically spoof private
singleton org.h2.engine.Engine#INSTANCE on node start via reflection
with our custom implementation that could invoke custom parsing when
needed.

Not that we need it now - just in case.

- Alex

2017-01-13 2:16 GMT+08:00 Alexander Paschenko <alexander.a.pasche...@gmail.com>:
> Sergi,
>
> OK, great. Still, what's up with CREATE TABLE? After a bit of code
> digging, I currently don't see major obstacles against registering
> query entities (i.e. type descriptors) on the fly - CREATE TABLE will
> essentially boil down to *GridQueryIndexing#registerType* call.
>
> But, as you have justly noted, we have to keep nodes joining the
> cluster up-to-date about what schemas and tables need to be created in
> order for those nodes to participate in distributed queries. And,
> correct me if I'm wrong, but this is relevant even outside of context
> of 2.0 and page memory and persistent stores, amirite?
>
> - Alex
>
> 2017-01-13 1:47 GMT+08:00 Sergi Vladykin <sergi.vlady...@gmail.com>:
>> The xml config was only for example. We can put in this configuration
>> string cache config parameters directly like this:
>>
>> CREATE SCHEMA "MyCacheName" WITH
>> "cacheMode=REPLICATED;atomicityMode=ATOMIC"
>>
>> Sergi
>>
>> 2017-01-12 20:21 GMT+03:00 Alexander Paschenko <
>> alexander.a.pasche...@gmail.com>:
>>
>>> Sergi, Alexey G.,
>>>
>>> I see your point and am rather inclined to agree that we should let
>>> current notion of "single schema - multiple tables" live.
>>>
>>> Still, if we create schema with cache config file, what's the whole
>>> point of SQL then if the user anyway has to write XML? This probably
>>> could be useful to propagate configuration to all cluster nodes tho.
>>>
>>> And if we skip CREATE TABLE now, it means that we leave user facing
>>> the need to write XML configuration, no other options. Is this what we
>>> want?
>>>
>>> Still I must admit that leaving user with his familiar XML stuff looks
>>> attractive - no messing with bunch of unknown new params, just write
>>> your XML and go. Also it's portable and allows to re-use
>>> configurations easily, so undoubtedly is a good approach from some
>>> point.
>>>
>>> - Alex
>>>
>>> 2017-01-12 23:51 GMT+08:00 Sergi Vladykin <sergi.vlady...@gmail.com>:
>>> > Hi,
>>> >
>>> > 1. For now I'm against inventing any custom SQL syntax and implementing
>>> > parsing.
>>> > Currently H2 supports the following syntax:
>>> >
>>> > CREATE TABLE test(...) WITH "myCustomParamString"
>>> >
>>> > This is enough for us to pass the needed parameters.
>>> >
>>> > 2. Agree with AG, we have to separate cache creation from table creation.
>>> > Cache == SQL schema for us. We just have to add the same WITH syntax in
>>> H2
>>> > for schema creation like this:
>>> >
>>> > CREATE SCHEMA "MyCacheName" WITH "cacheConfig=myCache.xml"
>>> >
>>> > 3. If we want to create tables then I suggest to put this functionality
>>> to
>>> > 2.0+PageMemory right away and think where and how we are going to store
>>> all
>>> > the related metadata.This is especially important for persistent
>>> storages.
>>> >
>>> > Sergi
>>> >
>>> >
>>> > 2017-01-12 16:56 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>:
>>> >
>>> >> I am afraid in this case user will have to define too much schemes -
>>> >> boilerplate.
>>> >> Does it make sense at all to pack multiple tuples into a single cache
>>> from
>>> >> user perspective?
>>> >>
>>> >> On Thu, Jan 12, 2017 at 4:40 PM, Alexey Goncharuk <
>>> >> alexey.goncha...@gmail.com> wrote:
>>> >>
>>> >> > Alexander,
>>> >> >
>>> >> > Will we keep the old option to have multiple tables in one cache? If
>>> so,
>>> >> > how will create table statement know which cache to choose?
>>> >> >
>>> >> > It seems to me that to be consistent with the current DML
>>> implementation
>>> >> we
&

Re: DDL implementation details

[Alexander Paschenko]

Sergi,

OK, great. Still, what's up with CREATE TABLE? After a bit of code
digging, I currently don't see major obstacles against registering
query entities (i.e. type descriptors) on the fly - CREATE TABLE will
essentially boil down to *GridQueryIndexing#registerType* call.

But, as you have justly noted, we have to keep nodes joining the
cluster up-to-date about what schemas and tables need to be created in
order for those nodes to participate in distributed queries. And,
correct me if I'm wrong, but this is relevant even outside of context
of 2.0 and page memory and persistent stores, amirite?

- Alex

2017-01-13 1:47 GMT+08:00 Sergi Vladykin <sergi.vlady...@gmail.com>:
> The xml config was only for example. We can put in this configuration
> string cache config parameters directly like this:
>
> CREATE SCHEMA "MyCacheName" WITH
> "cacheMode=REPLICATED;atomicityMode=ATOMIC"
>
> Sergi
>
> 2017-01-12 20:21 GMT+03:00 Alexander Paschenko <
> alexander.a.pasche...@gmail.com>:
>
>> Sergi, Alexey G.,
>>
>> I see your point and am rather inclined to agree that we should let
>> current notion of "single schema - multiple tables" live.
>>
>> Still, if we create schema with cache config file, what's the whole
>> point of SQL then if the user anyway has to write XML? This probably
>> could be useful to propagate configuration to all cluster nodes tho.
>>
>> And if we skip CREATE TABLE now, it means that we leave user facing
>> the need to write XML configuration, no other options. Is this what we
>> want?
>>
>> Still I must admit that leaving user with his familiar XML stuff looks
>> attractive - no messing with bunch of unknown new params, just write
>> your XML and go. Also it's portable and allows to re-use
>> configurations easily, so undoubtedly is a good approach from some
>> point.
>>
>> - Alex
>>
>> 2017-01-12 23:51 GMT+08:00 Sergi Vladykin <sergi.vlady...@gmail.com>:
>> > Hi,
>> >
>> > 1. For now I'm against inventing any custom SQL syntax and implementing
>> > parsing.
>> > Currently H2 supports the following syntax:
>> >
>> > CREATE TABLE test(...) WITH "myCustomParamString"
>> >
>> > This is enough for us to pass the needed parameters.
>> >
>> > 2. Agree with AG, we have to separate cache creation from table creation.
>> > Cache == SQL schema for us. We just have to add the same WITH syntax in
>> H2
>> > for schema creation like this:
>> >
>> > CREATE SCHEMA "MyCacheName" WITH "cacheConfig=myCache.xml"
>> >
>> > 3. If we want to create tables then I suggest to put this functionality
>> to
>> > 2.0+PageMemory right away and think where and how we are going to store
>> all
>> > the related metadata.This is especially important for persistent
>> storages.
>> >
>> > Sergi
>> >
>> >
>> > 2017-01-12 16:56 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>:
>> >
>> >> I am afraid in this case user will have to define too much schemes -
>> >> boilerplate.
>> >> Does it make sense at all to pack multiple tuples into a single cache
>> from
>> >> user perspective?
>> >>
>> >> On Thu, Jan 12, 2017 at 4:40 PM, Alexey Goncharuk <
>> >> alexey.goncha...@gmail.com> wrote:
>> >>
>> >> > Alexander,
>> >> >
>> >> > Will we keep the old option to have multiple tables in one cache? If
>> so,
>> >> > how will create table statement know which cache to choose?
>> >> >
>> >> > It seems to me that to be consistent with the current DML
>> implementation
>> >> we
>> >> > should have a CREATE SCHEMA statement which will define the cache and
>> >> cache
>> >> > configuration, and CREATE TABLE should specify the schema name.
>> >> >
>> >> > Otherwise, we should enforce the single type per cache rule at the
>> >> > configuration level and in runtime.
>> >> >
>> >> > As for affinity and primary key - agree with Vladimir.
>> >> >
>> >> > --
>> >> > AG
>> >> >
>> >> > 2017-01-12 11:41 GMT+03:00 Sergey Kozlov <skoz...@gridgain.com>:
>> >> >
>> >> > > As first stage of DDL we can implement following CREATE TABLE
>> statement
>> >> > > support:
>> >>

Re: DDL implementation details

[Alexander Paschenko]

ith using TABLESPACE (Postgess) or for
>> > > > CREATE
>> > > > > TABLE with using PARTITIONS (MySQL).
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Wed, Jan 11, 2017 at 10:05 PM, Vladimir Ozerov <
>> > > voze...@gridgain.com>
>> > > > > wrote:
>> > > > >
>> > > > > > I believe custom synthax and parsing is a *must* for us, as well
>> as
>> > > for
>> > > > > any
>> > > > > > distributed database. At the very least we need to specify
>> affinity
>> > > key
>> > > > > > column somehow. Any cache property can be specified at the very
>> end
>> > > of
>> > > > > > table definition. Key columns can be determined as the ones with
>> > > > PRIMARY
>> > > > > > KEY constraint (Alex K. idea) + affinity column(s):
>> > > > > >
>> > > > > > CREATE TABLE employee (
>> > > > > > id BIGINT PRIMARY KEY,
>> > > > > > dept_id BIGINT AFFINITY KEY,
>> > > > > > name VARCHAR(128),
>> > > > > > address VARCHAR(256)
>> > > > > > BACKUPS 2,
>> > > > > > ATOMICITY_MODE ATOMIC,
>> > > > > > );
>> > > > > >
>> > > > > > "id" and "dept_id" form key type, "name" and "address" form value
>> > > type.
>> > > > > >
>> > > > > > Vladimir.
>> > > > > >
>> > > > > > On Wed, Jan 11, 2017 at 9:08 PM, Alexey Kuznetsov <
>> > > > akuznet...@apache.org
>> > > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi, Alex!
>> > > > > > >
>> > > > > > > As far as I know most RDBMS allow something like: create table
>> t1
>> > > (id
>> > > > > > > integer primary key, )
>> > > > > > > How about to take as key field that marked as "primary key"?
>> > > > > > >
>> > > > > > > As of atomicity and replication - I think it is a cache
>> > properties
>> > > > and
>> > > > > > with
>> > > > > > > create table we will create "types" in cache. No?
>> > > > > > > I thought that cache it is a kind of "schema" in RDBMS.
>> > > > > > >
>> > > > > > > Could you describe what will be created with CREATE TABLE?
>> > > > > > >
>> > > > > > > On Thu, Jan 12, 2017 at 12:54 AM, Alexander Paschenko <
>> > > > > > > alexander.a.pasche...@gmail.com> wrote:
>> > > > > > >
>> > > > > > > > Hello Igniters,
>> > > > > > > >
>> > > > > > > > I would like to start discussion about implementation of SQL
>> > DDL
>> > > > > > > commands.
>> > > > > > > >
>> > > > > > > > At the first stage, the most important ones seem to be CREATE
>> > > TABLE
>> > > > > > > > (that will obviously correspond to creation of a cache) and
>> > > CREATE
>> > > > > > > > INDEX.
>> > > > > > > >
>> > > > > > > > Regarding first one: SQL command for CREATE TABLE does not
>> > > contain
>> > > > > any
>> > > > > > > > hints about cache settings (atomicity, replication, etc.), so
>> > > these
>> > > > > > > > will probably be defined by some configuration properties
>> (like
>> > > > > > > > ignite.ddl.default_cache_atomiticity, etc).
>> > > > > > > >
>> > > > > > > > Also it does not allow to distinguish between key and value
>> > > > columns -
>> > > > > > > > currently it is handled by keyFields property of QueryEntity,
>> > but
>> > > > it
>> > > > > > > > is unclear how to declare key fields via CREATE TABLE.
>> > > > > > > >
>> > > > > > > > So at a first glance it seems like we should either implement
>> > > some
>> > > > > > > > sort of custom parsing (I believe Sergi will be against it)
>> or
>> > > > > > > > introduce some kind of name prefix that would tell SQL engine
>> > > that
>> > > > > > > > certain column is a key field column.
>> > > > > > > >
>> > > > > > > > Of course, this problem disappears is key is of SQL type.
>> > > > > > > >
>> > > > > > > > Regarding CREATE INDEX: probably at first we will have to
>> > > implement
>> > > > > > > > this in "stop-the-world" manner, i.e. all cache will be
>> blocked
>> > > > > during
>> > > > > > > > the index's initial buildup.
>> > > > > > > >
>> > > > > > > > Any thoughts?
>> > > > > > > >
>> > > > > > > > Currently I'm working on parsing of those commands as that
>> will
>> > > be
>> > > > > > > > needed anyway and does not affect further implementation.
>> > > > > > > >
>> > > > > > > > - Alex
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > Alexey Kuznetsov
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Sergey Kozlov
>> > > > > GridGain Systems
>> > > > > www.gridgain.com
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Sergey Kozlov
>> > > GridGain Systems
>> > > www.gridgain.com
>> > >
>> >
>>

DDL implementation details

[Alexander Paschenko]

Hello Igniters,

I would like to start discussion about implementation of SQL DDL commands.

At the first stage, the most important ones seem to be CREATE TABLE
(that will obviously correspond to creation of a cache) and CREATE
INDEX.

Regarding first one: SQL command for CREATE TABLE does not contain any
hints about cache settings (atomicity, replication, etc.), so these
will probably be defined by some configuration properties (like
ignite.ddl.default_cache_atomiticity, etc).

Also it does not allow to distinguish between key and value columns -
currently it is handled by keyFields property of QueryEntity, but it
is unclear how to declare key fields via CREATE TABLE.

So at a first glance it seems like we should either implement some
sort of custom parsing (I believe Sergi will be against it) or
introduce some kind of name prefix that would tell SQL engine that
certain column is a key field column.

Of course, this problem disappears is key is of SQL type.

Regarding CREATE INDEX: probably at first we will have to implement
this in "stop-the-world" manner, i.e. all cache will be blocked during
the index's initial buildup.

Any thoughts?

Currently I'm working on parsing of those commands as that will be
needed anyway and does not affect further implementation.

- Alex

[jira] [Created] (IGNITE-4490) Optimize DML for fast INSERT and MERGE

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4490:
---

 Summary: Optimize DML for fast INSERT and MERGE
 Key: IGNITE-4490
 URL: https://issues.apache.org/jira/browse/IGNITE-4490
 Project: Ignite
  Issue Type: Improvement
  Components: SQL
Affects Versions: 1.8
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 1.8


It's possible to avoid any SQL querying and map some INSERT and MERGE 
statements to cache operations in a way similar to that of UPDATE and DELETE - 
i.e. don't make queries when there are no expressions to evaluate in the query 
and enhance update plans to perform direct cache operations when INSERT and 
MERGE affect columns _key and _value only.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4489) Maintain correct MERGE semantic in DML

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4489:
---

 Summary: Maintain correct MERGE semantic in DML
 Key: IGNITE-4489
 URL: https://issues.apache.org/jira/browse/IGNITE-4489
 Project: Ignite
  Issue Type: Bug
  Components: SQL
Affects Versions: 1.8
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 1.8


Currently it's impossible to MERGE object in UPDATE style - i.e. when key is 
present in cache, unaffected field values should be retained, and instead of 
building new object we should base it on previous one for given key.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: DML Documentation Readiness

[Alexander Paschenko]

Denis,

I have also added a doc titled 'Fast Updates' - it's a separate
document and it's hidden for now, please find the link to it in dash
at the bottom of 'SQL Grid' section on the left. It describes an
optimization technique that allows to run particular queries with
performance of bare cache operations reducing SQL related overhead to
nearly none at all. Hence I thought it was important to describe this
feature. Please review it and include into main doc as your heart
desires.

- Alex

2016-12-23 14:42 GMT+03:00 Alexander Paschenko
<alexander.a.pasche...@gmail.com>:
> Denis, Prachi,
>
> All links from ToC to subsections are broken on this page:
> http://apacheignite.gridgain.org/docs/dml
>
> - Alex
>
> 2016-12-23 3:54 GMT+03:00 Prachi Garg <pg...@gridgain.com>:
>> Sure, will look into it.
>>
>> -P
>>
>> On Thu, Dec 22, 2016 at 4:00 PM, Denis Magda <dma...@apache.org> wrote:
>>
>>> I’ve finished the review and released all the parts of DML documentation
>>> [1].
>>>
>>> Prachi, could you review whole SQL Grid documentation [2] including [1]?
>>> Also it will be great if you check this newly added section [3].
>>>
>>> [1] http://apacheignite.gridgain.org/docs/dml
>>> [2] http://apacheignite.gridgain.org/docs/sql-grid
>>> [3] http://apacheignite.gridgain.org/docs/binary-
>>> marshaller#handling-hash-code-generation-and-equals-execution
>>>
>>> —
>>> Denis
>>>
>>> On Dec 9, 2016, at 12:34 AM, Alexander Paschenko <
>>> alexander.a.pasche...@gmail.com> wrote:
>>>
>>> Denis,
>>>
>>> About your question: 'putIfAbsent' is used only when we have single
>>> key-value pair to process as long as we don't have method like 'put
>>> all if absent'. When there's a bunch of pairs, an entry processor is
>>> created for each pair to make its processing atomic (effectively CAS
>>> style processing), and all those processors are then executed via
>>> 'invokeAll'.
>>>
>>> - Alex
>>>
>>> 2016-12-09 8:17 GMT+03:00 Denis Magda <dma...@apache.org>:
>>>
>>> Alexander, thanks for all the material you’ve prepared so far.
>>>
>>> Actually, I’ve reviewed, restructured and released only a part of it [1].
>>>
>>> The rest, that is missing, requires more time from my side and I’ll be
>>> releasing additional blocks throughout the next week. The full
>>> documentation
>>> with the unreviewed parts is invisible and located here [2].
>>>
>>> Please don’t modify neither of the docs ([1] and [2]) until I’ve finished
>>> with the overall review. Send all your suggestions over the email.
>>>
>>> At the moment I have the following question. What cache method(s) do we
>>> convert INSERT statements to [3]? Is it ‘putIfAbsent’?
>>>
>>> [1] https://apacheignite.readme.io/docs/dml
>>> [2] https://apacheignite.readme.io/docs/distributed-dml-to-be-reviewe
>>> [3] https://apacheignite.readme.io/docs/dml#section-insert
>>>
>>> —
>>> Denis
>>>
>>> On Dec 8, 2016, at 1:35 AM, Alexander Paschenko
>>> <alexander.a.pasche...@gmail.com> wrote:
>>>
>>> Denis,
>>>
>>> it's all ready, JDBC driver page is also updated (I've created a
>>> hidden clone page, please find it on the left) as well as one on
>>> binary marshaller. Also I've renamed 'distributed DML' page to just
>>> 'DML' as long as there's much less to distribution of DML operations
>>> themselves than to queries and there's clearly no need to make another
>>> doc page to distinguish. Removed batches mention from limitations.
>>>
>>> - Alex
>>>
>>> 2016-12-08 5:26 GMT+03:00 Denis Magda <dma...@apache.org>:
>>>
>>> Alexander,
>>>
>>> How close are you to the finalization of the DML doc?
>>> https://apacheignite.readme.io/docs/distributed-dml
>>>
>>> Since we’re approaching 1.8 release I would like to do a final review of it
>>> polishing whatever is needed tomorrow.
>>>
>>> As I see that only the limitations section is left. The first note is that
>>> I
>>> wouldn’t mention “JDBC Batching mode” for now because this is something
>>> that
>>> doesn’t affect the usability and SQL API's scope of support.
>>>
>>> —
>>> Denis
>>>
>>>
>>>

[jira] [Created] (IGNITE-4486) Add serialVersionUID to AttributeNodeFilter

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4486:
---

 Summary: Add serialVersionUID to AttributeNodeFilter
 Key: IGNITE-4486
 URL: https://issues.apache.org/jira/browse/IGNITE-4486
 Project: Ignite
  Issue Type: Bug
  Components: general
Affects Versions: 1.8
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 1.8


Subj. - in particular, without it, TC build fails all at once



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Communication from JDBC/ODBC drivers

[Alexander Paschenko]

OK, will remove redundant query tasks creation for the case of DML
statements - they at least incur some GC overhead even if not context
switching.

- Alex

2016-12-20 12:49 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>:
> On Tue, Dec 20, 2016 at 1:23 AM, Alexander Paschenko <
> alexander.a.pasche...@gmail.com> wrote:
>
>> I see what you are suggesting and will remove this feature from the driver
>> if there are no objections. (Anyone?)
>> Still, for clarity and your better understanding I'd like to again point
>> out that there's no thread context switching or any Ignite imposed overhead
>> by default as even ignite compute ia not used — closure's call() method is
>> called directly in the same thread. Wrapped task isn't going anywhere in
>> local case. And it's the pattern that is used in Ignite code quite
>> extensively — if the task is local anyway, let's just run it "on the spot",
>> directly, without any unnecessary overhead.
>>
>>
> I disagree, there is an overhead, especially for relatively small batches.
> Why do you keep insisting on having this flawed design? Why not fix it
> properly by calling putAll(...) directly?
>
>
>> — Alex
>> 19 дек. 2016 г. 11:44 PM пользователь "Dmitriy Setrakyan" <
>> dsetrak...@apache.org> написал:
>>
>> > Alex,
>> >
>> > What Val and I are suggesting is that we get rid of any JdbcCallable
>> calls
>> > and invoke putAll(...) directly. Wrapping it into JdbcCallable will slow
>> us
>> > down and introduce unnecessary thread context switch.
>> >
>> > Do you agree?
>> >
>> > D.
>> >
>> > On Mon, Dec 19, 2016 at 11:01 AM, Valentin Kulichenko <
>> > valentin.kuliche...@gmail.com> wrote:
>> >
>> > > By "locally executed callable" I meant something like this:
>> > >
>> > > new JdbcCallable(...).call();
>> > >
>> > > So it's not a network call and it's not even going through
>> IgniteCompute.
>> > > It's just the logic is incapsulated in the callable to support remote
>> > > execution of the same logic.
>> > >
>> > > But anyway, +1 to removing nodeId property. It doesn't make much sense
>> > with
>> > > the new implementation of JDBC.
>> > >
>> > > -Val
>> > >
>> > > On Mon, Dec 19, 2016 at 10:15 AM, Alexander Paschenko <
>> > > alexander.a.pasche...@gmail.com> wrote:
>> > >
>> > > > When node is local, no network interaction occurs on query send.
>> Would
>> > be
>> > > > shame otherwise :)
>> > > >
>> > > > — Alex
>> > > > 19 дек. 2016 г. 8:36 PM пользователь "Denis Magda" <
>> dma...@apache.org>
>> > > > написал:
>> > > >
>> > > > > Dmitriy,
>> > > > >
>> > > > > According to Val and Alex P. explanations this happens when a
>> > specific
>> > > > > node id is set. I got confused by the code flow initially.
>> > > > >
>> > > > > —
>> > > > > Denis
>> > > > >
>> > > > > > On Dec 19, 2016, at 9:13 AM, Dmitriy Setrakyan <
>> > > dsetrak...@apache.org>
>> > > > > wrote:
>> > > > > >
>> > > > > > Thanks, Alex.
>> > > > > >
>> > > > > > Can you clarify if the putAll(...) call is made directly, or from
>> > > > locally
>> > > > > > executed callable? According to Denis, "even single (non batched)
>> > > > updates
>> > > > > > or queries are sent as callables", which should be fixed in my
>> > view.
>> > > > > >
>> > > > > > D.
>> > > > > >
>> > > > > > On Mon, Dec 19, 2016 at 3:32 AM, Alexander Paschenko <
>> > > > > > alexander.a.pasche...@gmail.com> wrote:
>> > > > > >
>> > > > > >> Dima, Val,
>> > > > > >>
>> > > > > >> Introduction of updates has not changed public API at all (only
>> > > > > >> configuration, in some cases), so they work in this respect
>> > exactly
>> > > > > >> like SELECTs - by default they run on client node started by the
>> > > > > >

Re: Batch DML queries design discussion

[Alexander Paschenko]

Auto flush freq is already there, I just forgot to mention it in the
comments. Will add the rest today.

— Alex
19 дек. 2016 г. 10:29 PM пользователь "Denis Magda" <dma...@apache.org>
написал:

> Alexander,
>
> A couple of comments in regards to the streaming mode.
>
> I would rename rename the existed property to “ignite.jdbc.streaming” and
> add additional ones that will help to manage and tune the streaming
> behavior:
> ignite.jdbc.streaming.perNodeBufferSize
> ignite.jdbc.streaming.perNodeParallelOperations
> ignite.jdbc.streaming.autoFlushFrequency
>
>
> Any other thoughts?
>
> —
> Denis
>
> > On Dec 19, 2016, at 8:02 AM, Alexander Paschenko <
> alexander.a.pasche...@gmail.com> wrote:
> >
> > OK folks, both data streamer support and batching support have been
> implemented.
> >
> > Resulting design fully conforms to what Dima suggested initially -
> > these two concepts are separated.
> >
> > Streamed statements are turned on by connection flag, stream auto
> > flush timeout can be tuned in the same way; these statements support
> > INSERT and MERGE w/o subquery as well as fast key bounded DELETE and
> > UPDATE; each prepared statement in streamed mode has its own streamer
> > object and their lifecycles are the same - on close, the statement
> > closes its streamer. Streaming mode is available only in "local" mode
> > of connection between JDBC driver and Ignite client (default mode when
> > JDBC driver creates Ignite client node by itself) - there would be no
> > sense in streaming if query args would have to travel over network.
> >
> > Batched statements sre used via conventional JDBC API (setArgs...
> > addBatch... executeBatch...), they also support INSERT and MERGE w/o
> > subquery as well as fast key (and, optionally, value) bounded DELETE
> > and UPDATE. These work in the similar manner to non batched statements
> > and likewise rely on traditional putAll/invokeAll routines.
> > Essentially, batching is just the way to pass a bigger map to
> > cache.putAll without writing single very long query. This works in
> > local as well as "remote" Ignite JDBC connectivity mode.
> >
> > More info (details are in the comments):
> >
> > Batching - https://issues.apache.org/jira/browse/IGNITE-4269
> > Streaming - https://issues.apache.org/jira/browse/IGNITE-4169
> >
> > Regards,
> > Alex
> >
> > 2016-12-10 23:39 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>:
> >> Alex,
> >>
> >> It seams to me that replace semantic can be implemented with
> >> StreamReceiver, no?
> >>
> >> D.
> >>
> >> On Sat, Dec 10, 2016 at 2:54 AM, Alexander Paschenko <
> >> alexander.a.pasche...@gmail.com> wrote:
> >>
> >>> Sorry, "no relation w/JDBC" in my previous message should read "no
> relation
> >>> w/JDBC batching".
> >>>
> >>> — Alex
> >>> 10 дек. 2016 г. 1:52 PM пользователь "Alexander Paschenko" <
> >>> alexander.a.pasche...@gmail.com> написал:
> >>>
> >>>> Dima,
> >>>>
> >>>> I would like to point out that data streamer support had already been
> >>>> implemented in the course of work on DML in 1.8 exactly as you are
> >>>> suggesting now (turned on via connection flag; allowed only MERGE —
> data
> >>>> streamer can't do putIfAbsent stuff, right?; absolutely no relation
> >>>> w/JDBC), *but* that patch had been reverted — by advice from Vlad
> which I
> >>>> believe had been agreed with you, so it didn't make it to 1.8 after
> all.
> >>>> Also, while it's possible to maintain INSERT vs MERGE semantic using
> >>>> streamer's allowOverwrite flag, I can't see how we could mimic UPDATE
> >>> here
> >>>> as long as the streamer does not put data to cache only in case when
> key
> >>> is
> >>>> present AND allowOverwrite is false, while UPDATE should not put
> anything
> >>>> when the key is *missing* — i.e., there's no way to emulate cache's
> >>>> *replace* operation semantic with streamer (update value only if key
> is
> >>>> present, otherwise do nothing).
> >>>>
> >>>> — Alex
> >>>> 9 дек. 2016 г. 10:00 PM пользователь "Dmitriy Setrakyan" <
> >>>> dsetrak...@apache.org> написал:
> >>>>
> >>>>> On Fri, Dec 9, 2016 at 12:4

Re: Communication from JDBC/ODBC drivers

[Alexander Paschenko]

I see what you are suggesting and will remove this feature from the driver
if there are no objections. (Anyone?)
Still, for clarity and your better understanding I'd like to again point
out that there's no thread context switching or any Ignite imposed overhead
by default as even ignite compute ia not used — closure's call() method is
called directly in the same thread. Wrapped task isn't going anywhere in
local case. And it's the pattern that is used in Ignite code quite
extensively — if the task is local anyway, let's just run it "on the spot",
directly, without any unnecessary overhead.

— Alex
19 дек. 2016 г. 11:44 PM пользователь "Dmitriy Setrakyan" <
dsetrak...@apache.org> написал:

> Alex,
>
> What Val and I are suggesting is that we get rid of any JdbcCallable calls
> and invoke putAll(...) directly. Wrapping it into JdbcCallable will slow us
> down and introduce unnecessary thread context switch.
>
> Do you agree?
>
> D.
>
> On Mon, Dec 19, 2016 at 11:01 AM, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > By "locally executed callable" I meant something like this:
> >
> > new JdbcCallable(...).call();
> >
> > So it's not a network call and it's not even going through IgniteCompute.
> > It's just the logic is incapsulated in the callable to support remote
> > execution of the same logic.
> >
> > But anyway, +1 to removing nodeId property. It doesn't make much sense
> with
> > the new implementation of JDBC.
> >
> > -Val
> >
> > On Mon, Dec 19, 2016 at 10:15 AM, Alexander Paschenko <
> > alexander.a.pasche...@gmail.com> wrote:
> >
> > > When node is local, no network interaction occurs on query send. Would
> be
> > > shame otherwise :)
> > >
> > > — Alex
> > > 19 дек. 2016 г. 8:36 PM пользователь "Denis Magda" <dma...@apache.org>
> > > написал:
> > >
> > > > Dmitriy,
> > > >
> > > > According to Val and Alex P. explanations this happens when a
> specific
> > > > node id is set. I got confused by the code flow initially.
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On Dec 19, 2016, at 9:13 AM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > > > wrote:
> > > > >
> > > > > Thanks, Alex.
> > > > >
> > > > > Can you clarify if the putAll(...) call is made directly, or from
> > > locally
> > > > > executed callable? According to Denis, "even single (non batched)
> > > updates
> > > > > or queries are sent as callables", which should be fixed in my
> view.
> > > > >
> > > > > D.
> > > > >
> > > > > On Mon, Dec 19, 2016 at 3:32 AM, Alexander Paschenko <
> > > > > alexander.a.pasche...@gmail.com> wrote:
> > > > >
> > > > >> Dima, Val,
> > > > >>
> > > > >> Introduction of updates has not changed public API at all (only
> > > > >> configuration, in some cases), so they work in this respect
> exactly
> > > > >> like SELECTs - by default they run on client node started by the
> > > > >> driver itself, but can be sent via the same callables mechanism to
> > any
> > > > >> remote node by its id.
> > > > >>
> > > > >> So Dima, you're right, currently it's possible to send query to
> any
> > > > >> given node. And, at the same time, currently by default everything
> > > > >> works exactly like you want it to - that is, any MERGE, batched or
> > > > >> not, boils down to putAll, and by default this call happens
> locally.
> > > > >>
> > > > >> - Alex
> > > > >>
> > > > >> 2016-12-17 18:47 GMT+03:00 Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >:
> > > > >>> On Fri, Dec 16, 2016 at 9:53 PM, Valentin Kulichenko <
> > > > >>> valentin.kuliche...@gmail.com> wrote:
> > > > >>>
> > > > >>>> I'm not sure about updates, but can tell about how selects are
> > > > >> implemented
> > > > >>>> there. Basically, there is an option to execute the query on a
> > > > >> particular
> > > > >>>> node specified by ignite.jdbc.nodeId property. Not sure why we
> > need
> > > > this
> 

Re: Communication from JDBC/ODBC drivers

[Alexander Paschenko]

When node is local, no network interaction occurs on query send. Would be
shame otherwise :)

— Alex
19 дек. 2016 г. 8:36 PM пользователь "Denis Magda" <dma...@apache.org>
написал:

> Dmitriy,
>
> According to Val and Alex P. explanations this happens when a specific
> node id is set. I got confused by the code flow initially.
>
> —
> Denis
>
> > On Dec 19, 2016, at 9:13 AM, Dmitriy Setrakyan <dsetrak...@apache.org>
> wrote:
> >
> > Thanks, Alex.
> >
> > Can you clarify if the putAll(...) call is made directly, or from locally
> > executed callable? According to Denis, "even single (non batched) updates
> > or queries are sent as callables", which should be fixed in my view.
> >
> > D.
> >
> > On Mon, Dec 19, 2016 at 3:32 AM, Alexander Paschenko <
> > alexander.a.pasche...@gmail.com> wrote:
> >
> >> Dima, Val,
> >>
> >> Introduction of updates has not changed public API at all (only
> >> configuration, in some cases), so they work in this respect exactly
> >> like SELECTs - by default they run on client node started by the
> >> driver itself, but can be sent via the same callables mechanism to any
> >> remote node by its id.
> >>
> >> So Dima, you're right, currently it's possible to send query to any
> >> given node. And, at the same time, currently by default everything
> >> works exactly like you want it to - that is, any MERGE, batched or
> >> not, boils down to putAll, and by default this call happens locally.
> >>
> >> - Alex
> >>
> >> 2016-12-17 18:47 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>:
> >>> On Fri, Dec 16, 2016 at 9:53 PM, Valentin Kulichenko <
> >>> valentin.kuliche...@gmail.com> wrote:
> >>>
> >>>> I'm not sure about updates, but can tell about how selects are
> >> implemented
> >>>> there. Basically, there is an option to execute the query on a
> >> particular
> >>>> node specified by ignite.jdbc.nodeId property. Not sure why we need
> this
> >>>> though, probably it's just leftover from the legacy version of the
> >> driver
> >>>> based on thin client.
> >>>>
> >>>> If the property is set, the callable is sent to a remote node. But if
> >> it is
> >>>> not, the same callable is created, but it is invoked directly on the
> >>>> embedded client which is the behavior that you expect. And it's the
> >> default
> >>>> one.
> >>>>
> >>>>
> >>> Ouch. If this is the reason, I would drop the nodeId property. I don't
> >>> think it makes sense and it significantly slows down the
> implementation.
> >>>
> >>>
> >>>> -Val
> >>>>
> >>>> On Fri, Dec 16, 2016 at 7:51 PM, Denis Magda <dma...@apache.org>
> wrote:
> >>>>
> >>>>> Frankly speaking, even single (non batched) updates or queries are
> >> sent
> >>>> as
> >>>>> callables. This is what I see in the code.
> >>>>> No idea what was the reason behind this design.
> >>>>>
> >>>>> Andrey G., Alex P. could you shed a light on this?
> >>>>>
> >>>>> —
> >>>>> Denis
> >>>>>
> >>>>>> On Dec 16, 2016, at 3:08 PM, Dmitriy Setrakyan <
> >> dsetrak...@apache.org>
> >>>>> wrote:
> >>>>>>
> >>>>>> To my understanding, we are implementing JDBC batches by sending a
> >>>>> callable
> >>>>>> to another node. If we already have a client node on the JDBC driver
> >>>>> side,
> >>>>>> why not just issue a putAll(...) call from the client?
> >>>>>>
> >>>>>> D.
> >>>>>>
> >>>>>> On Fri, Dec 16, 2016 at 3:02 PM, Denis Magda <dma...@apache.org>
> >>>> wrote:
> >>>>>>
> >>>>>>> Dmitriy,
> >>>>>>>
> >>>>>>> JDBC drivers spawns an Ignite client node and uses it for cluster
> >>>>>>> connectivity and queries execution. Queries issued over the JDBC
> >> are
> >>>>> turned
> >>>>>>> into SqlFieldsQueries and sent to the cluster in this form.
> >>>>>>>
> >>>>>>> ODBC driver works in a bit different way. It connects to the
> >> cluster
> >>>> via
> >>>>>>> ODBC processor that needs to be running on one of the nodes:
> >>>>>>> https://apacheignite.readme.io/docs/odbc-driver#cluster-
> >> configuration
> >>>> <
> >>>>>>> https://apacheignite.readme.io/docs/odbc-driver#cluster-
> >> configuration
> >>>>>
> >>>>>>>
> >>>>>>> —
> >>>>>>> Denis
> >>>>>>>
> >>>>>>>> On Dec 16, 2016, at 2:41 PM, Dmitriy Setrakyan <
> >>>> dsetrak...@apache.org>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Igniters,
> >>>>>>>>
> >>>>>>>> Can someone explain to me how Ignite executes SQL from JDBC and
> >> ODBC
> >>>>>>>> drivers? Do we start an Ignite client node on the driver side? Or
> >> do
> >>>> we
> >>>>>>> use
> >>>>>>>> some other protocol to send commands to one of the Ignite nodes?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> D.
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>
>
>

Re: Batch DML queries design discussion

[Alexander Paschenko]

OK folks, both data streamer support and batching support have been implemented.

Resulting design fully conforms to what Dima suggested initially -
these two concepts are separated.

Streamed statements are turned on by connection flag, stream auto
flush timeout can be tuned in the same way; these statements support
INSERT and MERGE w/o subquery as well as fast key bounded DELETE and
UPDATE; each prepared statement in streamed mode has its own streamer
object and their lifecycles are the same - on close, the statement
closes its streamer. Streaming mode is available only in "local" mode
of connection between JDBC driver and Ignite client (default mode when
JDBC driver creates Ignite client node by itself) - there would be no
sense in streaming if query args would have to travel over network.

Batched statements sre used via conventional JDBC API (setArgs...
addBatch... executeBatch...), they also support INSERT and MERGE w/o
subquery as well as fast key (and, optionally, value) bounded DELETE
and UPDATE. These work in the similar manner to non batched statements
and likewise rely on traditional putAll/invokeAll routines.
Essentially, batching is just the way to pass a bigger map to
cache.putAll without writing single very long query. This works in
local as well as "remote" Ignite JDBC connectivity mode.

More info (details are in the comments):

Batching - https://issues.apache.org/jira/browse/IGNITE-4269
Streaming - https://issues.apache.org/jira/browse/IGNITE-4169

Regards,
Alex

2016-12-10 23:39 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>:
> Alex,
>
> It seams to me that replace semantic can be implemented with
> StreamReceiver, no?
>
> D.
>
> On Sat, Dec 10, 2016 at 2:54 AM, Alexander Paschenko <
> alexander.a.pasche...@gmail.com> wrote:
>
>> Sorry, "no relation w/JDBC" in my previous message should read "no relation
>> w/JDBC batching".
>>
>> — Alex
>> 10 дек. 2016 г. 1:52 PM пользователь "Alexander Paschenko" <
>> alexander.a.pasche...@gmail.com> написал:
>>
>> > Dima,
>> >
>> > I would like to point out that data streamer support had already been
>> > implemented in the course of work on DML in 1.8 exactly as you are
>> > suggesting now (turned on via connection flag; allowed only MERGE — data
>> > streamer can't do putIfAbsent stuff, right?; absolutely no relation
>> > w/JDBC), *but* that patch had been reverted — by advice from Vlad which I
>> > believe had been agreed with you, so it didn't make it to 1.8 after all.
>> > Also, while it's possible to maintain INSERT vs MERGE semantic using
>> > streamer's allowOverwrite flag, I can't see how we could mimic UPDATE
>> here
>> > as long as the streamer does not put data to cache only in case when key
>> is
>> > present AND allowOverwrite is false, while UPDATE should not put anything
>> > when the key is *missing* — i.e., there's no way to emulate cache's
>> > *replace* operation semantic with streamer (update value only if key is
>> > present, otherwise do nothing).
>> >
>> > — Alex
>> > 9 дек. 2016 г. 10:00 PM пользователь "Dmitriy Setrakyan" <
>> > dsetrak...@apache.org> написал:
>> >
>> >> On Fri, Dec 9, 2016 at 12:45 AM, Vladimir Ozerov <voze...@gridgain.com>
>> >> wrote:
>> >>
>> >> > I already expressed my concern - this is counterintuitive approach.
>> >> Because
>> >> > without happens-before pure streaming model can be applied only on
>> >> > independent chunks of data. It mean that mentioned ETL use case is not
>> >> > feasible - ETL always depend on implicit or explicit links between
>> >> tables,
>> >> > and hence streaming is not applicable here. My question stands still -
>> >> what
>> >> > products except of possibly Ignite do this kind of JDBC streaming?
>> >> >
>> >>
>> >> Vova, we have 2 mechanisms in the product: IgniteCache.putAll() or
>> >> DataStreamer.addData().
>> >>
>> >> JDBC batching and putAll() are absolutely identical. If you see it as
>> >> counter-intuitive, I would ask for a concrete example.
>> >>
>> >> As far as links between data, Ignite does not have foreign-key
>> >> constraints,
>> >> so DataStreamer can insert data in any order (but again, not as part  of
>> >> JDBC batch).
>> >>
>> >>
>> >> >
>> >> > Another problem is that connection-wide property doesn't fit well in
>> >> JDBC
>&

Re: Communication from JDBC/ODBC drivers

[Alexander Paschenko]

Dima, Val,

Introduction of updates has not changed public API at all (only
configuration, in some cases), so they work in this respect exactly
like SELECTs - by default they run on client node started by the
driver itself, but can be sent via the same callables mechanism to any
remote node by its id.

So Dima, you're right, currently it's possible to send query to any
given node. And, at the same time, currently by default everything
works exactly like you want it to - that is, any MERGE, batched or
not, boils down to putAll, and by default this call happens locally.

- Alex

2016-12-17 18:47 GMT+03:00 Dmitriy Setrakyan :
> On Fri, Dec 16, 2016 at 9:53 PM, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
>> I'm not sure about updates, but can tell about how selects are implemented
>> there. Basically, there is an option to execute the query on a particular
>> node specified by ignite.jdbc.nodeId property. Not sure why we need this
>> though, probably it's just leftover from the legacy version of the driver
>> based on thin client.
>>
>> If the property is set, the callable is sent to a remote node. But if it is
>> not, the same callable is created, but it is invoked directly on the
>> embedded client which is the behavior that you expect. And it's the default
>> one.
>>
>>
> Ouch. If this is the reason, I would drop the nodeId property. I don't
> think it makes sense and it significantly slows down the implementation.
>
>
>> -Val
>>
>> On Fri, Dec 16, 2016 at 7:51 PM, Denis Magda  wrote:
>>
>> > Frankly speaking, even single (non batched) updates or queries are sent
>> as
>> > callables. This is what I see in the code.
>> > No idea what was the reason behind this design.
>> >
>> > Andrey G., Alex P. could you shed a light on this?
>> >
>> > —
>> > Denis
>> >
>> > > On Dec 16, 2016, at 3:08 PM, Dmitriy Setrakyan 
>> > wrote:
>> > >
>> > > To my understanding, we are implementing JDBC batches by sending a
>> > callable
>> > > to another node. If we already have a client node on the JDBC driver
>> > side,
>> > > why not just issue a putAll(...) call from the client?
>> > >
>> > > D.
>> > >
>> > > On Fri, Dec 16, 2016 at 3:02 PM, Denis Magda 
>> wrote:
>> > >
>> > >> Dmitriy,
>> > >>
>> > >> JDBC drivers spawns an Ignite client node and uses it for cluster
>> > >> connectivity and queries execution. Queries issued over the JDBC are
>> > turned
>> > >> into SqlFieldsQueries and sent to the cluster in this form.
>> > >>
>> > >> ODBC driver works in a bit different way. It connects to the cluster
>> via
>> > >> ODBC processor that needs to be running on one of the nodes:
>> > >> https://apacheignite.readme.io/docs/odbc-driver#cluster-configuration
>> <
>> > >> https://apacheignite.readme.io/docs/odbc-driver#cluster-configuration
>> >
>> > >>
>> > >> —
>> > >> Denis
>> > >>
>> > >>> On Dec 16, 2016, at 2:41 PM, Dmitriy Setrakyan <
>> dsetrak...@apache.org>
>> > >> wrote:
>> > >>>
>> > >>> Igniters,
>> > >>>
>> > >>> Can someone explain to me how Ignite executes SQL from JDBC and ODBC
>> > >>> drivers? Do we start an Ignite client node on the driver side? Or do
>> we
>> > >> use
>> > >>> some other protocol to send commands to one of the Ignite nodes?
>> > >>>
>> > >>> Thanks,
>> > >>> D.
>> > >>
>> > >>
>> >
>> >
>>

Re: Batch DML queries design discussion

[Alexander Paschenko]

Sorry, "no relation w/JDBC" in my previous message should read "no relation
w/JDBC batching".

— Alex
10 дек. 2016 г. 1:52 PM пользователь "Alexander Paschenko" <
alexander.a.pasche...@gmail.com> написал:

> Dima,
>
> I would like to point out that data streamer support had already been
> implemented in the course of work on DML in 1.8 exactly as you are
> suggesting now (turned on via connection flag; allowed only MERGE — data
> streamer can't do putIfAbsent stuff, right?; absolutely no relation
> w/JDBC), *but* that patch had been reverted — by advice from Vlad which I
> believe had been agreed with you, so it didn't make it to 1.8 after all.
> Also, while it's possible to maintain INSERT vs MERGE semantic using
> streamer's allowOverwrite flag, I can't see how we could mimic UPDATE here
> as long as the streamer does not put data to cache only in case when key is
> present AND allowOverwrite is false, while UPDATE should not put anything
> when the key is *missing* — i.e., there's no way to emulate cache's
> *replace* operation semantic with streamer (update value only if key is
> present, otherwise do nothing).
>
> — Alex
> 9 дек. 2016 г. 10:00 PM пользователь "Dmitriy Setrakyan" <
> dsetrak...@apache.org> написал:
>
>> On Fri, Dec 9, 2016 at 12:45 AM, Vladimir Ozerov <voze...@gridgain.com>
>> wrote:
>>
>> > I already expressed my concern - this is counterintuitive approach.
>> Because
>> > without happens-before pure streaming model can be applied only on
>> > independent chunks of data. It mean that mentioned ETL use case is not
>> > feasible - ETL always depend on implicit or explicit links between
>> tables,
>> > and hence streaming is not applicable here. My question stands still -
>> what
>> > products except of possibly Ignite do this kind of JDBC streaming?
>> >
>>
>> Vova, we have 2 mechanisms in the product: IgniteCache.putAll() or
>> DataStreamer.addData().
>>
>> JDBC batching and putAll() are absolutely identical. If you see it as
>> counter-intuitive, I would ask for a concrete example.
>>
>> As far as links between data, Ignite does not have foreign-key
>> constraints,
>> so DataStreamer can insert data in any order (but again, not as part  of
>> JDBC batch).
>>
>>
>> >
>> > Another problem is that connection-wide property doesn't fit well in
>> JDBC
>> > pooling model. Users will have use different connections for streaming
>> and
>> > non-streaming approaches.
>> >
>>
>> Using DataStreamer is not possible within JDBC batching paradigm, period.
>> I
>> wish we could drop the high-level-feels-good discussions altogether,
>> because it seems like we are spinning wheels here.
>>
>> There is no way to use the streamer in JDBC context, unless we add a
>> connection flag. Again, if you disagree, I would prefer to see a concrete
>> example explaining why.
>>
>>
>> > Please see how Oracle did that, this is precisely what I am talking
>> about:
>> > https://docs.oracle.com/cd/B28359_01/java.111/b31224/oraperf
>> .htm#i1056232
>> > Two batching modes - one with explicit flush, another one with implicit
>> > flush, when Oracle decides on it's own when it is better to communicate
>> the
>> > server. Batching mode can be declared globally or on per-statement
>> level.
>> > Simple and flexible.
>> >
>> >
>> > On Fri, Dec 9, 2016 at 4:40 AM, Dmitriy Setrakyan <
>> dsetrak...@apache.org>
>> > wrote:
>> >
>> > > Gents,
>> > >
>> > > As Sergi suggested, batching and streaming are very different
>> > semantically.
>> > >
>> > > To use standard JDBC batching, all we need to do is convert it to a
>> > > cache.putAll() method, as semantically a putAll(...) call is identical
>> > to a
>> > > JDBC batch. Of course, if we see and UPDATE with a WHERE clause in
>> > between,
>> > > then we may have to break a batch into several chunks and execute the
>> > > update in between. The DataStreamer should not be used here.
>> > >
>> > > I believe that for streaming we need to add a special JDBC/ODBC
>> > connection
>> > > flag. Whenever this flag is set to true, then we only should allow
>> INSERT
>> > > or single-UPDATE operations and use DataStreamer API internally. All
>> > > operations other than INSERT or single-UPDATE should be prohibited.
>> > >
&g

Re: Batch DML queries design discussion

[Alexander Paschenko]

hould not depend on each other at all:
> > > Batching
> > > > can work with or without Streaming, as well as Streaming can work
> with
> > or
> > > > without Batching.
> > > >
> > > > Your proposal is a set of non-obvious rules for them to work. I see
> no
> > > > reasons for these complications.
> > > >
> > > > Sergi
> > > >
> > > >
> > > > 2016-12-08 15:49 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>:
> > > >
> > > > > Sergi,
> > > > >
> > > > > If user call single *execute() *operation, than most likely it is
> not
> > > > > batching. We should not rely on strange case where user perform
> > > batching
> > > > > without using standard and well-adopted batching JDBC API. The main
> > > > problem
> > > > > with streamer is that it is async and hence break happens-before
> > > > guarantees
> > > > > in a single thread: SELECT after INSERT might not return inserted
> > > value.
> > > > >
> > > > > Honestly, I do not really understand why we are trying to
> re-invent a
> > > > > bicycle here. There is standard API - let's just use it and make
> > > flexible
> > > > > enough to take advantage of IgniteDataStreamer if needed.
> > > > >
> > > > > Is there any use case which is not covered with this solution? Or
> let
> > > me
> > > > > ask from the opposite side - are there any well-known JDBC drivers
> > > which
> > > > > perform batching/streaming from non-batched update statements?
> > > > >
> > > > > Vladimir.
> > > > >
> > > > > On Thu, Dec 8, 2016 at 3:38 PM, Sergi Vladykin <
> > > sergi.vlady...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Vladimir,
> > > > > >
> > > > > > I see no reason to forbid Streamer usage from non-batched
> statement
> > > > > > execution.
> > > > > > It is common that users already have their ETL tools and you
> can't
> > be
> > > > > sure
> > > > > > if they use batching or not.
> > > > > >
> > > > > > Alex,
> > > > > >
> > > > > > I guess we have to decide on Streaming first and then we will
> > discuss
> > > > > > Batching separately, ok? Because this decision may become
> important
> > > for
> > > > > > batching implementation.
> > > > > >
> > > > > > Sergi
> > > > > >
> > > > > > 2016-12-08 15:31 GMT+03:00 Andrey Gura <ag...@apache.org>:
> > > > > >
> > > > > > > Alex,
> > > > > > >
> > > > > > > In most cases JdbcQueryTask should be executed locally on
> client
> > > node
> > > > > > > started by JDBC driver.
> > > > > > >
> > > > > > > JdbcQueryTask.QueryResult res =
> > > > > > > loc ? qryTask.call() :
> > > > > > > ignite.compute(ignite.cluster().forNodeId(nodeId)).call(
> > qryTask);
> > > > > > >
> > > > > > > Is it valid behavior after introducing DML functionality?
> > > > > > >
> > > > > > > In cases when user wants to execute query on specific node he
> > > should
> > > > > > > fully understand what he wants and what can go in wrong way.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Dec 8, 2016 at 3:20 PM, Alexander Paschenko
> > > > > > > <alexander.a.pasche...@gmail.com> wrote:
> > > > > > > > Sergi,
> > > > > > > >
> > > > > > > > JDBC batching might work quite differently from driver to
> > driver.
> > > > > Say,
> > > > > > > > MySQL happily rewrites queries as I had suggested in the
> > > beginning
> > > > of
> > > > > > > > this thread (it's not the only strategy, but one of the
> > possible
> > > > > > > > options) - and, BTW, would like to hear at least an opinion
> > about
> > > > it.
> > > > > > > 

Re: DML Documentation Readiness

[Alexander Paschenko]

Denis,

About your question: 'putIfAbsent' is used only when we have single
key-value pair to process as long as we don't have method like 'put
all if absent'. When there's a bunch of pairs, an entry processor is
created for each pair to make its processing atomic (effectively CAS
style processing), and all those processors are then executed via
'invokeAll'.

- Alex

2016-12-09 8:17 GMT+03:00 Denis Magda <dma...@apache.org>:
> Alexander, thanks for all the material you’ve prepared so far.
>
> Actually, I’ve reviewed, restructured and released only a part of it [1].
>
> The rest, that is missing, requires more time from my side and I’ll be
> releasing additional blocks throughout the next week. The full documentation
> with the unreviewed parts is invisible and located here [2].
>
> Please don’t modify neither of the docs ([1] and [2]) until I’ve finished
> with the overall review. Send all your suggestions over the email.
>
> At the moment I have the following question. What cache method(s) do we
> convert INSERT statements to [3]? Is it ‘putIfAbsent’?
>
> [1] https://apacheignite.readme.io/docs/dml
> [2] https://apacheignite.readme.io/docs/distributed-dml-to-be-reviewe
> [3] https://apacheignite.readme.io/docs/dml#section-insert
>
> —
> Denis
>
> On Dec 8, 2016, at 1:35 AM, Alexander Paschenko
> <alexander.a.pasche...@gmail.com> wrote:
>
> Denis,
>
> it's all ready, JDBC driver page is also updated (I've created a
> hidden clone page, please find it on the left) as well as one on
> binary marshaller. Also I've renamed 'distributed DML' page to just
> 'DML' as long as there's much less to distribution of DML operations
> themselves than to queries and there's clearly no need to make another
> doc page to distinguish. Removed batches mention from limitations.
>
> - Alex
>
> 2016-12-08 5:26 GMT+03:00 Denis Magda <dma...@apache.org>:
>
> Alexander,
>
> How close are you to the finalization of the DML doc?
> https://apacheignite.readme.io/docs/distributed-dml
>
> Since we’re approaching 1.8 release I would like to do a final review of it
> polishing whatever is needed tomorrow.
>
> As I see that only the limitations section is left. The first note is that I
> wouldn’t mention “JDBC Batching mode” for now because this is something that
> doesn’t affect the usability and SQL API's scope of support.
>
> —
> Denis
>
>
>

Re: Batch DML queries design discussion

[Alexander Paschenko]

Sergi,

JDBC batching might work quite differently from driver to driver. Say,
MySQL happily rewrites queries as I had suggested in the beginning of
this thread (it's not the only strategy, but one of the possible
options) - and, BTW, would like to hear at least an opinion about it.

On your first approach, section before streamer: you suggest that we
send single statement and multiple param sets as a single query task,
am I right? (Just to make sure that I got you properly.) If so, do you
also mean that API (namely JdbcQueryTask) between server and client
should also change? Or should new API means be added to facilitate
batching tasks?

- Alex

2016-12-08 15:05 GMT+03:00 Sergi Vladykin <sergi.vlady...@gmail.com>:
> Guys,
>
> I discussed this feature with Dmitriy and we came to conclusion that
> batching in JDBC and Data Streaming in Ignite have different semantics and
> performance characteristics. Thus they are independent features (they may
> work together, may separately, but this is another story).
>
> Let me explain.
>
> This is how JDBC batching works:
> - Add N sets of parameters to a prepared statement.
> - Manually execute prepared statement.
> - Repeat until all the data is loaded.
>
>
> This is how data streamer works:
> - Keep adding data.
> - Streamer will buffer and load buffered per-node batches when they are big
> enough.
> - Close streamer to make sure that everything is loaded.
>
> As you can see we have a difference in semantics of when we send data: if
> in our JDBC we will allow sending batches to nodes without calling
> `execute` (and probably we will need to make `execute` to no-op here), then
> we are violating semantics of JDBC, if we will disallow this behavior, then
> this batching will underperform.
>
> Thus I suggest keeping these features (JDBC Batching and JDBC Streaming) as
> separate features.
>
> As I already said they can work together: Batching will batch parameters
> and on `execute` they will go to the Streamer in one shot and Streamer will
> deal with the rest.
>
> Sergi
>
>
>
>
>
>
>
> 2016-12-08 14:16 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>:
>
>> Hi Alex,
>>
>> To my understanding there are two possible approaches to batching in JDBC
>> layer:
>>
>> 1) Rely on default batching API. Specifically
>> *PreparedStatement.addBatch()* [1]
>> and others. This is nice and clear API, users are used to it, and it's
>> adoption will minimize user code changes when migrating from other JDBC
>> sources. We simply copy updates locally and then execute them all at once
>> with only a single network hop to servers. *IgniteDataStreamer* can be used
>> underneath.
>>
>> 2) Or we can have separate connection flag which will move all
>> INSERT/UPDATE/DELETE statements through streamer.
>>
>> I prefer the first approach
>>
>> Also we need to keep in mind that data streamer has poor performance when
>> adding single key-value pairs due to high overhead on concurrency and other
>> bookkeeping. Instead, it is better to pre-batch key-value pairs before
>> giving them to streamer.
>>
>> Vladimir.
>>
>> [1]
>> https://docs.oracle.com/javase/8/docs/api/java/sql/PreparedStatement.html#
>> addBatch--
>>
>> On Thu, Dec 8, 2016 at 1:21 PM, Alexander Paschenko <
>> alexander.a.pasche...@gmail.com> wrote:
>>
>> > Hello Igniters,
>> >
>> > One of the major improvements to DML has to be support of batch
>> > statements. I'd like to discuss its implementation. The suggested
>> > approach is to rewrite given query turning it from few INSERTs into
>> > single statement and processing arguments accordingly. I suggest this
>> > as long as the whole point of batching is to make as little
>> > interactions with cluster as possible and to make operations as
>> > condensed as possible, and in case of Ignite it means that we should
>> > send as little JdbcQueryTasks as possible. And, as long as a query
>> > task holds single query and its arguments, this approach will not
>> > require any changes to be done to current design and won't break any
>> > backward compatibility - all dirty work on rewriting will be done by
>> > JDBC driver.
>> > Without rewriting, we could introduce some new query task for batch
>> > operations, but that would make impossible sending such requests from
>> > newer clients to older servers (say, servers of version 1.8.0, which
>> > does not know about batching, let alone older versions).
>> > I'd like to hear comments and suggestions from the community. Thanks!
>> >
>> > - Alex
>> >
>>

Re: Batch DML queries design discussion

[Alexander Paschenko]

Vlad,

1. Of course API to my view should be well-known JDBC one - addBatch
and friends. Question was not about API but rather about
implementation. What do you mean by "copy locally and execute all at
once"?

2. As I see it, it does not contradict the 1st approach and could be
implemented on par as well.

Thanks,
Alex

2016-12-08 14:16 GMT+03:00 Vladimir Ozerov <voze...@gridgain.com>:
> Hi Alex,
>
> To my understanding there are two possible approaches to batching in JDBC
> layer:
>
> 1) Rely on default batching API. Specifically
> *PreparedStatement.addBatch()* [1]
> and others. This is nice and clear API, users are used to it, and it's
> adoption will minimize user code changes when migrating from other JDBC
> sources. We simply copy updates locally and then execute them all at once
> with only a single network hop to servers. *IgniteDataStreamer* can be used
> underneath.
>
> 2) Or we can have separate connection flag which will move all
> INSERT/UPDATE/DELETE statements through streamer.
>
> I prefer the first approach
>
> Also we need to keep in mind that data streamer has poor performance when
> adding single key-value pairs due to high overhead on concurrency and other
> bookkeeping. Instead, it is better to pre-batch key-value pairs before
> giving them to streamer.
>
> Vladimir.
>
> [1]
> https://docs.oracle.com/javase/8/docs/api/java/sql/PreparedStatement.html#addBatch--
>
> On Thu, Dec 8, 2016 at 1:21 PM, Alexander Paschenko <
> alexander.a.pasche...@gmail.com> wrote:
>
>> Hello Igniters,
>>
>> One of the major improvements to DML has to be support of batch
>> statements. I'd like to discuss its implementation. The suggested
>> approach is to rewrite given query turning it from few INSERTs into
>> single statement and processing arguments accordingly. I suggest this
>> as long as the whole point of batching is to make as little
>> interactions with cluster as possible and to make operations as
>> condensed as possible, and in case of Ignite it means that we should
>> send as little JdbcQueryTasks as possible. And, as long as a query
>> task holds single query and its arguments, this approach will not
>> require any changes to be done to current design and won't break any
>> backward compatibility - all dirty work on rewriting will be done by
>> JDBC driver.
>> Without rewriting, we could introduce some new query task for batch
>> operations, but that would make impossible sending such requests from
>> newer clients to older servers (say, servers of version 1.8.0, which
>> does not know about batching, let alone older versions).
>> I'd like to hear comments and suggestions from the community. Thanks!
>>
>> - Alex
>>

Batch DML queries design discussion

[Alexander Paschenko]

Hello Igniters,

One of the major improvements to DML has to be support of batch
statements. I'd like to discuss its implementation. The suggested
approach is to rewrite given query turning it from few INSERTs into
single statement and processing arguments accordingly. I suggest this
as long as the whole point of batching is to make as little
interactions with cluster as possible and to make operations as
condensed as possible, and in case of Ignite it means that we should
send as little JdbcQueryTasks as possible. And, as long as a query
task holds single query and its arguments, this approach will not
require any changes to be done to current design and won't break any
backward compatibility - all dirty work on rewriting will be done by
JDBC driver.
Without rewriting, we could introduce some new query task for batch
operations, but that would make impossible sending such requests from
newer clients to older servers (say, servers of version 1.8.0, which
does not know about batching, let alone older versions).
I'd like to hear comments and suggestions from the community. Thanks!

- Alex

[jira] [Created] (IGNITE-4383) Update section on binary marshaller in Apache Ignite Readme.io docs

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4383:
---

 Summary: Update section on binary marshaller in Apache Ignite 
Readme.io docs
 Key: IGNITE-4383
 URL: https://issues.apache.org/jira/browse/IGNITE-4383
 Project: Ignite
  Issue Type: Task
  Components: binary, documentation
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 1.8






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Apache Ignite 1.8 Release

[Alexander Paschenko]

>>>>
>> > >>>> Update for DML testing:
>> > >>>>
>> > >>>> IGNITE-4342 DML: clear() on atomic cache causes the exception from
>> > >>> previous
>> > >>>> failed DML statement <https://issues.apache.org/
>> > >> jira/browse/IGNITE-4342>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>> On Wed, Nov 30, 2016 at 10:18 PM, Denis Magda <dma...@apache.org>
>> > >> wrote:
>> > >>>>
>> > >>>>> Vladimir,
>> > >>>>>
>> > >>>>> Please add to the release notes information regarding this feature
>> > >>>>> https://issues.apache.org/jira/browse/IGNITE-2310 <
>> > >>>>> https://issues.apache.org/jira/browse/IGNITE-2310>
>> > >>>>>
>> > >>>>> Essentially, it allows locking a partition while a remote
>> computation
>> > >> is
>> > >>>>> being executed on a node. I’ve already documented the feature here.
>> > >>>>> http://apacheignite.gridgain.org/v1.7/docs/collocate-
>> > >>>>> compute-and-data#affinity-call-and-run-methods <
>> > >>>>> http://apacheignite.gridgain.org/v1.7/docs/collocate-
>> > >>>>> compute-and-data#affinity-call-and-run-methods>
>> > >>>>>
>> > >>>>> —
>> > >>>>> Denis
>> > >>>>>
>> > >>>>>> On Nov 25, 2016, at 4:08 AM, Vladimir Ozerov <
>> voze...@gridgain.com>
>> > >>>>> wrote:
>> > >>>>>>
>> > >>>>>> Folks,
>> > >>>>>>
>> > >>>>>> I need to create RELEASE NOTES. Please advise which notable
>> tickets
>> > >> you
>> > >>>>>> completed as a part of 1.8 release.
>> > >>>>>>
>> > >>>>>> Vladimir.
>> > >>>>>>
>> > >>>>>> On Fri, Nov 25, 2016 at 2:58 PM, Sergey Kozlov <
>> > skoz...@gridgain.com
>> > >>>
>> > >>>>> wrote:
>> > >>>>>>
>> > >>>>>>> Hi
>> > >>>>>>>
>> > >>>>>>> Could someone explain why Cassandra module splitted into three
>> > parts
>> > >>> in
>> > >>>>>>> optional directory for binary fabric build? At the moment I see
>> > >>>>> following
>> > >>>>>>> unclear points
>> > >>>>>>> 1. ignite-cassandra directory contains README.txt only
>> > >>>>>>> 2. Does ignite-cassandra-serializers depend on
>> > >> ignite-cassandra-store?
>> > >>>>> In
>> > >>>>>>> that case why do not make one module?
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>> On Fri, Nov 25, 2016 at 2:37 PM, Alexander Paschenko <
>> > >>>>>>> alexander.a.pasche...@gmail.com> wrote:
>> > >>>>>>>
>> > >>>>>>>> IGNITE-4303 is most likely fixed by IGNITE-4280 fix (already
>> > merged
>> > >>> in
>> > >>>>>>>> 1.8 branch).
>> > >>>>>>>>
>> > >>>>>>>> Meanwhile everything SQL/DML related seems to be in pull
>> > >>>>>>>> requests/reviewed/fixed/merged (no issues unapproached/not
>> fixed).
>> > >>>>>>>>
>> > >>>>>>>> - Alex
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> 2016-11-24 22:59 GMT+03:00 Sergey Kozlov <skoz...@gridgain.com
>> >:
>> > >>>>>>>>> Hi
>> > >>>>>>>>>
>> > >>>>>>>>> I found two issues for 1.8:
>> > >>>>>>>>> IGNITE-4304 ClusterTopologyCheckedException: Failed to send
>> > >> message
>> &g

[jira] [Created] (IGNITE-4363) Inner properties mutation broken in DML

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4363:
---

 Summary: Inner properties mutation broken in DML
 Key: IGNITE-4363
 URL: https://issues.apache.org/jira/browse/IGNITE-4363
 Project: Ignite
  Issue Type: Bug
  Components: binary, SQL
Affects Versions: 1.8
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 1.8






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4343) Check mutable entries for existence properly inside DML entry processors

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4343:
---

 Summary: Check mutable entries for existence properly inside DML 
entry processors
 Key: IGNITE-4343
 URL: https://issues.apache.org/jira/browse/IGNITE-4343
 Project: Ignite
  Issue Type: Improvement
  Components: SQL
Affects Versions: 1.8
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 1.8






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4340) Implicitly cast new column values to expected types on SQL UPDATE

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-4340:
---

 Summary: Implicitly cast new column values to expected types on 
SQL UPDATE
 Key: IGNITE-4340
 URL: https://issues.apache.org/jira/browse/IGNITE-4340
 Project: Ignite
  Issue Type: Improvement
  Components: SQL
Affects Versions: 1.8
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko
 Fix For: 1.8


When the following query is run,

{code:sql}
update AllTypes set longCol = 1 where _key = ?
{code}

it fails with exception

{noformat}
Suppressed: java.lang.ClassCastException: java.lang.Integer cannot be cast to 
java.lang.Long
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$RowDescriptor.wrap(IgniteH2Indexing.java:2960)
at 
org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2AbstractKeyValueRow.java:316)
at 
org.h2.index.BaseIndex.compareRows(BaseIndex.java:294)
at 
org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex$2.compare(GridH2TreeIndex.java:103)
at 
org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex$2.compare(GridH2TreeIndex.java:95)
at 
java.util.concurrent.ConcurrentSkipListMap$ComparableUsingComparator.compareTo(ConcurrentSkipListMap.java:647)
at 
java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:727)
at 
java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:850)
at 
java.util.concurrent.ConcurrentSkipListMap.put(ConcurrentSkipListMap.java:1645)
at 
org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.put(GridH2TreeIndex.java:362)
at 
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:566)
at 
org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:495)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:603)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:737)
at 
org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:431)
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.updateIndex(GridCacheMapEntry.java:4019)
at 
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2458)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2385)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1787)
{noformat}

It's due to that UPDATE's SELECT part selects 1 as int, and that's what we're 
setting to field. Problem can be solved by casting SELECTed values to the types 
that columns expect.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)