[jira] [Created] (IGNITE-6647) Web Console: Implement schema migration

[Alexey Kuznetsov (JIRA)]

Alexey Kuznetsov created IGNITE-6647:


 Summary: Web Console: Implement schema migration
 Key: IGNITE-6647
 URL: https://issues.apache.org/jira/browse/IGNITE-6647
 Project: Ignite
  Issue Type: Improvement
  Security Level: Public (Viewable by anyone)
  Components: wizards
Reporter: Alexey Kuznetsov
Assignee: Alexey Kuznetsov
 Fix For: 2.4


We need to support schema updates for Web Console new versions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: integrate with prestodb

[Denis Magda]

Shawn,

What's the purpose of Presto then if you consider holding all the data in
RAM? From what I see, Presto is intended for joining the data stored in
different storages.

As for the Ignite persistence, here are some performance hints you might
need to apply:
https://apacheignite.readme.io/docs/durable-memory-tuning

--
Denis


On Mon, Oct 16, 2017 at 7:42 PM, shawn.du  wrote:

> Hi Denis,
>
> We are evaluating this feature(our production use ignite 1.9 and we are
> testing ignite 2.2),  this do make things simple. But we don't want to loss
> performance.
> we need careful testing, seeing from our first round test result, the disk
> IO will be the bottleneck.
> The load average is higher than ignite 1.9 without this feature.  Also i
> don't know ignite load data from disk will fast enough
> comparing with decode the data in memory.
>
> Thanks
> Shawn
>
> On 10/17/2017 10:25，Denis Magda 
> wrote：
>
> Shawn,
>
> Then my suggestion would be to enable Ignite persistence [
> 1] that will store the whole data set you have. RAM will
> keep only a subset for the performance benefits. Ignite
> SQL is full supported for the persistence, you can even
> join data RAM and disk only data sets. Plus, your compression becomes 
> optional.
>
>
> [1] https://ignite.apache.org/features/persistence.html  ps://ignite.apache.org/features/persistence.html>
>
> —
> Denis
>
> > On Oct 16, 2017, at 7:18 PM, shawn.du  wrote:
> >
> > Hi Denis,
> >
> > Yes, We do want to limit the RAM to less than 64G.
> RAM resource is still an expensive resource.
> > If we store our data with ignite SQL queryable format,
> our data may use more than 640G. This is too expensive for us.
> > So we store data using binary format which works a
> bit like orc or parquet. Only several important columns are
> SQL queryable and the others are not. In this way, we do
> store using less RAMs, but we have to use map-reduce to
> query the data, which is a little bit of complex: Query
> in client with SQL, then submit jobs to ignite compute,
>  finally do some post aggregation in client.
> > This is why I want to have a try of Presto. We like SQL,
> we want all computation on server side.
> >
> > welcome your comments.
> >
> > Thanks
> > Shawn
> >
> > On 10/17/2017 07:57，Denis Magda  dma...@apache.org> wrote：
> > Hello Shawn,
> >
> > Do I understand properly that you have scarce RAM
> resources and think to exploit Presto as an alternative SQL engine in
> Ignite that queries both RAM and disk data sets? If that’s
> the case than just enable Ignite native persistence [1]
> and you’ll get all the data stored on disk and as much as
> you can afford in RAM. The SQL works over both tiers transparently for you.
>
> >
> > [1] https://ignite.apache.org/features/persistence.html <
> https://ignite.apache.org/features/persistence.html>  tps://ignite.apache.org/features/persistence.html  ps://ignite.apache.org/features/persistence.html>>
> >
> > —
> > Denis
> >
> > > On Oct 16, 2017, at 2:19 AM, Alexey Kukushkin  nale...@gmail.com > wrote:
> > >
> > > Cross-sending to the DEV community.
> > >
> > > On Mon, Oct 16, 2017 at 12:14 PM, shawn.du    >> wrote:
> > > Hi community,
> > >
> > > I am trying to implement a connector for presto to connect ignite.
> > > I think it will be a very interest thing to connect ignite and presto.
>
> > >
> > > In fact, currently we use ignite and it works very well.
>   but in order to save memory, we build compressed binary data.
> > > thus we cannot query them using SQL. We use ignite map-reduce to query 
> > > the data.
>
> > >
> > > Using presto, we may use SQL again. If it is fast
> enough, ignite will be our in memory storage and not
> responsible for computing or only for simple query.
> > > The only thing I concern about is presto is fast
> enough or not like Ignite. For now all ignite query cost
> less than 5 seconds and most are hundreds of milliseconds.
> > > Also presto provides a connector for redis.  I don't
> know community has interest to contribute to presto-ignite?
> > >
> > > Thanks
> > > Shawn
> > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > > Alexey
> >
>
>

Re: Indexing fields of non-POJO cache values

[Andrey Kornev]

Of course it does, Dmitriy! However as I suggested below, the feature should be 
optional. The current behavior (not requiring user classes on the server, etc.) 
would remain the default one.

Also, please realize that not everyone stores their data as POJOs or uses 
Ignite as a JDBC source -- the use cases that appear to have been the main 
focus of Ignite community lately.

Payloads with dynamic structures require more advanced mechanisms for indexing, 
for example, to avoid the overhead of duplicating the indexable fields as top 
level fields of the BinaryObjects. In cases where the cache sizes are in tens 
of millions of entries, the ability to generate index values on the fly rather 
than store them, would go a long way in terms of reducing memory utilization.

In Ignite community finds this feature generally useful, I'd be more than happy 
to contribute its implementation.

Regards
Andrey


From: Dmitriy Setrakyan 
Sent: Monday, October 16, 2017 6:14 PM
To: dev@ignite.apache.org
Subject: Re: Indexing fields of non-POJO cache values

On Mon, Oct 16, 2017 at 12:35 PM, Andrey Kornev 
wrote:

> [Crossposting to the dev list]
>
> Alexey,
>
> Yes, something like that, where the "reference"/"alias" is expressed as a
> piece of Java code (as part of QueryEntity definition, perhaps) that is
> invoked by Ignite at the cache entry indexing time.
>
> My point is that rather than limiting indexable fields only to predefined
> POJO attributes (or BinaryObject fields) Ignite could adopt a more general
> approach by allowing users designate an arbitrary piece of code (a
> lambda/closure) to be used as an index value extractor. In such case, the
> current functionality (extracting index values from POJO attributes)
> becomes just a special case that's supported by Ignite out of the box.
>

Andrey, this would require deserialization on the server side. It would
also require that user classes are present on the server side. Both of this
scenarios Ignite tries to avoid.

Makes sense?

Re: integrate with prestodb

[Denis Magda]

Shawn,

Then my suggestion would be to enable Ignite persistence [1] that will store 
the whole data set you have. RAM will keep only a subset for the performance 
benefits. Ignite SQL is full supported for the persistence, you can even join 
data RAM and disk only data sets. Plus, your compression becomes optional.

[1] https://ignite.apache.org/features/persistence.html 


—
Denis

> On Oct 16, 2017, at 7:18 PM, shawn.du  wrote:
> 
> Hi Denis,
> 
> Yes, We do want to limit the RAM to less than 64G.  RAM resource is still an 
> expensive resource.
> If we store our data with ignite SQL queryable format, our data may use more 
> than 640G. This is too expensive for us.
> So we store data using binary format which works a bit like orc or parquet. 
> Only several important columns are SQL queryable and the others are not. In 
> this way, we do store using less RAMs, but we have to use map-reduce to query 
> the data, which is a little bit of complex: Query in client with SQL, then 
> submit jobs to ignite compute, finally do some post aggregation in client.
> This is why I want to have a try of Presto. We like SQL, we want all 
> computation on server side. 
> 
> welcome your comments.
> 
> Thanks
> Shawn
> 
> On 10/17/2017 07:57，Denis Magda  
> wrote： 
> Hello Shawn, 
> 
> Do I understand properly that you have scarce RAM resources and think to 
> exploit Presto as an alternative SQL engine in Ignite that queries both RAM 
> and disk data sets? If that’s the case than just enable Ignite native 
> persistence [1] and you’ll get all the data stored on disk and as much as you 
> can afford in RAM. The SQL works over both tiers transparently for you. 
> 
> [1] https://ignite.apache.org/features/persistence.html 
>  
>  > 
> 
> — 
> Denis 
> 
> > On Oct 16, 2017, at 2:19 AM, Alexey Kukushkin  > > wrote: 
> >  
> > Cross-sending to the DEV community. 
> >  
> > On Mon, Oct 16, 2017 at 12:14 PM, shawn.du  >   > >> wrote: 
> > Hi community, 
> >  
> > I am trying to implement a connector for presto to connect ignite.  
> > I think it will be a very interest thing to connect ignite and presto. 
> >  
> > In fact, currently we use ignite and it works very well.  but in order to 
> > save memory, we build compressed binary data. 
> > thus we cannot query them using SQL. We use ignite map-reduce to query the 
> > data. 
> >  
> > Using presto, we may use SQL again. If it is fast enough, ignite will be 
> > our in memory storage and not responsible for computing or only for simple 
> > query. 
> > The only thing I concern about is presto is fast enough or not like Ignite. 
> > For now all ignite query cost less than 5 seconds and most are hundreds of 
> > milliseconds. 
> > Also presto provides a connector for redis.  I don't know community has 
> > interest to contribute to presto-ignite? 
> >  
> > Thanks 
> > Shawn 
> >  
> >  
> >  
> >  
> > --  
> > Best regards, 
> > Alexey 
> 

Re: Persistence per memory policy configuration

[Dmitriy Setrakyan]

Igniters, we should not have Swap in the property names. Swap is something
that behaves completely differently from Ignite persistence, and I don't
think it should be present within Ignite altogether.

D.

On Mon, Oct 16, 2017 at 11:17 AM, Vladimir Ozerov 
wrote:

> +1 to Ivan's suggestion.
>
> пн, 16 окт. 2017 г. в 21:00, Ivan Rakov :
>
> > *swapPath* is ok for me. It is also consistent with *walPath* and
> > *walArchivePath*.
> >
> > Regarding persistencePath/storagePath, I don't like the idea when path
> > for WAL is implicitly changed, especially when we have separate option
> > for it.
> > WAL and storage files are already located under same $IGNITE_HOME root.
> >  From user perspective, there's no need to change root for all
> > persistence-related directories as long as $IGNITE_HOME points to the
> > correct disk.
> >  From developer perspective, this change breaks backwards compatibility.
> > Maintaining backwards compatibility in fail-safe way (checking both
> > old-style and new-style paths) is complex and hard to maintain in the
> > codebase.
> >
> > Best Regards,
> > Ivan Rakov
> >
> > My vote is for *storagePath* and keeping behavior as is.
> >
> > On 16.10.2017 16:53, Pavel Tupitsyn wrote:
> > > Igniters, another thing to consider:
> > >
> > > DataRegionConfiguration.SwapFilePath should be SwapPath,
> > > since this is actually not a single file, but a directory path.
> > >
> > > On Fri, Oct 13, 2017 at 7:53 PM, Denis Magda 
> wrote:
> > >
> > >> Seems I've got what you’re talking about.
> > >>
> > >> I’ve tried to change the root directory (*persistencePath*) and saw
> that
> > >> only data/indexes were placed to it while wal stayed somewhere in my
> > work
> > >> dir. It works counterintuitive and causes non productive discussions
> > like
> > >> we are in arguing about *persistencePath* or *storagePath*. Neither
> name
> > >> fits this behavior.
> > >>
> > >> My suggestion will be the following:
> > >> - *persistencePath* refers to the path of all storage files
> > (data/indexes,
> > >> wal, archive). If the path is changed *all the files* will be under
> the
> > new
> > >> directory unless *setWalPath* and *setWalArchivePath* are set
> > *explicitly*.
> > >> - *setWalPath* overrides the default location of WAL (which is
> > >> setPersistencePath)
> > >> - *setWalArchivePath* overrides the default location of the archive
> > (which
> > >> is again has to be setPersistencePath).
> > >>
> > >> If we follow this approach the configuration and behavior becomes
> vivid.
> > >> Thoughts?
> > >>
> > >> —
> > >> Denis
> > >>
> > >>> On Oct 13, 2017, at 1:21 AM, Ivan Rakov 
> wrote:
> > >>>
> > >>> Denis,
> > >>>
> > >>> Data/index storage and WAL are located under the same root by
> default.
> > >>> However, this is not mandatory: *storagePath* and *walPath*
> properties
> > >> can contain both absolute and relative paths. If paths are absolute,
> > >> storage and WAL can reside on different devices, like this:
> >  storagePath: /storage1/NMVe_drive/storage
> >  walPath: /storage2/Big_SSD_drive/wal
> > >>> We even recommend this in tuning guide: https://apacheignite.readme.
> > >> io/docs/durable-memory-tuning
> > >>> That's why I think *persistencePath* is misleading.
> > >>>
> > >>> Best Regards,
> > >>> Ivan Rakov
> > >>>
> > >>> On 13.10.2017 5:03, Dmitriy Setrakyan wrote:
> >  On Thu, Oct 12, 2017 at 7:01 PM, Denis Magda 
> > >> wrote:
> > >  From what I see after running an example they are under the same
> > root
> > > folder and in different subdirectories. The root folder should be
> > >> defined
> > > by setPersistencePath as I guess.
> > >
> >  If that is the case, then you are right. Then we should not have
> >  storagePath or WalPath, and store them both under "persistencePath"
> > >> root.
> >  However, I would need Alexey Goncharuk or Ivan Rakov to confirm
> this.
> > 
> > >>
> >
> >
>

Re: Indexing fields of non-POJO cache values

[Dmitriy Setrakyan]

On Mon, Oct 16, 2017 at 12:35 PM, Andrey Kornev 
wrote:

> [Crossposting to the dev list]
>
> Alexey,
>
> Yes, something like that, where the "reference"/"alias" is expressed as a
> piece of Java code (as part of QueryEntity definition, perhaps) that is
> invoked by Ignite at the cache entry indexing time.
>
> My point is that rather than limiting indexable fields only to predefined
> POJO attributes (or BinaryObject fields) Ignite could adopt a more general
> approach by allowing users designate an arbitrary piece of code (a
> lambda/closure) to be used as an index value extractor. In such case, the
> current functionality (extracting index values from POJO attributes)
> becomes just a special case that's supported by Ignite out of the box.
>

Andrey, this would require deserialization on the server side. It would
also require that user classes are present on the server side. Both of this
scenarios Ignite tries to avoid.

Makes sense?

Re: integrate with prestodb

[Denis Magda]

Hello Shawn,

Do I understand properly that you have scarce RAM resources and think to 
exploit Presto as an alternative SQL engine in Ignite that queries both RAM and 
disk data sets? If that’s the case than just enable Ignite native persistence 
[1] and you’ll get all the data stored on disk and as much as you can afford in 
RAM. The SQL works over both tiers transparently for you.

[1] https://ignite.apache.org/features/persistence.html 


—
Denis

> On Oct 16, 2017, at 2:19 AM, Alexey Kukushkin  
> wrote:
> 
> Cross-sending to the DEV community.
> 
> On Mon, Oct 16, 2017 at 12:14 PM, shawn.du  > wrote:
> Hi community,
> 
> I am trying to implement a connector for presto to connect ignite. 
> I think it will be a very interest thing to connect ignite and presto.
> 
> In fact, currently we use ignite and it works very well.  but in order to 
> save memory, we build compressed binary data.
> thus we cannot query them using SQL. We use ignite map-reduce to query the 
> data.
> 
> Using presto, we may use SQL again. If it is fast enough, ignite will be our 
> in memory storage and not responsible for computing or only for simple query.
> The only thing I concern about is presto is fast enough or not like Ignite. 
> For now all ignite query cost less than 5 seconds and most are hundreds of 
> milliseconds.
> Also presto provides a connector for redis.  I don't know community has 
> interest to contribute to presto-ignite?
> 
> Thanks
> Shawn
> 
> 
> 
> 
> -- 
> Best regards,
> Alexey

Clarification on DROP TABLE command needed

[Denis Magda]

Alex P., Vladimir,

I’m finalizing DROP TABLE command documentation and unsure what happens with 
the following once the command is issued:

1. Do we destroy the underlying cache and purge all its content?

2. What happens to DML operations targeting a table that is somewhere in the 
middle of the drop process? Do we block such DML commands waiting while the 
DROP TABLE is over or kick them off right away with some special exception?

—
Denis

Re: Persistence per memory policy configuration

[Vladimir Ozerov]

+1 to Ivan's suggestion.

пн, 16 окт. 2017 г. в 21:00, Ivan Rakov :

> *swapPath* is ok for me. It is also consistent with *walPath* and
> *walArchivePath*.
>
> Regarding persistencePath/storagePath, I don't like the idea when path
> for WAL is implicitly changed, especially when we have separate option
> for it.
> WAL and storage files are already located under same $IGNITE_HOME root.
>  From user perspective, there's no need to change root for all
> persistence-related directories as long as $IGNITE_HOME points to the
> correct disk.
>  From developer perspective, this change breaks backwards compatibility.
> Maintaining backwards compatibility in fail-safe way (checking both
> old-style and new-style paths) is complex and hard to maintain in the
> codebase.
>
> Best Regards,
> Ivan Rakov
>
> My vote is for *storagePath* and keeping behavior as is.
>
> On 16.10.2017 16:53, Pavel Tupitsyn wrote:
> > Igniters, another thing to consider:
> >
> > DataRegionConfiguration.SwapFilePath should be SwapPath,
> > since this is actually not a single file, but a directory path.
> >
> > On Fri, Oct 13, 2017 at 7:53 PM, Denis Magda  wrote:
> >
> >> Seems I've got what you’re talking about.
> >>
> >> I’ve tried to change the root directory (*persistencePath*) and saw that
> >> only data/indexes were placed to it while wal stayed somewhere in my
> work
> >> dir. It works counterintuitive and causes non productive discussions
> like
> >> we are in arguing about *persistencePath* or *storagePath*. Neither name
> >> fits this behavior.
> >>
> >> My suggestion will be the following:
> >> - *persistencePath* refers to the path of all storage files
> (data/indexes,
> >> wal, archive). If the path is changed *all the files* will be under the
> new
> >> directory unless *setWalPath* and *setWalArchivePath* are set
> *explicitly*.
> >> - *setWalPath* overrides the default location of WAL (which is
> >> setPersistencePath)
> >> - *setWalArchivePath* overrides the default location of the archive
> (which
> >> is again has to be setPersistencePath).
> >>
> >> If we follow this approach the configuration and behavior becomes vivid.
> >> Thoughts?
> >>
> >> —
> >> Denis
> >>
> >>> On Oct 13, 2017, at 1:21 AM, Ivan Rakov  wrote:
> >>>
> >>> Denis,
> >>>
> >>> Data/index storage and WAL are located under the same root by default.
> >>> However, this is not mandatory: *storagePath* and *walPath* properties
> >> can contain both absolute and relative paths. If paths are absolute,
> >> storage and WAL can reside on different devices, like this:
>  storagePath: /storage1/NMVe_drive/storage
>  walPath: /storage2/Big_SSD_drive/wal
> >>> We even recommend this in tuning guide: https://apacheignite.readme.
> >> io/docs/durable-memory-tuning
> >>> That's why I think *persistencePath* is misleading.
> >>>
> >>> Best Regards,
> >>> Ivan Rakov
> >>>
> >>> On 13.10.2017 5:03, Dmitriy Setrakyan wrote:
>  On Thu, Oct 12, 2017 at 7:01 PM, Denis Magda 
> >> wrote:
> >  From what I see after running an example they are under the same
> root
> > folder and in different subdirectories. The root folder should be
> >> defined
> > by setPersistencePath as I guess.
> >
>  If that is the case, then you are right. Then we should not have
>  storagePath or WalPath, and store them both under "persistencePath"
> >> root.
>  However, I would need Alexey Goncharuk or Ivan Rakov to confirm this.
> 
> >>
>
>

Re: Persistence per memory policy configuration

[Ivan Rakov]

*swapPath* is ok for me. It is also consistent with *walPath* and 
*walArchivePath*.


Regarding persistencePath/storagePath, I don't like the idea when path 
for WAL is implicitly changed, especially when we have separate option 
for it.
WAL and storage files are already located under same $IGNITE_HOME root. 
From user perspective, there's no need to change root for all 
persistence-related directories as long as $IGNITE_HOME points to the 
correct disk.
From developer perspective, this change breaks backwards compatibility. 
Maintaining backwards compatibility in fail-safe way (checking both 
old-style and new-style paths) is complex and hard to maintain in the 
codebase.


Best Regards,
Ivan Rakov

My vote is for *storagePath* and keeping behavior as is.

On 16.10.2017 16:53, Pavel Tupitsyn wrote:

Igniters, another thing to consider:

DataRegionConfiguration.SwapFilePath should be SwapPath,
since this is actually not a single file, but a directory path.

On Fri, Oct 13, 2017 at 7:53 PM, Denis Magda  wrote:


Seems I've got what you’re talking about.

I’ve tried to change the root directory (*persistencePath*) and saw that
only data/indexes were placed to it while wal stayed somewhere in my work
dir. It works counterintuitive and causes non productive discussions like
we are in arguing about *persistencePath* or *storagePath*. Neither name
fits this behavior.

My suggestion will be the following:
- *persistencePath* refers to the path of all storage files (data/indexes,
wal, archive). If the path is changed *all the files* will be under the new
directory unless *setWalPath* and *setWalArchivePath* are set *explicitly*.
- *setWalPath* overrides the default location of WAL (which is
setPersistencePath)
- *setWalArchivePath* overrides the default location of the archive (which
is again has to be setPersistencePath).

If we follow this approach the configuration and behavior becomes vivid.
Thoughts?

—
Denis


On Oct 13, 2017, at 1:21 AM, Ivan Rakov  wrote:

Denis,

Data/index storage and WAL are located under the same root by default.
However, this is not mandatory: *storagePath* and *walPath* properties

can contain both absolute and relative paths. If paths are absolute,
storage and WAL can reside on different devices, like this:

storagePath: /storage1/NMVe_drive/storage
walPath: /storage2/Big_SSD_drive/wal

We even recommend this in tuning guide: https://apacheignite.readme.

io/docs/durable-memory-tuning

That's why I think *persistencePath* is misleading.

Best Regards,
Ivan Rakov

On 13.10.2017 5:03, Dmitriy Setrakyan wrote:

On Thu, Oct 12, 2017 at 7:01 PM, Denis Magda 

wrote:

 From what I see after running an example they are under the same root
folder and in different subdirectories. The root folder should be

defined

by setPersistencePath as I guess.


If that is the case, then you are right. Then we should not have
storagePath or WalPath, and store them both under "persistencePath"

root.

However, I would need Alexey Goncharuk or Ivan Rakov to confirm this.

[jira] [Created] (IGNITE-6645) Security issues in Ignite that allows users with write access to datagrid to execute arbitrary code

[Denis Magda (JIRA)]

Denis Magda created IGNITE-6645:
---

 Summary: Security issues in Ignite that allows users with write 
access to datagrid to execute arbitrary code
 Key: IGNITE-6645
 URL: https://issues.apache.org/jira/browse/IGNITE-6645
 Project: Ignite
  Issue Type: Bug
  Security Level: Public (Viewable by anyone)
Affects Versions: 1.0
Reporter: Denis Magda
Assignee: Yakov Zhdanov
Priority: Critical
 Fix For: 2.4


The security breach was reported by an end-user:
https://mail-search.apache.org/pmc/private-arch/ignite-private/201710.mbox/%3c7099cd44-92a7-4254-89c5-d8270b5a6...@apache.org%3e

Details shared by the user:
I would like to report some security issues that we found using the query 
language QL from lgtm.com. These are unsafe deserialization issues that allow 
users, possibly remote, that have rights to put entities on the datagrid to 
execute arbitrary code on an ignite server node.

As there are more than one of these issues, I will send them to you in separate 
emails.

The first issue affects the socket streaming server. The PoC code are included 
and are modifications of the `wordcount.socket` example in the examples 
package. 

A bit of set up is needed to see the full effect of code execution, so I will 
not include the details here, but if you want to try it out yourself, then 
please let me know and I can include the full PoC.

First add commons-beantil to the dependency, any version will work. Then 
download the file `obj`, which contains the serialized data of a malicious 
object. Change line 44 in `SocketStreamClient` so that it opens this file.

First start a server node using the example config `config/example-ignite.xml`, 
then start up the streaming server `SocketStreamerServer`. Now when you run 
`SocketStreamClient`, you will get an error, but somewhere in the stacktrace on 
the log in `SocketStreamerServer`, you will see this:

Caused by: java.lang.RuntimeException: InvocationTargetException: 
java.lang.reflect.InvocationTargetException
at 
org.apache.commons.beanutils.BeanComparator.compare(BeanComparator.java:171)
at java.util.PriorityQueue.siftDownUsingComparator(PriorityQueue.java:721)
at java.util.PriorityQueue.siftDown(PriorityQueue.java:687)
at java.util.PriorityQueue.heapify(PriorityQueue.java:736)
at java.util.PriorityQueue.readObject(PriorityQueue.java:795)

This shows that the node running the `SocketStreamerServer` is deserializing 
the payload object that I send it.

When properly set up, an attacker will have a remote ldap server that contains 
a second malicious Java object. Then when the above deserialization happens, an 
ldap look up will cause the second malicious object to be instantiated, which 
can then be used to execute arbitrary code. Also, although this exploit relies 
on having commons-beanutils to be on the classpath, there are other exploits 
that will work for different third party libraries, so it is not so much of a 
problem in commons-beanutils, but an issue in the handling of deserialization 
in ignite.

These results are using a slightly more ahead version of the QL library with we 
haven't made available on lgtm yet, but should be in a few days, if you are 
interested, I can share a link to the query and results to you when it is 
ready. Thanks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Avoid closing caches on client during cluster restart - determining it's the same cache

[Denis Magda]

Hello Ilya,

> 
> Or do you mean that we ignore cfg in getOrCreateCaches() if name is matched
> to already existing cache?

Yes, this is what I grasped from the source code.

—
Denis

> On Oct 16, 2017, at 2:29 AM, Ilya Kasnacheev  
> wrote:
> 
> Hello Denis!
> 
>> we do not check CacheConfiguration settings provided by the client
> 
> Are you sure about that? I've just tried a scenario where my client got
> bounced for not having matching EvictionPolicy compared to already deployed
> servers.
> 
> Or do you mean that we ignore cfg in getOrCreateCaches() if name is matched
> to already existing cache?
> 
> I am worried about a usability issue where a stale client occassionally
> writes entries of incorrect type into a cache or uses improper atomicity
> mode.
> 
> Regards,
> 
> -- 
> Ilya Kasnacheev
> 
> 2017-10-12 2:07 GMT+03:00 Denis Magda :
> 
>> Hello Ilya,
>> 
>> Isn’t the cache name itself sufficient to make all the validations? The
>> cache name is a unique parameter.
>> 
>> Furthermore, we do not check CacheConfiguration settings provided by the
>> client if CacheConfiguration.name is already deployed when the client tries
>> to get a reference to the cache (Ignite.getOrCreateCache(cacheCfg)).
>> Don’t think we should make any exception for the restart scenarios.
>> 
>> —
>> Denis
>> 
>> 
>>> On Oct 11, 2017, at 5:20 AM, Ilya Kasnacheev 
>> wrote:
>>> 
>>> Hello Igniters!
>>> 
>>> I'm working on IGNITE-2766. We want caches opened on client to survive
>>> cluster restart (all nodes down, then some nodes up). Currently Ignite
>>> compares deploymentId which will not match with new cluster's caches.
>>> 
>>> But still we want to make sure it's still the same cache in the new
>>> cluster, and not a different one with the same name. One obvious idea is
>> to
>>> look at CacheConfiguration. However I think a frequent scenario for
>> cluster
>>> restart is a minor change of cache configuration (e.g. add or remove
>>> backup), and it will be a pity to lose client caches in this case.
>>> 
>>> So, what fields in CacheConfiguration should we compare upon to figure
>> out
>>> if it's still the same cache? Name obviously. How does grpName work and
>>> should we compare on them too?
>>> 
>>> indexedTypes[] make sense because if it's a cache on different types it's
>>> not safe to be used where it was open. cache mode and atomicity mode make
>>> sence since cache should now be handled differently?
>>> 
>>> Any other fields in CacheConfiguration we should check because it's not
>>> safe to continue in case of mismatch? Some different approach?
>> Suggestions
>>> are kindly welcome
>>> 
>>> --
>>> Ilya Kasnacheev
>> 
>> 

[GitHub] ignite pull request #2865: ignite-5741: enable test after fix

[dspavlov]

GitHub user dspavlov opened a pull request:

https://github.com/apache/ignite/pull/2865

ignite-5741: enable test after fix

ignite-5741 was already implemented

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5741-enable-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2865.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2865


commit 78e059d16f4fc6d8430813ec9ad07f25e32d2be4
Author: dpavlov 
Date:   2017-10-16T17:06:43Z

ignite-5741: enable test after fix




---

[GitHub] ignite pull request #2864: IGNITE-6627 .NET: cache deserialization fails wit...

[apopovgg]

GitHub user apopovgg opened a pull request:

https://github.com/apache/ignite/pull/2864

IGNITE-6627 .NET: cache deserialization fails with complex value type…



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6627

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2864.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2864


commit 80bebfc0752fb75feb8eaea1771fb03ce01429ac
Author: apopov 
Date:   2017-10-16T16:10:16Z

IGNITE-6627 .NET: cache deserialization fails with complex value type & enum




---

[GitHub] ignite pull request #2863: Ignite 2.1.6.b1

[mcherkasov]

GitHub user mcherkasov opened a pull request:

https://github.com/apache/ignite/pull/2863

Ignite 2.1.6.b1



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2.1.6.b1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2863.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2863


commit 762b533d3666903dc4119b5a5252373354bb98e1
Author: sboikov 
Date:   2017-08-21T12:22:18Z

Merge remote-tracking branch 'community/ignite-2.1.4' into ignite-2.1.4

commit e9139d4916a22acab22008663e89f963f3d3a6b5
Author: sboikov 
Date:   2017-08-02T14:25:31Z

Call updateRebalanceVersion after evictions (was broken in 
c6fbe2d82a9f56f96c94551b09e85a12d192f32e);

(cherry picked from commit b277682)

commit 615a582b83d9acdd5703995f52f97e70027e0639
Author: Dmitriy Shabalin 
Date:   2017-08-21T12:34:59Z

IGNITE-4728 Fixed get params for saved state.
(cherry picked from commit 97b3cef)

commit 891a9e0001bd0d77e0bafc3ad109d04d02a9b7ff
Author: sboikov 
Date:   2017-08-21T10:21:44Z

ignite-6124 Merge exchanges for multiple discovery events

(cherry picked from commit bebf299799712b464ee0e3800752ecc07770d9f0)

commit 96d86acdfe1a251758c6915b473ceb20514cbbad
Author: sboikov 
Date:   2017-08-21T13:19:18Z

Merge remote-tracking branch 'community/ignite-2.1.4' into ignite-2.1.4

commit e9f729fcb9ba7b555aed246a834e385ce2c88795
Author: Alexey Goncharuk 
Date:   2017-08-21T10:49:22Z

ignite-5872 Replace standard java maps for partition counters with more 
effective data structures

(cherry picked from commit 918c409)

commit 8e672d4a83cb20136d375818505cacccf6c9b4fd
Author: sboikov 
Date:   2017-08-21T13:39:29Z

Changed ignite version.

commit fd5d83c44624baea9466c591f119195cb092df4c
Author: Alexey Kuznetsov 
Date:   2017-08-21T15:06:36Z

IGNITE-6104 Fixed target.
(cherry picked from commit 8775d2c)

commit cca9117c805d144ff451694e7324720f8d466d94
Author: sboikov 
Date:   2017-08-21T15:39:12Z

ignite-5872 Fixed backward compatibility

commit dc9b2660723b325e1d07991cd0936ee27f624a81
Author: sboikov 
Date:   2017-08-21T15:39:36Z

Merge remote-tracking branch 'community/ignite-2.1.4' into ignite-2.1.4

commit 1072654c8762113bc49a658f2898475381b47832
Author: Sergey Chugunov 
Date:   2017-08-21T17:05:05Z

IGNITE-6052 class rename was reverted

commit cf97f3dd903549d3b5968ceb5ca9eda29ef521cf
Author: Dmitriy Govorukhin 
Date:   2017-08-21T17:04:04Z

ignite-6035 Clear indexes on cache clear/destroy

Signed-off-by: Andrey Gura 

commit 47b84ec62fb988faad438bdfde4ee48165a4065f
Author: Andrey Gura 
Date:   2017-08-21T17:36:56Z

Visor compilation issues are fixed.

commit 6e0fd7222e64b3a5e05204b300b62f60e3142e40
Author: Ilya Lantukh 
Date:   2017-08-21T18:12:41Z

ignite-gg-12639 Fill factor metric is not working with enabled persistence

Signed-off-by: Andrey Gura 

commit bbafa1e946121b7143101d4060effc48ed74a0b3
Author: Alexey Kuznetsov 
Date:   2017-08-22T03:59:15Z

IGNITE-6136 Web Console: implemented universal version check.
(cherry picked from commit 1a42e78)

commit 2a2e8407aaa99a56f79a861fc1e201dd183eea11
Author: vsisko 
Date:   2017-08-22T04:08:21Z

IGNITE-6131 Visor Cmd: Fixed "cache -a" command output cache entries count.
(cherry picked from commit 1f5054a)

commit db64729fc9ebb0217f06b0cf9d5e53ab8d657510
Author: sboikov 
Date:   2017-08-22T08:29:32Z

ignite-6124 Fixed NPE in GridDhtPartitionsExchangeFuture.topologyVersion 
after future cleanup.

(cherry picked from commit 2c9057a)

commit 5b7724714264c14cc10f4b25abc9234387224e4b
Author: Ilya Lantukh 
Date:   2017-08-22T08:50:35Z

Fixed javadoc format.

commit 785a85eb0155444b3eef48cf373a990dc4c8c6dd
Author: sboikov 
Date:   2017-08-22T09:24:03Z

ignite-5872 GridDhtPartitionsSingleMessage.partitionUpdateCounters should 
not return null.

commit 6b506e774c59b64fc6254ea151699c852620a408
Author: sboikov 
Date:   2017-08-22T09:24:21Z

Merge remote-tracking branch 'community/ignite-2.1.4' into ignite-2.1.4

commit 160d9b7c707efc359b4014aa1a481dc0fbbf596f
Author: Ilya Lantukh 
Date:   2017-08-22T11:10:10Z

Fixed flaky test.

commit 9ed4b72044ba1b2c105761b431625736166af7e7
Author: Alexey 

[GitHub] ignite pull request #2862: IGNITE-5195 : DataStreamer remap optimization.

[ilantukh]

GitHub user ilantukh opened a pull request:

https://github.com/apache/ignite/pull/2862

IGNITE-5195 : DataStreamer remap optimization.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5195

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2862.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2862


commit caa7f7c833a162a01edacb241634515cdce67ec2
Author: Ilya Lantukh 
Date:   2017-10-16T14:56:18Z

ignite-5195 : DataStreamer remap optimization.




---

Re: Ignite ML Models improvements

[Yury Babak]

Vladisav,

Thanks for this idea. I`m not sure that this is priority task but anyway I
created separate ticket for it(IGNITE-6642).

Regards,
Yury.





--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Optimized DML execution: how to name it?

[Dmitriy Setrakyan]

On Mon, Oct 16, 2017 at 12:15 AM, Vladimir Ozerov 
wrote:

> Dmitriy,
>
> No :-) System properties is bad practice, because in some cases you cannot
> change them. Instead, this flag is available in private class
> SqlFIeldsQueryEx. So in case user desperately need it, we may advise him to
> use this class. But I hope it will not be needed.
>

Disagree. This is a temporary property and there is no reason to expose
users to external APIs. System property is the easiest to get rid of in the
future.

Re: Integration of Spark and Ignite. Prototype.

[Николай Ижиков]

Hello, guys.

I have created example application to run Ignite Data Frame on standalone
Spark cluster.
With Data Frame API implementation there are no requirements to have any
Ignite files on spark worker nodes.

I ran this application on the free dataset: ATP tennis match statistics.

data - https://github.com/nizhikov/atp_matches
app - https://github.com/nizhikov/ignite-spark-df-example

Valentin, do you have a chance to look at my changes?


2017-10-12 6:03 GMT+03:00 Valentin Kulichenko :

> Hi Nikolay,
>
> Sorry for delay on this, got a little swamped lately. I will do my best to
> review the code this week.
>
> -Val
>
> On Mon, Oct 9, 2017 at 11:48 AM, Николай Ижиков 
> wrote:
>
>> Hello, Valentin.
>>
>> Did you have a chance to look at my changes?
>>
>> Now I think I have done almost all required features.
>> I want to make some performance test to ensure my implementation work
>> properly with a significant amount of data.
>> And I definitely need some feedback for my changes.
>>
>>
>> 2017-10-09 18:45 GMT+03:00 Николай Ижиков :
>>
>>> Hello, guys.
>>>
>>> Which version of Spark do we want to use?
>>>
>>> 1. Currently, Ignite depends on Spark 2.1.0.
>>>
>>> * Can be run on JDK 7.
>>> * Still supported: 2.1.2 will be released soon.
>>>
>>> 2. Latest Spark version is 2.2.0.
>>>
>>> * Can be run only on JDK 8+
>>> * Released Jul 11, 2017.
>>> * Already supported by huge vendors(Amazon for example).
>>>
>>> Note that in IGNITE-3084 I implement some internal Spark API.
>>> So It will take some effort to switch between Spark 2.1 and 2.2
>>>
>>>
>>> 2017-09-27 2:20 GMT+03:00 Valentin Kulichenko <
>>> valentin.kuliche...@gmail.com>:
>>>
 I will review in the next few days.

 -Val

 On Tue, Sep 26, 2017 at 2:23 PM, Denis Magda  wrote:

 > Hello Nikolay,
 >
 > This is good news. Finally this capability is coming to Ignite.
 >
 > Val, Vladimir, could you do a preliminary review?
 >
 > Answering on your questions.
 >
 > 1. Yardstick should be enough for performance measurements. As a Spark
 > user, I will be curious to know what’s the point of this integration.
 > Probably we need to compare Spark + Ignite and Spark + Hive or Spark +
 > RDBMS cases.
 >
 > 2. If Spark community is reluctant let’s include the module in
 > ignite-spark integration.
 >
 > —
 > Denis
 >
 > > On Sep 25, 2017, at 11:14 AM, Николай Ижиков <
 nizhikov@gmail.com>
 > wrote:
 > >
 > > Hello, guys.
 > >
 > > Currently, I’m working on integration between Spark and Ignite [1].
 > >
 > > For now, I implement following:
 > >* Ignite DataSource implementation(IgniteRelationProvider)
 > >* DataFrame support for Ignite SQL table.
 > >* IgniteCatalog implementation for a transparent resolving of
 ignites
 > > SQL tables.
 > >
 > > Implementation of it can be found in PR [2]
 > > It would be great if someone provides feedback for a prototype.
 > >
 > > I made some examples in PR so you can see how API suppose to be
 used [3].
 > > [4].
 > >
 > > I need some advice. Can you help me?
 > >
 > > 1. How should this PR be tested?
 > >
 > > Of course, I need to provide some unit tests. But what about
 scalability
 > > tests, etc.
 > > Maybe we need some Yardstick benchmark or similar?
 > > What are your thoughts?
 > > Which scenarios should I consider in the first place?
 > >
 > > 2. Should we provide Spark Catalog implementation inside Ignite
 codebase?
 > >
 > > A current implementation of Spark Catalog based on *internal Spark
 API*.
 > > Spark community seems not interested in making Catalog API public or
 > > including Ignite Catalog in Spark code base [5], [6].
 > >
 > > *Should we include Spark internal API implementation inside Ignite
 code
 > > base?*
 > >
 > > Or should we consider to include Catalog implementation in some
 external
 > > module?
 > > That will be created and released outside Ignite?(we still can
 support
 > and
 > > develop it inside Ignite community).
 > >
 > > [1] https://issues.apache.org/jira/browse/IGNITE-3084
 > > [2] https://github.com/apache/ignite/pull/2742
 > > [3] https://github.com/apache/ignite/pull/2742/files#diff-
 > > f4ff509cef3018e221394474775e0905
 > > [4] https://github.com/apache/ignite/pull/2742/files#diff-
 > > f2b670497d81e780dfd5098c5dd8a89c
 > > [5] http://apache-spark-developers-list.1001551.n3.
 > > nabble.com/Spark-Core-Custom-Catalog-Integration-between-
 > > Apache-Ignite-and-Apache-Spark-td22452.html
 > > [6] https://issues.apache.org/jira/browse/SPARK-17767
 > >
 > > --
 > > Nikolay Izhikov
 > > 

[jira] [Created] (IGNITE-6642) Integration with PMML

[Yury Babak (JIRA)]

Yury Babak created IGNITE-6642:
--

 Summary: Integration with PMML
 Key: IGNITE-6642
 URL: https://issues.apache.org/jira/browse/IGNITE-6642
 Project: Ignite
  Issue Type: New Feature
  Security Level: Public (Viewable by anyone)
  Components: ml
Reporter: Yury Babak


PMML - Predictive Model Markup Language is XML based language which used in 
SPARK MLlib and others platforms.

Here some additional info about PMML:

(i) http://dmg.org/pmml/v4-3/GeneralStructure.html
(i) https://github.com/jpmml/jpmml-model



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6641) PageId handling improvement: PARTITION_ID masking

[Sergey Chugunov (JIRA)]

Sergey Chugunov created IGNITE-6641:
---

 Summary: PageId handling improvement: PARTITION_ID masking
 Key: IGNITE-6641
 URL: https://issues.apache.org/jira/browse/IGNITE-6641
 Project: Ignite
  Issue Type: Improvement
  Security Level: Public (Viewable by anyone)
  Components: cache
Reporter: Sergey Chugunov
 Fix For: 2.4


All in-memory caches share the same FreeList data structure which relies on 
pageId comparisons to manage its internal state.

As described in javadoc for *FullPageId* class each pageId contains a 
partitionID part which is not guaranteed to be constant and is going to be 
changed as part of a linked ticket.

Thus we need to exclude partitionID when using pageId for comparisons that 
expect pageId to be constant.

h2. Notes
This change is not applicable for persistent caches as they maintain separate 
FreeList for each partition so partitionID part is constant for pages from that 
partition.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Persistence per memory policy configuration

[Pavel Tupitsyn]

Igniters, another thing to consider:

DataRegionConfiguration.SwapFilePath should be SwapPath,
since this is actually not a single file, but a directory path.

On Fri, Oct 13, 2017 at 7:53 PM, Denis Magda  wrote:

> Seems I've got what you’re talking about.
>
> I’ve tried to change the root directory (*persistencePath*) and saw that
> only data/indexes were placed to it while wal stayed somewhere in my work
> dir. It works counterintuitive and causes non productive discussions like
> we are in arguing about *persistencePath* or *storagePath*. Neither name
> fits this behavior.
>
> My suggestion will be the following:
> - *persistencePath* refers to the path of all storage files (data/indexes,
> wal, archive). If the path is changed *all the files* will be under the new
> directory unless *setWalPath* and *setWalArchivePath* are set *explicitly*.
> - *setWalPath* overrides the default location of WAL (which is
> setPersistencePath)
> - *setWalArchivePath* overrides the default location of the archive (which
> is again has to be setPersistencePath).
>
> If we follow this approach the configuration and behavior becomes vivid.
> Thoughts?
>
> —
> Denis
>
> > On Oct 13, 2017, at 1:21 AM, Ivan Rakov  wrote:
> >
> > Denis,
> >
> > Data/index storage and WAL are located under the same root by default.
> > However, this is not mandatory: *storagePath* and *walPath* properties
> can contain both absolute and relative paths. If paths are absolute,
> storage and WAL can reside on different devices, like this:
> >
> >> storagePath: /storage1/NMVe_drive/storage
> >> walPath: /storage2/Big_SSD_drive/wal
> > We even recommend this in tuning guide: https://apacheignite.readme.
> io/docs/durable-memory-tuning
> >
> > That's why I think *persistencePath* is misleading.
> >
> > Best Regards,
> > Ivan Rakov
> >
> > On 13.10.2017 5:03, Dmitriy Setrakyan wrote:
> >> On Thu, Oct 12, 2017 at 7:01 PM, Denis Magda 
> wrote:
> >>
> >>> From what I see after running an example they are under the same root
> >>> folder and in different subdirectories. The root folder should be
> defined
> >>> by setPersistencePath as I guess.
> >>>
> >> If that is the case, then you are right. Then we should not have
> >> storagePath or WalPath, and store them both under "persistencePath"
> root.
> >> However, I would need Alexey Goncharuk or Ivan Rakov to confirm this.
> >>
> >
>
>

[jira] [Created] (IGNITE-6640) Introduction of models import/export

[Yury Babak (JIRA)]

Yury Babak created IGNITE-6640:
--

 Summary: Introduction of models import/export
 Key: IGNITE-6640
 URL: https://issues.apache.org/jira/browse/IGNITE-6640
 Project: Ignite
  Issue Type: New Feature
  Security Level: Public (Viewable by anyone)
  Components: ml
Reporter: Yury Babak
Assignee: Yury Babak


We need to add basic import/export functionality for ml models. We will start 
from simple binary save to file and load from file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2860: IGNITE-6637 Statements cache clear on cache destr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2860


---

Re: Ignite ML Models improvements

[Vladisav Jelisavcic]

Hi Artem,

I agree, it is not supported in many frameworks;
on the other hand, Spark MLlib supports it [1].

I think it is good thing to check with users if there is a need for this,
since we are already on the topic of model importing/exporting.

Best regards,
Vladisav

[1] https://spark.apache.org/docs/2.1.1/mllib-pmml-model-export.html



On Mon, Oct 16, 2017 at 2:57 PM, ArtemM  wrote:

> Hi, Vladislav,
>
> I'm not familiar with PMML, but quick googling has shown that Tensorflow,
> Keras, DL4j and some other major contemporary ML frameworks do not have
> support of it (at least native). So the question is why should we include
> it, can you please elaborate?
>
> Best regards, Artem.
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

[GitHub] ignite pull request #2861: Ignite 5608

[oleg-ostanin]

GitHub user oleg-ostanin opened a pull request:

https://github.com/apache/ignite/pull/2861

Ignite 5608



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5608

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2861.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2861


commit c5d6decfaf4c8be907975531f7d7bd41dda2da4e
Author: dpavlov 
Date:   2017-10-04T06:46:16Z

IGNITE-6285 Enhance persistent store paths handling - Fixes #2775.

Signed-off-by: Alexey Goncharuk 

commit 2e013fbe85da059580e97d57c15e9cbc92c0aa54
Author: dpavlov 
Date:   2017-10-04T12:05:48Z

IGNITE-6554 Atomic cache remove operations are not logged to WAL - Fixes 
#2800.

Signed-off-by: Alexey Goncharuk 

commit 959bf853b1a77fa674ac8115fcf57c31feb0e6c0
Author: oleg-ostanin 
Date:   2017-10-04T14:35:50Z

Removed excluding ML from examples/src

(cherry picked from commit 78f77b1)

commit d4e5d4fd3e989b69f7bfa49986ee8f8d04cac62f
Author: tledkov-gridgain 
Date:   2017-10-05T08:55:26Z

IGNITE-6529 JDBC: support column metadata 'nullable' property. This closes 
#2793.

commit 5e75aadba89761837fa0e87e36448086173a7e44
Author: oleg-ostanin 
Date:   2017-10-05T15:17:38Z

IGNITE-6561 added sqlline module

commit 2361eebffdf859e2cae278460e01e5f87b383f1f
Author: oleg-ostanin 
Date:   2017-10-06T10:02:54Z

IGNITE-6561 renamed script

commit 2d3726cf6a33a054724aa70b2d535b0e4b26ecb5
Author: oleg-ostanin 
Date:   2017-10-06T11:23:29Z

IGNITE-6561 edited script

commit b2127673104ed6525b2894fb702f467d74a84b32
Author: oleg-ostanin 
Date:   2017-10-06T13:17:09Z

IGNITE-6561 edited script parameters

commit 66521a62ddefa41b05f31c2ca7f2b7d2a3cb6616
Author: oleg-ostanin 
Date:   2017-10-07T08:50:03Z

IGNITE-6561 added header in pom

commit 58a5359643296ab778c5deb6f6f9d0e6933a5f65
Author: oleg-ostanin 
Date:   2017-10-09T09:14:21Z

IGNITE-6561 edited functions

commit 9f7c88a17b8ba6868287355cae1ff485d228a37a
Author: oleg-ostanin 
Date:   2017-10-09T11:40:30Z

IGNITE-6561 edited help

commit e2e45816ced6986dfc9ddf8fe54e17bf0ab8dd0f
Author: oleg-ostanin 
Date:   2017-10-09T12:04:45Z

IGNITE-6561 added jdbc link in help

commit 108d21c208777cb891186b5572855ab8e06cd94b
Author: oleg-ostanin 
Date:   2017-10-09T12:44:17Z

IGNITE-6561 added unknown argument handling

commit 6cddd085c643376a914be6745e319002689e6f10
Author: oleg-ostanin 
Date:   2017-10-09T16:53:38Z

IGNITE-6561 edited parser

commit 401f21cda3ed8c88b63285224a46ab40aec7b390
Author: oleg-ostanin 
Date:   2017-10-09T17:07:15Z

IGNITE-6561 fixed error handling

commit 689a5d4961d923f6bc64450c19f7f59163b15873
Author: oleg-ostanin 
Date:   2017-10-10T08:28:52Z

IGNITE-6561 edited help

commit 6be46b003f855cac8aacd13de8438a7afd3d4142
Author: oleg-ostanin 
Date:   2017-10-10T10:35:49Z

IGNITE-6561 edited parser

commit d98b8f0af16e50d8ce5a1caed636c8dea3638b74
Author: oleg-ostanin 
Date:   2017-10-10T10:39:06Z

IGNITE-6561 edited parser options

commit 06184b346c65225b3edf55a66bde5b0fa1e89242
Author: oleg-ostanin 
Date:   2017-10-10T12:18:14Z

IGNITE-6561 edited parser and help

commit f45110280b6296c65d1dfcb332420187f3c9e702
Author: oleg-ostanin 
Date:   2017-10-10T12:52:10Z

IGNITE-6561 edited ignitesql.sh

commit f36f04c2517541e50d5dab0b769b269ce5e8299d
Author: oleg-ostanin 
Date:   2017-10-11T08:00:44Z

IGNITE-6561 added .bat script

commit e4aa9eb14ae188127e4d54ef457ea8b06099bf93
Author: oleg-ostanin 
Date:   2017-10-11T09:11:12Z

IGNITE-6561 merged master

commit b02fe6403616b4f3d09c8b108f2255d751e00fc0
Author: oleg-ostanin 
Date:   2017-10-13T08:55:55Z

IGNITE-6561 edited bat file

commit 9439b4610420a1874167f2f06f1df74d7497dbb3
Author: oleg-ostanin 
Date:   2017-10-13T09:05:28Z

IGNITE-6561 edited help

commit 1e8fa199c94526d24797780fe33379f29ab65fac
Author: oleg-ostanin 
Date:   2017-10-13T09:26:42Z

IGNITE-6561 edited help

commit 3adb6fe09dcbd7aacd639ee8ff6a17838a85602c
Author: oleg-ostanin 
Date:   2017-10-13T10:23:24Z

IGNITE-6561 fixed bat

commit 

[GitHub] ignite pull request #2833: IGNITE-6362 NPE in Log4J2Logger

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2833


---

Re: Ignite ML Models improvements

[ArtemM]

Hi, Vladislav,

I'm not familiar with PMML, but quick googling has shown that Tensorflow,
Keras, DL4j and some other major contemporary ML frameworks do not have
support of it (at least native). So the question is why should we include
it, can you please elaborate?

Best regards, Artem.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Ignite ML Models improvements

[Vladisav Jelisavcic]

Hi Yury,

since we're already on topic for importing/exporting models,
just as a thought: maybe we should add support for PMML [1] [2]?

Best regards,
Vladisav

[1] http://dmg.org/pmml/v4-1/GeneralStructure.html
[2] https://github.com/jpmml/jpmml-model

On Mon, Oct 16, 2017 at 2:20 PM, Yury Babak  wrote:

> Hi all.
>
> I want to improve our models. Currently our models can predict result and
> be
> combined with other models, thats all. So I will add import/export
> functionality, check serialization performance and add some related tests.
>
> And also I want to ask community - may be we need something else from our
> models?
>
> Thanks,
> Yury
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

[jira] [Created] (IGNITE-6639) Ignite node can try to join to itself

[Mikhail Cherkasov (JIRA)]

Mikhail Cherkasov created IGNITE-6639:
-

 Summary: Ignite node can try to join to itself
 Key: IGNITE-6639
 URL: https://issues.apache.org/jira/browse/IGNITE-6639
 Project: Ignite
  Issue Type: Bug
  Security Level: Public (Viewable by anyone)
  Components: general
Affects Versions: 2.3
Reporter: Mikhail Cherkasov
Assignee: Mikhail Cherkasov
 Fix For: 2.4






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6638) Add an ability for auto activate BLT

[Sergey Puchnin (JIRA)]

Sergey Puchnin created IGNITE-6638:
--

 Summary: Add an ability for auto activate BLT
 Key: IGNITE-6638
 URL: https://issues.apache.org/jira/browse/IGNITE-6638
 Project: Ignite
  Issue Type: Improvement
  Components: persistence
Affects Versions: 2.1
Reporter: Sergey Puchnin
Assignee: Dmitriy Pavlov
 Fix For: 2.1


Currently, the WAL iterator can be obtained from the WAL manager when an Ignite 
node is up and running.
However, it may be extremely useful to read WAL in an 'offline' mode, when a 
node is not started up. This may be required for crash analysis or to export 
committed data to some external systems.

In the future we can make this even a public interface, however as a starting 
point, I would like to keep it in private package because moving to the public 
package will require Iterator and records to be public too.

So, as a starting point, we need:
 * An object that will allow us to get WALIterator instances (probably, should 
be closeable)
 * A method on this object which will create an iterator by a file name or file 
names

Using this object should not require an active Ignite instance running.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2860: IGNITE-6637 Statements cache clear on cache destr...

[alexpaschenko]

GitHub user alexpaschenko opened a pull request:

https://github.com/apache/ignite/pull/2860

IGNITE-6637 Statements cache clear on cache destroy.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6637

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2860.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2860


commit 0f2acd36c851a4f4ebdf5eff976e94289aa09bf2
Author: Alexander Paschenko 
Date:   2017-10-16T12:18:46Z

IGNITE-6637 Statements cache clear on cache destroy.




---

Node-wide and cluster-wide health reflection and reporting

[Ilya Kasnacheev]

Hello Igniters,

A fair share of problems with Ignite deployment stems from the fact that
Ignite node doesn't know if it is working properly or it is experiencing
problems.
Cluster doesn't know either if it has malfunctioning nodes, and if there
are, which ones.

For example, a recent case, if node fails to deserialize cache discovery
message batch due to missing class referenced by cache store policy, an
exception is logged, and then client is created without any caches
available to it. Application has no way to figure out the exact problem, or
indeed that there is a problem, until some developer looks at logs and
greps Exception (which is also a skill not available to anyone).

Another case, sometimes Discovery connection is established with a new
node, but it has trouble creating Communication connections. Right now,
this node will hang the whole on join because it will try to re-partition
system cache in progress. If not for system cache, we could accept a node
into topology without checking that we can open Communication connections
to it.

Another trouble is, when node A cannot connect to node B, we can't know for
sure which one is faulty. We would really want to kick faulty nodes out,
but we may end up kicking A when B was the culprit, because we could only
use internal state of A while making the decision.

My proposal - let's add a check list! It is an object with a lot of boolean
fields (Futures, even better) which correspond to various stages that node
must accomplish during joining.
It will be like:
- Discovered nodes and joined ring - check.
- Could open Communication to all other nodes - check.
- Received and populated cache discovery messages - check.

When all check boxes are in, node sends its checklist via ring. If a node
failed to submit its check list, it's eventually kicked out of topology.

We can have a configuration setting specifying whether a node which failed
some checks may still join.

Moreover, we can continue to monitor various health-related metrics in one
place.

E.g. are tasks run in striped pool in the last five minutes? Do we have
alive threads in striped pool? Do we have discovery thread running? Other
kinds of threads and pools, similarly? We have some of those checks but
their results should be gathered in one place.

This will save us from scenario where Ignite instance throws Error in a
thread, thread is stopped and is never restored, node is in unusable state
but that's not apparent to the cluster.

In future we can even fix some problems, for example, by restarting failed
threads. But we need to measure this first.

Maybe there's something like this already, it should just be more visible?


I gather that this should be implemented by people above me in skill.


Regards,

-- 
Ilya Kasnacheev

[GitHub] ignite pull request #2810: IGNITE-6562: Dynamic service deployment should us...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2810


---

[jira] [Created] (IGNITE-6637) No proper cleanup of statements cache is done on table drop

[Alexander Paschenko (JIRA)]

Alexander Paschenko created IGNITE-6637:
---

 Summary: No proper cleanup of statements cache is done on table 
drop
 Key: IGNITE-6637
 URL: https://issues.apache.org/jira/browse/IGNITE-6637
 Project: Ignite
  Issue Type: Bug
  Security Level: Public (Viewable by anyone)
  Components: sql
Reporter: Alexander Paschenko
Assignee: Alexander Paschenko


We should cleanup statements cache whenever cache is deregistered - otherwise 
it's possible to retrieve from statements cache a statement that is prepared 
from H2 perspective but may not be executed.
Reproducer attached.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-6636) BinaryStream position integer overflow

[Alexandr Kuramshin (JIRA)]

Alexandr Kuramshin created IGNITE-6636:
--

 Summary: BinaryStream position integer overflow
 Key: IGNITE-6636
 URL: https://issues.apache.org/jira/browse/IGNITE-6636
 Project: Ignite
  Issue Type: Bug
  Security Level: Public (Viewable by anyone)
  Components: general
Affects Versions: 2.2
Reporter: Alexandr Kuramshin


There were some issues with negative {{BinaryAbstractStream#pos}} value.

We may get stack trace like that
{noformat}
java.lang.ArrayIndexOutOfBoundsException: -2142240123
at 
org.apache.ignite.internal.binary.streams.BinaryHeapOutputStream.writeByteAndShift(BinaryHeapOutputStream.java)
at 
org.apache.ignite.internal.binary.streams.BinaryAbstractOutputStream.writeByte(BinaryAbstractOutputStream.java)
at 
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java)
at 
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java)
{noformat}

The worst of it is that the {{ArrayIndexOutOfBoundsException}} has been thrown 
on the next write to the stream, and upon stack unwinding we couldn't know 
which object actually cause the overflow.

I've to suggest to check all updates to the {{BinaryAbstractStream#pos}} and 
throw exception right after the change.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2859: IGNITE-6635

[devozerov]

GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/2859

IGNITE-6635



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6635

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2859.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2859


commit b807a7f0ff33de416287b506927188b077744109
Author: devozerov 
Date:   2017-10-16T08:02:23Z

WIP.

commit d40025def33eb46d85abb60457ce0adc110b8ac2
Author: devozerov 
Date:   2017-10-16T08:10:31Z

Key-only row.

commit da96581de2e45fcfb8fea7215f84ddb1610b327a
Author: devozerov 
Date:   2017-10-16T08:10:40Z

Key-only row.

commit 4732faa5c4ad0a636a494fcf6a5da7bdef57f7d3
Author: devozerov 
Date:   2017-10-16T08:14:05Z

It compiles.

commit fe602efca5b873397cd956fad0b9e73ef4e0d603
Author: devozerov 
Date:   2017-10-16T08:21:23Z

Hidden public fields.

commit 3f79ee7437117e786fdfbb188a6f13ddb88a27d6
Author: devozerov 
Date:   2017-10-16T08:48:40Z

WIP.

commit 020367e724621e772735de24f2eb14e12f08e256
Author: devozerov 
Date:   2017-10-16T10:39:42Z

Merge branch 'master' into inline-separate-value-enum

commit e8207f82c5436616e5c31f9193ee243bd18a914a
Author: devozerov 
Date:   2017-10-16T10:58:11Z

Reworked sort order.




---

[jira] [Created] (IGNITE-6635) Minimize H2 usages in InlineIndexHelper

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-6635:
---

 Summary: Minimize H2 usages in InlineIndexHelper
 Key: IGNITE-6635
 URL: https://issues.apache.org/jira/browse/IGNITE-6635
 Project: Ignite
  Issue Type: Task
  Security Level: Public (Viewable by anyone)
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
 Fix For: 2.4


In order to extract index from {{H2Tree}} we need to rework 
{{InlineIndexHelper}} to minimize H2 classes usage. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2857: IGNITE-6634

[devozerov]

Github user devozerov closed the pull request at:

https://github.com/apache/ignite/pull/2857


---

[GitHub] ignite pull request #2858: IGNITE-6521 Update JVM options (default and recom...

[akuramshingg]

GitHub user akuramshingg opened a pull request:

https://github.com/apache/ignite/pull/2858

IGNITE-6521 Update JVM options (default and recommendations) for better 
performance



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6521

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2858.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2858


commit 8f2490fd64f3a0935b7633c800d804e3a2c53940
Author: Alexandr Kuramshin 
Date:   2017-10-16T09:45:13Z

IGNITE-6521 Update JVM options (default and recommendations) for better 
performance




---

[GitHub] ignite pull request #2856: IGNITE-6632

[devozerov]

Github user devozerov closed the pull request at:

https://github.com/apache/ignite/pull/2856


---

Re: Avoid closing caches on client during cluster restart - determining it's the same cache

[Ilya Kasnacheev]

Hello Denis!

> we do not check CacheConfiguration settings provided by the client

Are you sure about that? I've just tried a scenario where my client got
bounced for not having matching EvictionPolicy compared to already deployed
servers.

Or do you mean that we ignore cfg in getOrCreateCaches() if name is matched
to already existing cache?

I am worried about a usability issue where a stale client occassionally
writes entries of incorrect type into a cache or uses improper atomicity
mode.

Regards,

-- 
Ilya Kasnacheev

2017-10-12 2:07 GMT+03:00 Denis Magda :

> Hello Ilya,
>
> Isn’t the cache name itself sufficient to make all the validations? The
> cache name is a unique parameter.
>
> Furthermore, we do not check CacheConfiguration settings provided by the
> client if CacheConfiguration.name is already deployed when the client tries
> to get a reference to the cache (Ignite.getOrCreateCache(cacheCfg)).
> Don’t think we should make any exception for the restart scenarios.
>
> —
> Denis
>
>
> > On Oct 11, 2017, at 5:20 AM, Ilya Kasnacheev 
> wrote:
> >
> > Hello Igniters!
> >
> > I'm working on IGNITE-2766. We want caches opened on client to survive
> > cluster restart (all nodes down, then some nodes up). Currently Ignite
> > compares deploymentId which will not match with new cluster's caches.
> >
> > But still we want to make sure it's still the same cache in the new
> > cluster, and not a different one with the same name. One obvious idea is
> to
> > look at CacheConfiguration. However I think a frequent scenario for
> cluster
> > restart is a minor change of cache configuration (e.g. add or remove
> > backup), and it will be a pity to lose client caches in this case.
> >
> > So, what fields in CacheConfiguration should we compare upon to figure
> out
> > if it's still the same cache? Name obviously. How does grpName work and
> > should we compare on them too?
> >
> > indexedTypes[] make sense because if it's a cache on different types it's
> > not safe to be used where it was open. cache mode and atomicity mode make
> > sence since cache should now be handled differently?
> >
> > Any other fields in CacheConfiguration we should check because it's not
> > safe to continue in case of mismatch? Some different approach?
> Suggestions
> > are kindly welcome
> >
> > --
> > Ilya Kasnacheev
>
>

[GitHub] ignite pull request #2857: IGNITE-6634

[devozerov]

GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/2857

IGNITE-6634



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6634

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2857.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2857


commit 593a7795738b453a6579faf078da1f4996f6f44d
Author: devozerov 
Date:   2017-10-16T09:25:32Z

Done.




---

[jira] [Created] (IGNITE-6634) SQL: move IgniteDistributedJoinTestSuite to query suite(s)

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-6634:
---

 Summary: SQL: move IgniteDistributedJoinTestSuite to query suite(s)
 Key: IGNITE-6634
 URL: https://issues.apache.org/jira/browse/IGNITE-6634
 Project: Ignite
  Issue Type: Task
  Security Level: Public (Viewable by anyone)
  Components: sql
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
 Fix For: 2.4


Currently this suite is executed as a part of continuous queries tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: integrate with prestodb

[Alexey Kukushkin]

Cross-sending to the DEV community.

On Mon, Oct 16, 2017 at 12:14 PM, shawn.du  wrote:

> Hi community,
>
> I am trying to implement a connector for presto to connect ignite.
> I think it will be a very interest thing to connect ignite and presto.
>
> In fact, currently we use ignite and it works very well.  but in order to
> save memory, we build compressed binary data.
> thus we cannot query them using SQL. We use ignite map-reduce to query the
> data.
>
> Using presto, we may use SQL again. If it is fast enough, ignite will be
> our in memory storage and not responsible for computing or only for simple
> query.
> The only thing I concern about is presto is fast enough or not like
> Ignite. For now all ignite query cost less than 5 seconds and most are
> hundreds of milliseconds.
> Also presto provides a connector for redis.  I don't know community has
> interest to contribute to presto-ignite?
>
> Thanks
> Shawn
>
>


-- 
Best regards,
Alexey

[jira] [Created] (IGNITE-6633) Repair basic SQL functionality

[Alexei Scherbakov (JIRA)]

Alexei Scherbakov created IGNITE-6633:
-

 Summary: Repair basic SQL functionality
 Key: IGNITE-6633
 URL: https://issues.apache.org/jira/browse/IGNITE-6633
 Project: Ignite
  Issue Type: Improvement
  Security Level: Public (Viewable by anyone)
  Components: sql
Reporter: Alexei Scherbakov
 Fix For: 2.4


For a long time SQL engine has known limitation (H2 related) [1]

This is huge usability issue, because proposed workaround requires query 
rewriting and is difficult to implement in some cases, e.g. when some kind of 
query builder is used.

I suggest to fix it at last.

[1] 
https://apacheignite.readme.io/v2.1/docs/sql-performance-and-debugging#sql-performance-and-usability-considerations



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2856: IGNITE-6632

[devozerov]

GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/2856

IGNITE-6632



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-6632

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2856.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2856


commit b807a7f0ff33de416287b506927188b077744109
Author: devozerov 
Date:   2017-10-16T08:02:23Z

WIP.

commit d40025def33eb46d85abb60457ce0adc110b8ac2
Author: devozerov 
Date:   2017-10-16T08:10:31Z

Key-only row.

commit da96581de2e45fcfb8fea7215f84ddb1610b327a
Author: devozerov 
Date:   2017-10-16T08:10:40Z

Key-only row.

commit 4732faa5c4ad0a636a494fcf6a5da7bdef57f7d3
Author: devozerov 
Date:   2017-10-16T08:14:05Z

It compiles.

commit fe602efca5b873397cd956fad0b9e73ef4e0d603
Author: devozerov 
Date:   2017-10-16T08:21:23Z

Hidden public fields.




---

[jira] [Created] (IGNITE-6632) SQL: simplify GridH2Row infrastructure

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-6632:
---

 Summary: SQL: simplify GridH2Row infrastructure
 Key: IGNITE-6632
 URL: https://issues.apache.org/jira/browse/IGNITE-6632
 Project: Ignite
  Issue Type: Task
  Security Level: Public (Viewable by anyone)
  Components: sql
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
 Fix For: 2.4


Currently we have a lot of {{GridH2Row}} implementations. But in reality we 
need only two - for normal row and for deleted row. 

Need to refactor the rest rows so that they do not extend {{GridH2Row}}. 
Without it we will not be able to decouple H2 index from H2 infrastructure.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #2855: IGNITE-6631

[devozerov]

Github user devozerov closed the pull request at:

https://github.com/apache/ignite/pull/2855


---

Re: Optimized DML execution: how to name it?

[Vladimir Ozerov]

Dmitriy,

No :-) System properties is bad practice, because in some cases you cannot
change them. Instead, this flag is available in private class
SqlFIeldsQueryEx. So in case user desperately need it, we may advise him to
use this class. But I hope it will not be needed.

On Sun, Oct 15, 2017 at 5:09 PM, Dmitriy Setrakyan 
wrote:

> On Sun, Oct 15, 2017 at 12:27 AM, Vladimir Ozerov 
> wrote:
>
> > No, it is not exposed to public API. Didn't want to add another flag to
> > SqlFieldsQuery.
> >
>
> I think we should add a system property. This way you don't have to add any
> API flags. Agree?
>