Re: [DISCUSS] Webinar for Ignite Persistent Store walk-through

[Evans Ye]

+1

2017-06-10 6:19 GMT+08:00 Valentin Kulichenko :

> +1
>
> On Fri, Jun 9, 2017 at 3:15 PM, Andrey Mashenkov <
> andrey.mashen...@gmail.com
> > wrote:
>
> > +1
> >
> > 10 июня 2017 г. 0:08 пользователь "William Do" 
> > написал:
> >
> > > +1
> > >
> > > On 9 June 2017 at 21:37, Dmitriy Setrakyan 
> > wrote:
> > >
> > > > Hm... we have only 3 community members who are interested so far.
> > Anyone
> > > > else who may be willing to attend?
> > > >
> > > > On Fri, Jun 9, 2017 at 12:03 AM, Sergi Vladykin <
> > > sergi.vlady...@gmail.com>
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Sergi
> > > > >
> > > > > 2017-06-08 23:03 GMT+03:00 Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >:
> > > > >
> > > > > > +1 (I will attend)
> > > > > >
> > > > > > On Thu, Jun 8, 2017 at 1:02 PM, Konstantin Boudnik <
> c...@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > > > That'd be great! Thank you!
> > > > > > > --
> > > > > > >   Take care,
> > > > > > > Konstantin (Cos) Boudnik
> > > > > > > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> > > > > > >
> > > > > > > Disclaimer: Opinions expressed in this email are those of the
> > > author,
> > > > > > > and do not necessarily represent the views of any company the
> > > author
> > > > > > > might be affiliated with at the moment of writing.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jun 8, 2017 at 12:54 PM, Denis Magda <
> dma...@apache.org>
> > > > > wrote:
> > > > > > > > Igniters,
> > > > > > > >
> > > > > > > > What’d you think if we arrange an internal webinar for our
> > > > community
> > > > > to
> > > > > > > walk through the features, capabilities and implementation
> > details
> > > of
> > > > > the
> > > > > > > Ignite Persistent Store [1]? That should help us understanding
> > the
> > > > > > donation
> > > > > > > better.
> > > > > > > >
> > > > > > > > Please reply if you will be happy to attend.
> > > > > > > >
> > > > > > > > [1] https://apacheignite.readme.io/docs/distributed-
> > > > persistent-store
> > > > > <
> > > > > > > https://apacheignite.readme.io/docs/distributed-
> persistent-store
> > >
> > > > > > > >
> > > > > > > > —
> > > > > > > > Denis
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Webinar for Ignite Persistent Store walk-through

[Valentin Kulichenko]

+1

On Fri, Jun 9, 2017 at 3:15 PM, Andrey Mashenkov  wrote:

> +1
>
> 10 июня 2017 г. 0:08 пользователь "William Do" 
> написал:
>
> > +1
> >
> > On 9 June 2017 at 21:37, Dmitriy Setrakyan 
> wrote:
> >
> > > Hm... we have only 3 community members who are interested so far.
> Anyone
> > > else who may be willing to attend?
> > >
> > > On Fri, Jun 9, 2017 at 12:03 AM, Sergi Vladykin <
> > sergi.vlady...@gmail.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > Sergi
> > > >
> > > > 2017-06-08 23:03 GMT+03:00 Dmitriy Setrakyan  >:
> > > >
> > > > > +1 (I will attend)
> > > > >
> > > > > On Thu, Jun 8, 2017 at 1:02 PM, Konstantin Boudnik  >
> > > > wrote:
> > > > >
> > > > > > That'd be great! Thank you!
> > > > > > --
> > > > > >   Take care,
> > > > > > Konstantin (Cos) Boudnik
> > > > > > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> > > > > >
> > > > > > Disclaimer: Opinions expressed in this email are those of the
> > author,
> > > > > > and do not necessarily represent the views of any company the
> > author
> > > > > > might be affiliated with at the moment of writing.
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 8, 2017 at 12:54 PM, Denis Magda 
> > > > wrote:
> > > > > > > Igniters,
> > > > > > >
> > > > > > > What’d you think if we arrange an internal webinar for our
> > > community
> > > > to
> > > > > > walk through the features, capabilities and implementation
> details
> > of
> > > > the
> > > > > > Ignite Persistent Store [1]? That should help us understanding
> the
> > > > > donation
> > > > > > better.
> > > > > > >
> > > > > > > Please reply if you will be happy to attend.
> > > > > > >
> > > > > > > [1] https://apacheignite.readme.io/docs/distributed-
> > > persistent-store
> > > > <
> > > > > > https://apacheignite.readme.io/docs/distributed-persistent-store
> >
> > > > > > >
> > > > > > > —
> > > > > > > Denis
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Webinar for Ignite Persistent Store walk-through

[Andrey Mashenkov]

+1

10 июня 2017 г. 0:08 пользователь "William Do" 
написал:

> +1
>
> On 9 June 2017 at 21:37, Dmitriy Setrakyan  wrote:
>
> > Hm... we have only 3 community members who are interested so far. Anyone
> > else who may be willing to attend?
> >
> > On Fri, Jun 9, 2017 at 12:03 AM, Sergi Vladykin <
> sergi.vlady...@gmail.com>
> > wrote:
> >
> > > +1
> > >
> > > Sergi
> > >
> > > 2017-06-08 23:03 GMT+03:00 Dmitriy Setrakyan :
> > >
> > > > +1 (I will attend)
> > > >
> > > > On Thu, Jun 8, 2017 at 1:02 PM, Konstantin Boudnik 
> > > wrote:
> > > >
> > > > > That'd be great! Thank you!
> > > > > --
> > > > >   Take care,
> > > > > Konstantin (Cos) Boudnik
> > > > > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> > > > >
> > > > > Disclaimer: Opinions expressed in this email are those of the
> author,
> > > > > and do not necessarily represent the views of any company the
> author
> > > > > might be affiliated with at the moment of writing.
> > > > >
> > > > >
> > > > > On Thu, Jun 8, 2017 at 12:54 PM, Denis Magda 
> > > wrote:
> > > > > > Igniters,
> > > > > >
> > > > > > What’d you think if we arrange an internal webinar for our
> > community
> > > to
> > > > > walk through the features, capabilities and implementation details
> of
> > > the
> > > > > Ignite Persistent Store [1]? That should help us understanding the
> > > > donation
> > > > > better.
> > > > > >
> > > > > > Please reply if you will be happy to attend.
> > > > > >
> > > > > > [1] https://apacheignite.readme.io/docs/distributed-
> > persistent-store
> > > <
> > > > > https://apacheignite.readme.io/docs/distributed-persistent-store>
> > > > > >
> > > > > > —
> > > > > > Denis
> > > > >
> > > >
> > >
> >
>

Re: Configuration issues with IGFS in Ignite 1.9

[Konstantin Boudnik]

Sorry, please disregard the last one - that's my omission. service.sh
script is still fine!
--
  Take care,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Fri, Jun 9, 2017 at 12:19 PM, Konstantin Boudnik  wrote:
> Vladimir,
>
> one more issue - it seems that ignite script (bin/ignite.sh) doesn't
> respond to 'stop' command anymore as it was the case in the earlier
> versions of it. As the result, linux init scripts have no control over
> it rather than simply kill the process. Is it the intention or am I
> missing something?
>
> Thank you very much!
>   Cos
>
> --
>   Take care,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
>
>
> On Fri, Jun 9, 2017 at 11:05 AM, Konstantin Boudnik  wrote:
>> Yey! That did the trick, thank you for your help! It'd be great to
>> have this fix for sure!
>> --
>>   Take care,
>> Konstantin (Cos) Boudnik
>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>
>> Disclaimer: Opinions expressed in this email are those of the author,
>> and do not necessarily represent the views of any company the author
>> might be affiliated with at the moment of writing.
>>
>>
>> On Fri, Jun 9, 2017 at 6:41 AM, Vladimir Ozerov  wrote:
>>> Cos,
>>>
>>> This is a problem with our URI parsing. Please add a slash to the end and
>>> it should work: "hdfs://0f6e4ed13002.bigtop.apache.org:8020*/*". I'll
>>> create a ticket for this.
>>>
>>> On Thu, Jun 8, 2017 at 9:28 PM, Konstantin Boudnik  wrote:
>>>
 Thank you for your help, Vladimir. Unfortunately, I am still seeing
 the same behavior while attempting to bring up the service. The full
 element's configuration could be found at [1]
 Can we  perhaps get on a skype/hangout for a few minutes so we can
 resolve this quickly? Unless, I am doing something silly and missing
 it, of course.

 That'd be a great help!

 [1] https://gist.github.com/c0s/cfb2b78a87f4f7f3ef4c1fa0bb5c4dea

 --
   Take care,
 Konstantin (Cos) Boudnik
 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

 Disclaimer: Opinions expressed in this email are those of the author,
 and do not necessarily represent the views of any company the author
 might be affiliated with at the moment of writing.


 On Thu, Jun 8, 2017 at 1:23 AM, Vladimir Ozerov 
 wrote:
 > Hi Cos,
 >
 > Please try this one:
 >
 > 
 > >>> > class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
 > 
 > >>> > class="org.apache.ignite.hadoop.fs.CachingHadoopFileSystemFactory">
 > 
 > 
 > 
 > /etc/hadoop/conf/core-site.xml
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 >
 > Vladimir.
 >
 > On Wed, Jun 7, 2017 at 11:40 PM, Konstantin Boudnik 
 wrote:
 >
 >> Guys,
 >>
 >> after upgrading the Bigtop stack from Ignite 1.5 to 1.9 we are seeing
 >> a number of issues introduced by changed script interfaces and
 >> similar. However, the one which I still can not fix so far is this.
 >> The secondary file system is configured like this:
 >>
 >> 
 >>   
 >> >>> >> parent="igfsCfgBase">
 >>   
 >>
 >>   
 >>   
 >>   
 >>
 >>   
 >>   
 >> 
 >>   
 >>   
 >>   
 >> 
 >>   
 >>
 >>   
 >> >>> >> class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileS
 ystem">
 >>   >>> >> value='hdfs://0f6e4ed13002.bigtop.apache.org:8020'/>
 >>   >>> >> value='/etc/hadoop/conf/core-site.xml'/>
 >>   
 >> 
 >>   
 >>
 >> And we are getting the following error message:
 >>
 >> Caused by: java.lang.IllegalArgumentException: Can not create a Path
 >> from an empty string
 >> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:126)
 >> at org.apache.hadoop.fs.Path.(Path.java:134)
 >> at org.apache.ignite.internal.processors.hadoop.impl.delegate.
 >> HadoopBasicFileSystemFactoryDelegate.start(
 HadoopBasicFileSystemFactoryDe
 >> 

Re: [DISCUSS] Webinar for Ignite Persistent Store walk-through

[Dmitriy Setrakyan]

Hm... we have only 3 community members who are interested so far. Anyone
else who may be willing to attend?

On Fri, Jun 9, 2017 at 12:03 AM, Sergi Vladykin 
wrote:

> +1
>
> Sergi
>
> 2017-06-08 23:03 GMT+03:00 Dmitriy Setrakyan :
>
> > +1 (I will attend)
> >
> > On Thu, Jun 8, 2017 at 1:02 PM, Konstantin Boudnik 
> wrote:
> >
> > > That'd be great! Thank you!
> > > --
> > >   Take care,
> > > Konstantin (Cos) Boudnik
> > > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> > >
> > > Disclaimer: Opinions expressed in this email are those of the author,
> > > and do not necessarily represent the views of any company the author
> > > might be affiliated with at the moment of writing.
> > >
> > >
> > > On Thu, Jun 8, 2017 at 12:54 PM, Denis Magda 
> wrote:
> > > > Igniters,
> > > >
> > > > What’d you think if we arrange an internal webinar for our community
> to
> > > walk through the features, capabilities and implementation details of
> the
> > > Ignite Persistent Store [1]? That should help us understanding the
> > donation
> > > better.
> > > >
> > > > Please reply if you will be happy to attend.
> > > >
> > > > [1] https://apacheignite.readme.io/docs/distributed-persistent-store
> <
> > > https://apacheignite.readme.io/docs/distributed-persistent-store>
> > > >
> > > > —
> > > > Denis
> > >
> >
>

Re: Configuration issues with IGFS in Ignite 1.9

[Konstantin Boudnik]

Vladimir,

one more issue - it seems that ignite script (bin/ignite.sh) doesn't
respond to 'stop' command anymore as it was the case in the earlier
versions of it. As the result, linux init scripts have no control over
it rather than simply kill the process. Is it the intention or am I
missing something?

Thank you very much!
  Cos

--
  Take care,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Fri, Jun 9, 2017 at 11:05 AM, Konstantin Boudnik  wrote:
> Yey! That did the trick, thank you for your help! It'd be great to
> have this fix for sure!
> --
>   Take care,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
>
>
> On Fri, Jun 9, 2017 at 6:41 AM, Vladimir Ozerov  wrote:
>> Cos,
>>
>> This is a problem with our URI parsing. Please add a slash to the end and
>> it should work: "hdfs://0f6e4ed13002.bigtop.apache.org:8020*/*". I'll
>> create a ticket for this.
>>
>> On Thu, Jun 8, 2017 at 9:28 PM, Konstantin Boudnik  wrote:
>>
>>> Thank you for your help, Vladimir. Unfortunately, I am still seeing
>>> the same behavior while attempting to bring up the service. The full
>>> element's configuration could be found at [1]
>>> Can we  perhaps get on a skype/hangout for a few minutes so we can
>>> resolve this quickly? Unless, I am doing something silly and missing
>>> it, of course.
>>>
>>> That'd be a great help!
>>>
>>> [1] https://gist.github.com/c0s/cfb2b78a87f4f7f3ef4c1fa0bb5c4dea
>>>
>>> --
>>>   Take care,
>>> Konstantin (Cos) Boudnik
>>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>>
>>> Disclaimer: Opinions expressed in this email are those of the author,
>>> and do not necessarily represent the views of any company the author
>>> might be affiliated with at the moment of writing.
>>>
>>>
>>> On Thu, Jun 8, 2017 at 1:23 AM, Vladimir Ozerov 
>>> wrote:
>>> > Hi Cos,
>>> >
>>> > Please try this one:
>>> >
>>> > 
>>> > >> > class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
>>> > 
>>> > >> > class="org.apache.ignite.hadoop.fs.CachingHadoopFileSystemFactory">
>>> > 
>>> > 
>>> > 
>>> > /etc/hadoop/conf/core-site.xml
>>> > 
>>> > 
>>> > 
>>> > 
>>> > 
>>> > 
>>> > 
>>> >
>>> > Vladimir.
>>> >
>>> > On Wed, Jun 7, 2017 at 11:40 PM, Konstantin Boudnik 
>>> wrote:
>>> >
>>> >> Guys,
>>> >>
>>> >> after upgrading the Bigtop stack from Ignite 1.5 to 1.9 we are seeing
>>> >> a number of issues introduced by changed script interfaces and
>>> >> similar. However, the one which I still can not fix so far is this.
>>> >> The secondary file system is configured like this:
>>> >>
>>> >> 
>>> >>   
>>> >> >> >> parent="igfsCfgBase">
>>> >>   
>>> >>
>>> >>   
>>> >>   
>>> >>   
>>> >>
>>> >>   
>>> >>   
>>> >> 
>>> >>   
>>> >>   
>>> >>   
>>> >> 
>>> >>   
>>> >>
>>> >>   
>>> >> >> >> class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileS
>>> ystem">
>>> >>   >> >> value='hdfs://0f6e4ed13002.bigtop.apache.org:8020'/>
>>> >>   >> >> value='/etc/hadoop/conf/core-site.xml'/>
>>> >>   
>>> >> 
>>> >>   
>>> >>
>>> >> And we are getting the following error message:
>>> >>
>>> >> Caused by: java.lang.IllegalArgumentException: Can not create a Path
>>> >> from an empty string
>>> >> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:126)
>>> >> at org.apache.hadoop.fs.Path.(Path.java:134)
>>> >> at org.apache.ignite.internal.processors.hadoop.impl.delegate.
>>> >> HadoopBasicFileSystemFactoryDelegate.start(
>>> HadoopBasicFileSystemFactoryDe
>>> >> legate
>>> >> .java:165)
>>> >> at org.apache.ignite.internal.processors.hadoop.impl.delegate.
>>> >> HadoopCachingFileSystemFactoryDelegate.start(
>>> >> HadoopCachingFileSystemFactoryDele
>>> >> gate.java:58)
>>> >> at org.apache.ignite.internal.processors.hadoop.impl.delegate.
>>> >> HadoopIgfsSecondaryFileSystemDelegateImpl.start(
>>> >> HadoopIgfsSecondaryFileSystemDe
>>> >> legateImpl.java:375)
>>> >> at org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileS
>>> >> ystem.start(IgniteHadoopIgfsSecondaryFileSystem.java:261)
>>> >> at 

[jira] [Created] (IGNITE-5467) Exception in communication SPI can stall the cluster

[Valentin Kulichenko (JIRA)]

Valentin Kulichenko created IGNITE-5467:
---

 Summary: Exception in communication SPI can stall the cluster
 Key: IGNITE-5467
 URL: https://issues.apache.org/jira/browse/IGNITE-5467
 Project: Ignite
  Issue Type: Bug
  Components: general
Affects Versions: 2.0
Reporter: Valentin Kulichenko
 Fix For: 2.1


Test attached.

If there is an exception in {{CommunicationSpi#sendMessage}} while sending 
response for a cache operation, it can stall whole cluster forever (operation 
doesn't complete, new nodes can't join, etc.).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Configuration issues with IGFS in Ignite 1.9

[Konstantin Boudnik]

Yey! That did the trick, thank you for your help! It'd be great to
have this fix for sure!
--
  Take care,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Fri, Jun 9, 2017 at 6:41 AM, Vladimir Ozerov  wrote:
> Cos,
>
> This is a problem with our URI parsing. Please add a slash to the end and
> it should work: "hdfs://0f6e4ed13002.bigtop.apache.org:8020*/*". I'll
> create a ticket for this.
>
> On Thu, Jun 8, 2017 at 9:28 PM, Konstantin Boudnik  wrote:
>
>> Thank you for your help, Vladimir. Unfortunately, I am still seeing
>> the same behavior while attempting to bring up the service. The full
>> element's configuration could be found at [1]
>> Can we  perhaps get on a skype/hangout for a few minutes so we can
>> resolve this quickly? Unless, I am doing something silly and missing
>> it, of course.
>>
>> That'd be a great help!
>>
>> [1] https://gist.github.com/c0s/cfb2b78a87f4f7f3ef4c1fa0bb5c4dea
>>
>> --
>>   Take care,
>> Konstantin (Cos) Boudnik
>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>
>> Disclaimer: Opinions expressed in this email are those of the author,
>> and do not necessarily represent the views of any company the author
>> might be affiliated with at the moment of writing.
>>
>>
>> On Thu, Jun 8, 2017 at 1:23 AM, Vladimir Ozerov 
>> wrote:
>> > Hi Cos,
>> >
>> > Please try this one:
>> >
>> > 
>> > > > class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
>> > 
>> > > > class="org.apache.ignite.hadoop.fs.CachingHadoopFileSystemFactory">
>> > 
>> > 
>> > 
>> > /etc/hadoop/conf/core-site.xml
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> > 
>> >
>> > Vladimir.
>> >
>> > On Wed, Jun 7, 2017 at 11:40 PM, Konstantin Boudnik 
>> wrote:
>> >
>> >> Guys,
>> >>
>> >> after upgrading the Bigtop stack from Ignite 1.5 to 1.9 we are seeing
>> >> a number of issues introduced by changed script interfaces and
>> >> similar. However, the one which I still can not fix so far is this.
>> >> The secondary file system is configured like this:
>> >>
>> >> 
>> >>   
>> >> > >> parent="igfsCfgBase">
>> >>   
>> >>
>> >>   
>> >>   
>> >>   
>> >>
>> >>   
>> >>   
>> >> 
>> >>   
>> >>   
>> >>   
>> >> 
>> >>   
>> >>
>> >>   
>> >> > >> class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileS
>> ystem">
>> >>   > >> value='hdfs://0f6e4ed13002.bigtop.apache.org:8020'/>
>> >>   > >> value='/etc/hadoop/conf/core-site.xml'/>
>> >>   
>> >> 
>> >>   
>> >>
>> >> And we are getting the following error message:
>> >>
>> >> Caused by: java.lang.IllegalArgumentException: Can not create a Path
>> >> from an empty string
>> >> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:126)
>> >> at org.apache.hadoop.fs.Path.(Path.java:134)
>> >> at org.apache.ignite.internal.processors.hadoop.impl.delegate.
>> >> HadoopBasicFileSystemFactoryDelegate.start(
>> HadoopBasicFileSystemFactoryDe
>> >> legate
>> >> .java:165)
>> >> at org.apache.ignite.internal.processors.hadoop.impl.delegate.
>> >> HadoopCachingFileSystemFactoryDelegate.start(
>> >> HadoopCachingFileSystemFactoryDele
>> >> gate.java:58)
>> >> at org.apache.ignite.internal.processors.hadoop.impl.delegate.
>> >> HadoopIgfsSecondaryFileSystemDelegateImpl.start(
>> >> HadoopIgfsSecondaryFileSystemDe
>> >> legateImpl.java:375)
>> >> at org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileS
>> >> ystem.start(IgniteHadoopIgfsSecondaryFileSystem.java:261)
>> >> at org.apache.ignite.internal.processors.igfs.IgfsImpl.<
>> >> init>(IgfsImpl.java:186)
>> >> at org.apache.ignite.internal.processors.igfs.IgfsContext.<
>> >> init>(IgfsContext.java:101)
>> >> at org.apache.ignite.internal.processors.igfs.IgfsProcessor.
>> >> start(IgfsProcessor.java:130)
>> >> at org.apache.ignite.internal.IgniteKernal.startProcessor(
>> >> IgniteKernal.java:1644)
>> >> at org.apache.ignite.internal.IgniteKernal.start(
>> >> IgniteKernal.java:903)
>> >> ... 10 more
>> >> Failed to start grid: Can not create a Path from an empty string
>> >>
>> >> Removing the value from cfgPath doesn't change anything. I'd
>> >> appreciate if someone can point me to the right direction: current
>> >> config examples aren't helping much either. Thanks!
>> >> --
>> >>   Take care,
>> >> Konstantin (Cos) Boudnik
>> >> 2CAC 

[jira] [Created] (IGNITE-5466) Web Console: New Configuration Screen

[Vica Abramova (JIRA)]

Vica Abramova created IGNITE-5466:
-

 Summary: Web Console: New Configuration Screen
 Key: IGNITE-5466
 URL: https://issues.apache.org/jira/browse/IGNITE-5466
 Project: Ignite
  Issue Type: Task
  Components: UI, wizards
Reporter: Vica Abramova
Assignee: Ilya Borisov






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] ignite pull request #2108: IGNITE-5455 .NET: Fix binary hash code calculatio...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2108


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-5465) CREATE TABLE: add multithreaded tests

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-5465:
---

 Summary: CREATE TABLE: add multithreaded tests
 Key: IGNITE-5465
 URL: https://issues.apache.org/jira/browse/IGNITE-5465
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Vladimir Ozerov
Assignee: Alexander Paschenko
 Fix For: 2.1


We need to add multithreaded tests for {{CREATE TABLE}} and {{DROP TABLE}} 
commands in a similar way as we did that for indexes 
({{DynamicIndexAbstractConcurrentSelfTest}}. 

Let's test the following case:
- Create and drop of the same several tables from different threads and 
different nodes (including a client). 
- Success criteria: nothing hangs and in the end table/cache is either doesn't 
exist, or exists and operational (verify through DML and SELECT queries).

AFAIK there is a bug in how we start caches dynamically, so honestly there is a 
high chance that this test will not work as expected due to non-SQL bugs. So at 
the very least we must ensure that it either works, or doesn't work due to a 
problem outside of DDL engine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-5464) DML: Can not insert arrays

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-5464:
--

 Summary: DML: Can not insert arrays
 Key: IGNITE-5464
 URL: https://issues.apache.org/jira/browse/IGNITE-5464
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.0
Reporter: Pavel Tupitsyn
 Fix For: 2.1


Reproducer:
{code}
QueryEntity queryEntity = new QueryEntity(Integer[].class.getName(), 
String.class.getName());
queryEntity.setTableName("ints");
CacheConfiguration ccfg = new 
CacheConfiguration().setName("foo").setQueryEntities(Arrays.asList(queryEntity));
IgniteCache cache = ignite.createCache(ccfg);

cache.put(1, new Integer[] {1, 2}); // works

cache.query(new SqlFieldsQuery("insert into ints (_key, _val) values 
(?, ?)")
.setArgs(1, new Integer[] {1, 2})).getAll();  // throws

{code}

Exception:
{code}
Exception in thread "main" javax.cache.CacheException: class 
org.apache.ignite.IgniteException: Failed to execute SQL query.
at 
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:818)
at TestDml.main(TestDml.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: class org.apache.ignite.IgniteException: Failed to execute SQL query.
at 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor$3.iterator(DmlStatementsProcessor.java:356)
at 
org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator(QueryCursorImpl.java:94)
at 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.doInsert(DmlStatementsProcessor.java:827)
at 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.executeUpdateStatement(DmlStatementsProcessor.java:369)
at 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.updateSqlFields(DmlStatementsProcessor.java:164)
at 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.updateSqlFieldsTwoStep(DmlStatementsProcessor.java:198)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryTwoStep(IgniteH2Indexing.java:1659)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:1659)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor$4.applyx(GridQueryProcessor.java:1657)
at 
org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:36)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:2103)
at 
org.apache.ignite.internal.processors.query.GridQueryProcessor.queryTwoStep(GridQueryProcessor.java:1657)
at 
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:806)
... 6 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to execute 
SQL query.
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:1226)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:1278)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.access$1300(IgniteH2Indexing.java:241)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$4.iterator(IgniteH2Indexing.java:1081)
at 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$4.iterator(IgniteH2Indexing.java:1069)
at 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor$3.iterator(DmlStatementsProcessor.java:353)
... 18 more
Caused by: org.h2.jdbc.JdbcSQLException: Hexadecimal string with odd number of 
characters: "1"; SQL statement:
SELECT
TABLE._KEY,
TABLE._VAL
FROM TABLE(_KEY OTHER=(?1,), _VAL VARCHAR=(?2,)) [90003-195]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
at org.h2.message.DbException.get(DbException.java:179)
at org.h2.message.DbException.get(DbException.java:155)
at org.h2.util.StringUtils.convertHexToBytes(StringUtils.java:930)
at org.h2.value.Value.convertTo(Value.java:957)
at org.h2.table.Column.convert(Column.java:167)
at org.h2.expression.TableFunction.getTable(TableFunction.java:118)
at org.h2.expression.TableFunction.getValue(TableFunction.java:41)
at org.h2.table.FunctionTable.getValueResultSet(FunctionTable.java:218)
at 

[jira] [Created] (IGNITE-5463) Weird CPU load reported

[Yakov Zhdanov (JIRA)]

Yakov Zhdanov created IGNITE-5463:
-

 Summary: Weird CPU load reported
 Key: IGNITE-5463
 URL: https://issues.apache.org/jira/browse/IGNITE-5463
 Project: Ignite
  Issue Type: Bug
Reporter: Yakov Zhdanov


{noformat}
2017-06-09 16:27:55,827 INFO [IgniteKernal%187358542614215] - 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=56495ffd, name=187358542614215, uptime=00:46:42:936]
^-- H/N/C [hosts=1, nodes=1, CPUs=4]
^-- CPU [cur=100%, avg=12.85%, GC=125.47%]
^-- PageMemory [pages=314674]
^-- Heap [used=4777MB, free=12.52%, comm=5461MB]
^-- Non heap [used=71MB, free=-1%, comm=77MB]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=1, qSize=0]
^-- Outbound messages queue [size=0]
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-5462) JDBC thin driver: add complex DDL+DML test.

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-5462:


 Summary: JDBC thin driver: add complex DDL+DML test.
 Key: IGNITE-5462
 URL: https://issues.apache.org/jira/browse/IGNITE-5462
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.0
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.2


We need a test that will test data flow behavior in chain of DML+DDL operations 
as a whole through JDBC interface.

Related issue for similar test through pure Ignite API: IGNITE-5449



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-5461) Visor show wrong statistics for off heap memory

[Mikhail Cherkasov (JIRA)]

Mikhail Cherkasov created IGNITE-5461:
-

 Summary: Visor show wrong statistics for off heap memory
 Key: IGNITE-5461
 URL: https://issues.apache.org/jira/browse/IGNITE-5461
 Project: Ignite
  Issue Type: Bug
Reporter: Mikhail Cherkasov
Assignee: Alexey Kuznetsov


Visor show that data is stored in Heap, while the data is in off heap:

Total: 1
Heap: 1
Off-Heap: 0
Off-Heap Memory: 0

while:
cache.localPeek("Key1", ONHEAP) == null
cache.localPeek("Key1", OFFHEAP) == Value

reproducer is attached.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Configuration issues with IGFS in Ignite 1.9

[Vladimir Ozerov]

Cos,

This is a problem with our URI parsing. Please add a slash to the end and
it should work: "hdfs://0f6e4ed13002.bigtop.apache.org:8020*/*". I'll
create a ticket for this.

On Thu, Jun 8, 2017 at 9:28 PM, Konstantin Boudnik  wrote:

> Thank you for your help, Vladimir. Unfortunately, I am still seeing
> the same behavior while attempting to bring up the service. The full
> element's configuration could be found at [1]
> Can we  perhaps get on a skype/hangout for a few minutes so we can
> resolve this quickly? Unless, I am doing something silly and missing
> it, of course.
>
> That'd be a great help!
>
> [1] https://gist.github.com/c0s/cfb2b78a87f4f7f3ef4c1fa0bb5c4dea
>
> --
>   Take care,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
>
>
> On Thu, Jun 8, 2017 at 1:23 AM, Vladimir Ozerov 
> wrote:
> > Hi Cos,
> >
> > Please try this one:
> >
> > 
> >  > class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
> > 
> >  > class="org.apache.ignite.hadoop.fs.CachingHadoopFileSystemFactory">
> > 
> > 
> > 
> > /etc/hadoop/conf/core-site.xml
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >
> > Vladimir.
> >
> > On Wed, Jun 7, 2017 at 11:40 PM, Konstantin Boudnik 
> wrote:
> >
> >> Guys,
> >>
> >> after upgrading the Bigtop stack from Ignite 1.5 to 1.9 we are seeing
> >> a number of issues introduced by changed script interfaces and
> >> similar. However, the one which I still can not fix so far is this.
> >> The secondary file system is configured like this:
> >>
> >> 
> >>   
> >>  >> parent="igfsCfgBase">
> >>   
> >>
> >>   
> >>   
> >>   
> >>
> >>   
> >>   
> >> 
> >>   
> >>   
> >>   
> >> 
> >>   
> >>
> >>   
> >>  >> class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileS
> ystem">
> >>>> value='hdfs://0f6e4ed13002.bigtop.apache.org:8020'/>
> >>>> value='/etc/hadoop/conf/core-site.xml'/>
> >>   
> >> 
> >>   
> >>
> >> And we are getting the following error message:
> >>
> >> Caused by: java.lang.IllegalArgumentException: Can not create a Path
> >> from an empty string
> >> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:126)
> >> at org.apache.hadoop.fs.Path.(Path.java:134)
> >> at org.apache.ignite.internal.processors.hadoop.impl.delegate.
> >> HadoopBasicFileSystemFactoryDelegate.start(
> HadoopBasicFileSystemFactoryDe
> >> legate
> >> .java:165)
> >> at org.apache.ignite.internal.processors.hadoop.impl.delegate.
> >> HadoopCachingFileSystemFactoryDelegate.start(
> >> HadoopCachingFileSystemFactoryDele
> >> gate.java:58)
> >> at org.apache.ignite.internal.processors.hadoop.impl.delegate.
> >> HadoopIgfsSecondaryFileSystemDelegateImpl.start(
> >> HadoopIgfsSecondaryFileSystemDe
> >> legateImpl.java:375)
> >> at org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileS
> >> ystem.start(IgniteHadoopIgfsSecondaryFileSystem.java:261)
> >> at org.apache.ignite.internal.processors.igfs.IgfsImpl.<
> >> init>(IgfsImpl.java:186)
> >> at org.apache.ignite.internal.processors.igfs.IgfsContext.<
> >> init>(IgfsContext.java:101)
> >> at org.apache.ignite.internal.processors.igfs.IgfsProcessor.
> >> start(IgfsProcessor.java:130)
> >> at org.apache.ignite.internal.IgniteKernal.startProcessor(
> >> IgniteKernal.java:1644)
> >> at org.apache.ignite.internal.IgniteKernal.start(
> >> IgniteKernal.java:903)
> >> ... 10 more
> >> Failed to start grid: Can not create a Path from an empty string
> >>
> >> Removing the value from cfgPath doesn't change anything. I'd
> >> appreciate if someone can point me to the right direction: current
> >> config examples aren't helping much either. Thanks!
> >> --
> >>   Take care,
> >> Konstantin (Cos) Boudnik
> >> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >>
> >> Disclaimer: Opinions expressed in this email are those of the author,
> >> and do not necessarily represent the views of any company the author
> >> might be affiliated with at the moment of writing.
> >>
>

[GitHub] ignite pull request #2115: IGNITE-5204: Fixed NPE on certain data with index...

[skalashnikov]

GitHub user skalashnikov opened a pull request:

https://github.com/apache/ignite/pull/2115

IGNITE-5204: Fixed NPE on certain data with index inlining



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5204

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2115.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2115


commit 62aa323162ee7d195574b771c47dfb8bbc5ed58d
Author: skalashnikov 
Date:   2017-06-09T13:31:48Z

IGNITE-5204: Fixed NPE on certain data with index inlining




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2114: IGNITE-5449

[alexpaschenko]

GitHub user alexpaschenko opened a pull request:

https://github.com/apache/ignite/pull/2114

IGNITE-5449



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5449

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2114.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2114


commit 88cccd006995c5806192c8219e287ee7c8c7d45f
Author: Alexander Paschenko 
Date:   2017-06-09T04:34:34Z

Test

commit 9a1f51fd58cc40098312bbe82402dd2bab35a5f2
Author: Alexander Paschenko 
Date:   2017-06-09T13:26:49Z

IGNITE-5449 DML+DDL complex test.

commit 091d6dfe81ab02b97067dc05bb6a30a994248116
Author: Alexander Paschenko 
Date:   2017-06-09T13:27:45Z

Merge branch 'ignite-5499' into ignite-5449




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2103: IGNITE-5431 Better handling of cache SQL flag inc...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2103


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #2113: IGNITE-5458

[devozerov]

GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/2113

IGNITE-5458



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5458

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2113.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2113


commit d52223aaa212f42fdccbba5262d3925715bd445d
Author: devozerov 
Date:   2017-06-08T11:51:40Z

WIP.

commit c765f54f387177c5d2d76c0b47a0bb2a3d5d3c1a
Author: devozerov 
Date:   2017-06-09T07:11:36Z

Merge branch 'master' into vozerov-affinity

commit 4269eee2a43674b4a24b81ee3b09210b9c0eb202
Author: devozerov 
Date:   2017-06-09T07:28:34Z

WIP.

commit 54afab2b9b934e9dae96c5eb095ab4a344633b34
Author: devozerov 
Date:   2017-06-09T07:35:48Z

Returning old mappings.

commit 78276a21878559e14b8cf9d0407cda62aebf1bb0
Author: devozerov 
Date:   2017-06-09T07:38:54Z

WIP.

commit bc4a4c7774a1d1be2a2437fcfb241b5d6c966cd9
Author: devozerov 
Date:   2017-06-09T07:40:39Z

WIP.

commit 53011359a1ccc85825ff09fbb28be2e57e6c78d6
Author: devozerov 
Date:   2017-06-09T07:42:04Z

WIP.

commit 4c7e3653eb3d7abef843b2f9bab1da9519660d72
Author: devozerov 
Date:   2017-06-09T07:59:42Z

WIP.

commit bc952483e626c6599faad41f2b3bb30162f407bb
Author: devozerov 
Date:   2017-06-09T09:30:40Z

WIP.

commit ae7bd7c1767fbd7018cd193a235aea8cabb8ee11
Author: devozerov 
Date:   2017-06-09T09:44:14Z

WIP.

commit 36265d1dbf91b4a795b324626aa5ce94add6b7c7
Author: devozerov 
Date:   2017-06-09T09:45:49Z

Property.

commit 5bb8fb008e7365d2b3b489bb5e047a1dc9be9038
Author: devozerov 
Date:   2017-06-09T09:48:54Z

Removed affinitydynamic table affinity mapper.

commit 38597d1910a2ddaaf65c81e71d7e86609e20a8b2
Author: devozerov 
Date:   2017-06-09T09:50:40Z

MInors.

commit 6cb550053e95d3529771bb16ccf648f3c0066720
Author: devozerov 
Date:   2017-06-09T09:53:24Z

WIP.

commit 72dfc1d86bdef2e7e620967f1f025198fff16143
Author: devozerov 
Date:   2017-06-09T10:22:43Z

WIP.

commit 386f614170e42bf8089536d6f944098caae9eb68
Author: devozerov 
Date:   2017-06-09T10:33:54Z

Fixed affinity key field name resolution for DML.

commit a491d1d607560ffa25f4e4021ba731adbccc482a
Author: devozerov 
Date:   2017-06-09T11:06:05Z

WIP.

commit 5665fe2c0e7583f0cc80f3d1a63af7d9011f5e27
Author: devozerov 
Date:   2017-06-09T11:10:12Z

Appears to work somehow.

commit ec175880c753f8f70fb66e7c9428c6bcad364ad6
Author: devozerov 
Date:   2017-06-09T11:16:29Z

Fixes.

commit 4d50bc5d9020135e06a7c9d3b9dde6337eec4b9c
Author: devozerov 
Date:   2017-06-09T11:26:57Z

Fixing tests.

commit 9686fbaf4f082bac0a49d54fde86f2a1137e26a0
Author: devozerov 
Date:   2017-06-09T11:37:07Z

Fixing tests.

commit 1ceb9a1321e8b947b6350192af1d2e50429d4bc3
Author: devozerov 
Date:   2017-06-09T11:42:14Z

Merge branch 'master' into ignite-5458




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-5460) Web Console: add cache group

[Alexey Kuznetsov (JIRA)]

Alexey Kuznetsov created IGNITE-5460:


 Summary: Web Console: add cache group
 Key: IGNITE-5460
 URL: https://issues.apache.org/jira/browse/IGNITE-5460
 Project: Ignite
  Issue Type: Task
  Components: wizards
Affects Versions: 2.1
Reporter: Alexey Kuznetsov
Assignee: Alexey Kuznetsov
 Fix For: 2.1


In 2.1 cache group was added, we need to show it on Web Console



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] ignite pull request #2112: IGNITE-5339: JDBC thin driver: validate complianc...

[tledkov-gridgain]

GitHub user tledkov-gridgain opened a pull request:

https://github.com/apache/ignite/pull/2112

IGNITE-5339: JDBC thin driver: validate compliance



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-5339

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2112.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2112


commit 313fdc4eb9e3e101a829269b5d1c3ac58c619d9e
Author: skalashnikov 
Date:   2017-05-30T15:06:29Z

IGNITE-5339: Added draft tests for majority of methods in jdbc Connection 
interface

commit 6fc4715336140fe9e76935b51c29746d7e436a88
Author: skalashnikov 
Date:   2017-06-01T09:57:14Z

Merge branch 'master' of https://github.com/apache/ignite into ignite-5339

commit 600c2884004a80e2988f5f8a2477c63fc23cdcc1
Author: skalashnikov 
Date:   2017-06-01T11:32:26Z

IGNITE-5339 Some more Connection methods covered

commit 558d1f288e38f0fb5cb16c62ffc48eb34a12d44b
Author: skalashnikov 
Date:   2017-06-01T12:29:04Z

IGNITE-5339: work in progress

commit cadf9595a1ea7717459a125fa4e9071e34b050db
Author: skalashnikov 
Date:   2017-06-01T13:44:12Z

IGNITE-5339 Work in progress

commit 154cbde6d051aa511f50af8160154d75d7b2856a
Author: skalashnikov 
Date:   2017-06-02T11:30:58Z

IGNITE-5339 WIP

commit b6b971c8510892349e575c923bd51e53b04dafe6
Author: skalashnikov 
Date:   2017-06-02T11:31:29Z

Merge branch 'master' of https://github.com/apache/ignite into ignite-5339

commit 5acfeebfff5db2cf56454088f7f0c60739aa6161
Author: skalashnikov 
Date:   2017-06-02T11:49:50Z

Merge branch 'master' of https://github.com/apache/ignite into ignite-5339

commit 4ae23d4cee9852eefff4237772693ba83bfd4984
Author: skalashnikov 
Date:   2017-06-02T14:26:44Z

IGNITE-5339 WIP

commit e08c85673e9e52018c6feb98b248b46ec81b3de9
Author: skalashnikov 
Date:   2017-06-02T16:52:21Z

IGNITE-5339: added some statement method tests

commit ca0dc56a2a4d6130135e5a1bb7cf7fd19fcd36cd
Author: skalashnikov 
Date:   2017-06-02T16:53:28Z

Merge branch 'master' of https://github.com/apache/ignite into ignite-5339

commit 67c419d81720d3a5fbbb85cec7925c9b590e6cb5
Author: skalashnikov 
Date:   2017-06-05T10:44:07Z

Merge branch 'master' of https://github.com/apache/ignite into ignite-5339

commit eaf03ba107c7fa1335ee5406d8f822b7462e78b7
Author: skalashnikov 
Date:   2017-06-05T12:20:53Z

IGNITE-5339 work in progress

commit a2860d561d9fbfde53b26eae4f8fc403b1cccff3
Author: tledkov-gridgain 
Date:   2017-06-06T11:56:39Z

Merge branch 'master' into ignite-5339

# Conflicts:
#   
modules/clients/src/test/java/org/apache/ignite/jdbc/thin/JdbcThinConnectionSelfTest.java

commit d3b104c6ddaf9f29a2533305231d5cd14329d447
Author: tledkov-gridgain 
Date:   2017-06-06T11:57:40Z

IGNITE-5339: argument check fix

commit d250d102ac7832b967fe28a72863425169c93e05
Author: tledkov-gridgain 
Date:   2017-06-06T12:19:52Z

IGNITE-5339: fix result set type, concurrency, holdability support

commit d90279ecde286ac1f082939091da90b09ef8aeb8
Author: tledkov-gridgain 
Date:   2017-06-06T12:25:59Z

IGNITE-5339: fix result set type, concurrency, holdability support

commit 984f3c05c7d0d5f3a6b1eccb6562003b48855178
Author: tledkov-gridgain 
Date:   2017-06-06T14:11:07Z

IGNITE-5339: some fixes

commit 66f2c2453ae77764d32d33cea37c6dfb42dd444a
Author: tledkov-gridgain 
Date:   2017-06-06T14:35:25Z

IGNITE-5339: failed with tickets

commit 3181cab663e971944ee54cd6bf2e22cf2e38b81e
Author: tledkov-gridgain 
Date:   2017-06-07T10:46:49Z

IGNITE-5339: save the progress on Statement

commit efa60bf231017c74e9f3af47886ab18f35615d1b
Author: tledkov-gridgain 
Date:   2017-06-07T10:47:30Z

IGNITE-5339: cleanup

commit 5e14627b136b49c0a50e49a3402b6f4f503041c2
Author: tledkov-gridgain 
Date:   2017-06-07T12:13:59Z

IGNITE-5339: save the progress on Statement

commit 33f01e439ee33c6dbacad2ac83f8c50c7eee51d6
Author: tledkov-gridgain 
Date:   2017-06-07T12:28:13Z

IGNITE-5339: save the progress on Statement

commit c89f64f598cc908fad9e66d80ed18b93f1bcaf50
Author: tledkov-gridgain 
Date:   2017-06-07T12:43:09Z

Merge remote-tracking branch 

[GitHub] ignite pull request #2109: IGNITE-4431: Include date into the default format...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2109


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Data compression in Ignite 2.0

[Sergey Kozlov]

Hi

* "Per-field compression" is applicable for huge BLOB fields and will
impose the restrictions like unable ot index such fields, slower getting
data, potential OOM issues if compression ration is too high.
But for some cases it makes sense

On Fri, Jun 9, 2017 at 11:11 AM, Антон Чураев  wrote:

> Seems that Dmitry is referring to transparent data encryption. It is used
> throughout the whale database industry.
>
> 2017-06-09 10:50 GMT+03:00 Vladimir Ozerov :
>
> > Dima,
> >
> > Encryption of certain fields is as bad as compression. First, it is a
> huge
> > change, which makes already complex binary protocol even more complex.
> > Second, it have to be ported to CPP, .NET platforms, as well as to JDBC
> and
> > ODBC.
> > Last, but the most important - this is not our headache to encrypt
> > sensitive data. This is user responsibility. Nobody in a sane mind will
> > store passwords in plain form. Instead, user should encrypt it on his
> own,
> > choosing proper encryption parameters - algorithms, key lengths, salts,
> > etc.. How are you going to expose this in API or configuration?
> >
> > We should not implement data encryption on binary level, this is out of
> > question. Encryption should be implemented on application level (user
> > efforts), transport layer (SSL - we already have it), and possibly on
> > disk-level (there are tools for this already).
> >
> >
> > On Fri, Jun 9, 2017 at 9:06 AM, Vyacheslav Daradur 
> > wrote:
> >
> > > >> which is much less useful.
> > > I note, in some cases there is profit more than twice per size of an
> > > object.
> > >
> > > >> Would it be possible to change your implementation to handle the
> > > encryption instead?
> > > Yes, of cource, there's not much difference between compression and
> > > encryption, including in my implementation of per-field-compression.
> > >
> > > 2017-06-09 8:55 GMT+03:00 Dmitriy Setrakyan :
> > >
> > > > Vyacheslav,
> > > >
> > > > When this feature started out as data compression in Ignite, it
> sounded
> > > > very useful. Now it is unfolding as a per-field compression, which is
> > > much
> > > > less useful. In fact, it is questionable whether it is useful at all.
> > The
> > > > fact that this feature is implemented does not make it mandatory for
> > the
> > > > community to accept it.
> > > >
> > > > However, as I mentioned before, per-field encryption is very useful,
> as
> > > it
> > > > would allow users automatically encrypt certain sensitive fields,
> like
> > > > passwords, credit card numbers, etc. There is not much conceptual
> > > > difference between compressing a field vs encrypting a field. Would
> it
> > be
> > > > possible to change your implementation to handle the encryption
> > instead?
> > > >
> > > > D.
> > > >
> > > > On Thu, Jun 8, 2017 at 10:42 PM, Vyacheslav Daradur <
> > daradu...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Guys, I want to be clear:
> > > > > * "Per-field compression" design is the result of a research of the
> > > > binary
> > > > > infrastructure of Ignite and some other its places (querying,
> > indexing,
> > > > > etc.)
> > > > > * Full-compression of object will be more effective, but in this
> case
> > > > there
> > > > > is no capability with querying and indexing (or there is large
> > overhead
> > > > by
> > > > > way of decompressing of full object (or caches pages) on demand)
> > > > > * "Per-field compression" is a one of ways to implement the
> > compression
> > > > > feature
> > > > >
> > > > > I'm new to Ignite also I can be mistaken in some things.
> > > > > Last 3-4 month I've tryed to start dicussion about a design, but
> > nobody
> > > > > answers nothing (except Dmitry and Valentin who was interested how
> it
> > > > > works).
> > > > > But I understand that this is community and nobody is obliged to
> > > anybody.
> > > > >
> > > > > There are strong Ignite experts.
> > > > > If they can help me and community with a design of the compression
> > > > feature
> > > > > it will be great.
> > > > > At the moment I have a desire and time to be engaged in development
> > of
> > > > > compression feature in Ignite.
> > > > > Let's use this opportunity :)
> > > > >
> > > > > 2017-06-09 5:36 GMT+03:00 Dmitriy Setrakyan  >:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > I have never seen a single Ignite user asking about compressing a
> > > > single
> > > > > > field. However, we have had requests to secure certain fields,
> e.g.
> > > > > > passwords.
> > > > > >
> > > > > > I personally do not think per-field compression is needed, unless
> > we
> > > > can
> > > > > > point out some concrete real life use cases.
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Thu, Jun 8, 2017 at 3:42 AM, Vyacheslav Daradur <
> > > > daradu...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Anton,
> > > > > > >
> > > > > > > >> I thought that if 

Re: Data compression in Ignite 2.0

[Sergey Kozlov]

Hi

* "Per-field compression" is applicable for huge BLOB fields and will
impose the restrictions like unable ot index such fields, slower getting
data, potential OOM issues if compression ration is too high.
But for some cases it makes sense

On Fri, Jun 9, 2017 at 11:11 AM, Антон Чураев  wrote:

> Seems that Dmitry is referring to transparent data encryption. It is used
> throughout the whale database industry.
>
> 2017-06-09 10:50 GMT+03:00 Vladimir Ozerov :
>
> > Dima,
> >
> > Encryption of certain fields is as bad as compression. First, it is a
> huge
> > change, which makes already complex binary protocol even more complex.
> > Second, it have to be ported to CPP, .NET platforms, as well as to JDBC
> and
> > ODBC.
> > Last, but the most important - this is not our headache to encrypt
> > sensitive data. This is user responsibility. Nobody in a sane mind will
> > store passwords in plain form. Instead, user should encrypt it on his
> own,
> > choosing proper encryption parameters - algorithms, key lengths, salts,
> > etc.. How are you going to expose this in API or configuration?
> >
> > We should not implement data encryption on binary level, this is out of
> > question. Encryption should be implemented on application level (user
> > efforts), transport layer (SSL - we already have it), and possibly on
> > disk-level (there are tools for this already).
> >
> >
> > On Fri, Jun 9, 2017 at 9:06 AM, Vyacheslav Daradur 
> > wrote:
> >
> > > >> which is much less useful.
> > > I note, in some cases there is profit more than twice per size of an
> > > object.
> > >
> > > >> Would it be possible to change your implementation to handle the
> > > encryption instead?
> > > Yes, of cource, there's not much difference between compression and
> > > encryption, including in my implementation of per-field-compression.
> > >
> > > 2017-06-09 8:55 GMT+03:00 Dmitriy Setrakyan :
> > >
> > > > Vyacheslav,
> > > >
> > > > When this feature started out as data compression in Ignite, it
> sounded
> > > > very useful. Now it is unfolding as a per-field compression, which is
> > > much
> > > > less useful. In fact, it is questionable whether it is useful at all.
> > The
> > > > fact that this feature is implemented does not make it mandatory for
> > the
> > > > community to accept it.
> > > >
> > > > However, as I mentioned before, per-field encryption is very useful,
> as
> > > it
> > > > would allow users automatically encrypt certain sensitive fields,
> like
> > > > passwords, credit card numbers, etc. There is not much conceptual
> > > > difference between compressing a field vs encrypting a field. Would
> it
> > be
> > > > possible to change your implementation to handle the encryption
> > instead?
> > > >
> > > > D.
> > > >
> > > > On Thu, Jun 8, 2017 at 10:42 PM, Vyacheslav Daradur <
> > daradu...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Guys, I want to be clear:
> > > > > * "Per-field compression" design is the result of a research of the
> > > > binary
> > > > > infrastructure of Ignite and some other its places (querying,
> > indexing,
> > > > > etc.)
> > > > > * Full-compression of object will be more effective, but in this
> case
> > > > there
> > > > > is no capability with querying and indexing (or there is large
> > overhead
> > > > by
> > > > > way of decompressing of full object (or caches pages) on demand)
> > > > > * "Per-field compression" is a one of ways to implement the
> > compression
> > > > > feature
> > > > >
> > > > > I'm new to Ignite also I can be mistaken in some things.
> > > > > Last 3-4 month I've tryed to start dicussion about a design, but
> > nobody
> > > > > answers nothing (except Dmitry and Valentin who was interested how
> it
> > > > > works).
> > > > > But I understand that this is community and nobody is obliged to
> > > anybody.
> > > > >
> > > > > There are strong Ignite experts.
> > > > > If they can help me and community with a design of the compression
> > > > feature
> > > > > it will be great.
> > > > > At the moment I have a desire and time to be engaged in development
> > of
> > > > > compression feature in Ignite.
> > > > > Let's use this opportunity :)
> > > > >
> > > > > 2017-06-09 5:36 GMT+03:00 Dmitriy Setrakyan  >:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > I have never seen a single Ignite user asking about compressing a
> > > > single
> > > > > > field. However, we have had requests to secure certain fields,
> e.g.
> > > > > > passwords.
> > > > > >
> > > > > > I personally do not think per-field compression is needed, unless
> > we
> > > > can
> > > > > > point out some concrete real life use cases.
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Thu, Jun 8, 2017 at 3:42 AM, Vyacheslav Daradur <
> > > > daradu...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Anton,
> > > > > > >
> > > > > > > >> I thought that if 

Re: Data compression in Ignite 2.0

[Sergi Vladykin]

+1 to Vladimir. Fields encryption is a user responsibility. I see no reason
to introduce additional complexity to Ignite.

Sergi

2017-06-09 11:11 GMT+03:00 Антон Чураев :

> Seems that Dmitry is referring to transparent data encryption. It is used
> throughout the whale database industry.
>
> 2017-06-09 10:50 GMT+03:00 Vladimir Ozerov :
>
> > Dima,
> >
> > Encryption of certain fields is as bad as compression. First, it is a
> huge
> > change, which makes already complex binary protocol even more complex.
> > Second, it have to be ported to CPP, .NET platforms, as well as to JDBC
> and
> > ODBC.
> > Last, but the most important - this is not our headache to encrypt
> > sensitive data. This is user responsibility. Nobody in a sane mind will
> > store passwords in plain form. Instead, user should encrypt it on his
> own,
> > choosing proper encryption parameters - algorithms, key lengths, salts,
> > etc.. How are you going to expose this in API or configuration?
> >
> > We should not implement data encryption on binary level, this is out of
> > question. Encryption should be implemented on application level (user
> > efforts), transport layer (SSL - we already have it), and possibly on
> > disk-level (there are tools for this already).
> >
> >
> > On Fri, Jun 9, 2017 at 9:06 AM, Vyacheslav Daradur 
> > wrote:
> >
> > > >> which is much less useful.
> > > I note, in some cases there is profit more than twice per size of an
> > > object.
> > >
> > > >> Would it be possible to change your implementation to handle the
> > > encryption instead?
> > > Yes, of cource, there's not much difference between compression and
> > > encryption, including in my implementation of per-field-compression.
> > >
> > > 2017-06-09 8:55 GMT+03:00 Dmitriy Setrakyan :
> > >
> > > > Vyacheslav,
> > > >
> > > > When this feature started out as data compression in Ignite, it
> sounded
> > > > very useful. Now it is unfolding as a per-field compression, which is
> > > much
> > > > less useful. In fact, it is questionable whether it is useful at all.
> > The
> > > > fact that this feature is implemented does not make it mandatory for
> > the
> > > > community to accept it.
> > > >
> > > > However, as I mentioned before, per-field encryption is very useful,
> as
> > > it
> > > > would allow users automatically encrypt certain sensitive fields,
> like
> > > > passwords, credit card numbers, etc. There is not much conceptual
> > > > difference between compressing a field vs encrypting a field. Would
> it
> > be
> > > > possible to change your implementation to handle the encryption
> > instead?
> > > >
> > > > D.
> > > >
> > > > On Thu, Jun 8, 2017 at 10:42 PM, Vyacheslav Daradur <
> > daradu...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Guys, I want to be clear:
> > > > > * "Per-field compression" design is the result of a research of the
> > > > binary
> > > > > infrastructure of Ignite and some other its places (querying,
> > indexing,
> > > > > etc.)
> > > > > * Full-compression of object will be more effective, but in this
> case
> > > > there
> > > > > is no capability with querying and indexing (or there is large
> > overhead
> > > > by
> > > > > way of decompressing of full object (or caches pages) on demand)
> > > > > * "Per-field compression" is a one of ways to implement the
> > compression
> > > > > feature
> > > > >
> > > > > I'm new to Ignite also I can be mistaken in some things.
> > > > > Last 3-4 month I've tryed to start dicussion about a design, but
> > nobody
> > > > > answers nothing (except Dmitry and Valentin who was interested how
> it
> > > > > works).
> > > > > But I understand that this is community and nobody is obliged to
> > > anybody.
> > > > >
> > > > > There are strong Ignite experts.
> > > > > If they can help me and community with a design of the compression
> > > > feature
> > > > > it will be great.
> > > > > At the moment I have a desire and time to be engaged in development
> > of
> > > > > compression feature in Ignite.
> > > > > Let's use this opportunity :)
> > > > >
> > > > > 2017-06-09 5:36 GMT+03:00 Dmitriy Setrakyan  >:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > I have never seen a single Ignite user asking about compressing a
> > > > single
> > > > > > field. However, we have had requests to secure certain fields,
> e.g.
> > > > > > passwords.
> > > > > >
> > > > > > I personally do not think per-field compression is needed, unless
> > we
> > > > can
> > > > > > point out some concrete real life use cases.
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Thu, Jun 8, 2017 at 3:42 AM, Vyacheslav Daradur <
> > > > daradu...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Anton,
> > > > > > >
> > > > > > > >> I thought that if there will storing compressed data in the
> > > > memory,
> > > > > > data
> > > > > > > >> will transmit over wire in 

Re: Data compression in Ignite 2.0

[Антон Чураев]

Seems that Dmitry is referring to transparent data encryption. It is used
throughout the whale database industry.

2017-06-09 10:50 GMT+03:00 Vladimir Ozerov :

> Dima,
>
> Encryption of certain fields is as bad as compression. First, it is a huge
> change, which makes already complex binary protocol even more complex.
> Second, it have to be ported to CPP, .NET platforms, as well as to JDBC and
> ODBC.
> Last, but the most important - this is not our headache to encrypt
> sensitive data. This is user responsibility. Nobody in a sane mind will
> store passwords in plain form. Instead, user should encrypt it on his own,
> choosing proper encryption parameters - algorithms, key lengths, salts,
> etc.. How are you going to expose this in API or configuration?
>
> We should not implement data encryption on binary level, this is out of
> question. Encryption should be implemented on application level (user
> efforts), transport layer (SSL - we already have it), and possibly on
> disk-level (there are tools for this already).
>
>
> On Fri, Jun 9, 2017 at 9:06 AM, Vyacheslav Daradur 
> wrote:
>
> > >> which is much less useful.
> > I note, in some cases there is profit more than twice per size of an
> > object.
> >
> > >> Would it be possible to change your implementation to handle the
> > encryption instead?
> > Yes, of cource, there's not much difference between compression and
> > encryption, including in my implementation of per-field-compression.
> >
> > 2017-06-09 8:55 GMT+03:00 Dmitriy Setrakyan :
> >
> > > Vyacheslav,
> > >
> > > When this feature started out as data compression in Ignite, it sounded
> > > very useful. Now it is unfolding as a per-field compression, which is
> > much
> > > less useful. In fact, it is questionable whether it is useful at all.
> The
> > > fact that this feature is implemented does not make it mandatory for
> the
> > > community to accept it.
> > >
> > > However, as I mentioned before, per-field encryption is very useful, as
> > it
> > > would allow users automatically encrypt certain sensitive fields, like
> > > passwords, credit card numbers, etc. There is not much conceptual
> > > difference between compressing a field vs encrypting a field. Would it
> be
> > > possible to change your implementation to handle the encryption
> instead?
> > >
> > > D.
> > >
> > > On Thu, Jun 8, 2017 at 10:42 PM, Vyacheslav Daradur <
> daradu...@gmail.com
> > >
> > > wrote:
> > >
> > > > Guys, I want to be clear:
> > > > * "Per-field compression" design is the result of a research of the
> > > binary
> > > > infrastructure of Ignite and some other its places (querying,
> indexing,
> > > > etc.)
> > > > * Full-compression of object will be more effective, but in this case
> > > there
> > > > is no capability with querying and indexing (or there is large
> overhead
> > > by
> > > > way of decompressing of full object (or caches pages) on demand)
> > > > * "Per-field compression" is a one of ways to implement the
> compression
> > > > feature
> > > >
> > > > I'm new to Ignite also I can be mistaken in some things.
> > > > Last 3-4 month I've tryed to start dicussion about a design, but
> nobody
> > > > answers nothing (except Dmitry and Valentin who was interested how it
> > > > works).
> > > > But I understand that this is community and nobody is obliged to
> > anybody.
> > > >
> > > > There are strong Ignite experts.
> > > > If they can help me and community with a design of the compression
> > > feature
> > > > it will be great.
> > > > At the moment I have a desire and time to be engaged in development
> of
> > > > compression feature in Ignite.
> > > > Let's use this opportunity :)
> > > >
> > > > 2017-06-09 5:36 GMT+03:00 Dmitriy Setrakyan :
> > > >
> > > > > Igniters,
> > > > >
> > > > > I have never seen a single Ignite user asking about compressing a
> > > single
> > > > > field. However, we have had requests to secure certain fields, e.g.
> > > > > passwords.
> > > > >
> > > > > I personally do not think per-field compression is needed, unless
> we
> > > can
> > > > > point out some concrete real life use cases.
> > > > >
> > > > > D.
> > > > >
> > > > > On Thu, Jun 8, 2017 at 3:42 AM, Vyacheslav Daradur <
> > > daradu...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Anton,
> > > > > >
> > > > > > >> I thought that if there will storing compressed data in the
> > > memory,
> > > > > data
> > > > > > >> will transmit over wire in compression too. Is it right?
> > > > > >
> > > > > > In per-field compression case - yes.
> > > > > >
> > > > > > 2017-06-08 13:36 GMT+03:00 Антон Чураев :
> > > > > >
> > > > > > > Guys, could you please help me.
> > > > > > > I thought that if there will storing compressed data in the
> > memory,
> > > > > data
> > > > > > > will transmit over wire in compression too. Is it right?
> > > > > > >
> > > > > > > 2017-06-08 13:30 GMT+03:00 

Re: Data compression in Ignite 2.0

[Vyacheslav Daradur]

Vladimir,

>>  Nobody in a sane mind will
>> store passwords in plain form. Instead, user should encrypt it on his
own,
>> choosing proper encryption parameters - algorithms, key lengths, salts,
etc..
Sounds reasonable to me.
But if someone want to have this feature OOTB we can continue discussion,
may implement it in some other way.

>> How are you going to expose this in API or configuration?
Just for example: we can provide a plugable interface in the
IgniteConfiguration (or other place), which user will able to implement.

About compression, you wrote:
>> 2) Storing data in memory - here the much simpler step would be to full
>> compression on per-cache basis rather than dealing with per-fields case.
Could you explain your idea? How we can implement it and how it will able
to be compatible with querying and indexing?
Thanks in advance.


2017-06-09 10:50 GMT+03:00 Vladimir Ozerov :

> Dima,
>
> Encryption of certain fields is as bad as compression. First, it is a huge
> change, which makes already complex binary protocol even more complex.
> Second, it have to be ported to CPP, .NET platforms, as well as to JDBC and
> ODBC.
> Last, but the most important - this is not our headache to encrypt
> sensitive data. This is user responsibility. Nobody in a sane mind will
> store passwords in plain form. Instead, user should encrypt it on his own,
> choosing proper encryption parameters - algorithms, key lengths, salts,
> etc.. How are you going to expose this in API or configuration?
>
> We should not implement data encryption on binary level, this is out of
> question. Encryption should be implemented on application level (user
> efforts), transport layer (SSL - we already have it), and possibly on
> disk-level (there are tools for this already).
>
>
> On Fri, Jun 9, 2017 at 9:06 AM, Vyacheslav Daradur 
> wrote:
>
> > >> which is much less useful.
> > I note, in some cases there is profit more than twice per size of an
> > object.
> >
> > >> Would it be possible to change your implementation to handle the
> > encryption instead?
> > Yes, of cource, there's not much difference between compression and
> > encryption, including in my implementation of per-field-compression.
> >
> > 2017-06-09 8:55 GMT+03:00 Dmitriy Setrakyan :
> >
> > > Vyacheslav,
> > >
> > > When this feature started out as data compression in Ignite, it sounded
> > > very useful. Now it is unfolding as a per-field compression, which is
> > much
> > > less useful. In fact, it is questionable whether it is useful at all.
> The
> > > fact that this feature is implemented does not make it mandatory for
> the
> > > community to accept it.
> > >
> > > However, as I mentioned before, per-field encryption is very useful, as
> > it
> > > would allow users automatically encrypt certain sensitive fields, like
> > > passwords, credit card numbers, etc. There is not much conceptual
> > > difference between compressing a field vs encrypting a field. Would it
> be
> > > possible to change your implementation to handle the encryption
> instead?
> > >
> > > D.
> > >
> > > On Thu, Jun 8, 2017 at 10:42 PM, Vyacheslav Daradur <
> daradu...@gmail.com
> > >
> > > wrote:
> > >
> > > > Guys, I want to be clear:
> > > > * "Per-field compression" design is the result of a research of the
> > > binary
> > > > infrastructure of Ignite and some other its places (querying,
> indexing,
> > > > etc.)
> > > > * Full-compression of object will be more effective, but in this case
> > > there
> > > > is no capability with querying and indexing (or there is large
> overhead
> > > by
> > > > way of decompressing of full object (or caches pages) on demand)
> > > > * "Per-field compression" is a one of ways to implement the
> compression
> > > > feature
> > > >
> > > > I'm new to Ignite also I can be mistaken in some things.
> > > > Last 3-4 month I've tryed to start dicussion about a design, but
> nobody
> > > > answers nothing (except Dmitry and Valentin who was interested how it
> > > > works).
> > > > But I understand that this is community and nobody is obliged to
> > anybody.
> > > >
> > > > There are strong Ignite experts.
> > > > If they can help me and community with a design of the compression
> > > feature
> > > > it will be great.
> > > > At the moment I have a desire and time to be engaged in development
> of
> > > > compression feature in Ignite.
> > > > Let's use this opportunity :)
> > > >
> > > > 2017-06-09 5:36 GMT+03:00 Dmitriy Setrakyan :
> > > >
> > > > > Igniters,
> > > > >
> > > > > I have never seen a single Ignite user asking about compressing a
> > > single
> > > > > field. However, we have had requests to secure certain fields, e.g.
> > > > > passwords.
> > > > >
> > > > > I personally do not think per-field compression is needed, unless
> we
> > > can
> > > > > point out some concrete real life use cases.
> > > > >
> > > > > D.
> > > > >
> > > > > On Thu, Jun 8, 

Re: Data compression in Ignite 2.0

[Vladimir Ozerov]

Dima,

Encryption of certain fields is as bad as compression. First, it is a huge
change, which makes already complex binary protocol even more complex.
Second, it have to be ported to CPP, .NET platforms, as well as to JDBC and
ODBC.
Last, but the most important - this is not our headache to encrypt
sensitive data. This is user responsibility. Nobody in a sane mind will
store passwords in plain form. Instead, user should encrypt it on his own,
choosing proper encryption parameters - algorithms, key lengths, salts,
etc.. How are you going to expose this in API or configuration?

We should not implement data encryption on binary level, this is out of
question. Encryption should be implemented on application level (user
efforts), transport layer (SSL - we already have it), and possibly on
disk-level (there are tools for this already).


On Fri, Jun 9, 2017 at 9:06 AM, Vyacheslav Daradur 
wrote:

> >> which is much less useful.
> I note, in some cases there is profit more than twice per size of an
> object.
>
> >> Would it be possible to change your implementation to handle the
> encryption instead?
> Yes, of cource, there's not much difference between compression and
> encryption, including in my implementation of per-field-compression.
>
> 2017-06-09 8:55 GMT+03:00 Dmitriy Setrakyan :
>
> > Vyacheslav,
> >
> > When this feature started out as data compression in Ignite, it sounded
> > very useful. Now it is unfolding as a per-field compression, which is
> much
> > less useful. In fact, it is questionable whether it is useful at all. The
> > fact that this feature is implemented does not make it mandatory for the
> > community to accept it.
> >
> > However, as I mentioned before, per-field encryption is very useful, as
> it
> > would allow users automatically encrypt certain sensitive fields, like
> > passwords, credit card numbers, etc. There is not much conceptual
> > difference between compressing a field vs encrypting a field. Would it be
> > possible to change your implementation to handle the encryption instead?
> >
> > D.
> >
> > On Thu, Jun 8, 2017 at 10:42 PM, Vyacheslav Daradur  >
> > wrote:
> >
> > > Guys, I want to be clear:
> > > * "Per-field compression" design is the result of a research of the
> > binary
> > > infrastructure of Ignite and some other its places (querying, indexing,
> > > etc.)
> > > * Full-compression of object will be more effective, but in this case
> > there
> > > is no capability with querying and indexing (or there is large overhead
> > by
> > > way of decompressing of full object (or caches pages) on demand)
> > > * "Per-field compression" is a one of ways to implement the compression
> > > feature
> > >
> > > I'm new to Ignite also I can be mistaken in some things.
> > > Last 3-4 month I've tryed to start dicussion about a design, but nobody
> > > answers nothing (except Dmitry and Valentin who was interested how it
> > > works).
> > > But I understand that this is community and nobody is obliged to
> anybody.
> > >
> > > There are strong Ignite experts.
> > > If they can help me and community with a design of the compression
> > feature
> > > it will be great.
> > > At the moment I have a desire and time to be engaged in development of
> > > compression feature in Ignite.
> > > Let's use this opportunity :)
> > >
> > > 2017-06-09 5:36 GMT+03:00 Dmitriy Setrakyan :
> > >
> > > > Igniters,
> > > >
> > > > I have never seen a single Ignite user asking about compressing a
> > single
> > > > field. However, we have had requests to secure certain fields, e.g.
> > > > passwords.
> > > >
> > > > I personally do not think per-field compression is needed, unless we
> > can
> > > > point out some concrete real life use cases.
> > > >
> > > > D.
> > > >
> > > > On Thu, Jun 8, 2017 at 3:42 AM, Vyacheslav Daradur <
> > daradu...@gmail.com>
> > > > wrote:
> > > >
> > > > > Anton,
> > > > >
> > > > > >> I thought that if there will storing compressed data in the
> > memory,
> > > > data
> > > > > >> will transmit over wire in compression too. Is it right?
> > > > >
> > > > > In per-field compression case - yes.
> > > > >
> > > > > 2017-06-08 13:36 GMT+03:00 Антон Чураев :
> > > > >
> > > > > > Guys, could you please help me.
> > > > > > I thought that if there will storing compressed data in the
> memory,
> > > > data
> > > > > > will transmit over wire in compression too. Is it right?
> > > > > >
> > > > > > 2017-06-08 13:30 GMT+03:00 Vyacheslav Daradur <
> daradu...@gmail.com
> > >:
> > > > > >
> > > > > > > Vladimir,
> > > > > > >
> > > > > > > The main problem which I'am trying to solve is storing data in
> > > memory
> > > > > in
> > > > > > a
> > > > > > > compression form via Ignite.
> > > > > > > The main goal is using memory more effectivelly.
> > > > > > >
> > > > > > > >> here the much simpler step would be to full
> > > > > > > compression on per-cache basis 

[jira] [Created] (IGNITE-5459) Deprecate IgniteConfiguration.marshaller property

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-5459:
---

 Summary: Deprecate IgniteConfiguration.marshaller property
 Key: IGNITE-5459
 URL: https://issues.apache.org/jira/browse/IGNITE-5459
 Project: Ignite
  Issue Type: Task
  Components: general
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
 Fix For: 2.1


{{IgniteConfiguration.marshaller}} property should be deprecated. We have a lot 
of components which heavily depend on BinaryMarshaller (e.g. ,NET, CPP, ODBC), 
so there is no need to allow users override it.

Let's remove it in 3.0.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-5458) Add CacheKeyConfiguration to CacheConfiguration

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-5458:
---

 Summary: Add CacheKeyConfiguration to CacheConfiguration
 Key: IGNITE-5458
 URL: https://issues.apache.org/jira/browse/IGNITE-5458
 Project: Ignite
  Issue Type: Task
  Components: cache
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
 Fix For: 2.1


Currently affinity column of certain type is not known on startup, as types are 
registered dynamically. For this reason affinity column could be resolved 
incorrectly if needed before type is registered. E.g. this is the case for our 
H2 tables, which need affinity key column on startup.

This could be resolved by setting {{IgniteConfiguration.cacheKeyConfiguration}} 
property, which maps type name to affinity key column name. However, this 
mechanism is not flexible enough because mappings cannot be changed in runtime.

We should add similar property to cache configuration and use it to resolve 
affinity column. Ideally, affinity key should be removed from {{BinaryType}} at 
all.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Deprecate IgniteConfiguration.marshaller and make it no-op!

[Vladimir Ozerov]

Igniters,

SQL do work without BinaryMarshaller. And I was wrong about DML - it
appears to work as well. Let's just deprecate this.

On Fri, Jun 9, 2017 at 9:55 AM, Semyon Boikov  wrote:

> As far as I know before 2.0 SQL worked without BinaryMarshaller, is this
> support was removed intentionally? Also I think it is possible to implement
> DML without BinaryMashaller as well?
>
> I'm not against Vladimir's suggestion, but I think it make sense to ask on
> user list to know if somebody uses custom marshallers.
>
> Thanks
>
> On Fri, Jun 9, 2017 at 3:15 AM, Dmitriy Setrakyan 
> wrote:
>
> > On Thu, Jun 8, 2017 at 2:30 PM, Vladimir Ozerov 
> > wrote:
> >
> > > Denis,
> > >
> > > I think we should not provide distinction between "data grid", "compute
> > > grid" and other use case. We should have consistent product which just
> > > works.
> > >
> > > Anyway,I am OK to just deprecate these property for 2.0 and remove 3.0
> if
> > > community thinks that custom marshallers are still needed.
> > >
> >
> > Vladimir, I tend to agree with you here. +1 on deprecating it.
> >
>

Re: [DISCUSS] Webinar for Ignite Persistent Store walk-through

[Sergi Vladykin]

+1

Sergi

2017-06-08 23:03 GMT+03:00 Dmitriy Setrakyan :

> +1 (I will attend)
>
> On Thu, Jun 8, 2017 at 1:02 PM, Konstantin Boudnik  wrote:
>
> > That'd be great! Thank you!
> > --
> >   Take care,
> > Konstantin (Cos) Boudnik
> > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >
> > Disclaimer: Opinions expressed in this email are those of the author,
> > and do not necessarily represent the views of any company the author
> > might be affiliated with at the moment of writing.
> >
> >
> > On Thu, Jun 8, 2017 at 12:54 PM, Denis Magda  wrote:
> > > Igniters,
> > >
> > > What’d you think if we arrange an internal webinar for our community to
> > walk through the features, capabilities and implementation details of the
> > Ignite Persistent Store [1]? That should help us understanding the
> donation
> > better.
> > >
> > > Please reply if you will be happy to attend.
> > >
> > > [1] https://apacheignite.readme.io/docs/distributed-persistent-store <
> > https://apacheignite.readme.io/docs/distributed-persistent-store>
> > >
> > > —
> > > Denis
> >
>

Re: Deprecate IgniteConfiguration.marshaller and make it no-op!

[Semyon Boikov]

As far as I know before 2.0 SQL worked without BinaryMarshaller, is this
support was removed intentionally? Also I think it is possible to implement
DML without BinaryMashaller as well?

I'm not against Vladimir's suggestion, but I think it make sense to ask on
user list to know if somebody uses custom marshallers.

Thanks

On Fri, Jun 9, 2017 at 3:15 AM, Dmitriy Setrakyan 
wrote:

> On Thu, Jun 8, 2017 at 2:30 PM, Vladimir Ozerov 
> wrote:
>
> > Denis,
> >
> > I think we should not provide distinction between "data grid", "compute
> > grid" and other use case. We should have consistent product which just
> > works.
> >
> > Anyway,I am OK to just deprecate these property for 2.0 and remove 3.0 if
> > community thinks that custom marshallers are still needed.
> >
>
> Vladimir, I tend to agree with you here. +1 on deprecating it.
>

Re: Data compression in Ignite 2.0

[Vyacheslav Daradur]

>> which is much less useful.
I note, in some cases there is profit more than twice per size of an object.

>> Would it be possible to change your implementation to handle the
encryption instead?
Yes, of cource, there's not much difference between compression and
encryption, including in my implementation of per-field-compression.

2017-06-09 8:55 GMT+03:00 Dmitriy Setrakyan :

> Vyacheslav,
>
> When this feature started out as data compression in Ignite, it sounded
> very useful. Now it is unfolding as a per-field compression, which is much
> less useful. In fact, it is questionable whether it is useful at all. The
> fact that this feature is implemented does not make it mandatory for the
> community to accept it.
>
> However, as I mentioned before, per-field encryption is very useful, as it
> would allow users automatically encrypt certain sensitive fields, like
> passwords, credit card numbers, etc. There is not much conceptual
> difference between compressing a field vs encrypting a field. Would it be
> possible to change your implementation to handle the encryption instead?
>
> D.
>
> On Thu, Jun 8, 2017 at 10:42 PM, Vyacheslav Daradur 
> wrote:
>
> > Guys, I want to be clear:
> > * "Per-field compression" design is the result of a research of the
> binary
> > infrastructure of Ignite and some other its places (querying, indexing,
> > etc.)
> > * Full-compression of object will be more effective, but in this case
> there
> > is no capability with querying and indexing (or there is large overhead
> by
> > way of decompressing of full object (or caches pages) on demand)
> > * "Per-field compression" is a one of ways to implement the compression
> > feature
> >
> > I'm new to Ignite also I can be mistaken in some things.
> > Last 3-4 month I've tryed to start dicussion about a design, but nobody
> > answers nothing (except Dmitry and Valentin who was interested how it
> > works).
> > But I understand that this is community and nobody is obliged to anybody.
> >
> > There are strong Ignite experts.
> > If they can help me and community with a design of the compression
> feature
> > it will be great.
> > At the moment I have a desire and time to be engaged in development of
> > compression feature in Ignite.
> > Let's use this opportunity :)
> >
> > 2017-06-09 5:36 GMT+03:00 Dmitriy Setrakyan :
> >
> > > Igniters,
> > >
> > > I have never seen a single Ignite user asking about compressing a
> single
> > > field. However, we have had requests to secure certain fields, e.g.
> > > passwords.
> > >
> > > I personally do not think per-field compression is needed, unless we
> can
> > > point out some concrete real life use cases.
> > >
> > > D.
> > >
> > > On Thu, Jun 8, 2017 at 3:42 AM, Vyacheslav Daradur <
> daradu...@gmail.com>
> > > wrote:
> > >
> > > > Anton,
> > > >
> > > > >> I thought that if there will storing compressed data in the
> memory,
> > > data
> > > > >> will transmit over wire in compression too. Is it right?
> > > >
> > > > In per-field compression case - yes.
> > > >
> > > > 2017-06-08 13:36 GMT+03:00 Антон Чураев :
> > > >
> > > > > Guys, could you please help me.
> > > > > I thought that if there will storing compressed data in the memory,
> > > data
> > > > > will transmit over wire in compression too. Is it right?
> > > > >
> > > > > 2017-06-08 13:30 GMT+03:00 Vyacheslav Daradur  >:
> > > > >
> > > > > > Vladimir,
> > > > > >
> > > > > > The main problem which I'am trying to solve is storing data in
> > memory
> > > > in
> > > > > a
> > > > > > compression form via Ignite.
> > > > > > The main goal is using memory more effectivelly.
> > > > > >
> > > > > > >> here the much simpler step would be to full
> > > > > > compression on per-cache basis rather than dealing with
> per-fields
> > > > case.
> > > > > >
> > > > > > Please explain your idea. Compess data by memory-page?
> > > > > > Is it compatible with quering and indexing?
> > > > > >
> > > > > > >> In the end, if user would like to compress particular field,
> he
> > > can
> > > > > > always to it on his own
> > > > > > I think we mustn't think in this way, if user need something he
> > > trying
> > > > to
> > > > > > choose a tool which has this feature OOTB.
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2017-06-08 12:53 GMT+03:00 Vladimir Ozerov  >:
> > > > > >
> > > > > > > Igniters,
> > > > > > >
> > > > > > > Honestly I still do not see how to apply it gracefully this
> > feature
> > > > ti
> > > > > > > Ignite. And overall approach to compress only particular fields
> > > looks
> > > > > > > overcomplicated to me. Remember, that our main use case is an
> > > > > application
> > > > > > > without classes on the server. It means that any kind of
> > > annotations
> > > > > are
> > > > > > > inapplicable. To be more precise: proper API should be
> > implemented
> > > to
> > > > > > > handle