Re[2]: Cache scan efficiency

[Zhenya Stanilovsky]

hi, but how to deal with page replacements, which Dmitriy Pavlov mentioned?
this approach would be efficient if all data fits into memory, may be better to 
have method to pin some critical caches?


>Среда, 19 сентября 2018, 0:26 +03:00 от Dmitriy Pavlov :
>
>Even better, if RAM is exhausted page replacement process will be started.
>https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Pagereplacement(rotationwithdisk
> )
>
>Effect of the preloading will be still markable, but not as excelled as
>with full-fitting into RAM. Later I can review or improve javadocs if it is
>necessary.
>
>ср, 19 сент. 2018 г. в 0:18, Denis Magda < dma...@apache.org >:
>
>> Agree, it's just a matter of the documentation. If a user stores 100% in
>> RAM and on disk, and just wants to warm RAM up after a restart then he
>> knows everything will fit there. If during the preloading we detect that
>> the RAM is exhausted we can halt it and print out a warning.
>>
>> --
>> Denis
>>
>> On Tue, Sep 18, 2018 at 2:10 PM Dmitriy Pavlov < dpavlov@gmail.com >
>> wrote:
>>
>> > Hi,
>> >
>> > I totally support the idea of cache preload.
>> >
>> > IMO it can be expanded. We can iterate over local partitions of the cache
>> > group and preload each.
>> >
>> > But it should be really clear documented methods so a user can be aware
>> of
>> > the benefits of such method (e.g. if RAM region is big enough, etc).
>> >
>> > Sincerely,
>> > Dmitriy Pavlov
>> >
>> > вт, 18 сент. 2018 г. в 21:36, Denis Magda < dma...@apache.org >:
>> >
>> > > Folks,
>> > >
>> > > Since we're adding a method that would preload a certain partition, can
>> > we
>> > > add the one which will preload the whole cache? Ignite persistence
>> users
>> > > I've been working with look puzzled once they realize there is no way
>> to
>> > > warm up RAM after the restart. There are use cases that require this.
>> > >
>> > > Can the current optimizations be expanded to the cache preloading use
>> > case?
>> > >
>> > > --
>> > > Denis
>> > >
>> > > On Tue, Sep 18, 2018 at 3:58 AM Alexei Scherbakov <
>> > >  alexey.scherbak...@gmail.com > wrote:
>> > >
>> > > > Summing up, I suggest adding new public
>> > > > method IgniteCache.preloadPartition(partId).
>> > > >
>> > > > I will start preparing PR for IGNITE-8873
>> > > > < https://issues.apache.org/jira/browse/IGNITE-8873 > if no more
>> > > objections
>> > > > follow.
>> > > >
>> > > >
>> > > >
>> > > > вт, 18 сент. 2018 г. в 10:50, Alexey Goncharuk <
>> > >  alexey.goncha...@gmail.com
>> > > > >:
>> > > >
>> > > > > Dmitriy,
>> > > > >
>> > > > > In my understanding, the proper fix for the scan query looks like a
>> > big
>> > > > > change and it is unlikely that we include it in Ignite 2.7. On the
>> > > other
>> > > > > hand, the method suggested by Alexei is quite simple  and it
>> > definitely
>> > > > > fits Ignite 2.7, which will provide a better user experience. Even
>> > > > having a
>> > > > > proper scan query implemented this method can be useful in some
>> > > specific
>> > > > > scenarios, so we will not have to deprecate it.
>> > > > >
>> > > > > --AG
>> > > > >
>> > > > > пн, 17 сент. 2018 г. в 19:15, Dmitriy Pavlov <
>>  dpavlov@gmail.com
>> > >:
>> > > > >
>> > > > > > As I understood it is not a hack, it is an advanced feature for
>> > > warming
>> > > > > up
>> > > > > > the partition. We can build warm-up of the overall cache by
>> calling
>> > > its
>> > > > > > partitions warm-up. Users often ask about this feature and are
>> not
>> > > > > > confident with our lazy upload.
>> > > > > >
>> > > > > > Please correct me if I misunderstood the idea.
>> > > > > >
>> > > > > > пн, 17 сент. 2018 г. в 18:37, Dmitriy Setrakyan <
>> > >  dsetrak...@apache.org
>> > > > >:
>> > > > > >
>> > > > > > > I would rather fix the scan than hack the scan. Is there any
>> > > > technical
>> > > > > > > reason for hacking it now instead of fixing it properly? Can
>> some
>> > > of
>> > > > > the
>> > > > > > > experts in this thread provide an estimate of complexity and
>> > > > difference
>> > > > > > in
>> > > > > > > work that would be required for each approach?
>> > > > > > >
>> > > > > > > D.
>> > > > > > >
>> > > > > > > On Mon, Sep 17, 2018 at 4:42 PM Alexey Goncharuk <
>> > > > > > >  alexey.goncha...@gmail.com >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > I think it would be beneficial for some Ignite users if we
>> > added
>> > > > > such a
>> > > > > > > > partition warmup method to the public API. The method should
>> be
>> > > > > > > > well-documented and state that it may invalidate existing
>> page
>> > > > cache.
>> > > > > > It
>> > > > > > > > will be a very effective instrument until we add the proper
>> > scan
>> > > > > > ability
>> > > > > > > > that Vladimir was referring to.
>> > > > > > > >
>> > > > > > > > пн, 17 сент. 2018 г. в 13:05, Maxim Muzafarov <
>> > >  maxmu...@gmail.com
>> > > > >:
>> > > > > > > >
>> > > > 

[GitHub] ignite pull request #4789: Add TypeScript support to web console frontend

[Klaster1]

GitHub user Klaster1 opened a pull request:

https://github.com/apache/ignite/pull/4789

Add TypeScript support to web console frontend



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9552

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4789.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4789


commit 0a6424d6bec9cae48125fcd48a71a4b19bccb6c9
Author: Ilya Borisov 
Date:   2018-09-13T06:57:22Z

IGNITE-9552 Add babel typescript preset.

commit a1b47d497b6181e99477ae1a172d5f7d6c018385
Author: Ilya Borisov 
Date:   2018-09-13T06:57:42Z

IGNITE-9552 Add .ts extension to webpack config.

commit 5698f76997378bc5a5952c7bd5daadaf5415a0ff
Author: Ilya Borisov 
Date:   2018-09-13T06:58:24Z

IGNITE-9552 Don't use .js extension in imports.

commit 6286da65f0ef6bb3ce5612abb4ac5c0d2011a121
Author: Ilya Borisov 
Date:   2018-09-13T07:25:20Z

IGNITE-9552 Add .ts extension to webpack resolve aliases.

commit 8f5cde1df8dd5dedff75b84cffb0342941707697
Author: Ilya Borisov 
Date:   2018-09-13T09:07:04Z

IGNITE-9552 Disable eslint loader for .ts files.

commit a81b551e63b2b1ed995c52bbfae102db497ac54b
Author: Ilya Borisov 
Date:   2018-09-13T09:40:40Z

IGNITE-9552 Covert page-signin to TS.

commit 5624cd1711e9b432f630b9d498cadfeddec9d3d6
Author: Ilya Borisov 
Date:   2018-09-18T08:13:28Z

Eslint wip

commit d6fd460686568e948c6aa0d85a5f8947397edad1
Author: Ilya Borisov 
Date:   2018-09-18T10:02:30Z

IGNITE-9552 Update eslint formatter plugin.

commit 3096a84fc99b252adc6281e572570aedd20a5006
Author: Ilya Borisov 
Date:   2018-09-18T10:03:11Z

IGNITE-9552 Make eslint work with TS.

commit 89b5c819682561b1bfabf3f891bee30761b4a8ae
Author: Ilya Borisov 
Date:   2018-09-18T10:03:48Z

IGNITE-9552 Fix eslint issues in app code.

commit 1c16994c96292a3cbd94a6d31f27c9d24cc54528
Author: Ilya Borisov 
Date:   2018-09-19T04:49:35Z

IGNITE-9552 Disable emission of transpiled TS.




---

Re: About work around for https://issues.apache.org/jira/browse/IGNITE-5795

[kcheng.mvp]

I checked the source code just now and found *AffinityKey* is using
*@AffinityKeyMapped*
that's to  say *AffinityKey* has the same issue. 
am I right?



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

About work around for https://issues.apache.org/jira/browse/IGNITE-5795

[kcheng.mvp]

I ran into the same issue as
http://apache-ignite-users.70518.x6.nabble.com/SQL-SELECT-with-AffinityKeyMapped-no-results-td23141.html#a24232


from the document https://apacheignite.readme.io/docs/affinity-collocation,
there are two ways to achieve *Affinity*.  I would like to confirm that use
*AffinityKey* has the same issue or not?


Thanks.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Python thin client

[Prachi Garg]

Hi Dmitry,

Not sure if you are aware but all documentation for the thin clients will
be moved to readme.io. Here is the ticket for Python thin clients -
https://issues.apache.org/jira/browse/IGNITE-9522

I am already working on it. So if you make any changes to the docs in your
repo, please let me know so that I don't miss bringing them over to
readme.io

-Prachi

On Tue, Sep 18, 2018 at 4:08 AM, Igor Sapego  wrote:

> Great job.
>
> Best Regards,
> Igor
>
>
> On Tue, Sep 18, 2018 at 11:35 AM Dmitry Melnichuk <
> dmitry.melnic...@nobitlost.com> wrote:
>
> > Igor,
> >
> > All examples are in 'ignite/modules/platforms/python/examples'.
> >
> > I put examples in separate Python files mostly to be able to
> > automatically confirm their operability. All the lengthy explanations
> > and cross-references are in the main documentation:
> >
> >
> > https://apache-ignite-binary-protocol-client.readthedocs.
> io/en/latest/examples.html
> >
> > To make it easier for end users to navigate in pyignite directories, I
> > added a small README file to the 'examples' directory with a short
> > description and a link to the docs:
> >
> >
> > https://github.com/nobitlost/ignite/blob/ignite-7782/
> modules/platforms/python/examples/readme.md
> >
> > If you think this README is lacking something regardless of the main
> > pyignite docs, please let me know.
> >
> > On 9/17/18 8:59 PM, Igor Sapego wrote:
> > > Dmitry,
> > >
> > > Sorry, I was not clear enough. What I mean is that Ignite distributed
> > > by both source and binary releases. Binary releases contain platforms
> > > code, which is needed to write your own application in the language,
> > > but does not contain developer stuff, such as tests, documentation
> > > generating scripts, etc.
> > >
> > > Of course, it is more common to use package managers when
> > > developing your application, and of course, we are going to support
> > > this approach. But still, we provide a way for a user to get any client
> > > without any package manager.
> > >
> > > I like the idea of supplying whl package in a binary release, though it
> > > looks like it's going to take more effort to implement it. But I
> believe
> > > except for the whl package, we will need to supply examples and
> > > instructions, how user can run them.
> > >
> > > What do you think?
> > >
> > > Best Regards,
> > > Igor
> > >
> > >
> > > On Sat, Sep 15, 2018 at 7:03 AM Dmitry Melnichuk <
> > > dmitry.melnic...@nobitlost.com> wrote:
> > >
> > >> Igor,
> > >>
> > >> I am in doubt here because I am not fully comprehend the meaning of
> > >> "binary release". But if it is somehow related to the "distribution"
> > >> thing, I would dare to suggest the following options:
> > >>
> > >> 1. Copy nothing. Just do
> > >>
> > >> ```
> > >> $ python setup.py bdist_wheel
> > >> $ twine upload dist/*
> > >> ```
> > >>
> > >> during the build process (or separately) and let PyPI handle the
> > >> distribution.
> > >>
> > >> This is the most natural and user-friendly way of distributing Python
> > >> packages. End user might only do
> > >>
> > >> ```
> > >> $ pip install pyignite
> > >> ```
> > >>
> > >> as it is suggested by my readme file.
> > >>
> > >> 2. Supply the wheel package. It is the file 'pyignite-*.whl' from
> 'dist'
> > >> directory, that should appear there as the "python setup.py
> bdist_wheel"
> > >> command finishes. Actually, it can be combined with the first option.
> > >>
> > >> The wheel can then be installed by the end user:
> > >>
> > >> ```
> > >> $ pip install pyignite-*.whl
> > >> ```
> > >>
> > >> 3. Supply the following files:
> > >>
> > >> ignite/modules/platforms/python/pyignite/**
> > >> ignite/modules/platforms/python/requirements/**
> > >> ignite/modules/platforms/python/LICENSE
> > >> ignite/modules/platforms/python/README.md
> > >> ignite/modules/platforms/python/setup.py
> > >>
> > >> (** stands for "all the files and sub-folders recursively, starting
> from
> > >> this folder".)
> > >>
> > >> It is not uncommon or wrong to distribute Python programs as text
> > >> without prior packaging, but, in my personal experience, this is a way
> > >> more suitable for one-time scripts or proprietary programs. For
> example,
> > >> most of Python web apps relies on git for deployment, without any
> > >> packaging subsystem.
> > >>
> > >> Here's a few words about wheels:
> > >>
> > >> https://wheel.readthedocs.io/
> > >>
> > >> Here's about uploading to PyPI with twine:
> > >>
> > >>
> > >>
> > https://packaging.python.org/tutorials/packaging-projects/#
> uploading-the-distribution-archives
> > >>
> > >> On 9/14/18 9:05 PM, Igor Sapego wrote:
> > >>> Ok, good.
> > >>>
> > >>> Now, what is about installation? Which directories/files
> > >>> need to be copied to ignite's binary release?
> > >>>
> > >>> Best Regards,
> > >>> Igor
> > >>>
> > >>
> > >
> >
> >
>

Re: PHP thin client

[Prachi Garg]

Hi Stephan, Alexey

That's exactly what readme.io contains - installation instructions,
configuration, and examples for key-value, sql, etc. for thin clients. For
example, see these documentation pages for Node.js (currently hidden in the
latest version of the doc) :

https://apacheignite.readme.io/v2.6/docs/nodejs-thin-client
https://apacheignite.readme.io/v2.6/docs/nodejs-thin-client-initialization-and-configuration
https://apacheignite.readme.io/v2.6/docs/nodejs-thin-client-key-value
https://apacheignite.readme.io/v2.6/docs/nodejs-thin-client-sql
https://apacheignite.readme.io/v2.6/docs/nodejs-thin-client-binary-types
https://apacheignite.readme.io/v2.6/docs/nodejs-thin-client-security

This how Python and PHP thin clients will also be documented on readme.io

-Prachi




On Tue, Sep 18, 2018 at 3:26 AM, Stepan Pilshchikov <
pilshchikov@gmail.com> wrote:

> > You know, I'm confused with all this documentation stuff...
> > For nodejs client all docs were moved from the repo to readme.io but
> the
> > readme.md keeps the installation instructions (duplicated with
> > readme.io). Probably, that's ok.
> > Will add similar short readme.md to the PHP PR.
>
> Its good
>
> What i think (and how it partially now):
> All user documentation should be on readme.io (how to install, use it,
> configurate, description for examples)
> All developers documentation (how to release, how to start develop) and(!)
> basic description should be in repository
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Re: Cache scan efficiency

[Dmitriy Pavlov]

Even better, if RAM is exhausted page replacement process will be started.
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Pagereplacement(rotationwithdisk)

Effect of the preloading will be still markable, but not as excelled as
with full-fitting into RAM. Later I can review or improve javadocs if it is
necessary.

ср, 19 сент. 2018 г. в 0:18, Denis Magda :

> Agree, it's just a matter of the documentation. If a user stores 100% in
> RAM and on disk, and just wants to warm RAM up after a restart then he
> knows everything will fit there. If during the preloading we detect that
> the RAM is exhausted we can halt it and print out a warning.
>
> --
> Denis
>
> On Tue, Sep 18, 2018 at 2:10 PM Dmitriy Pavlov 
> wrote:
>
> > Hi,
> >
> > I totally support the idea of cache preload.
> >
> > IMO it can be expanded. We can iterate over local partitions of the cache
> > group and preload each.
> >
> > But it should be really clear documented methods so a user can be aware
> of
> > the benefits of such method (e.g. if RAM region is big enough, etc).
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > вт, 18 сент. 2018 г. в 21:36, Denis Magda :
> >
> > > Folks,
> > >
> > > Since we're adding a method that would preload a certain partition, can
> > we
> > > add the one which will preload the whole cache? Ignite persistence
> users
> > > I've been working with look puzzled once they realize there is no way
> to
> > > warm up RAM after the restart. There are use cases that require this.
> > >
> > > Can the current optimizations be expanded to the cache preloading use
> > case?
> > >
> > > --
> > > Denis
> > >
> > > On Tue, Sep 18, 2018 at 3:58 AM Alexei Scherbakov <
> > > alexey.scherbak...@gmail.com> wrote:
> > >
> > > > Summing up, I suggest adding new public
> > > > method IgniteCache.preloadPartition(partId).
> > > >
> > > > I will start preparing PR for IGNITE-8873
> > > >  if no more
> > > objections
> > > > follow.
> > > >
> > > >
> > > >
> > > > вт, 18 сент. 2018 г. в 10:50, Alexey Goncharuk <
> > > alexey.goncha...@gmail.com
> > > > >:
> > > >
> > > > > Dmitriy,
> > > > >
> > > > > In my understanding, the proper fix for the scan query looks like a
> > big
> > > > > change and it is unlikely that we include it in Ignite 2.7. On the
> > > other
> > > > > hand, the method suggested by Alexei is quite simple  and it
> > definitely
> > > > > fits Ignite 2.7, which will provide a better user experience. Even
> > > > having a
> > > > > proper scan query implemented this method can be useful in some
> > > specific
> > > > > scenarios, so we will not have to deprecate it.
> > > > >
> > > > > --AG
> > > > >
> > > > > пн, 17 сент. 2018 г. в 19:15, Dmitriy Pavlov <
> dpavlov@gmail.com
> > >:
> > > > >
> > > > > > As I understood it is not a hack, it is an advanced feature for
> > > warming
> > > > > up
> > > > > > the partition. We can build warm-up of the overall cache by
> calling
> > > its
> > > > > > partitions warm-up. Users often ask about this feature and are
> not
> > > > > > confident with our lazy upload.
> > > > > >
> > > > > > Please correct me if I misunderstood the idea.
> > > > > >
> > > > > > пн, 17 сент. 2018 г. в 18:37, Dmitriy Setrakyan <
> > > dsetrak...@apache.org
> > > > >:
> > > > > >
> > > > > > > I would rather fix the scan than hack the scan. Is there any
> > > > technical
> > > > > > > reason for hacking it now instead of fixing it properly? Can
> some
> > > of
> > > > > the
> > > > > > > experts in this thread provide an estimate of complexity and
> > > > difference
> > > > > > in
> > > > > > > work that would be required for each approach?
> > > > > > >
> > > > > > > D.
> > > > > > >
> > > > > > > On Mon, Sep 17, 2018 at 4:42 PM Alexey Goncharuk <
> > > > > > > alexey.goncha...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I think it would be beneficial for some Ignite users if we
> > added
> > > > > such a
> > > > > > > > partition warmup method to the public API. The method should
> be
> > > > > > > > well-documented and state that it may invalidate existing
> page
> > > > cache.
> > > > > > It
> > > > > > > > will be a very effective instrument until we add the proper
> > scan
> > > > > > ability
> > > > > > > > that Vladimir was referring to.
> > > > > > > >
> > > > > > > > пн, 17 сент. 2018 г. в 13:05, Maxim Muzafarov <
> > > maxmu...@gmail.com
> > > > >:
> > > > > > > >
> > > > > > > > > Folks,
> > > > > > > > >
> > > > > > > > > Such warming up can be an effective technique for
> performing
> > > > > > > calculations
> > > > > > > > > which required large cache
> > > > > > > > > data reads, but I think it's the single narrow use case of
> > all
> > > > over
> > > > > > > > Ignite
> > > > > > > > > store usages. Like all other
> > > > > > > > > powerfull techniques, we should use it wisely. In the
> general
> > > > > case, I
> > > > > > > > think
> > 

Re: Cache scan efficiency

[Denis Magda]

Agree, it's just a matter of the documentation. If a user stores 100% in
RAM and on disk, and just wants to warm RAM up after a restart then he
knows everything will fit there. If during the preloading we detect that
the RAM is exhausted we can halt it and print out a warning.

--
Denis

On Tue, Sep 18, 2018 at 2:10 PM Dmitriy Pavlov 
wrote:

> Hi,
>
> I totally support the idea of cache preload.
>
> IMO it can be expanded. We can iterate over local partitions of the cache
> group and preload each.
>
> But it should be really clear documented methods so a user can be aware of
> the benefits of such method (e.g. if RAM region is big enough, etc).
>
> Sincerely,
> Dmitriy Pavlov
>
> вт, 18 сент. 2018 г. в 21:36, Denis Magda :
>
> > Folks,
> >
> > Since we're adding a method that would preload a certain partition, can
> we
> > add the one which will preload the whole cache? Ignite persistence users
> > I've been working with look puzzled once they realize there is no way to
> > warm up RAM after the restart. There are use cases that require this.
> >
> > Can the current optimizations be expanded to the cache preloading use
> case?
> >
> > --
> > Denis
> >
> > On Tue, Sep 18, 2018 at 3:58 AM Alexei Scherbakov <
> > alexey.scherbak...@gmail.com> wrote:
> >
> > > Summing up, I suggest adding new public
> > > method IgniteCache.preloadPartition(partId).
> > >
> > > I will start preparing PR for IGNITE-8873
> > >  if no more
> > objections
> > > follow.
> > >
> > >
> > >
> > > вт, 18 сент. 2018 г. в 10:50, Alexey Goncharuk <
> > alexey.goncha...@gmail.com
> > > >:
> > >
> > > > Dmitriy,
> > > >
> > > > In my understanding, the proper fix for the scan query looks like a
> big
> > > > change and it is unlikely that we include it in Ignite 2.7. On the
> > other
> > > > hand, the method suggested by Alexei is quite simple  and it
> definitely
> > > > fits Ignite 2.7, which will provide a better user experience. Even
> > > having a
> > > > proper scan query implemented this method can be useful in some
> > specific
> > > > scenarios, so we will not have to deprecate it.
> > > >
> > > > --AG
> > > >
> > > > пн, 17 сент. 2018 г. в 19:15, Dmitriy Pavlov  >:
> > > >
> > > > > As I understood it is not a hack, it is an advanced feature for
> > warming
> > > > up
> > > > > the partition. We can build warm-up of the overall cache by calling
> > its
> > > > > partitions warm-up. Users often ask about this feature and are not
> > > > > confident with our lazy upload.
> > > > >
> > > > > Please correct me if I misunderstood the idea.
> > > > >
> > > > > пн, 17 сент. 2018 г. в 18:37, Dmitriy Setrakyan <
> > dsetrak...@apache.org
> > > >:
> > > > >
> > > > > > I would rather fix the scan than hack the scan. Is there any
> > > technical
> > > > > > reason for hacking it now instead of fixing it properly? Can some
> > of
> > > > the
> > > > > > experts in this thread provide an estimate of complexity and
> > > difference
> > > > > in
> > > > > > work that would be required for each approach?
> > > > > >
> > > > > > D.
> > > > > >
> > > > > > On Mon, Sep 17, 2018 at 4:42 PM Alexey Goncharuk <
> > > > > > alexey.goncha...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I think it would be beneficial for some Ignite users if we
> added
> > > > such a
> > > > > > > partition warmup method to the public API. The method should be
> > > > > > > well-documented and state that it may invalidate existing page
> > > cache.
> > > > > It
> > > > > > > will be a very effective instrument until we add the proper
> scan
> > > > > ability
> > > > > > > that Vladimir was referring to.
> > > > > > >
> > > > > > > пн, 17 сент. 2018 г. в 13:05, Maxim Muzafarov <
> > maxmu...@gmail.com
> > > >:
> > > > > > >
> > > > > > > > Folks,
> > > > > > > >
> > > > > > > > Such warming up can be an effective technique for performing
> > > > > > calculations
> > > > > > > > which required large cache
> > > > > > > > data reads, but I think it's the single narrow use case of
> all
> > > over
> > > > > > > Ignite
> > > > > > > > store usages. Like all other
> > > > > > > > powerfull techniques, we should use it wisely. In the general
> > > > case, I
> > > > > > > think
> > > > > > > > we should consider other
> > > > > > > > techniques mentioned by Vladimir and may create something
> like
> > > > > `global
> > > > > > > > statistics of cache data usage`
> > > > > > > > to choose the best technique in each case.
> > > > > > > >
> > > > > > > > For instance, it's not obvious what would take longer:
> > > multi-block
> > > > > > reads
> > > > > > > or
> > > > > > > > 50 single-block reads issues
> > > > > > > > sequentially. It strongly depends on used hardware under the
> > hood
> > > > and
> > > > > > > might
> > > > > > > > depend on workload system
> > > > > > > > resources (CPU-intensive calculations and I\O access) as
> well.
> > > But
> > > > > > > > `statistics` will help us to choose
> > > > > > > > the 

Re: Cache scan efficiency

[Dmitriy Pavlov]

Hi,

I totally support the idea of cache preload.

IMO it can be expanded. We can iterate over local partitions of the cache
group and preload each.

But it should be really clear documented methods so a user can be aware of
the benefits of such method (e.g. if RAM region is big enough, etc).

Sincerely,
Dmitriy Pavlov

вт, 18 сент. 2018 г. в 21:36, Denis Magda :

> Folks,
>
> Since we're adding a method that would preload a certain partition, can we
> add the one which will preload the whole cache? Ignite persistence users
> I've been working with look puzzled once they realize there is no way to
> warm up RAM after the restart. There are use cases that require this.
>
> Can the current optimizations be expanded to the cache preloading use case?
>
> --
> Denis
>
> On Tue, Sep 18, 2018 at 3:58 AM Alexei Scherbakov <
> alexey.scherbak...@gmail.com> wrote:
>
> > Summing up, I suggest adding new public
> > method IgniteCache.preloadPartition(partId).
> >
> > I will start preparing PR for IGNITE-8873
> >  if no more
> objections
> > follow.
> >
> >
> >
> > вт, 18 сент. 2018 г. в 10:50, Alexey Goncharuk <
> alexey.goncha...@gmail.com
> > >:
> >
> > > Dmitriy,
> > >
> > > In my understanding, the proper fix for the scan query looks like a big
> > > change and it is unlikely that we include it in Ignite 2.7. On the
> other
> > > hand, the method suggested by Alexei is quite simple  and it definitely
> > > fits Ignite 2.7, which will provide a better user experience. Even
> > having a
> > > proper scan query implemented this method can be useful in some
> specific
> > > scenarios, so we will not have to deprecate it.
> > >
> > > --AG
> > >
> > > пн, 17 сент. 2018 г. в 19:15, Dmitriy Pavlov :
> > >
> > > > As I understood it is not a hack, it is an advanced feature for
> warming
> > > up
> > > > the partition. We can build warm-up of the overall cache by calling
> its
> > > > partitions warm-up. Users often ask about this feature and are not
> > > > confident with our lazy upload.
> > > >
> > > > Please correct me if I misunderstood the idea.
> > > >
> > > > пн, 17 сент. 2018 г. в 18:37, Dmitriy Setrakyan <
> dsetrak...@apache.org
> > >:
> > > >
> > > > > I would rather fix the scan than hack the scan. Is there any
> > technical
> > > > > reason for hacking it now instead of fixing it properly? Can some
> of
> > > the
> > > > > experts in this thread provide an estimate of complexity and
> > difference
> > > > in
> > > > > work that would be required for each approach?
> > > > >
> > > > > D.
> > > > >
> > > > > On Mon, Sep 17, 2018 at 4:42 PM Alexey Goncharuk <
> > > > > alexey.goncha...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I think it would be beneficial for some Ignite users if we added
> > > such a
> > > > > > partition warmup method to the public API. The method should be
> > > > > > well-documented and state that it may invalidate existing page
> > cache.
> > > > It
> > > > > > will be a very effective instrument until we add the proper scan
> > > > ability
> > > > > > that Vladimir was referring to.
> > > > > >
> > > > > > пн, 17 сент. 2018 г. в 13:05, Maxim Muzafarov <
> maxmu...@gmail.com
> > >:
> > > > > >
> > > > > > > Folks,
> > > > > > >
> > > > > > > Such warming up can be an effective technique for performing
> > > > > calculations
> > > > > > > which required large cache
> > > > > > > data reads, but I think it's the single narrow use case of all
> > over
> > > > > > Ignite
> > > > > > > store usages. Like all other
> > > > > > > powerfull techniques, we should use it wisely. In the general
> > > case, I
> > > > > > think
> > > > > > > we should consider other
> > > > > > > techniques mentioned by Vladimir and may create something like
> > > > `global
> > > > > > > statistics of cache data usage`
> > > > > > > to choose the best technique in each case.
> > > > > > >
> > > > > > > For instance, it's not obvious what would take longer:
> > multi-block
> > > > > reads
> > > > > > or
> > > > > > > 50 single-block reads issues
> > > > > > > sequentially. It strongly depends on used hardware under the
> hood
> > > and
> > > > > > might
> > > > > > > depend on workload system
> > > > > > > resources (CPU-intensive calculations and I\O access) as well.
> > But
> > > > > > > `statistics` will help us to choose
> > > > > > > the right way.
> > > > > > >
> > > > > > >
> > > > > > > On Sun, 16 Sep 2018 at 23:59 Dmitriy Pavlov <
> > dpavlov@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Alexei,
> > > > > > > >
> > > > > > > > I did not find any PRs associated with the ticket for check
> > code
> > > > > > changes
> > > > > > > > behind this idea. Are there any PRs?
> > > > > > > >
> > > > > > > > If we create some forwards scan of pages, it should be a very
> > > > > > > intellectual
> > > > > > > > algorithm including a lot of parameters (how much RAM is
> free,
> > > how
> > > > > > > probably
> > > > > > > > we will need 

Re: Cache scan efficiency

[Denis Magda]

Folks,

Since we're adding a method that would preload a certain partition, can we
add the one which will preload the whole cache? Ignite persistence users
I've been working with look puzzled once they realize there is no way to
warm up RAM after the restart. There are use cases that require this.

Can the current optimizations be expanded to the cache preloading use case?

--
Denis

On Tue, Sep 18, 2018 at 3:58 AM Alexei Scherbakov <
alexey.scherbak...@gmail.com> wrote:

> Summing up, I suggest adding new public
> method IgniteCache.preloadPartition(partId).
>
> I will start preparing PR for IGNITE-8873
>  if no more objections
> follow.
>
>
>
> вт, 18 сент. 2018 г. в 10:50, Alexey Goncharuk  >:
>
> > Dmitriy,
> >
> > In my understanding, the proper fix for the scan query looks like a big
> > change and it is unlikely that we include it in Ignite 2.7. On the other
> > hand, the method suggested by Alexei is quite simple  and it definitely
> > fits Ignite 2.7, which will provide a better user experience. Even
> having a
> > proper scan query implemented this method can be useful in some specific
> > scenarios, so we will not have to deprecate it.
> >
> > --AG
> >
> > пн, 17 сент. 2018 г. в 19:15, Dmitriy Pavlov :
> >
> > > As I understood it is not a hack, it is an advanced feature for warming
> > up
> > > the partition. We can build warm-up of the overall cache by calling its
> > > partitions warm-up. Users often ask about this feature and are not
> > > confident with our lazy upload.
> > >
> > > Please correct me if I misunderstood the idea.
> > >
> > > пн, 17 сент. 2018 г. в 18:37, Dmitriy Setrakyan  >:
> > >
> > > > I would rather fix the scan than hack the scan. Is there any
> technical
> > > > reason for hacking it now instead of fixing it properly? Can some of
> > the
> > > > experts in this thread provide an estimate of complexity and
> difference
> > > in
> > > > work that would be required for each approach?
> > > >
> > > > D.
> > > >
> > > > On Mon, Sep 17, 2018 at 4:42 PM Alexey Goncharuk <
> > > > alexey.goncha...@gmail.com>
> > > > wrote:
> > > >
> > > > > I think it would be beneficial for some Ignite users if we added
> > such a
> > > > > partition warmup method to the public API. The method should be
> > > > > well-documented and state that it may invalidate existing page
> cache.
> > > It
> > > > > will be a very effective instrument until we add the proper scan
> > > ability
> > > > > that Vladimir was referring to.
> > > > >
> > > > > пн, 17 сент. 2018 г. в 13:05, Maxim Muzafarov  >:
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > > Such warming up can be an effective technique for performing
> > > > calculations
> > > > > > which required large cache
> > > > > > data reads, but I think it's the single narrow use case of all
> over
> > > > > Ignite
> > > > > > store usages. Like all other
> > > > > > powerfull techniques, we should use it wisely. In the general
> > case, I
> > > > > think
> > > > > > we should consider other
> > > > > > techniques mentioned by Vladimir and may create something like
> > > `global
> > > > > > statistics of cache data usage`
> > > > > > to choose the best technique in each case.
> > > > > >
> > > > > > For instance, it's not obvious what would take longer:
> multi-block
> > > > reads
> > > > > or
> > > > > > 50 single-block reads issues
> > > > > > sequentially. It strongly depends on used hardware under the hood
> > and
> > > > > might
> > > > > > depend on workload system
> > > > > > resources (CPU-intensive calculations and I\O access) as well.
> But
> > > > > > `statistics` will help us to choose
> > > > > > the right way.
> > > > > >
> > > > > >
> > > > > > On Sun, 16 Sep 2018 at 23:59 Dmitriy Pavlov <
> dpavlov@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi Alexei,
> > > > > > >
> > > > > > > I did not find any PRs associated with the ticket for check
> code
> > > > > changes
> > > > > > > behind this idea. Are there any PRs?
> > > > > > >
> > > > > > > If we create some forwards scan of pages, it should be a very
> > > > > > intellectual
> > > > > > > algorithm including a lot of parameters (how much RAM is free,
> > how
> > > > > > probably
> > > > > > > we will need next page, etc). We had the private talk about
> such
> > > idea
> > > > > > some
> > > > > > > time ago.
> > > > > > >
> > > > > > > By my experience, Linux systems already do such forward reading
> > of
> > > > file
> > > > > > > data (for corresponding sequential flagged file descriptors),
> but
> > > > some
> > > > > > > prefetching of data at the level of application may be useful
> for
> > > > > > O_DIRECT
> > > > > > > file descriptors.
> > > > > > >
> > > > > > > And one more concern from me is about selecting a right place
> in
> > > the
> > > > > > system
> > > > > > > to do such prefetch.
> > > > > > >
> > > > > > > Sincerely,
> > > > > > > Dmitriy Pavlov
> > > > > > >
> > > > > > > вс, 16 сент. 2018 

[GitHub] ignite pull request #4788: IGNITE-9514: Refactor tests and common test time

[zaleslaw]

GitHub user zaleslaw opened a pull request:

https://github.com/apache/ignite/pull/4788

IGNITE-9514: Refactor tests and common test time



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9514

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4788.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4788


commit 8d20b3ec8889cd4628866e7f70ff41d3790a0adb
Author: Zinoviev Alexey 
Date:   2018-09-18T16:32:59Z

Reduce amount of partitions

commit da1ffa0eb9f78d7b83698a9ecbb05fcc348eaaba
Author: Zinoviev Alexey 
Date:   2018-09-18T17:53:56Z

Fixed codestyle in tests

commit 924060adfc49b94fc788dd95a270947a4622b0a9
Author: Zinoviev Alexey 
Date:   2018-09-18T18:21:03Z

Refactor Trainer tests




---

Re: Apache Ignite 2.7 release

[Dmitriy Pavlov]

Hi Paul.

There are 2 PRs linked to that ticket. Who is reviewing your changes?

Branch for 2.7 is still master, so if your changes are reviewed and
accepted soon it will be in 2.7.

Sincerely,
Dmitriy Pavlov

вт, 18 сент. 2018 г. в 16:22, Paul Anderson :

> Hi, may I ask for IGNITE-9298 to be included in 2.7 pls
>
> On Tue, Sep 18, 2018 at 1:03 PM Nikolay Izhikov 
> wrote:
>
> > Hello, folks.
> >
> > Thanks for the comments.
> >
> > I will follow them.
> >
> > В Вт, 18/09/2018 в 13:31 +0300, Anton Vinogradov пишет:
> > > Nikolay,
> > >
> > > 1) *Do not* create ignite-2.7 branch until we're not started
> preparation
> > to
> > > real 2.7.
> > > Use some temporary branch for this check instead, eg.
> > > ignite-2.7-release-test
> > >
> > > 2) Please make sure you'll not cause real release actions (maven
> release
> > > and so on).
> > > Perform only vote_* steps.
> > >
> > > 3) Make sure you'll remove all tags, branches, and other RC artifacts
> > after
> > > check.
> > >
> > > 4) Mark this release as RC0 to make sure it will be clear to everybody
> > that
> > > it's a check.
> > >
> > >
> > > вт, 18 сент. 2018 г. в 13:24, Dmitriy Setrakyan  >:
> > >
> > > > If it is an Ignite release, then it has to pass through the vote. If
> > not,
> > > > then you can do the test without publishing or uploading the release.
> > > >
> > > > D.
> > > >
> > > > On Tue, Sep 18, 2018 at 1:18 PM Petr Ivanov 
> > wrote:
> > > >
> > > > > Ok.
> > > > >
> > > > > In case of TC questions — ask me.
> > > > >
> > > > >
> > > > >
> > > > > > On 18 Sep 2018, at 13:16, Nikolay Izhikov 
> > wrote:
> > > > > >
> > > > > > Hello, Petr.
> > > > > >
> > > > > > I want to make ignite-2.7 branch today.
> > > > > > And execute release procedure based on this branch.
> > > > > >
> > > > > > However, ignite-2.7 branch will be copy of master until code
> freeze
> > > >
> > > > date.
> > > > > >
> > > > > > В Вт, 18/09/2018 в 13:13 +0300, Petr Ivanov пишет:
> > > > > > > Will it be just a test or there is already ignite-2.7 branch?
> > > > > > >
> > > > > > > Fabric removal related TC modifications are not ready yet, and
> > code is
> > > > >
> > > > > not in master.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > On 18 Sep 2018, at 13:07, Nikolay Izhikov <
> nizhi...@apache.org
> > >
> > > >
> > > > wrote:
> > > > > > > >
> > > > > > > > Hello, Igniters.
> > > > > > > >
> > > > > > > > I want to start and release procedures and make an RC1 build.
> > > > > > > >
> > > > > > > > It has a 2 intention:
> > > > > > > >
> > > > > > > > 1. I want to walk through all release steps to make sure they
> > all
> > > > >
> > > > > works for me.
> > > > > > > > So I will be fully ready on release date.
> > > > > > > >
> > > > > > > > 2. We have updated some dependencies in 2.7 and we need to
> > make sure
> > > > >
> > > > > binary build is still workable.
> > > > > > > >
> > > > > > > > Any objections?
> > > > > > > >
> > > > > > > >
> > > > > > > > В Пт, 14/09/2018 в 18:52 +0300, Alexey Goncharuk пишет:
> > > > > > > > > We already have all the mechanics in place to work with
> > properties -
> > > > >
> > > > > we use
> > > > > > > > > ignite.build and ignite.revision from ignite.properties
> > which are
> > > > >
> > > > > adjusted
> > > > > > > > > during the build in the binary package.
> > > > > > > > >
> > > > > > > > > Should I create the ticket if there are no objections?
> > > > > > > > >
> > > > > > > > > пт, 14 сент. 2018 г. в 13:22, Ilya Kasnacheev <
> > > > >
> > > > > ilya.kasnach...@gmail.com>:
> > > > > > > > >
> > > > > > > > > > Hello!
> > > > > > > > > >
> > > > > > > > > > So now there's an issue that this script makes source
> > change after
> > > > >
> > > > > every
> > > > > > > > > > build, show up in git status.
> > > > > > > > > >
> > > > > > > > > > What we could do to it:
> > > > > > > > > > - Commit the changes after the build, once. In hopes that
> > it won't
> > > > >
> > > > > change
> > > > > > > > > > very often. With benefit that we could do that right now,
> > before
> > > >
> > > > the
> > > > > code
> > > > > > > > > > freeze.
> > > > > > > > > > - Move these values to a properties file from both
> pom.xml
> > and
> > > > > > > > > > IgniteProvider.java. Any problems with this approach?
> > We'll just
> > > > >
> > > > > read them
> > > > > > > > > > from classpath properties file.
> > > > > > > > > > - Update the links in the file once and remove them from
> > build
> > > > >
> > > > > process. Why
> > > > > > > > > > were they added to build process in the first place - to
> > make them
> > > > > > > > > > configurable during build?
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > --
> > > > > > > > > > Ilya Kasnacheev
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > вт, 11 сент. 2018 г. в 5:53, Roman Shtykh <
> > rsht...@yahoo.com>:
> > > > > > > > > >
> > > > > > > > > > > Ilya,
> > > > > > > > > > >
> > > > > > > > > > > The "latest" version is the default, and resolved by

Re: The future of Affinity / Topology concepts and possible PME optimizations.

[Dmitriy Setrakyan]

Ilya, I would suggest that you discuss everything here on the dev@ list,
including suggestions how to split the work.

D.

On Tue, Sep 18, 2018 at 6:34 PM Ilya Lantukh  wrote:

> Thanks for the feedback!
>
> I agree that we should start with the simplest optimizations, but it seems
> that decoupling affinity/topology versions is necessary before we can make
> any progress here, and this is a rather complex change all over the code.
>
> If anyone wants to help, please contact me privately and we will discuss
> how this work can be split.
>
> Denis Magda, do you think we should create IEP for these optimizations?
>
> On Tue, Sep 18, 2018 at 5:59 PM, Maxim Muzafarov 
> wrote:
>
> > Ilya,
> >
> >
> > > 3. Start node in baseline: both affinity and topology versions should
> be
> > incremented, but it might be possible to optimize PME for such case and
> > avoid cluster-wide freeze. Partition assignments for such node are
> already
> > calculated, so we can simply put them all into MOVING state. However, it
> > might take significant effort to avoid race conditions and redesign our
> > architecture.
> >
> > As you mentioned all assignments are already calculated. So as another
> > suggestion,
> > we can introduce a new `intermediate` state of such joined nodes. Being
> in
> > this state
> > node recovers all data from their local storage, preloads whole missed
> > partition
> > data from the cluster (probably on some point in time), creates and
> > preloads missed
> > in-memory and persistent caches. And only after these recovery such node
> > will fire
> > discovery join event and affinity\topology version may be incremented. I
> > think this
> > approach can help to reduce the further rebalance time.
> > WDYT?
> >
> >
> >
> > On Tue, 18 Sep 2018 at 16:31 Alexey Goncharuk <
> alexey.goncha...@gmail.com>
> > wrote:
> >
> > > Ilya,
> > >
> > > This is a great idea, but before we can ultimately decouple the
> affinity
> > > version from the topology version, we need to fix a few things with
> > > baseline topology first. Currently the in-memory caches are not using
> the
> > > baseline topology. We are going to fix this as a part of IEP-4 Phase II
> > > (baseline auto-adjust). Once fixed, we can safely assume that
> > > out-of-baseline node does not affect affinity distribution.
> > >
> > > Agree with Dmitriy that we should start with simpler optimizations
> first.
> > >
> > > чт, 13 сент. 2018 г. в 15:58, Ilya Lantukh :
> > >
> > > > Igniters,
> > > >
> > > > As most of you know, Ignite has a concept of AffinityTopologyVersion,
> > > which
> > > > is associated with nodes that are currently present in topology and a
> > > > global cluster state (active/inactive, baseline topology, started
> > > caches).
> > > > Modification of either of them involves process called Partition Map
> > > > Exchange (PME) and results in new AffinityTopologyVersion. At that
> > moment
> > > > all new cache and compute grid operations are globally "frozen". This
> > > might
> > > > lead to indeterminate cache downtimes.
> > > >
> > > > However, our recent changes (esp. introduction of Baseline Topology)
> > > caused
> > > > me to re-think those concept. Currently there are many cases when we
> > > > trigger PME, but it isn't necessary. For example, adding/removing
> > client
> > > > node or server node not in BLT should never cause partition map
> > > > modifications. Those events modify the *topology*, but *affinity* in
> > > > unaffected. On the other hand, there are events that affect only
> > > *affinity*
> > > > - most straightforward example is CacheAffinityChange event, which is
> > > > triggered after rebalance is finished to assign new primary/backup
> > nodes.
> > > > So the term *AffinityTopologyVersion* now looks weird - it tries to
> > > "merge"
> > > > two entities that aren't always related. To me it makes sense to
> > > introduce
> > > > separate *AffinityVersion *and *TopologyVersion*, review all events
> > that
> > > > currently modify AffinityTopologyVersion and split them into 3
> > > categories:
> > > > those that modify only AffinityVersion, only TopologyVersion and
> both.
> > It
> > > > will allow us to process such events using different mechanics and
> > avoid
> > > > redundant steps, and also reconsider mapping of operations - some
> will
> > be
> > > > mapped to topology, others - to affinity.
> > > >
> > > > Here is my view about how different event types theoretically can be
> > > > optimized:
> > > > 1. Client node start / stop: as stated above, no PME is needed,
> ticket
> > > > https://issues.apache.org/jira/browse/IGNITE-9558 is already in
> > > progress.
> > > > 2. Server node start / stop not from baseline: should be similar to
> the
> > > > previous case, since nodes outside of baseline cannot be partition
> > > owners.
> > > > 3. Start node in baseline: both affinity and topology versions should
> > be
> > > > incremented, but it might be possible to optimize PME for such case
> and
> > > > avoid 

Re: .net decimal -> Ignite is being stored as Other

[wayne theron]

Thanks, I was then incorrectly told to come here.

On 18 Sep 2018, 17:21, at 17:21, Dmitriy Pavlov  wrote:
>Let me explain. User list intented for user and developers
>communication.
>Igniters monitor user list. And why should we hide answers to this
>issue
>from other users?
>
>The dev. list related to contribution-related questions. For example,
>if
>someone has PR fixing issue in the product. So he or she will use dev.
>list.
>
>In this example case too deep technical question about how to make a
>contribution as perfect as possible is not interesting for users.
>
>Hope it makes sense for you.
>
>вт, 18 сент. 2018 г. в 19:11, wayne theron :
>
>> Is this not the ignite dev user forum. I am programming tables in
>ignite
>> so why do I need to go there when they always tell me to come here
>for dev
>> related questions?
>>
>> The issue is code and likely ignites ability to map. Net decimal to
>java
>> big decimal. I really don't think the user group is the place to ask,
>> really?
>>
>>
>>
>> On 18 Sep 2018, 16:55, at 16:55, Dmitriy Pavlov
>
>> wrote:
>> >Hi, is it related to some contribution? If not, I propose
>> >u...@ignite.apache.org as a better place to ask.
>> >
>> >вт, 18 сент. 2018 г. в 18:43, wt :
>> >
>> >> i have the following class
>> >>
>> >> [QuerySqlField]
>> >> public int vd { get; set; }
>> >> [QuerySqlField]
>> >> public long sharesinindex { get; set; }
>> >> [QuerySqlField]
>> >> public string name { get; set; }
>> >> [QuerySqlField]
>> >> public string isin { get; set; }
>> >> [QuerySqlField]
>> >> public string sedol { get; set; }
>> >> [QuerySqlField]
>> >> public string ric { get; set; }
>> >> [QuerySqlField]
>> >> public decimal close { get; set; }
>> >> [QuerySqlField]
>> >> public decimal rate { get; set; }
>> >>
>> >> when i configure ignite with this class in .net and start the
>server
>> >it
>> >> correctly sets all the other fields just not the decimals. The
>> >> documentation
>> >> states
>> >>
>> >> DECIMAL
>> >> Possible values: Data type with fixed precision and scale.
>> >>
>> >> Mapped to:
>> >>
>> >> Java/JDBC: java.math.BigDecimal
>> >> .NET/C#: decimal
>> >> C/C++: ignite::Decimal
>> >> ODBC: SQL_DECIMAL
>> >>
>> >>
>> >> Why is Ignite not mapping this correctly ->  tables.png
>> >> <
>> >>
>> >
>>
>http://apache-ignite-developers.2346864.n4.nabble.com/file/t604/tables.png
>> >
>> >>
>> >>
>> >>
>> >> here is the config
>> >>
>> >> var cfg = new IgniteConfiguration
>> >> {
>> >> DiscoverySpi = new TcpDiscoverySpi
>> >> {
>> >> IpFinder = new TcpDiscoveryStaticIpFinder
>> >> {
>> >> Endpoints = new[] {
>"127.0.0.1:47500..47509"
>> >}
>> >> },
>> >> SocketTimeout = TimeSpan.FromSeconds(0.3)
>> >> },
>> >> CacheConfiguration = new[]
>> >> {
>> >> new CacheConfiguration("IndexComposition")
>> >> {
>> >> SqlSchema = "IndexComposition",CacheMode =
>> >> CacheMode.Replicated,
>> >> QueryEntities = new []
>> >> {
>> >> new
>> >> QueryEntity(typeof(int),typeof(IndexComposition))
>> >> }
>> >> }
>> >> }
>> >>
>> >> };
>> >>
>> >>
>> >>
>> >> --
>> >> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>> >>
>>

Re: .net decimal -> Ignite is being stored as Other

[Dmitriy Pavlov]

Let me explain. User list intented for user and developers communication.
Igniters monitor user list. And why should we hide answers to this issue
from other users?

The dev. list related to contribution-related questions. For example, if
someone has PR fixing issue in the product. So he or she will use dev.
list.

In this example case too deep technical question about how to make a
contribution as perfect as possible is not interesting for users.

Hope it makes sense for you.

вт, 18 сент. 2018 г. в 19:11, wayne theron :

> Is this not the ignite dev user forum. I am programming tables in ignite
> so why do I need to go there when they always tell me to come here for dev
> related questions?
>
> The issue is code and likely ignites ability to map. Net decimal to java
> big decimal. I really don't think the user group is the place to ask,
> really?
>
>
>
> On 18 Sep 2018, 16:55, at 16:55, Dmitriy Pavlov 
> wrote:
> >Hi, is it related to some contribution? If not, I propose
> >u...@ignite.apache.org as a better place to ask.
> >
> >вт, 18 сент. 2018 г. в 18:43, wt :
> >
> >> i have the following class
> >>
> >> [QuerySqlField]
> >> public int vd { get; set; }
> >> [QuerySqlField]
> >> public long sharesinindex { get; set; }
> >> [QuerySqlField]
> >> public string name { get; set; }
> >> [QuerySqlField]
> >> public string isin { get; set; }
> >> [QuerySqlField]
> >> public string sedol { get; set; }
> >> [QuerySqlField]
> >> public string ric { get; set; }
> >> [QuerySqlField]
> >> public decimal close { get; set; }
> >> [QuerySqlField]
> >> public decimal rate { get; set; }
> >>
> >> when i configure ignite with this class in .net and start the server
> >it
> >> correctly sets all the other fields just not the decimals. The
> >> documentation
> >> states
> >>
> >> DECIMAL
> >> Possible values: Data type with fixed precision and scale.
> >>
> >> Mapped to:
> >>
> >> Java/JDBC: java.math.BigDecimal
> >> .NET/C#: decimal
> >> C/C++: ignite::Decimal
> >> ODBC: SQL_DECIMAL
> >>
> >>
> >> Why is Ignite not mapping this correctly ->  tables.png
> >> <
> >>
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/file/t604/tables.png
> >
> >>
> >>
> >>
> >> here is the config
> >>
> >> var cfg = new IgniteConfiguration
> >> {
> >> DiscoverySpi = new TcpDiscoverySpi
> >> {
> >> IpFinder = new TcpDiscoveryStaticIpFinder
> >> {
> >> Endpoints = new[] { "127.0.0.1:47500..47509"
> >}
> >> },
> >> SocketTimeout = TimeSpan.FromSeconds(0.3)
> >> },
> >> CacheConfiguration = new[]
> >> {
> >> new CacheConfiguration("IndexComposition")
> >> {
> >> SqlSchema = "IndexComposition",CacheMode =
> >> CacheMode.Replicated,
> >> QueryEntities = new []
> >> {
> >> new
> >> QueryEntity(typeof(int),typeof(IndexComposition))
> >> }
> >> }
> >> }
> >>
> >> };
> >>
> >>
> >>
> >> --
> >> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >>
>

Re: .net decimal -> Ignite is being stored as Other

[wayne theron]

Is this not the ignite dev user forum. I am programming tables in ignite so why 
do I need to go there when they always tell me to come here for dev related 
questions?

The issue is code and likely ignites ability to map. Net decimal to java big 
decimal. I really don't think the user group is the place to ask, really?



On 18 Sep 2018, 16:55, at 16:55, Dmitriy Pavlov  wrote:
>Hi, is it related to some contribution? If not, I propose
>u...@ignite.apache.org as a better place to ask.
>
>вт, 18 сент. 2018 г. в 18:43, wt :
>
>> i have the following class
>>
>> [QuerySqlField]
>> public int vd { get; set; }
>> [QuerySqlField]
>> public long sharesinindex { get; set; }
>> [QuerySqlField]
>> public string name { get; set; }
>> [QuerySqlField]
>> public string isin { get; set; }
>> [QuerySqlField]
>> public string sedol { get; set; }
>> [QuerySqlField]
>> public string ric { get; set; }
>> [QuerySqlField]
>> public decimal close { get; set; }
>> [QuerySqlField]
>> public decimal rate { get; set; }
>>
>> when i configure ignite with this class in .net and start the server
>it
>> correctly sets all the other fields just not the decimals. The
>> documentation
>> states
>>
>> DECIMAL
>> Possible values: Data type with fixed precision and scale.
>>
>> Mapped to:
>>
>> Java/JDBC: java.math.BigDecimal
>> .NET/C#: decimal
>> C/C++: ignite::Decimal
>> ODBC: SQL_DECIMAL
>>
>>
>> Why is Ignite not mapping this correctly ->  tables.png
>> <
>>
>http://apache-ignite-developers.2346864.n4.nabble.com/file/t604/tables.png>
>>
>>
>>
>> here is the config
>>
>> var cfg = new IgniteConfiguration
>> {
>> DiscoverySpi = new TcpDiscoverySpi
>> {
>> IpFinder = new TcpDiscoveryStaticIpFinder
>> {
>> Endpoints = new[] { "127.0.0.1:47500..47509"
>}
>> },
>> SocketTimeout = TimeSpan.FromSeconds(0.3)
>> },
>> CacheConfiguration = new[]
>> {
>> new CacheConfiguration("IndexComposition")
>> {
>> SqlSchema = "IndexComposition",CacheMode =
>> CacheMode.Replicated,
>> QueryEntities = new []
>> {
>> new
>> QueryEntity(typeof(int),typeof(IndexComposition))
>> }
>> }
>> }
>>
>> };
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>>

Re: .net decimal -> Ignite is being stored as Other

[Dmitriy Pavlov]

Hi, is it related to some contribution? If not, I propose
u...@ignite.apache.org as a better place to ask.

вт, 18 сент. 2018 г. в 18:43, wt :

> i have the following class
>
> [QuerySqlField]
> public int vd { get; set; }
> [QuerySqlField]
> public long sharesinindex { get; set; }
> [QuerySqlField]
> public string name { get; set; }
> [QuerySqlField]
> public string isin { get; set; }
> [QuerySqlField]
> public string sedol { get; set; }
> [QuerySqlField]
> public string ric { get; set; }
> [QuerySqlField]
> public decimal close { get; set; }
> [QuerySqlField]
> public decimal rate { get; set; }
>
> when i configure ignite with this class in .net and start the server it
> correctly sets all the other fields just not the decimals. The
> documentation
> states
>
> DECIMAL
> Possible values: Data type with fixed precision and scale.
>
> Mapped to:
>
> Java/JDBC: java.math.BigDecimal
> .NET/C#: decimal
> C/C++: ignite::Decimal
> ODBC: SQL_DECIMAL
>
>
> Why is Ignite not mapping this correctly ->  tables.png
> <
> http://apache-ignite-developers.2346864.n4.nabble.com/file/t604/tables.png>
>
>
>
> here is the config
>
> var cfg = new IgniteConfiguration
> {
> DiscoverySpi = new TcpDiscoverySpi
> {
> IpFinder = new TcpDiscoveryStaticIpFinder
> {
> Endpoints = new[] { "127.0.0.1:47500..47509" }
> },
> SocketTimeout = TimeSpan.FromSeconds(0.3)
> },
> CacheConfiguration = new[]
> {
> new CacheConfiguration("IndexComposition")
> {
> SqlSchema = "IndexComposition",CacheMode =
> CacheMode.Replicated,
> QueryEntities = new []
> {
> new
> QueryEntity(typeof(int),typeof(IndexComposition))
> }
> }
> }
>
> };
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

.net decimal -> Ignite is being stored as Other

[wt]

i have the following class

[QuerySqlField]
public int vd { get; set; }
[QuerySqlField]
public long sharesinindex { get; set; }
[QuerySqlField]
public string name { get; set; }
[QuerySqlField]
public string isin { get; set; }
[QuerySqlField]
public string sedol { get; set; }
[QuerySqlField]
public string ric { get; set; }
[QuerySqlField]
public decimal close { get; set; }
[QuerySqlField]
public decimal rate { get; set; }

when i configure ignite with this class in .net and start the server it
correctly sets all the other fields just not the decimals. The documentation
states

DECIMAL
Possible values: Data type with fixed precision and scale.

Mapped to:

Java/JDBC: java.math.BigDecimal
.NET/C#: decimal
C/C++: ignite::Decimal
ODBC: SQL_DECIMAL


Why is Ignite not mapping this correctly ->  tables.png
  


here is the config

var cfg = new IgniteConfiguration
{
DiscoverySpi = new TcpDiscoverySpi
{
IpFinder = new TcpDiscoveryStaticIpFinder
{
Endpoints = new[] { "127.0.0.1:47500..47509" }
},
SocketTimeout = TimeSpan.FromSeconds(0.3)
},
CacheConfiguration = new[]
{
new CacheConfiguration("IndexComposition")
{
SqlSchema = "IndexComposition",CacheMode =
CacheMode.Replicated,
QueryEntities = new []
{
new
QueryEntity(typeof(int),typeof(IndexComposition))
}
}
}

};



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: The future of Affinity / Topology concepts and possible PME optimizations.

[Ilya Lantukh]

Thanks for the feedback!

I agree that we should start with the simplest optimizations, but it seems
that decoupling affinity/topology versions is necessary before we can make
any progress here, and this is a rather complex change all over the code.

If anyone wants to help, please contact me privately and we will discuss
how this work can be split.

Denis Magda, do you think we should create IEP for these optimizations?

On Tue, Sep 18, 2018 at 5:59 PM, Maxim Muzafarov  wrote:

> Ilya,
>
>
> > 3. Start node in baseline: both affinity and topology versions should be
> incremented, but it might be possible to optimize PME for such case and
> avoid cluster-wide freeze. Partition assignments for such node are already
> calculated, so we can simply put them all into MOVING state. However, it
> might take significant effort to avoid race conditions and redesign our
> architecture.
>
> As you mentioned all assignments are already calculated. So as another
> suggestion,
> we can introduce a new `intermediate` state of such joined nodes. Being in
> this state
> node recovers all data from their local storage, preloads whole missed
> partition
> data from the cluster (probably on some point in time), creates and
> preloads missed
> in-memory and persistent caches. And only after these recovery such node
> will fire
> discovery join event and affinity\topology version may be incremented. I
> think this
> approach can help to reduce the further rebalance time.
> WDYT?
>
>
>
> On Tue, 18 Sep 2018 at 16:31 Alexey Goncharuk 
> wrote:
>
> > Ilya,
> >
> > This is a great idea, but before we can ultimately decouple the affinity
> > version from the topology version, we need to fix a few things with
> > baseline topology first. Currently the in-memory caches are not using the
> > baseline topology. We are going to fix this as a part of IEP-4 Phase II
> > (baseline auto-adjust). Once fixed, we can safely assume that
> > out-of-baseline node does not affect affinity distribution.
> >
> > Agree with Dmitriy that we should start with simpler optimizations first.
> >
> > чт, 13 сент. 2018 г. в 15:58, Ilya Lantukh :
> >
> > > Igniters,
> > >
> > > As most of you know, Ignite has a concept of AffinityTopologyVersion,
> > which
> > > is associated with nodes that are currently present in topology and a
> > > global cluster state (active/inactive, baseline topology, started
> > caches).
> > > Modification of either of them involves process called Partition Map
> > > Exchange (PME) and results in new AffinityTopologyVersion. At that
> moment
> > > all new cache and compute grid operations are globally "frozen". This
> > might
> > > lead to indeterminate cache downtimes.
> > >
> > > However, our recent changes (esp. introduction of Baseline Topology)
> > caused
> > > me to re-think those concept. Currently there are many cases when we
> > > trigger PME, but it isn't necessary. For example, adding/removing
> client
> > > node or server node not in BLT should never cause partition map
> > > modifications. Those events modify the *topology*, but *affinity* in
> > > unaffected. On the other hand, there are events that affect only
> > *affinity*
> > > - most straightforward example is CacheAffinityChange event, which is
> > > triggered after rebalance is finished to assign new primary/backup
> nodes.
> > > So the term *AffinityTopologyVersion* now looks weird - it tries to
> > "merge"
> > > two entities that aren't always related. To me it makes sense to
> > introduce
> > > separate *AffinityVersion *and *TopologyVersion*, review all events
> that
> > > currently modify AffinityTopologyVersion and split them into 3
> > categories:
> > > those that modify only AffinityVersion, only TopologyVersion and both.
> It
> > > will allow us to process such events using different mechanics and
> avoid
> > > redundant steps, and also reconsider mapping of operations - some will
> be
> > > mapped to topology, others - to affinity.
> > >
> > > Here is my view about how different event types theoretically can be
> > > optimized:
> > > 1. Client node start / stop: as stated above, no PME is needed, ticket
> > > https://issues.apache.org/jira/browse/IGNITE-9558 is already in
> > progress.
> > > 2. Server node start / stop not from baseline: should be similar to the
> > > previous case, since nodes outside of baseline cannot be partition
> > owners.
> > > 3. Start node in baseline: both affinity and topology versions should
> be
> > > incremented, but it might be possible to optimize PME for such case and
> > > avoid cluster-wide freeze. Partition assignments for such node are
> > already
> > > calculated, so we can simply put them all into MOVING state. However,
> it
> > > might take significant effort to avoid race conditions and redesign our
> > > architecture.
> > > 4. Cache start / stop: starting or stopping one cache doesn't modify
> > > partition maps for other caches. It should be possible to change this
> > > procedure to skip PME and perform all 

[GitHub] ignite pull request #4787: IGNITE-9639 Flaky failure of SqlSystemViewsSelfTe...

[alex-plekhanov]

GitHub user alex-plekhanov opened a pull request:

https://github.com/apache/ignite/pull/4787

IGNITE-9639 Flaky failure of SqlSystemViewsSelfTest



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alex-plekhanov/ignite ignite-9639

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4787.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4787


commit 80715bf1e721e9c72b98d7764d57aff8e0a5570a
Author: Aleksey Plekhanov 
Date:   2018-09-18T15:24:08Z

IGNITE-9639 Flaky failure of SqlSystemViewsSelfTest

commit b836a154864dbbd2edaaf97c5728a09f0fce4c9a
Author: Aleksey Plekhanov 
Date:   2018-09-18T15:26:10Z

IGNITE-9639 For TC run




---

[jira] [Created] (IGNITE-9640) [TC Bot] Determine repetitive failure types by analyzing build log

[Andrey Kuznetsov (JIRA)]

Andrey Kuznetsov created IGNITE-9640:


 Summary: [TC Bot] Determine repetitive failure types by analyzing 
build log
 Key: IGNITE-9640
 URL: https://issues.apache.org/jira/browse/IGNITE-9640
 Project: Ignite
  Issue Type: Task
Reporter: Andrey Kuznetsov


When someone is analyzing flaky test failure, it's important to distinguish 
between newly created failure and pre-existing one. In the latter case, the bot 
should not attract contributor's attention to the test.

In more detail, TC build log fragments starts with identical substrings for 
identical failures very often, e.g.

{noformat}
junit.framework.AssertionFailedError
at 
org.apache.ignite.internal.GridVersionSelfTest.testVersions(GridVersionSelfTest.java:54)
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4782: IGNITE-9631: Timeouts in ZookeeperDIscoverySpi te...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4782


---

[jira] [Created] (IGNITE-9639) Flaky failure of SqlSystemViewsSelfTest

[Aleksey Plekhanov (JIRA)]

Aleksey Plekhanov created IGNITE-9639:
-

 Summary: Flaky failure of SqlSystemViewsSelfTest 
 Key: IGNITE-9639
 URL: https://issues.apache.org/jira/browse/IGNITE-9639
 Project: Ignite
  Issue Type: Task
Reporter: Aleksey Plekhanov
Assignee: Aleksey Plekhanov


SqlSystemViewsSelfTest fails sometimes with the following error:

{noformat}
[2018-09-18 08:23:56,275][ERROR][main][root] Test failed.
javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: 
Failed to register MBean for component: 
org.apache.ignite.internal.processors.cache.CacheLocalMetricsMXBeanImpl@58f7fe1b
at 
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1326)
at 
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:3161)
at 
org.apache.ignite.internal.processors.query.SqlSystemViewsSelfTest.execSql(SqlSystemViewsSelfTest.java:76)
at 
org.apache.ignite.internal.processors.query.SqlSystemViewsSelfTest.testNodesViews(SqlSystemViewsSelfTest.java:386)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2177)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:143)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:2092)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to register 
MBean for component: 
org.apache.ignite.internal.processors.cache.CacheLocalMetricsMXBeanImpl@58f7fe1b
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.registerMbean(GridCacheProcessor.java:4312)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.createCache(GridCacheProcessor.java:1727)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1982)
at 
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.processClientCacheStartRequests(CacheAffinitySharedManager.java:439)
at 
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.processClientCachesChanges(CacheAffinitySharedManager.java:633)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.processCustomExchangeTask(GridCacheProcessor.java:379)
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.processCustomTask(GridCachePartitionExchangeManager.java:2393)
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2529)
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2457)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
... 1 more
Caused by: javax.management.InstanceAlreadyExistsException: 
org.apache:clsLdr=5c080ef3,igniteInstanceName=query.SqlSystemViewsSelfTest1,group=default,name="org.apache.ignite.internal.processors.cache.CacheLocalMetricsMXBeanImpl"
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at 
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at 
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at 
org.apache.ignite.internal.util.IgniteUtils.registerMBean(IgniteUtils.java:4614)
at 
org.apache.ignite.internal.util.IgniteUtils.registerMBean(IgniteUtils.java:4585)
at 
org.apache.ignite.internal.processors.cache.GridCacheProcessor.registerMbean(GridCacheProcessor.java:4308)
... 10 more
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Exchange stucks while node restoring state from WAL

[Maxim Muzafarov]

Igniters,

I would like this issue to be part of release 2.7. There is not much time
left
considering that I will have to correct comments after the review.

Will anyone help with review?

PR: https://github.com/apache/ignite/pull/4520/files
Upsource: https://reviews.ignite.apache.org/ignite/review/IGNT-CR-727
JIRA: https://issues.apache.org/jira/browse/IGNITE-7196

On Thu, 13 Sep 2018 at 11:27 Maxim Muzafarov  wrote:

> Igniters,
>
> I need your help with the review of this patch. In general, it will help
> to reduce PME duration. I've moved binary
> memory recovery at the moment of node startup as we've discussed it with
> Pavel previously this topic.
>
> Changes are relatively small (+299 −95) and they are ready. I've left a
> comment in JIRA with high-level
> implementation details. Hope, this will help with the review [1].
>
> Please, take a look.
> PR: https://github.com/apache/ignite/pull/4520/files
> Upsource: https://reviews.ignite.apache.org/ignite/review/IGNT-CR-727
> JIRA: https://issues.apache.org/jira/browse/IGNITE-7196
> TC Run All: [2]
>
>
> [1]
> https://issues.apache.org/jira/browse/IGNITE-7196?focusedCommentId=16613175=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16613175
> [2]
> https://ci.ignite.apache.org/viewLog.html?buildId=1835822=buildResultsDiv=IgniteTests24Java8_RunAll
>
>
> On Tue, 28 Aug 2018 at 17:59 Pavel Kovalenko  wrote:
>
>> Hello Maxim,
>>
>> I think you're going in the right direction, but why you perform binary
>> recovery only in the case when a node in the baseline?
>> I think that node can also perform binary recovery before the first
>> exchange is started in case of
>> 1) First activation
>> 2) Baseline change
>>
>> In all of these cases, you need just local available CacheDescriptors to
>> restore binary state from WAL.
>> I would like also see a test when Ignite is stopped in the middle of a
>> checkpoint and binary recovery succeed before PME in that case also.
>>
>>
>> ср, 22 авг. 2018 г. в 15:31, Pavel Kovalenko :
>>
>> > Hello Maxim,
>> >
>> > Thank you for your work. I will review your changes within the next
>> > several days.
>> >
>> > 2018-08-22 11:12 GMT+03:00 Maxim Muzafarov :
>> >
>> >> Pavel,
>> >>
>> >> Thank you for so detailed introduction into the root of the problem of
>> >> ticket IGNITE-7196.
>> >> As you mentioned before, we can divide this ticket into two sub-tasks.
>> So,
>> >> I will
>> >> file new issue for `Logical consistency recovery`. I think it's more
>> >> convenient way
>> >> for reviewing and merging each of these high level Steps (PRs).
>> >>
>> >> Currently, let's focus on `Physical(binary) consistency recovery` as
>> the
>> >> most critical
>> >> part of this issue and synchronize our vision and vision of other
>> >> community
>> >> members
>> >> on implementation details of Step 1. From this point everything what
>> I'm
>> >> saying is about
>> >> only binary recovery.
>> >>
>> >>
>> >> I think PR is almost ready (I need to check TC carefully).
>> >> Can you look at my changes? Am I going the right way?
>> >>
>> >> PR: https://github.com/apache/ignite/pull/4520
>> >> Upsource: https://reviews.ignite.apache.org/ignite/review/IGNT-CR-727
>> >> JIRA: https://issues.apache.org/jira/browse/IGNITE-7196
>> >>
>> >>
>> >> These are my milestones which I've tried to solve:
>> >> 1. On first time cluster activation no changes required. We should keep
>> >> this behavior.
>> >>
>> >> 2. `startMemoryPolicies` earlier if need.
>> >> Now at the master branch data regions starts at onActivate method
>> called.
>> >> If we are  trying
>> >> to restore binary state on node reconnects to cluster we should start
>> >> regions earlier.
>> >>
>> >> 3. `suspendLogging` method added to switch off handlers.
>> >> Keep fast PDS cleanup when joining is not in BLT (or belongs to another
>> >> BLT).
>> >> Restore memory and metastorage initialization required `WALWriter` and
>> >> `FileWriteHandle`
>> >> to be started. They can lock directory\files and block cleanups.
>> >>
>> >> 4. `isBaselineNode` check before resumming logging.
>> >> The same as point 3. Local node gets discovery data and determines
>> >> #isLocalNodeNotInBaseline.
>> >> If node not in current baseline it performs cleanup local PDS and we
>> >> should
>> >> reset state of
>> >> previously initialized at startup metastorage instance.
>> >>
>> >> 5. `cacheProcessorStarted` callback added to restore memory at startup.
>> >> We can restore memory only after having info about all
>> >> `CacheGroupDescriptor`. To achieve this
>> >> we should do restoring at the moment `GridCacheProcessor` started.
>> >>
>> >> 6. Move `fsync` for `MemoryRecoveryRecord` at node startup.
>> >> We are fsync'ing it inside running PME. Suppose, it used for recovery
>> >> actions and\or control
>> >> cluster state outside Ignite project (methods `nodeStart` and
>> >> `nodeStartedPointers` of this
>> >> are not used at Ignite project code). I've moved them at the node
>> startup,
>> 

Re: The future of Affinity / Topology concepts and possible PME optimizations.

[Maxim Muzafarov]

Ilya,


> 3. Start node in baseline: both affinity and topology versions should be
incremented, but it might be possible to optimize PME for such case and
avoid cluster-wide freeze. Partition assignments for such node are already
calculated, so we can simply put them all into MOVING state. However, it
might take significant effort to avoid race conditions and redesign our
architecture.

As you mentioned all assignments are already calculated. So as another
suggestion,
we can introduce a new `intermediate` state of such joined nodes. Being in
this state
node recovers all data from their local storage, preloads whole missed
partition
data from the cluster (probably on some point in time), creates and
preloads missed
in-memory and persistent caches. And only after these recovery such node
will fire
discovery join event and affinity\topology version may be incremented. I
think this
approach can help to reduce the further rebalance time.
WDYT?



On Tue, 18 Sep 2018 at 16:31 Alexey Goncharuk 
wrote:

> Ilya,
>
> This is a great idea, but before we can ultimately decouple the affinity
> version from the topology version, we need to fix a few things with
> baseline topology first. Currently the in-memory caches are not using the
> baseline topology. We are going to fix this as a part of IEP-4 Phase II
> (baseline auto-adjust). Once fixed, we can safely assume that
> out-of-baseline node does not affect affinity distribution.
>
> Agree with Dmitriy that we should start with simpler optimizations first.
>
> чт, 13 сент. 2018 г. в 15:58, Ilya Lantukh :
>
> > Igniters,
> >
> > As most of you know, Ignite has a concept of AffinityTopologyVersion,
> which
> > is associated with nodes that are currently present in topology and a
> > global cluster state (active/inactive, baseline topology, started
> caches).
> > Modification of either of them involves process called Partition Map
> > Exchange (PME) and results in new AffinityTopologyVersion. At that moment
> > all new cache and compute grid operations are globally "frozen". This
> might
> > lead to indeterminate cache downtimes.
> >
> > However, our recent changes (esp. introduction of Baseline Topology)
> caused
> > me to re-think those concept. Currently there are many cases when we
> > trigger PME, but it isn't necessary. For example, adding/removing client
> > node or server node not in BLT should never cause partition map
> > modifications. Those events modify the *topology*, but *affinity* in
> > unaffected. On the other hand, there are events that affect only
> *affinity*
> > - most straightforward example is CacheAffinityChange event, which is
> > triggered after rebalance is finished to assign new primary/backup nodes.
> > So the term *AffinityTopologyVersion* now looks weird - it tries to
> "merge"
> > two entities that aren't always related. To me it makes sense to
> introduce
> > separate *AffinityVersion *and *TopologyVersion*, review all events that
> > currently modify AffinityTopologyVersion and split them into 3
> categories:
> > those that modify only AffinityVersion, only TopologyVersion and both. It
> > will allow us to process such events using different mechanics and avoid
> > redundant steps, and also reconsider mapping of operations - some will be
> > mapped to topology, others - to affinity.
> >
> > Here is my view about how different event types theoretically can be
> > optimized:
> > 1. Client node start / stop: as stated above, no PME is needed, ticket
> > https://issues.apache.org/jira/browse/IGNITE-9558 is already in
> progress.
> > 2. Server node start / stop not from baseline: should be similar to the
> > previous case, since nodes outside of baseline cannot be partition
> owners.
> > 3. Start node in baseline: both affinity and topology versions should be
> > incremented, but it might be possible to optimize PME for such case and
> > avoid cluster-wide freeze. Partition assignments for such node are
> already
> > calculated, so we can simply put them all into MOVING state. However, it
> > might take significant effort to avoid race conditions and redesign our
> > architecture.
> > 4. Cache start / stop: starting or stopping one cache doesn't modify
> > partition maps for other caches. It should be possible to change this
> > procedure to skip PME and perform all necessary actions (compute
> affinity,
> > start/stop cache contexts on each node) in background, but it looks like
> a
> > very complex modification too.
> > 5. Rebalance finish: it seems possible to design a "lightweight" PME for
> > this case as well. If there were no node failures (and if there were, PME
> > should be triggered and rebalance should be cancelled anyways) all
> > partition states are already known by coordinator. Furthermore, no new
> > MOVING or OWNING node for any partition is introduced, so all previous
> > mappings should still be valid.
> >
> > For the latter complex cases in might be necessary to introduce "is
> > compatible" relationship between 

[GitHub] ignite pull request #4738: IGNITE-9330

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4738


---

[GitHub] ignite pull request #4786: IGNITE-8570

[xtern]

GitHub user xtern opened a pull request:

https://github.com/apache/ignite/pull/4786

IGNITE-8570



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xtern/ignite IGNITE-8570

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4786.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4786


commit d9bcad3752feb73988576fb4cc5181c5a7baff0d
Author: voipp 
Date:   2018-06-28T20:14:54Z

logger implemented

commit cbb1442ac9cf3a01fad3b7690cc1266ec6f61b55
Author: voipp 
Date:   2018-07-05T16:18:52Z

added listener for null-regexp. Removed waitFor

commit 53e4dd678376f0a5a530921e033660bee95f0787
Author: pereslegin-pa 
Date:   2018-09-18T13:41:52Z

IGNITE-8570 Code cleanup.




---

[GitHub] SomeFire opened a new pull request #13: Green table header for PRs without possible blockers

[GitBox]

SomeFire opened a new pull request #13: Green table header for PRs without 
possible blockers
URL: https://github.com/apache/ignite-teamcity-bot/pull/13
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Re: MTCGA: SqlSystemViewsSelfTest is flaky

[Alex Plehanov]

Ok, I will take a look at this test tomorrow.


вт, 18 сент. 2018 г. в 16:40, Alexey Goncharuk :

> Igniters,
>
> I noticed a pretty high raise in the SqlSystemViewsSelfTest failure rate
> recently [1]. There is a chance that the failure was introduced by another
> SQL system view ticket [2].
>
> Alexey, as an author of the change and the test, would you mind taking a
> look? The test failure looks simple (it fails with the MXBean registration
> failure).
>
> Thanks,
> --AG
>
> [1]
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7838705496181695292=%3Cdefault%3E=testDetails
> [2] https://issues.apache.org/jira/browse/IGNITE-9366
>

MTCGA: SqlSystemViewsSelfTest is flaky

[Alexey Goncharuk]

Igniters,

I noticed a pretty high raise in the SqlSystemViewsSelfTest failure rate
recently [1]. There is a chance that the failure was introduced by another
SQL system view ticket [2].

Alexey, as an author of the change and the test, would you mind taking a
look? The test failure looks simple (it fails with the MXBean registration
failure).

Thanks,
--AG

[1]
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7838705496181695292=%3Cdefault%3E=testDetails
[2] https://issues.apache.org/jira/browse/IGNITE-9366

Re: The future of Affinity / Topology concepts and possible PME optimizations.

[Alexey Goncharuk]

Ilya,

This is a great idea, but before we can ultimately decouple the affinity
version from the topology version, we need to fix a few things with
baseline topology first. Currently the in-memory caches are not using the
baseline topology. We are going to fix this as a part of IEP-4 Phase II
(baseline auto-adjust). Once fixed, we can safely assume that
out-of-baseline node does not affect affinity distribution.

Agree with Dmitriy that we should start with simpler optimizations first.

чт, 13 сент. 2018 г. в 15:58, Ilya Lantukh :

> Igniters,
>
> As most of you know, Ignite has a concept of AffinityTopologyVersion, which
> is associated with nodes that are currently present in topology and a
> global cluster state (active/inactive, baseline topology, started caches).
> Modification of either of them involves process called Partition Map
> Exchange (PME) and results in new AffinityTopologyVersion. At that moment
> all new cache and compute grid operations are globally "frozen". This might
> lead to indeterminate cache downtimes.
>
> However, our recent changes (esp. introduction of Baseline Topology) caused
> me to re-think those concept. Currently there are many cases when we
> trigger PME, but it isn't necessary. For example, adding/removing client
> node or server node not in BLT should never cause partition map
> modifications. Those events modify the *topology*, but *affinity* in
> unaffected. On the other hand, there are events that affect only *affinity*
> - most straightforward example is CacheAffinityChange event, which is
> triggered after rebalance is finished to assign new primary/backup nodes.
> So the term *AffinityTopologyVersion* now looks weird - it tries to "merge"
> two entities that aren't always related. To me it makes sense to introduce
> separate *AffinityVersion *and *TopologyVersion*, review all events that
> currently modify AffinityTopologyVersion and split them into 3 categories:
> those that modify only AffinityVersion, only TopologyVersion and both. It
> will allow us to process such events using different mechanics and avoid
> redundant steps, and also reconsider mapping of operations - some will be
> mapped to topology, others - to affinity.
>
> Here is my view about how different event types theoretically can be
> optimized:
> 1. Client node start / stop: as stated above, no PME is needed, ticket
> https://issues.apache.org/jira/browse/IGNITE-9558 is already in progress.
> 2. Server node start / stop not from baseline: should be similar to the
> previous case, since nodes outside of baseline cannot be partition owners.
> 3. Start node in baseline: both affinity and topology versions should be
> incremented, but it might be possible to optimize PME for such case and
> avoid cluster-wide freeze. Partition assignments for such node are already
> calculated, so we can simply put them all into MOVING state. However, it
> might take significant effort to avoid race conditions and redesign our
> architecture.
> 4. Cache start / stop: starting or stopping one cache doesn't modify
> partition maps for other caches. It should be possible to change this
> procedure to skip PME and perform all necessary actions (compute affinity,
> start/stop cache contexts on each node) in background, but it looks like a
> very complex modification too.
> 5. Rebalance finish: it seems possible to design a "lightweight" PME for
> this case as well. If there were no node failures (and if there were, PME
> should be triggered and rebalance should be cancelled anyways) all
> partition states are already known by coordinator. Furthermore, no new
> MOVING or OWNING node for any partition is introduced, so all previous
> mappings should still be valid.
>
> For the latter complex cases in might be necessary to introduce "is
> compatible" relationship between affinity versions. Operation needs to be
> remapped only if new version isn't compatible with the previous one.
>
> Please share your thoughts.
>
> --
> Best regards,
> Ilya
>

[jira] [Created] (IGNITE-9638) .Net: JVM keeps track of CLR Threads, even when they are finished

[Ilya Kasnacheev (JIRA)]

Ilya Kasnacheev created IGNITE-9638:
---

 Summary: .Net: JVM keeps track of CLR Threads, even when they are 
finished 
 Key: IGNITE-9638
 URL: https://issues.apache.org/jira/browse/IGNITE-9638
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Affects Versions: 2.6
Reporter: Ilya Kasnacheev


When you create a Thread in C#, JVM creates corresponding thread "Thread-" 
which is visible in jstack. When C# joins this thread, it is not removed from 
JVM and is kept around. This means that jstack may show thousands of threads 
which are not there. Reproducer is attached. It is presumed that memory will be 
exhausted eventually.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Apache Ignite 2.7 release

[Paul Anderson]

Hi, may I ask for IGNITE-9298 to be included in 2.7 pls

On Tue, Sep 18, 2018 at 1:03 PM Nikolay Izhikov  wrote:

> Hello, folks.
>
> Thanks for the comments.
>
> I will follow them.
>
> В Вт, 18/09/2018 в 13:31 +0300, Anton Vinogradov пишет:
> > Nikolay,
> >
> > 1) *Do not* create ignite-2.7 branch until we're not started preparation
> to
> > real 2.7.
> > Use some temporary branch for this check instead, eg.
> > ignite-2.7-release-test
> >
> > 2) Please make sure you'll not cause real release actions (maven release
> > and so on).
> > Perform only vote_* steps.
> >
> > 3) Make sure you'll remove all tags, branches, and other RC artifacts
> after
> > check.
> >
> > 4) Mark this release as RC0 to make sure it will be clear to everybody
> that
> > it's a check.
> >
> >
> > вт, 18 сент. 2018 г. в 13:24, Dmitriy Setrakyan :
> >
> > > If it is an Ignite release, then it has to pass through the vote. If
> not,
> > > then you can do the test without publishing or uploading the release.
> > >
> > > D.
> > >
> > > On Tue, Sep 18, 2018 at 1:18 PM Petr Ivanov 
> wrote:
> > >
> > > > Ok.
> > > >
> > > > In case of TC questions — ask me.
> > > >
> > > >
> > > >
> > > > > On 18 Sep 2018, at 13:16, Nikolay Izhikov 
> wrote:
> > > > >
> > > > > Hello, Petr.
> > > > >
> > > > > I want to make ignite-2.7 branch today.
> > > > > And execute release procedure based on this branch.
> > > > >
> > > > > However, ignite-2.7 branch will be copy of master until code freeze
> > >
> > > date.
> > > > >
> > > > > В Вт, 18/09/2018 в 13:13 +0300, Petr Ivanov пишет:
> > > > > > Will it be just a test or there is already ignite-2.7 branch?
> > > > > >
> > > > > > Fabric removal related TC modifications are not ready yet, and
> code is
> > > >
> > > > not in master.
> > > > > >
> > > > > >
> > > > > >
> > > > > > > On 18 Sep 2018, at 13:07, Nikolay Izhikov  >
> > >
> > > wrote:
> > > > > > >
> > > > > > > Hello, Igniters.
> > > > > > >
> > > > > > > I want to start and release procedures and make an RC1 build.
> > > > > > >
> > > > > > > It has a 2 intention:
> > > > > > >
> > > > > > > 1. I want to walk through all release steps to make sure they
> all
> > > >
> > > > works for me.
> > > > > > > So I will be fully ready on release date.
> > > > > > >
> > > > > > > 2. We have updated some dependencies in 2.7 and we need to
> make sure
> > > >
> > > > binary build is still workable.
> > > > > > >
> > > > > > > Any objections?
> > > > > > >
> > > > > > >
> > > > > > > В Пт, 14/09/2018 в 18:52 +0300, Alexey Goncharuk пишет:
> > > > > > > > We already have all the mechanics in place to work with
> properties -
> > > >
> > > > we use
> > > > > > > > ignite.build and ignite.revision from ignite.properties
> which are
> > > >
> > > > adjusted
> > > > > > > > during the build in the binary package.
> > > > > > > >
> > > > > > > > Should I create the ticket if there are no objections?
> > > > > > > >
> > > > > > > > пт, 14 сент. 2018 г. в 13:22, Ilya Kasnacheev <
> > > >
> > > > ilya.kasnach...@gmail.com>:
> > > > > > > >
> > > > > > > > > Hello!
> > > > > > > > >
> > > > > > > > > So now there's an issue that this script makes source
> change after
> > > >
> > > > every
> > > > > > > > > build, show up in git status.
> > > > > > > > >
> > > > > > > > > What we could do to it:
> > > > > > > > > - Commit the changes after the build, once. In hopes that
> it won't
> > > >
> > > > change
> > > > > > > > > very often. With benefit that we could do that right now,
> before
> > >
> > > the
> > > > code
> > > > > > > > > freeze.
> > > > > > > > > - Move these values to a properties file from both pom.xml
> and
> > > > > > > > > IgniteProvider.java. Any problems with this approach?
> We'll just
> > > >
> > > > read them
> > > > > > > > > from classpath properties file.
> > > > > > > > > - Update the links in the file once and remove them from
> build
> > > >
> > > > process. Why
> > > > > > > > > were they added to build process in the first place - to
> make them
> > > > > > > > > configurable during build?
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > --
> > > > > > > > > Ilya Kasnacheev
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > вт, 11 сент. 2018 г. в 5:53, Roman Shtykh <
> rsht...@yahoo.com>:
> > > > > > > > >
> > > > > > > > > > Ilya,
> > > > > > > > > >
> > > > > > > > > > The "latest" version is the default, and resolved by
> > > > > > > > > > https://ignite.apache.org/latest which is used by our
> web site
> > > >
> > > > when a
> > > > > > > > > > user download the latest Ignite version. And I think
> this is the
> > > > > > > > >
> > > > > > > > > authority
> > > > > > > > > > to judge of the latest official release (pom.xml you
> suggest can
> > > >
> > > > have
> > > > > > > > > > SNAPSHOTs etc.).
> > > > > > > > > > Also, as I explained during our review sessions,
> > >
> > > ignite-mesos-2.6.0
> > > > is a
> > > > > > > > > > driver and doesn't mean you need to have Ignite 2.6.0.
> User can
> > >
> > > run
> > > 

Re: Using materialised views for queries with joins

[Sergi Vladykin]

Sven,

Support of materialized views sounds like a huge project. I would not
expect it to appear in Ignite soon.

As far as I see you have problems with data collocation. If you can not
store the original data in replicated caches, then these views will be huge
as well and thus must be partitioned with some collocation. So it does not
change the picture too much actually.

I would suggest to go for a manual denormalization: put the same value with
different affinity keys.

Sergi

пн, 17 сент. 2018 г. в 14:22, Sven Beauprez :

> Hi Dmitry,
>
> Yes we can use those solutions in some cases, but not always.
>
> Replication is indeed the simplest, but sadly enough replicated caches are
> too much overhead. We have often a minimum of 12 nodes and all data must
> stay in sync 12x then. We do use it for small caches that don't need a lot
> of updates.
>
> We use colocation all over the place. Colocation based on affinity keys is
> not possible though for distinct data sets with only some very specific
> relationships with _some other_ dataset, well known before hand.
>  (eg. for example -not our exact use case which is more complex- items in
> a shopping basket with items from product inventory, both are in different
> caches managed on other nodes and it is not possible to denormalize such
> that the shopping basket knows the amount of availble items)
>
>
> Regards,
>
> Sven
>
>
>
>
> SVEN BEAUPREZ
>
> L e a d   A r c h i t e c t
>
>
>
> De Kleetlaan 5, B-1831 Diegem
>
> www.theglue.com 
>
> On 17/09/2018, 10:37, "Dmitriy Setrakyan"  wrote:
>
> Hi Sven,
>
> I will let others comment on the materialized view suggestion, but I
> have
> another question.
>
> *As we all know, joins are a nightmare in a distributed system*
>
>
> Have you considered collocation or replication? If a table is
> replicated,
> then it will be present on all the nodes and all joins will be fast.
> If two
> partitioned tables are colocated based on some affinity key, then
> joins on
> that affinity key will be fast as well.
>
> Both, colocation and replication are supported by Ignite. Will any of
> these
> approaches work for you?
>
> D.
>
> On Mon, Sep 17, 2018 at 11:04 AM Sven Beauprez <
> sven.beaup...@theglue.com>
> wrote:
>
> > All,
> >
> >
> >
> > We are in a situation where we have to query data over several
> caches. As
> > we all know, joins are a nightmare in a distributed system and I
> know there
> > are other means like denormalisation, but it is not sufficient
> anymore in
> > some cases we have and we need the joins.
> >
> >
> >
> > We mainly work in an OLTP context, where queries are known in
> advance (ie
> > dev time) and inpsired by following blog of several years ago, I was
> > wondering if the concept of “materialized views” could make it into
> Apache
> > Ignite.
> >
> > (
> >
> https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/
> > )
> >
> >
> >
> > It would work as follows:
> >
> >- A query must register itself in Ignite at startup time (eg. via
> >configuration) or during run time (eg. API call)
> >- The registered query is parsed and a new “view” cache is created
> >which will ‘cache’ the resultset of the query (could take a
> while, but
> >intermediate status can be “warming up” and “hot” when ready)
> >- All caches involved in the joins are now monitored for CUD
> >operations and relevant data is stored in the new “view” cache so
> the view
> >gets updated in real time
> >- All operations must be ACID compliant
> >- The view is queried via a very trivial select statement
> >
> >
> >
> > Would that be feasible as a new feature?
> >
> >
> >
> >
> >
> > Regards,
> >
> >
> >
> > Sven
> >
> >
> >
> >
> >
> >
> >
> > [image: cid:image001.png@01D3007B.4D007790]
> >
> >
> >
> > SVEN BEAUPREZ
> >
> >
> >
> > L e a d   A r c h i t e c t
> >
> >
> >
> > De Kleetlaan 5, B-1831 Diegem
> >
> > www.theglue.com
> >
>
>
>

[GitHub] ignite pull request #4753: IGNITE-9585 Error message sometimes refers nonexi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4753


---

Re: The future of Affinity / Topology concepts and possible PME optimizations.

[Anton Vinogradov]

Ilya,

I like your idea.
Let's create IEP and jira issues.
I will be glad to take a part in this journey once we'll discuss every
optimization in details.


чт, 13 сент. 2018 г. в 22:51, Dmitriy Setrakyan :

> Ilya,
>
> Thanks for the detailed explanation. Everything you suggested makes sense.
> Needless to say, PME effect on the grid should be minimized. Let's start
> with the simpler and less risky changes first.
>
> D.
>
> On Thu, Sep 13, 2018 at 5:58 AM Ilya Lantukh 
> wrote:
>
> > Igniters,
> >
> > As most of you know, Ignite has a concept of AffinityTopologyVersion,
> which
> > is associated with nodes that are currently present in topology and a
> > global cluster state (active/inactive, baseline topology, started
> caches).
> > Modification of either of them involves process called Partition Map
> > Exchange (PME) and results in new AffinityTopologyVersion. At that moment
> > all new cache and compute grid operations are globally "frozen". This
> might
> > lead to indeterminate cache downtimes.
> >
> > However, our recent changes (esp. introduction of Baseline Topology)
> caused
> > me to re-think those concept. Currently there are many cases when we
> > trigger PME, but it isn't necessary. For example, adding/removing client
> > node or server node not in BLT should never cause partition map
> > modifications. Those events modify the *topology*, but *affinity* in
> > unaffected. On the other hand, there are events that affect only
> *affinity*
> > - most straightforward example is CacheAffinityChange event, which is
> > triggered after rebalance is finished to assign new primary/backup nodes.
> > So the term *AffinityTopologyVersion* now looks weird - it tries to
> "merge"
> > two entities that aren't always related. To me it makes sense to
> introduce
> > separate *AffinityVersion *and *TopologyVersion*, review all events that
> > currently modify AffinityTopologyVersion and split them into 3
> categories:
> > those that modify only AffinityVersion, only TopologyVersion and both. It
> > will allow us to process such events using different mechanics and avoid
> > redundant steps, and also reconsider mapping of operations - some will be
> > mapped to topology, others - to affinity.
> >
> > Here is my view about how different event types theoretically can be
> > optimized:
> > 1. Client node start / stop: as stated above, no PME is needed, ticket
> > https://issues.apache.org/jira/browse/IGNITE-9558 is already in
> progress.
> > 2. Server node start / stop not from baseline: should be similar to the
> > previous case, since nodes outside of baseline cannot be partition
> owners.
> > 3. Start node in baseline: both affinity and topology versions should be
> > incremented, but it might be possible to optimize PME for such case and
> > avoid cluster-wide freeze. Partition assignments for such node are
> already
> > calculated, so we can simply put them all into MOVING state. However, it
> > might take significant effort to avoid race conditions and redesign our
> > architecture.
> > 4. Cache start / stop: starting or stopping one cache doesn't modify
> > partition maps for other caches. It should be possible to change this
> > procedure to skip PME and perform all necessary actions (compute
> affinity,
> > start/stop cache contexts on each node) in background, but it looks like
> a
> > very complex modification too.
> > 5. Rebalance finish: it seems possible to design a "lightweight" PME for
> > this case as well. If there were no node failures (and if there were, PME
> > should be triggered and rebalance should be cancelled anyways) all
> > partition states are already known by coordinator. Furthermore, no new
> > MOVING or OWNING node for any partition is introduced, so all previous
> > mappings should still be valid.
> >
> > For the latter complex cases in might be necessary to introduce "is
> > compatible" relationship between affinity versions. Operation needs to be
> > remapped only if new version isn't compatible with the previous one.
> >
> > Please share your thoughts.
> >
> > --
> > Best regards,
> > Ilya
> >
>

[GitHub] ignite pull request #4785: IGNITE-9550 Get operation returns null for a lost...

[ibessonov]

GitHub user ibessonov opened a pull request:

https://github.com/apache/ignite/pull/4785

IGNITE-9550 Get operation returns null for a lost partition with READ_SAFE 
policy



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9550

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4785.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4785


commit 09187b0525f5198d9822bae2b254c0761bf90e46
Author: ibessonov 
Date:   2018-09-14T15:58:10Z

IGNITE-9550 Added test that reproduces the problem reliably.

commit 106a0d606f5187498ae6983a9f501ff8576c32f5
Author: ibessonov 
Date:   2018-09-18T12:02:04Z

Merge branch 'master' into ignite-9550

commit 5e3d17c0ee0c0b987972b0344b7566a59387316c
Author: ibessonov 
Date:   2018-09-18T12:17:29Z

IGNITE-9550 Supposed fix. Documentation and minor cleanup is required




---

[GitHub] ignite pull request #4749: IGNITE-9022: Changed class label output in SVM

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4749


---

[GitHub] ignite pull request #4715: IGNITE-9158: Added Pipeline

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4715


---

[jira] [Created] (IGNITE-9637) SQL: Add indexes to data load benchmarks

[Pavel Kuznetsov (JIRA)]

Pavel Kuznetsov created IGNITE-9637:
---

 Summary: SQL: Add indexes to data load benchmarks
 Key: IGNITE-9637
 URL: https://issues.apache.org/jira/browse/IGNITE-9637
 Project: Ignite
  Issue Type: Bug
Reporter: Pavel Kuznetsov
Assignee: Pavel Kuznetsov


We want to measure data load with indexes.

We already have table with several fileds. We need benchmark parameter that 
handles number of indexes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9636) CacheBaselineTopologyTest#testClusterActiveWhileBaselineChanging is flaky in master

[Alexey Goncharuk (JIRA)]

Alexey Goncharuk created IGNITE-9636:


 Summary: 
CacheBaselineTopologyTest#testClusterActiveWhileBaselineChanging is flaky in 
master
 Key: IGNITE-9636
 URL: https://issues.apache.org/jira/browse/IGNITE-9636
 Project: Ignite
  Issue Type: Test
Reporter: Alexey Goncharuk


The test asynchronously sets baseline topology and while the transition happens 
checks that public API active state returns true. Sometimes this fails because 
publicApiActiveState(false) checks for transition and may return false if 
transition is in progress.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Python thin client

[Igor Sapego]

Great job.

Best Regards,
Igor


On Tue, Sep 18, 2018 at 11:35 AM Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:

> Igor,
>
> All examples are in 'ignite/modules/platforms/python/examples'.
>
> I put examples in separate Python files mostly to be able to
> automatically confirm their operability. All the lengthy explanations
> and cross-references are in the main documentation:
>
>
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/examples.html
>
> To make it easier for end users to navigate in pyignite directories, I
> added a small README file to the 'examples' directory with a short
> description and a link to the docs:
>
>
> https://github.com/nobitlost/ignite/blob/ignite-7782/modules/platforms/python/examples/readme.md
>
> If you think this README is lacking something regardless of the main
> pyignite docs, please let me know.
>
> On 9/17/18 8:59 PM, Igor Sapego wrote:
> > Dmitry,
> >
> > Sorry, I was not clear enough. What I mean is that Ignite distributed
> > by both source and binary releases. Binary releases contain platforms
> > code, which is needed to write your own application in the language,
> > but does not contain developer stuff, such as tests, documentation
> > generating scripts, etc.
> >
> > Of course, it is more common to use package managers when
> > developing your application, and of course, we are going to support
> > this approach. But still, we provide a way for a user to get any client
> > without any package manager.
> >
> > I like the idea of supplying whl package in a binary release, though it
> > looks like it's going to take more effort to implement it. But I believe
> > except for the whl package, we will need to supply examples and
> > instructions, how user can run them.
> >
> > What do you think?
> >
> > Best Regards,
> > Igor
> >
> >
> > On Sat, Sep 15, 2018 at 7:03 AM Dmitry Melnichuk <
> > dmitry.melnic...@nobitlost.com> wrote:
> >
> >> Igor,
> >>
> >> I am in doubt here because I am not fully comprehend the meaning of
> >> "binary release". But if it is somehow related to the "distribution"
> >> thing, I would dare to suggest the following options:
> >>
> >> 1. Copy nothing. Just do
> >>
> >> ```
> >> $ python setup.py bdist_wheel
> >> $ twine upload dist/*
> >> ```
> >>
> >> during the build process (or separately) and let PyPI handle the
> >> distribution.
> >>
> >> This is the most natural and user-friendly way of distributing Python
> >> packages. End user might only do
> >>
> >> ```
> >> $ pip install pyignite
> >> ```
> >>
> >> as it is suggested by my readme file.
> >>
> >> 2. Supply the wheel package. It is the file 'pyignite-*.whl' from 'dist'
> >> directory, that should appear there as the "python setup.py bdist_wheel"
> >> command finishes. Actually, it can be combined with the first option.
> >>
> >> The wheel can then be installed by the end user:
> >>
> >> ```
> >> $ pip install pyignite-*.whl
> >> ```
> >>
> >> 3. Supply the following files:
> >>
> >> ignite/modules/platforms/python/pyignite/**
> >> ignite/modules/platforms/python/requirements/**
> >> ignite/modules/platforms/python/LICENSE
> >> ignite/modules/platforms/python/README.md
> >> ignite/modules/platforms/python/setup.py
> >>
> >> (** stands for "all the files and sub-folders recursively, starting from
> >> this folder".)
> >>
> >> It is not uncommon or wrong to distribute Python programs as text
> >> without prior packaging, but, in my personal experience, this is a way
> >> more suitable for one-time scripts or proprietary programs. For example,
> >> most of Python web apps relies on git for deployment, without any
> >> packaging subsystem.
> >>
> >> Here's a few words about wheels:
> >>
> >> https://wheel.readthedocs.io/
> >>
> >> Here's about uploading to PyPI with twine:
> >>
> >>
> >>
> https://packaging.python.org/tutorials/packaging-projects/#uploading-the-distribution-archives
> >>
> >> On 9/14/18 9:05 PM, Igor Sapego wrote:
> >>> Ok, good.
> >>>
> >>> Now, what is about installation? Which directories/files
> >>> need to be copied to ignite's binary release?
> >>>
> >>> Best Regards,
> >>> Igor
> >>>
> >>
> >
>
>

[jira] [Created] (IGNITE-9635) Eternal initial partition map exchange

[Alexey Platonov (JIRA)]

Alexey Platonov created IGNITE-9635:
---

 Summary: Eternal initial partition map exchange
 Key: IGNITE-9635
 URL: https://issues.apache.org/jira/browse/IGNITE-9635
 Project: Ignite
  Issue Type: Bug
Reporter: Alexey Platonov


Sometimes test suites times out on test class 
GridCacheReplicatedNodeRestartSelfTest. Finally, some test from this class 
blocked on awaiting node starting due to eternal initial partition map exchange.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9634) [ML] Trainers as pipeline parameters that can be varied

[Aleksey Zinoviev (JIRA)]

Aleksey Zinoviev created IGNITE-9634:


 Summary: [ML] Trainers as pipeline parameters that can be varied
 Key: IGNITE-9634
 URL: https://issues.apache.org/jira/browse/IGNITE-9634
 Project: Ignite
  Issue Type: Sub-task
  Components: ml
Affects Versions: 2.8
Reporter: Aleksey Zinoviev
Assignee: Aleksey Zinoviev
 Fix For: 2.8


Based 
http://apache-ignite-developers.2346864.n4.nabble.com/ML-New-Feature-Trainers-as-pipeline-parameters-that-can-be-varied-td35132.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ML] [New Feature] Trainers as pipeline parameters that can be varied

[Alexey Zinoviev]

I think it's great proposal, I've created a ticket
https://issues.apache.org/jira/browse/IGNITE-9634



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[jira] [Created] (IGNITE-9633) [ML] Hyperparameter tuning improvements umbrella ticket

[Aleksey Zinoviev (JIRA)]

Aleksey Zinoviev created IGNITE-9633:


 Summary: [ML] Hyperparameter tuning improvements umbrella ticket
 Key: IGNITE-9633
 URL: https://issues.apache.org/jira/browse/IGNITE-9633
 Project: Ignite
  Issue Type: New Feature
  Components: ml
Affects Versions: 2.8
Reporter: Aleksey Zinoviev
Assignee: Aleksey Zinoviev
 Fix For: 2.8


Umbrella ticket for all hyperparameter tuning improvements



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Cache scan efficiency

[Dmitriy Setrakyan]

On Tue, Sep 18, 2018 at 1:58 PM Alexei Scherbakov <
alexey.scherbak...@gmail.com> wrote:

> Summing up, I suggest adding new public
> method IgniteCache.preloadPartition(partId).
>
> I will start preparing PR for IGNITE-8873
>  if no more objections
> follow.
>


Alexey, let's make sure we document this feature very well in Javadoc, as
well as in public readme.io documentation. Also, all cache iterator methods
and SCAN queries should be documented, suggesting when the partitions
should be preloaded to achieve better performance.

D.

Re: Apache Ignite 2.7 release

[Nikolay Izhikov]

Hello, folks.

Thanks for the comments.

I will follow them.

В Вт, 18/09/2018 в 13:31 +0300, Anton Vinogradov пишет:
> Nikolay,
> 
> 1) *Do not* create ignite-2.7 branch until we're not started preparation to
> real 2.7.
> Use some temporary branch for this check instead, eg.
> ignite-2.7-release-test
> 
> 2) Please make sure you'll not cause real release actions (maven release
> and so on).
> Perform only vote_* steps.
> 
> 3) Make sure you'll remove all tags, branches, and other RC artifacts after
> check.
> 
> 4) Mark this release as RC0 to make sure it will be clear to everybody that
> it's a check.
> 
> 
> вт, 18 сент. 2018 г. в 13:24, Dmitriy Setrakyan :
> 
> > If it is an Ignite release, then it has to pass through the vote. If not,
> > then you can do the test without publishing or uploading the release.
> > 
> > D.
> > 
> > On Tue, Sep 18, 2018 at 1:18 PM Petr Ivanov  wrote:
> > 
> > > Ok.
> > > 
> > > In case of TC questions — ask me.
> > > 
> > > 
> > > 
> > > > On 18 Sep 2018, at 13:16, Nikolay Izhikov  wrote:
> > > > 
> > > > Hello, Petr.
> > > > 
> > > > I want to make ignite-2.7 branch today.
> > > > And execute release procedure based on this branch.
> > > > 
> > > > However, ignite-2.7 branch will be copy of master until code freeze
> > 
> > date.
> > > > 
> > > > В Вт, 18/09/2018 в 13:13 +0300, Petr Ivanov пишет:
> > > > > Will it be just a test or there is already ignite-2.7 branch?
> > > > > 
> > > > > Fabric removal related TC modifications are not ready yet, and code is
> > > 
> > > not in master.
> > > > > 
> > > > > 
> > > > > 
> > > > > > On 18 Sep 2018, at 13:07, Nikolay Izhikov 
> > 
> > wrote:
> > > > > > 
> > > > > > Hello, Igniters.
> > > > > > 
> > > > > > I want to start and release procedures and make an RC1 build.
> > > > > > 
> > > > > > It has a 2 intention:
> > > > > > 
> > > > > > 1. I want to walk through all release steps to make sure they all
> > > 
> > > works for me.
> > > > > > So I will be fully ready on release date.
> > > > > > 
> > > > > > 2. We have updated some dependencies in 2.7 and we need to make sure
> > > 
> > > binary build is still workable.
> > > > > > 
> > > > > > Any objections?
> > > > > > 
> > > > > > 
> > > > > > В Пт, 14/09/2018 в 18:52 +0300, Alexey Goncharuk пишет:
> > > > > > > We already have all the mechanics in place to work with 
> > > > > > > properties -
> > > 
> > > we use
> > > > > > > ignite.build and ignite.revision from ignite.properties which are
> > > 
> > > adjusted
> > > > > > > during the build in the binary package.
> > > > > > > 
> > > > > > > Should I create the ticket if there are no objections?
> > > > > > > 
> > > > > > > пт, 14 сент. 2018 г. в 13:22, Ilya Kasnacheev <
> > > 
> > > ilya.kasnach...@gmail.com>:
> > > > > > > 
> > > > > > > > Hello!
> > > > > > > > 
> > > > > > > > So now there's an issue that this script makes source change 
> > > > > > > > after
> > > 
> > > every
> > > > > > > > build, show up in git status.
> > > > > > > > 
> > > > > > > > What we could do to it:
> > > > > > > > - Commit the changes after the build, once. In hopes that it 
> > > > > > > > won't
> > > 
> > > change
> > > > > > > > very often. With benefit that we could do that right now, before
> > 
> > the
> > > code
> > > > > > > > freeze.
> > > > > > > > - Move these values to a properties file from both pom.xml and
> > > > > > > > IgniteProvider.java. Any problems with this approach? We'll just
> > > 
> > > read them
> > > > > > > > from classpath properties file.
> > > > > > > > - Update the links in the file once and remove them from build
> > > 
> > > process. Why
> > > > > > > > were they added to build process in the first place - to make 
> > > > > > > > them
> > > > > > > > configurable during build?
> > > > > > > > 
> > > > > > > > Regards,
> > > > > > > > --
> > > > > > > > Ilya Kasnacheev
> > > > > > > > 
> > > > > > > > 
> > > > > > > > вт, 11 сент. 2018 г. в 5:53, Roman Shtykh :
> > > > > > > > 
> > > > > > > > > Ilya,
> > > > > > > > > 
> > > > > > > > > The "latest" version is the default, and resolved by
> > > > > > > > > https://ignite.apache.org/latest which is used by our web site
> > > 
> > > when a
> > > > > > > > > user download the latest Ignite version. And I think this is 
> > > > > > > > > the
> > > > > > > > 
> > > > > > > > authority
> > > > > > > > > to judge of the latest official release (pom.xml you suggest 
> > > > > > > > > can
> > > 
> > > have
> > > > > > > > > SNAPSHOTs etc.).
> > > > > > > > > Also, as I explained during our review sessions,
> > 
> > ignite-mesos-2.6.0
> > > is a
> > > > > > > > > driver and doesn't mean you need to have Ignite 2.6.0. User 
> > > > > > > > > can
> > 
> > run
> > > any
> > > > > > > > > version of Ignite he/she specifies. By default, it's "latest" 
> > > > > > > > > but
> > 
> > a
> > > user
> > > > > > > > > can specify any version needed, even from a non-archive URL.
> > > > > > > > > 
> > > > > > > > > In short, what we have now
> > > > > > > > > 1. mesos 

Re: Upsource lags and connection timeouts

[Anton Vinogradov]

Also, I see following on each resolve attempt.
UpsourceRequestExceptionImpl: 108: Internal error: An error occurred during
flushing data to database



вт, 18 сент. 2018 г. в 13:56, Anton Vinogradov :

> Folks,
>
> Who is responsible for Upsouce [1]?
> I see a performance issues last week.
> Can we check Upsource health?
>
> [1] https://reviews.ignite.apache.org
>

Re: Cache scan efficiency

[Alexei Scherbakov]

Summing up, I suggest adding new public
method IgniteCache.preloadPartition(partId).

I will start preparing PR for IGNITE-8873
 if no more objections
follow.



вт, 18 сент. 2018 г. в 10:50, Alexey Goncharuk :

> Dmitriy,
>
> In my understanding, the proper fix for the scan query looks like a big
> change and it is unlikely that we include it in Ignite 2.7. On the other
> hand, the method suggested by Alexei is quite simple  and it definitely
> fits Ignite 2.7, which will provide a better user experience. Even having a
> proper scan query implemented this method can be useful in some specific
> scenarios, so we will not have to deprecate it.
>
> --AG
>
> пн, 17 сент. 2018 г. в 19:15, Dmitriy Pavlov :
>
> > As I understood it is not a hack, it is an advanced feature for warming
> up
> > the partition. We can build warm-up of the overall cache by calling its
> > partitions warm-up. Users often ask about this feature and are not
> > confident with our lazy upload.
> >
> > Please correct me if I misunderstood the idea.
> >
> > пн, 17 сент. 2018 г. в 18:37, Dmitriy Setrakyan :
> >
> > > I would rather fix the scan than hack the scan. Is there any technical
> > > reason for hacking it now instead of fixing it properly? Can some of
> the
> > > experts in this thread provide an estimate of complexity and difference
> > in
> > > work that would be required for each approach?
> > >
> > > D.
> > >
> > > On Mon, Sep 17, 2018 at 4:42 PM Alexey Goncharuk <
> > > alexey.goncha...@gmail.com>
> > > wrote:
> > >
> > > > I think it would be beneficial for some Ignite users if we added
> such a
> > > > partition warmup method to the public API. The method should be
> > > > well-documented and state that it may invalidate existing page cache.
> > It
> > > > will be a very effective instrument until we add the proper scan
> > ability
> > > > that Vladimir was referring to.
> > > >
> > > > пн, 17 сент. 2018 г. в 13:05, Maxim Muzafarov :
> > > >
> > > > > Folks,
> > > > >
> > > > > Such warming up can be an effective technique for performing
> > > calculations
> > > > > which required large cache
> > > > > data reads, but I think it's the single narrow use case of all over
> > > > Ignite
> > > > > store usages. Like all other
> > > > > powerfull techniques, we should use it wisely. In the general
> case, I
> > > > think
> > > > > we should consider other
> > > > > techniques mentioned by Vladimir and may create something like
> > `global
> > > > > statistics of cache data usage`
> > > > > to choose the best technique in each case.
> > > > >
> > > > > For instance, it's not obvious what would take longer: multi-block
> > > reads
> > > > or
> > > > > 50 single-block reads issues
> > > > > sequentially. It strongly depends on used hardware under the hood
> and
> > > > might
> > > > > depend on workload system
> > > > > resources (CPU-intensive calculations and I\O access) as well. But
> > > > > `statistics` will help us to choose
> > > > > the right way.
> > > > >
> > > > >
> > > > > On Sun, 16 Sep 2018 at 23:59 Dmitriy Pavlov  >
> > > > wrote:
> > > > >
> > > > > > Hi Alexei,
> > > > > >
> > > > > > I did not find any PRs associated with the ticket for check code
> > > > changes
> > > > > > behind this idea. Are there any PRs?
> > > > > >
> > > > > > If we create some forwards scan of pages, it should be a very
> > > > > intellectual
> > > > > > algorithm including a lot of parameters (how much RAM is free,
> how
> > > > > probably
> > > > > > we will need next page, etc). We had the private talk about such
> > idea
> > > > > some
> > > > > > time ago.
> > > > > >
> > > > > > By my experience, Linux systems already do such forward reading
> of
> > > file
> > > > > > data (for corresponding sequential flagged file descriptors), but
> > > some
> > > > > > prefetching of data at the level of application may be useful for
> > > > > O_DIRECT
> > > > > > file descriptors.
> > > > > >
> > > > > > And one more concern from me is about selecting a right place in
> > the
> > > > > system
> > > > > > to do such prefetch.
> > > > > >
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > > > >
> > > > > > вс, 16 сент. 2018 г. в 19:54, Vladimir Ozerov <
> > voze...@gridgain.com
> > > >:
> > > > > >
> > > > > > > HI Alex,
> > > > > > >
> > > > > > > This is good that you observed speedup. But I do not think this
> > > > > solution
> > > > > > > works for the product in general case. Amount of RAM is
> limited,
> > > and
> > > > > > even a
> > > > > > > single partition may need more space than RAM available.
> Moving a
> > > lot
> > > > > of
> > > > > > > pages to page memory for scan means that you evict a lot of
> other
> > > > > pages,
> > > > > > > what will ultimately lead to bad performance of subsequent
> > queries
> > > > and
> > > > > > > defeat LRU algorithms, which are of great improtance for good
> > > > database
> > > > > > > performance.
> > > > > > >
> > > > > > > Database 

Upsource lags and connection timeouts

[Anton Vinogradov]

Folks,

Who is responsible for Upsouce [1]?
I see a performance issues last week.
Can we check Upsource health?

[1] https://reviews.ignite.apache.org

[GitHub] ignite pull request #4784: IGNITE-9381 fix, reproducer were added

[antonovsergey93]

GitHub user antonovsergey93 opened a pull request:

https://github.com/apache/ignite/pull/4784

IGNITE-9381 fix, reproducer were added



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9381

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4784.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4784


commit 2179d49e9402190016ac33df58dcc6a14c340811
Author: Sergey Antonov 
Date:   2018-09-18T10:44:29Z

IGNITE-9381 fix, reproducer were added




---

[GitHub] ignite pull request #4783: Ignite 9309 a

[oleg-ostanin]

GitHub user oleg-ostanin opened a pull request:

https://github.com/apache/ignite/pull/4783

Ignite 9309 a



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9309-a

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4783.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4783


commit 25178213f331045ef37f37bc90c828d8c3be1291
Author: oleg-ostanin 
Date:   2018-09-17T17:13:16Z

IGNITE-9309 added excange version check

commit 61721bc2e1d93d6a2446b942aa94a193c6072d7e
Author: oleg-ostanin 
Date:   2018-09-18T10:36:50Z

IGNITE-9309 checking exchangeVer




---

Re: Apache Ignite 2.7 release

[Anton Vinogradov]

Nikolay,

1) *Do not* create ignite-2.7 branch until we're not started preparation to
real 2.7.
Use some temporary branch for this check instead, eg.
ignite-2.7-release-test

2) Please make sure you'll not cause real release actions (maven release
and so on).
Perform only vote_* steps.

3) Make sure you'll remove all tags, branches, and other RC artifacts after
check.

4) Mark this release as RC0 to make sure it will be clear to everybody that
it's a check.


вт, 18 сент. 2018 г. в 13:24, Dmitriy Setrakyan :

> If it is an Ignite release, then it has to pass through the vote. If not,
> then you can do the test without publishing or uploading the release.
>
> D.
>
> On Tue, Sep 18, 2018 at 1:18 PM Petr Ivanov  wrote:
>
> > Ok.
> >
> > In case of TC questions — ask me.
> >
> >
> >
> > > On 18 Sep 2018, at 13:16, Nikolay Izhikov  wrote:
> > >
> > > Hello, Petr.
> > >
> > > I want to make ignite-2.7 branch today.
> > > And execute release procedure based on this branch.
> > >
> > > However, ignite-2.7 branch will be copy of master until code freeze
> date.
> > >
> > > В Вт, 18/09/2018 в 13:13 +0300, Petr Ivanov пишет:
> > >> Will it be just a test or there is already ignite-2.7 branch?
> > >>
> > >> Fabric removal related TC modifications are not ready yet, and code is
> > not in master.
> > >>
> > >>
> > >>
> > >>> On 18 Sep 2018, at 13:07, Nikolay Izhikov 
> wrote:
> > >>>
> > >>> Hello, Igniters.
> > >>>
> > >>> I want to start and release procedures and make an RC1 build.
> > >>>
> > >>> It has a 2 intention:
> > >>>
> > >>> 1. I want to walk through all release steps to make sure they all
> > works for me.
> > >>> So I will be fully ready on release date.
> > >>>
> > >>> 2. We have updated some dependencies in 2.7 and we need to make sure
> > binary build is still workable.
> > >>>
> > >>> Any objections?
> > >>>
> > >>>
> > >>> В Пт, 14/09/2018 в 18:52 +0300, Alexey Goncharuk пишет:
> >  We already have all the mechanics in place to work with properties -
> > we use
> >  ignite.build and ignite.revision from ignite.properties which are
> > adjusted
> >  during the build in the binary package.
> > 
> >  Should I create the ticket if there are no objections?
> > 
> >  пт, 14 сент. 2018 г. в 13:22, Ilya Kasnacheev <
> > ilya.kasnach...@gmail.com>:
> > 
> > > Hello!
> > >
> > > So now there's an issue that this script makes source change after
> > every
> > > build, show up in git status.
> > >
> > > What we could do to it:
> > > - Commit the changes after the build, once. In hopes that it won't
> > change
> > > very often. With benefit that we could do that right now, before
> the
> > code
> > > freeze.
> > > - Move these values to a properties file from both pom.xml and
> > > IgniteProvider.java. Any problems with this approach? We'll just
> > read them
> > > from classpath properties file.
> > > - Update the links in the file once and remove them from build
> > process. Why
> > > were they added to build process in the first place - to make them
> > > configurable during build?
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > вт, 11 сент. 2018 г. в 5:53, Roman Shtykh :
> > >
> > >> Ilya,
> > >>
> > >> The "latest" version is the default, and resolved by
> > >> https://ignite.apache.org/latest which is used by our web site
> > when a
> > >> user download the latest Ignite version. And I think this is the
> > >
> > > authority
> > >> to judge of the latest official release (pom.xml you suggest can
> > have
> > >> SNAPSHOTs etc.).
> > >> Also, as I explained during our review sessions,
> ignite-mesos-2.6.0
> > is a
> > >> driver and doesn't mean you need to have Ignite 2.6.0. User can
> run
> > any
> > >> version of Ignite he/she specifies. By default, it's "latest" but
> a
> > user
> > >> can specify any version needed, even from a non-archive URL.
> > >>
> > >> In short, what we have now
> > >> 1. mesos driver (ignite-mesos-x.x.x) will use "latest" version by
> > default
> > >> -> it will try to resolve the latest officially releases version
> of
> > >
> > > Apache
> > >> Ignite, find the closest mirror and download Ignite in a minute.
> If
> > the
> > >> version resolution fails, we fall back to the slow apache archive
> > (as you
> > >> suggest; in my opinion we better fail-fast instead of waiting for
> > hours
> > >
> > > to
> > >> download, so the user can choose another download option (3))
> > >> 2. If the user specifies the version explicitly, it goes to the
> slow
> > >> apache archive.
> > >> 3. The user can put ignite zip file on his/her http server and
> > provide
> > >
> > > the
> > >> URL as a parameter to the driver, if options 1 and 2 don't work.
> > >>
> > >> As you see, there are 3 options. And I just fix the 1st one 

Re: PHP thin client

[Stepan Pilshchikov]

> You know, I'm confused with all this documentation stuff... 
> For nodejs client all docs were moved from the repo to readme.io but the 
> readme.md keeps the installation instructions (duplicated with 
> readme.io). Probably, that's ok. 
> Will add similar short readme.md to the PHP PR.

Its good

What i think (and how it partially now): 
All user documentation should be on readme.io (how to install, use it,
configurate, description for examples)
All developers documentation (how to release, how to start develop) and(!)
basic description should be in repository




--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: Apache Ignite 2.7 release

[Dmitriy Setrakyan]

If it is an Ignite release, then it has to pass through the vote. If not,
then you can do the test without publishing or uploading the release.

D.

On Tue, Sep 18, 2018 at 1:18 PM Petr Ivanov  wrote:

> Ok.
>
> In case of TC questions — ask me.
>
>
>
> > On 18 Sep 2018, at 13:16, Nikolay Izhikov  wrote:
> >
> > Hello, Petr.
> >
> > I want to make ignite-2.7 branch today.
> > And execute release procedure based on this branch.
> >
> > However, ignite-2.7 branch will be copy of master until code freeze date.
> >
> > В Вт, 18/09/2018 в 13:13 +0300, Petr Ivanov пишет:
> >> Will it be just a test or there is already ignite-2.7 branch?
> >>
> >> Fabric removal related TC modifications are not ready yet, and code is
> not in master.
> >>
> >>
> >>
> >>> On 18 Sep 2018, at 13:07, Nikolay Izhikov  wrote:
> >>>
> >>> Hello, Igniters.
> >>>
> >>> I want to start and release procedures and make an RC1 build.
> >>>
> >>> It has a 2 intention:
> >>>
> >>> 1. I want to walk through all release steps to make sure they all
> works for me.
> >>> So I will be fully ready on release date.
> >>>
> >>> 2. We have updated some dependencies in 2.7 and we need to make sure
> binary build is still workable.
> >>>
> >>> Any objections?
> >>>
> >>>
> >>> В Пт, 14/09/2018 в 18:52 +0300, Alexey Goncharuk пишет:
>  We already have all the mechanics in place to work with properties -
> we use
>  ignite.build and ignite.revision from ignite.properties which are
> adjusted
>  during the build in the binary package.
> 
>  Should I create the ticket if there are no objections?
> 
>  пт, 14 сент. 2018 г. в 13:22, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com>:
> 
> > Hello!
> >
> > So now there's an issue that this script makes source change after
> every
> > build, show up in git status.
> >
> > What we could do to it:
> > - Commit the changes after the build, once. In hopes that it won't
> change
> > very often. With benefit that we could do that right now, before the
> code
> > freeze.
> > - Move these values to a properties file from both pom.xml and
> > IgniteProvider.java. Any problems with this approach? We'll just
> read them
> > from classpath properties file.
> > - Update the links in the file once and remove them from build
> process. Why
> > were they added to build process in the first place - to make them
> > configurable during build?
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > вт, 11 сент. 2018 г. в 5:53, Roman Shtykh :
> >
> >> Ilya,
> >>
> >> The "latest" version is the default, and resolved by
> >> https://ignite.apache.org/latest which is used by our web site
> when a
> >> user download the latest Ignite version. And I think this is the
> >
> > authority
> >> to judge of the latest official release (pom.xml you suggest can
> have
> >> SNAPSHOTs etc.).
> >> Also, as I explained during our review sessions, ignite-mesos-2.6.0
> is a
> >> driver and doesn't mean you need to have Ignite 2.6.0. User can run
> any
> >> version of Ignite he/she specifies. By default, it's "latest" but a
> user
> >> can specify any version needed, even from a non-archive URL.
> >>
> >> In short, what we have now
> >> 1. mesos driver (ignite-mesos-x.x.x) will use "latest" version by
> default
> >> -> it will try to resolve the latest officially releases version of
> >
> > Apache
> >> Ignite, find the closest mirror and download Ignite in a minute. If
> the
> >> version resolution fails, we fall back to the slow apache archive
> (as you
> >> suggest; in my opinion we better fail-fast instead of waiting for
> hours
> >
> > to
> >> download, so the user can choose another download option (3))
> >> 2. If the user specifies the version explicitly, it goes to the slow
> >> apache archive.
> >> 3. The user can put ignite zip file on his/her http server and
> provide
> >
> > the
> >> URL as a parameter to the driver, if options 1 and 2 don't work.
> >>
> >> As you see, there are 3 options. And I just fix the 1st one with
> >> https://issues.apache.org/jira/browse/IGNITE-9388 and don't change
> the
> >> original logic (which I find reasonable) documented on our site -- I
> >
> > don't
> >> see how it blocks anything.
> >>
> >> Roman Shtykh
> >>
> >>
> >> On Monday, September 10, 2018, 6:16:15 p.m. GMT+9, Ilya Kasnacheev <
> >> ilya.kasnach...@gmail.com> wrote:
> >>
> >>
> >> Hello!
> >>
> >> There's still two issues with the submission.
> >>
> >> The first one is that we're downloading "latest" version from
> preferred
> >> mirror but a specified version, such as "2.6", we're also going to
> >
> > download
> >> from "slow" archive.apache.org/dist.
> >> That's a great limitation for this change, since 

Re: Apache Ignite 2.7 release

[Petr Ivanov]

Ok.

In case of TC questions — ask me.



> On 18 Sep 2018, at 13:16, Nikolay Izhikov  wrote:
> 
> Hello, Petr.
> 
> I want to make ignite-2.7 branch today.
> And execute release procedure based on this branch.
> 
> However, ignite-2.7 branch will be copy of master until code freeze date.
> 
> В Вт, 18/09/2018 в 13:13 +0300, Petr Ivanov пишет:
>> Will it be just a test or there is already ignite-2.7 branch?
>> 
>> Fabric removal related TC modifications are not ready yet, and code is not 
>> in master.
>> 
>> 
>> 
>>> On 18 Sep 2018, at 13:07, Nikolay Izhikov  wrote:
>>> 
>>> Hello, Igniters.
>>> 
>>> I want to start and release procedures and make an RC1 build.
>>> 
>>> It has a 2 intention:
>>> 
>>> 1. I want to walk through all release steps to make sure they all works for 
>>> me.
>>> So I will be fully ready on release date.
>>> 
>>> 2. We have updated some dependencies in 2.7 and we need to make sure binary 
>>> build is still workable.
>>> 
>>> Any objections?
>>> 
>>> 
>>> В Пт, 14/09/2018 в 18:52 +0300, Alexey Goncharuk пишет:
 We already have all the mechanics in place to work with properties - we use
 ignite.build and ignite.revision from ignite.properties which are adjusted
 during the build in the binary package.
 
 Should I create the ticket if there are no objections?
 
 пт, 14 сент. 2018 г. в 13:22, Ilya Kasnacheev :
 
> Hello!
> 
> So now there's an issue that this script makes source change after every
> build, show up in git status.
> 
> What we could do to it:
> - Commit the changes after the build, once. In hopes that it won't change
> very often. With benefit that we could do that right now, before the code
> freeze.
> - Move these values to a properties file from both pom.xml and
> IgniteProvider.java. Any problems with this approach? We'll just read them
> from classpath properties file.
> - Update the links in the file once and remove them from build process. 
> Why
> were they added to build process in the first place - to make them
> configurable during build?
> 
> Regards,
> --
> Ilya Kasnacheev
> 
> 
> вт, 11 сент. 2018 г. в 5:53, Roman Shtykh :
> 
>> Ilya,
>> 
>> The "latest" version is the default, and resolved by
>> https://ignite.apache.org/latest which is used by our web site when a
>> user download the latest Ignite version. And I think this is the
> 
> authority
>> to judge of the latest official release (pom.xml you suggest can have
>> SNAPSHOTs etc.).
>> Also, as I explained during our review sessions, ignite-mesos-2.6.0 is a
>> driver and doesn't mean you need to have Ignite 2.6.0. User can run any
>> version of Ignite he/she specifies. By default, it's "latest" but a user
>> can specify any version needed, even from a non-archive URL.
>> 
>> In short, what we have now
>> 1. mesos driver (ignite-mesos-x.x.x) will use "latest" version by default
>> -> it will try to resolve the latest officially releases version of
> 
> Apache
>> Ignite, find the closest mirror and download Ignite in a minute. If the
>> version resolution fails, we fall back to the slow apache archive (as you
>> suggest; in my opinion we better fail-fast instead of waiting for hours
> 
> to
>> download, so the user can choose another download option (3))
>> 2. If the user specifies the version explicitly, it goes to the slow
>> apache archive.
>> 3. The user can put ignite zip file on his/her http server and provide
> 
> the
>> URL as a parameter to the driver, if options 1 and 2 don't work.
>> 
>> As you see, there are 3 options. And I just fix the 1st one with
>> https://issues.apache.org/jira/browse/IGNITE-9388 and don't change the
>> original logic (which I find reasonable) documented on our site -- I
> 
> don't
>> see how it blocks anything.
>> 
>> Roman Shtykh
>> 
>> 
>> On Monday, September 10, 2018, 6:16:15 p.m. GMT+9, Ilya Kasnacheev <
>> ilya.kasnach...@gmail.com> wrote:
>> 
>> 
>> Hello!
>> 
>> There's still two issues with the submission.
>> 
>> The first one is that we're downloading "latest" version from preferred
>> mirror but a specified version, such as "2.6", we're also going to
> 
> download
>> from "slow" archive.apache.org/dist.
>> That's a great limitation for this change, since most real deployments of
>> Apache Ignite will have their Ignite version pegged to a specific
> 
> release.
>> But in this case there's no win in download speed.
>> *In my opinion it is a blocker.*
>> 
>> The second one is that we can't download anything when we failed to
>> resolve "latest". My idea is that we should try and download last known
>> version in this case, which can be pushed to source from pom.xml, as we

Re: Apache Ignite 2.7 release

[Nikolay Izhikov]

Hello, Petr.

I want to make ignite-2.7 branch today.
And execute release procedure based on this branch.

However, ignite-2.7 branch will be copy of master until code freeze date.

В Вт, 18/09/2018 в 13:13 +0300, Petr Ivanov пишет:
> Will it be just a test or there is already ignite-2.7 branch?
> 
> Fabric removal related TC modifications are not ready yet, and code is not in 
> master.
> 
> 
> 
> > On 18 Sep 2018, at 13:07, Nikolay Izhikov  wrote:
> > 
> > Hello, Igniters.
> > 
> > I want to start and release procedures and make an RC1 build.
> > 
> > It has a 2 intention:
> > 
> > 1. I want to walk through all release steps to make sure they all works for 
> > me.
> > So I will be fully ready on release date.
> > 
> > 2. We have updated some dependencies in 2.7 and we need to make sure binary 
> > build is still workable.
> > 
> > Any objections?
> > 
> > 
> > В Пт, 14/09/2018 в 18:52 +0300, Alexey Goncharuk пишет:
> > > We already have all the mechanics in place to work with properties - we 
> > > use
> > > ignite.build and ignite.revision from ignite.properties which are adjusted
> > > during the build in the binary package.
> > > 
> > > Should I create the ticket if there are no objections?
> > > 
> > > пт, 14 сент. 2018 г. в 13:22, Ilya Kasnacheev :
> > > 
> > > > Hello!
> > > > 
> > > > So now there's an issue that this script makes source change after every
> > > > build, show up in git status.
> > > > 
> > > > What we could do to it:
> > > > - Commit the changes after the build, once. In hopes that it won't 
> > > > change
> > > > very often. With benefit that we could do that right now, before the 
> > > > code
> > > > freeze.
> > > > - Move these values to a properties file from both pom.xml and
> > > > IgniteProvider.java. Any problems with this approach? We'll just read 
> > > > them
> > > > from classpath properties file.
> > > > - Update the links in the file once and remove them from build process. 
> > > > Why
> > > > were they added to build process in the first place - to make them
> > > > configurable during build?
> > > > 
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > > 
> > > > 
> > > > вт, 11 сент. 2018 г. в 5:53, Roman Shtykh :
> > > > 
> > > > > Ilya,
> > > > > 
> > > > > The "latest" version is the default, and resolved by
> > > > > https://ignite.apache.org/latest which is used by our web site when a
> > > > > user download the latest Ignite version. And I think this is the
> > > > 
> > > > authority
> > > > > to judge of the latest official release (pom.xml you suggest can have
> > > > > SNAPSHOTs etc.).
> > > > > Also, as I explained during our review sessions, ignite-mesos-2.6.0 
> > > > > is a
> > > > > driver and doesn't mean you need to have Ignite 2.6.0. User can run 
> > > > > any
> > > > > version of Ignite he/she specifies. By default, it's "latest" but a 
> > > > > user
> > > > > can specify any version needed, even from a non-archive URL.
> > > > > 
> > > > > In short, what we have now
> > > > > 1. mesos driver (ignite-mesos-x.x.x) will use "latest" version by 
> > > > > default
> > > > > -> it will try to resolve the latest officially releases version of
> > > > 
> > > > Apache
> > > > > Ignite, find the closest mirror and download Ignite in a minute. If 
> > > > > the
> > > > > version resolution fails, we fall back to the slow apache archive (as 
> > > > > you
> > > > > suggest; in my opinion we better fail-fast instead of waiting for 
> > > > > hours
> > > > 
> > > > to
> > > > > download, so the user can choose another download option (3))
> > > > > 2. If the user specifies the version explicitly, it goes to the slow
> > > > > apache archive.
> > > > > 3. The user can put ignite zip file on his/her http server and provide
> > > > 
> > > > the
> > > > > URL as a parameter to the driver, if options 1 and 2 don't work.
> > > > > 
> > > > > As you see, there are 3 options. And I just fix the 1st one with
> > > > > https://issues.apache.org/jira/browse/IGNITE-9388 and don't change the
> > > > > original logic (which I find reasonable) documented on our site -- I
> > > > 
> > > > don't
> > > > > see how it blocks anything.
> > > > > 
> > > > > Roman Shtykh
> > > > > 
> > > > > 
> > > > > On Monday, September 10, 2018, 6:16:15 p.m. GMT+9, Ilya Kasnacheev <
> > > > > ilya.kasnach...@gmail.com> wrote:
> > > > > 
> > > > > 
> > > > > Hello!
> > > > > 
> > > > > There's still two issues with the submission.
> > > > > 
> > > > > The first one is that we're downloading "latest" version from 
> > > > > preferred
> > > > > mirror but a specified version, such as "2.6", we're also going to
> > > > 
> > > > download
> > > > > from "slow" archive.apache.org/dist.
> > > > > That's a great limitation for this change, since most real 
> > > > > deployments of
> > > > > Apache Ignite will have their Ignite version pegged to a specific
> > > > 
> > > > release.
> > > > > But in this case there's no win in download speed.
> > > > > *In my opinion it is a blocker.*
> > > > 

[GitHub] ignite pull request #4782: IGNITE-9631: Timeouts in ZookeeperDIscoverySpi te...

[avplatonov]

GitHub user avplatonov opened a pull request:

https://github.com/apache/ignite/pull/4782

IGNITE-9631: Timeouts in ZookeeperDIscoverySpi test suite



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9631

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4782.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4782


commit f66af0f4707d8df84f9d552a865b75bf303ddb93
Author: Alexey Platonov 
Date:   2018-09-18T09:47:48Z

add timeouts to test




---

Re: Apache Ignite 2.7 release

[Petr Ivanov]

Will it be just a test or there is already ignite-2.7 branch?

Fabric removal related TC modifications are not ready yet, and code is not in 
master.



> On 18 Sep 2018, at 13:07, Nikolay Izhikov  wrote:
> 
> Hello, Igniters.
> 
> I want to start and release procedures and make an RC1 build.
> 
> It has a 2 intention:
> 
> 1. I want to walk through all release steps to make sure they all works for 
> me.
> So I will be fully ready on release date.
> 
> 2. We have updated some dependencies in 2.7 and we need to make sure binary 
> build is still workable.
> 
> Any objections?
> 
> 
> В Пт, 14/09/2018 в 18:52 +0300, Alexey Goncharuk пишет:
>> We already have all the mechanics in place to work with properties - we use
>> ignite.build and ignite.revision from ignite.properties which are adjusted
>> during the build in the binary package.
>> 
>> Should I create the ticket if there are no objections?
>> 
>> пт, 14 сент. 2018 г. в 13:22, Ilya Kasnacheev :
>> 
>>> Hello!
>>> 
>>> So now there's an issue that this script makes source change after every
>>> build, show up in git status.
>>> 
>>> What we could do to it:
>>> - Commit the changes after the build, once. In hopes that it won't change
>>> very often. With benefit that we could do that right now, before the code
>>> freeze.
>>> - Move these values to a properties file from both pom.xml and
>>> IgniteProvider.java. Any problems with this approach? We'll just read them
>>> from classpath properties file.
>>> - Update the links in the file once and remove them from build process. Why
>>> were they added to build process in the first place - to make them
>>> configurable during build?
>>> 
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>> 
>>> 
>>> вт, 11 сент. 2018 г. в 5:53, Roman Shtykh :
>>> 
 Ilya,
 
 The "latest" version is the default, and resolved by
 https://ignite.apache.org/latest which is used by our web site when a
 user download the latest Ignite version. And I think this is the
>>> 
>>> authority
 to judge of the latest official release (pom.xml you suggest can have
 SNAPSHOTs etc.).
 Also, as I explained during our review sessions, ignite-mesos-2.6.0 is a
 driver and doesn't mean you need to have Ignite 2.6.0. User can run any
 version of Ignite he/she specifies. By default, it's "latest" but a user
 can specify any version needed, even from a non-archive URL.
 
 In short, what we have now
 1. mesos driver (ignite-mesos-x.x.x) will use "latest" version by default
 -> it will try to resolve the latest officially releases version of
>>> 
>>> Apache
 Ignite, find the closest mirror and download Ignite in a minute. If the
 version resolution fails, we fall back to the slow apache archive (as you
 suggest; in my opinion we better fail-fast instead of waiting for hours
>>> 
>>> to
 download, so the user can choose another download option (3))
 2. If the user specifies the version explicitly, it goes to the slow
 apache archive.
 3. The user can put ignite zip file on his/her http server and provide
>>> 
>>> the
 URL as a parameter to the driver, if options 1 and 2 don't work.
 
 As you see, there are 3 options. And I just fix the 1st one with
 https://issues.apache.org/jira/browse/IGNITE-9388 and don't change the
 original logic (which I find reasonable) documented on our site -- I
>>> 
>>> don't
 see how it blocks anything.
 
 Roman Shtykh
 
 
 On Monday, September 10, 2018, 6:16:15 p.m. GMT+9, Ilya Kasnacheev <
 ilya.kasnach...@gmail.com> wrote:
 
 
 Hello!
 
 There's still two issues with the submission.
 
 The first one is that we're downloading "latest" version from preferred
 mirror but a specified version, such as "2.6", we're also going to
>>> 
>>> download
 from "slow" archive.apache.org/dist.
 That's a great limitation for this change, since most real deployments of
 Apache Ignite will have their Ignite version pegged to a specific
>>> 
>>> release.
 But in this case there's no win in download speed.
 *In my opinion it is a blocker.*
 
 The second one is that we can't download anything when we failed to
 resolve "latest". My idea is that we should try and download last known
 version in this case, which can be pushed to source from pom.xml, as we
 already do with URLs. So if you could not resolve "latest" you will
 download 2.7.0.
 
 Buuut, maybe it's not necessary, maybe we should just *discourage
 "latest"*, which is in my opinion almost always a bad idea.
 
 WDYT?
 
 Regards,
 --
 Ilya Kasnacheev
 
 
 вс, 9 сент. 2018 г. в 5:47, Roman Shtykh :
 
 Hi Ilya,
 
 Sorry, missed that.
 Added now.
 
 --
 Roman Shtykh
 
 
 On Thursday, September 6, 2018, 6:16:58 p.m. GMT+9, Ilya Kasnacheev <
 ilya.kasnach...@gmail.com> wrote:
 
 
 

Re: Apache Ignite 2.7 release

[Nikolay Izhikov]

Hello, Igniters.

I want to start and release procedures and make an RC1 build.

It has a 2 intention:

1. I want to walk through all release steps to make sure they all works for me.
So I will be fully ready on release date.

2. We have updated some dependencies in 2.7 and we need to make sure binary 
build is still workable.

Any objections?


В Пт, 14/09/2018 в 18:52 +0300, Alexey Goncharuk пишет:
> We already have all the mechanics in place to work with properties - we use
> ignite.build and ignite.revision from ignite.properties which are adjusted
> during the build in the binary package.
> 
> Should I create the ticket if there are no objections?
> 
> пт, 14 сент. 2018 г. в 13:22, Ilya Kasnacheev :
> 
> > Hello!
> > 
> > So now there's an issue that this script makes source change after every
> > build, show up in git status.
> > 
> > What we could do to it:
> > - Commit the changes after the build, once. In hopes that it won't change
> > very often. With benefit that we could do that right now, before the code
> > freeze.
> > - Move these values to a properties file from both pom.xml and
> > IgniteProvider.java. Any problems with this approach? We'll just read them
> > from classpath properties file.
> > - Update the links in the file once and remove them from build process. Why
> > were they added to build process in the first place - to make them
> > configurable during build?
> > 
> > Regards,
> > --
> > Ilya Kasnacheev
> > 
> > 
> > вт, 11 сент. 2018 г. в 5:53, Roman Shtykh :
> > 
> > > Ilya,
> > > 
> > > The "latest" version is the default, and resolved by
> > > https://ignite.apache.org/latest which is used by our web site when a
> > > user download the latest Ignite version. And I think this is the
> > 
> > authority
> > > to judge of the latest official release (pom.xml you suggest can have
> > > SNAPSHOTs etc.).
> > > Also, as I explained during our review sessions, ignite-mesos-2.6.0 is a
> > > driver and doesn't mean you need to have Ignite 2.6.0. User can run any
> > > version of Ignite he/she specifies. By default, it's "latest" but a user
> > > can specify any version needed, even from a non-archive URL.
> > > 
> > > In short, what we have now
> > > 1. mesos driver (ignite-mesos-x.x.x) will use "latest" version by default
> > > -> it will try to resolve the latest officially releases version of
> > 
> > Apache
> > > Ignite, find the closest mirror and download Ignite in a minute. If the
> > > version resolution fails, we fall back to the slow apache archive (as you
> > > suggest; in my opinion we better fail-fast instead of waiting for hours
> > 
> > to
> > > download, so the user can choose another download option (3))
> > > 2. If the user specifies the version explicitly, it goes to the slow
> > > apache archive.
> > > 3. The user can put ignite zip file on his/her http server and provide
> > 
> > the
> > > URL as a parameter to the driver, if options 1 and 2 don't work.
> > > 
> > > As you see, there are 3 options. And I just fix the 1st one with
> > > https://issues.apache.org/jira/browse/IGNITE-9388 and don't change the
> > > original logic (which I find reasonable) documented on our site -- I
> > 
> > don't
> > > see how it blocks anything.
> > > 
> > > Roman Shtykh
> > > 
> > > 
> > > On Monday, September 10, 2018, 6:16:15 p.m. GMT+9, Ilya Kasnacheev <
> > > ilya.kasnach...@gmail.com> wrote:
> > > 
> > > 
> > > Hello!
> > > 
> > > There's still two issues with the submission.
> > > 
> > > The first one is that we're downloading "latest" version from preferred
> > > mirror but a specified version, such as "2.6", we're also going to
> > 
> > download
> > > from "slow" archive.apache.org/dist.
> > > That's a great limitation for this change, since most real deployments of
> > > Apache Ignite will have their Ignite version pegged to a specific
> > 
> > release.
> > > But in this case there's no win in download speed.
> > > *In my opinion it is a blocker.*
> > > 
> > > The second one is that we can't download anything when we failed to
> > > resolve "latest". My idea is that we should try and download last known
> > > version in this case, which can be pushed to source from pom.xml, as we
> > > already do with URLs. So if you could not resolve "latest" you will
> > > download 2.7.0.
> > > 
> > > Buuut, maybe it's not necessary, maybe we should just *discourage
> > > "latest"*, which is in my opinion almost always a bad idea.
> > > 
> > > WDYT?
> > > 
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > > 
> > > 
> > > вс, 9 сент. 2018 г. в 5:47, Roman Shtykh :
> > > 
> > > Hi Ilya,
> > > 
> > > Sorry, missed that.
> > > Added now.
> > > 
> > > --
> > > Roman Shtykh
> > > 
> > > 
> > > On Thursday, September 6, 2018, 6:16:58 p.m. GMT+9, Ilya Kasnacheev <
> > > ilya.kasnach...@gmail.com> wrote:
> > > 
> > > 
> > > Hello!
> > > 
> > > The last of my requests still standing is that we should fall-back to
> > > single URL download in case of error with 'latest' version. Everything
> > 
> > 

[GitHub] ignite pull request #4781: IGNITE-9629 JDBCv2: supports types conversions at...

[tledkov-gridgain]

GitHub user tledkov-gridgain opened a pull request:

https://github.com/apache/ignite/pull/4781

IGNITE-9629 JDBCv2: supports types conversions at the JdbcResultSet



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9629

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4781.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4781


commit 62cf76d910af0e6d57327ee3af6c2a4ba9621f0d
Author: tledkov-gridgain 
Date:   2018-09-18T10:01:17Z

IGNITE-9629 JDBCv2: supports types conversions at the JdbcResultSet




---

[GitHub] ignite pull request #4780: IGNITE-9628 Avoid using sun.reflect.generics.refl...

[dmitrievanthony]

GitHub user dmitrievanthony opened a pull request:

https://github.com/apache/ignite/pull/4780

IGNITE-9628 Avoid using sun.reflect.generics.reflectiveObjects package in 
ML module

in ML module.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9628

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4780.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4780


commit 87dc630c26eb23b3bd2407724f24992323492622
Author: Anton Dmitriev 
Date:   2018-09-18T09:58:33Z

IGNITE-9628 Avoid using sun.reflect.generics.reflectiveObjects package
in ML module.




---

[GitHub] ignite pull request #4778: IGNITE-9625 Fix ML javadoc

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4778


---

[GitHub] ignite pull request #4714: IGNITE-9441 Do not fail on zero CRC check in last...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4714


---

Re: PHP thin client

[Alexey Kosenchuk]

But now in PR we have no readme at all
Agree with examples on readme.io, but other information (how to install, how
to configurate, install from sources, basic description) may be in sources,
it help a lot
Or it also should be on readme.io?

This question because in other platforms have readme exist and they have
this basic information in it


You know, I'm confused with all this documentation stuff...
For nodejs client all docs were moved from the repo to readme.io but the 
readme.md keeps the installation instructions (duplicated with 
readme.io). Probably, that's ok.

Will add similar short readme.md to the PHP PR.

Thanks,
-Alexey

[jira] [Created] (IGNITE-9632) Add support of trivial IN-operation for partition extraction

[Sergey Grimstad (JIRA)]

Sergey Grimstad created IGNITE-9632:
---

 Summary: Add support of trivial IN-operation for partition 
extraction
 Key: IGNITE-9632
 URL: https://issues.apache.org/jira/browse/IGNITE-9632
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Affects Versions: 2.7
Reporter: Sergey Grimstad
Assignee: Sergey Grimstad
 Fix For: 2.7






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9631) Timeouts in ZookeeperDIscoverySpi test suite

[Alexey Platonov (JIRA)]

Alexey Platonov created IGNITE-9631:
---

 Summary: Timeouts in ZookeeperDIscoverySpi test suite
 Key: IGNITE-9631
 URL: https://issues.apache.org/jira/browse/IGNITE-9631
 Project: Ignite
  Issue Type: Bug
Reporter: Alexey Platonov
Assignee: Alexey Platonov


Sometimes this test suite times out. Failed test examples:

[https://ci.ignite.apache.org/viewLog.html?buildId=1862120=buildResultsDiv=IgniteTests24Java8_ZooKeeperDiscovery2]

[https://ci.ignite.apache.org/viewLog.html?buildId=1839809=buildResultsDiv=IgniteTests24Java8_ZooKeeperDiscovery2]

It seems that test class GridCachePartitionedNodeRestartTest times out and 
block all suite.

We need to add timeouts into test for suite timeouts preventing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9630) Partitions intersection for AND condition of same key

[Sergey Grimstad (JIRA)]

Sergey Grimstad created IGNITE-9630:
---

 Summary: Partitions intersection for AND condition of same key
 Key: IGNITE-9630
 URL: https://issues.apache.org/jira/browse/IGNITE-9630
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Affects Versions: 2.7
Reporter: Sergey Grimstad
Assignee: Sergey Grimstad
 Fix For: 2.7


GridSqlQuerySplitter when extractPartition for AND operation type should 
calculate intersection of resolved partitions for the same key



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: PHP thin client

[Stepan Pilshchikov]

> 1. Because finally the docs should not be in the repo (unfortunately) but 
> on readme.io. 
> There is a task for that:
> https://issues.apache.org/jira/browse/IGNITE-9523
> Added the link in the jira. 

Ok

> 2. Examples in readme.md are more like code snippets (mostly not big) 
> intended to help better understanding the usage of the client along the 
> reading. 
> Examples in php/examples are more meaningful and fully cover the 
> examples in readme.md. 

Oh, now i get it
But now in PR we have no readme at all
Agree with examples on readme.io, but other information (how to install, how
to configurate, install from sources, basic description) may be in sources,
it help a lot
Or it also should be on readme.io?

This question because in other platforms have readme exist and they have
this basic information in it



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: [MTCGA]: new failures in builds [1888723] needs to be handled

[Roman Kondakov]

Hi, Dmitriy!

Vladimir assured me in a private conversation that both API parity and 
MemoryMetrics fixes will be made soon.


Thank you!
--
Kind Regards
Roman Kondakov

On 17.09.2018 18:55, Dmitriy Pavlov wrote:

Hi Roman Kondakov, Vladimir Ozerov,

is it possible and is it reasonable to fix tests faster than full support
in https://issues.apache.org/jira/browse/IGNITE-9390 ?

E.g. can the Parity test be fixed using
https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Tests+How+To#IgniteTestsHowTo-Testof.NETAPIparitywithJavaAPI
  recommendations?

Sincerely,
Dmitriy Pavlov

пн, 17 сент. 2018 г. в 17:25, :


Hi Ignite Developer,

I am MTCGA.Bot, and I've detected some issue on TeamCity to be addressed.
I hope you can help.

  *New test failure in master
IgniteConfigurationParityTest.TestIgniteConfiguration
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=7406999233661835096=%3Cdefault%3E=testDetails

  *New test failure in master DataRegionMetricsTest.TestMemoryMetrics
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=6742613397597284603=%3Cdefault%3E=testDetails

  *New test failure in master MemoryMetricsTest.TestMemoryMetrics
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-7558087625238261420=%3Cdefault%3E=testDetails
  Changes may led to failure were done by
  - kondakov87
http://ci.ignite.apache.org/viewModification.html?modId=831703=false

 - If your changes can led to this failure(s), please create issue
with label MakeTeamCityGreenAgain and assign it to you.
 -- If you have fix, please set ticket to PA state and write to dev
list fix is ready
 -- For case fix will require some time please mute test and set
label Muted_Test to issue
 - If you know which change caused failure please contact change
author directly
 - If you don't know which change caused failure please send
message to dev list to find out
Should you have any questions please contact dev@ignite.apache.org
Best Regards,
MTCGA.Bot
Notification generated at Mon Sep 17 17:25:16 MSK 2018

[jira] [Created] (IGNITE-9629) JDBCv2: JdbcThinResultSet must support types conversions

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-9629:


 Summary: JDBCv2: JdbcThinResultSet must support types conversions
 Key: IGNITE-9629
 URL: https://issues.apache.org/jira/browse/IGNITE-9629
 Project: Ignite
  Issue Type: Bug
  Components: jdbc
Affects Versions: 2.6
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.7


Now {{JdbcResultSet}} doesn't support data types transformation.

e.g.: if the attribute type is SHORT
all the methods below must return valid value::
{{getByte, getShort, getInt, getLong, getFloat, getDouble, getBigDecimal, 
getBoolean, getString, getNString}}

See the table B-6 at the JDBC spec  (see also [8.9.5 and 
8.9.6|http://docs.oracle.com/javase/6/docs/technotes/guides/jdbc/getstart/mapping.html#996857]).





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9628) Java 9: ML module compilation failure

[Peter Ivanov (JIRA)]

Peter Ivanov created IGNITE-9628:


 Summary: Java 9: ML module compilation failure
 Key: IGNITE-9628
 URL: https://issues.apache.org/jira/browse/IGNITE-9628
 Project: Ignite
  Issue Type: Task
Affects Versions: 2.6
Reporter: Peter Ivanov
 Fix For: 2.7


{code}
[17:56:31]  [ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.7.0:compile (default-compile) 
on project ignite-ml: Compilation failure: Compilation failure: 
[17:56:31]  [ERROR] 
/data/teamcity/work/9198da4c51c3e112/modules/ml/src/main/java/org/apache/ignite/ml/tree/randomforest/data/impurity/MSEHistogram.java:[28,28]
 package sun.reflect.generics.reflectiveObjects is not visible
[17:56:31]  [ERROR]   (package sun.reflect.generics.reflectiveObjects is 
declared in module java.base, which does not export it to the unnamed module)
[17:56:31]  [ERROR] 
/data/teamcity/work/9198da4c51c3e112/modules/ml/src/main/java/org/apache/ignite/ml/tree/randomforest/data/impurity/GiniHistogram.java:[34,28]
 package sun.reflect.generics.reflectiveObjects is not visible
[17:56:31]  [ERROR]   (package sun.reflect.generics.reflectiveObjects is 
declared in module java.base, which does not export it to the unnamed module)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: PHP thin client

[Alexey Kosenchuk]

Hi Stepan,


1. Why exists second branches with all documentation? I mean they looks good
together, why need to separate?


Because finally the docs should not be in the repo (unfortunately) but 
on readme.io.

There is a task for that: https://issues.apache.org/jira/browse/IGNITE-9523


Also, one place when i found this branch it is this thread, not in jira, not
in a github


Added the link in the jira.


2. Examples in readme.md have better look in php/examples. I mean
php/examples have one set of examples and readme have second one, they not
match and they both looks good. Why doesn't add readme examples in
php/examples?


Examples in readme.md are more like code snippets (mostly not big) 
intended to help better understanding the usage of the client along the 
reading.
Examples in php/examples are more meaningful and fully cover the 
examples in readme.md.


Thanks,
-Alexey

[GitHub] ignite pull request #4779: IGNITE-9162 fixing condition is added

[SGrimstad]

GitHub user SGrimstad opened a pull request:

https://github.com/apache/ignite/pull/4779

IGNITE-9162 fixing condition is added



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite IGNITE-9162

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4779.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4779


commit b8312c14d4c63e1471504d1c152344114f01387b
Author: SGrimstad 
Date:   2018-09-18T08:51:26Z

IGNITE-9162 fixing condition is added




---

[jira] [Created] (IGNITE-9627) TcpCommunicationSpiSkipMessageSendTest.testClientSegmented is flaky in master

[Amelchev Nikita (JIRA)]

Amelchev Nikita created IGNITE-9627:
---

 Summary: 
TcpCommunicationSpiSkipMessageSendTest.testClientSegmented is flaky in master
 Key: IGNITE-9627
 URL: https://issues.apache.org/jira/browse/IGNITE-9627
 Project: Ignite
  Issue Type: Bug
Reporter: Amelchev Nikita
Assignee: Amelchev Nikita


The test is flaky in master. Also, it uses the default failure handler and can 
halt JVM.
Example of fail: [TC 
build|https://ci.ignite.apache.org/viewLog.html?buildId=1895268=buildResultsDiv=IgniteTests24Java8_Spi#testNameId-3225493048753223945].
 Log:
{noformat}
[2018-09-18 06:12:42,466][ERROR][main][root] Test failed.
junit.framework.AssertionFailedError: Client wasn't segmented.
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.TestCase.fail(TestCase.java:227)
at 
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpiSkipMessageSendTest.testClientSegmented(TcpCommunicationSpiSkipMessageSendTest.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2177)
at 
org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:143)
at 
org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:2092)
at java.lang.Thread.run(Thread.java:748)
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9626) Applying WAL updates ignores evicition policy

[Pavel Vinokurov (JIRA)]

Pavel Vinokurov created IGNITE-9626:
---

 Summary: Applying WAL updates ignores evicition policy
 Key: IGNITE-9626
 URL: https://issues.apache.org/jira/browse/IGNITE-9626
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.6
Reporter: Pavel Vinokurov
Assignee: Pavel Vinokurov
 Attachments: IgniteExpirationWitPeristanceReproducer.java

Steps to reproduce:
1. Add record for cache obtained by ignite.cache().withExpiryPolicy().
2. Stops node before checkpoint.
3. Start node and get record for cache after specified duration.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4778: IGNITE-9625 Fix ML javadoc

[dmitrievanthony]

GitHub user dmitrievanthony opened a pull request:

https://github.com/apache/ignite/pull/4778

IGNITE-9625 Fix ML javadoc



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9625

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4778.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4778


commit afb9c38daed5970dfc22b703e4e9d22dcc57faee
Author: Anton Dmitriev 
Date:   2018-09-18T08:37:39Z

IGNITE-9625 Fix javadoc.




---

Re: Python thin client

[Dmitry Melnichuk]

Igor,

All examples are in 'ignite/modules/platforms/python/examples'.

I put examples in separate Python files mostly to be able to 
automatically confirm their operability. All the lengthy explanations 
and cross-references are in the main documentation:


https://apache-ignite-binary-protocol-client.readthedocs.io/en/latest/examples.html

To make it easier for end users to navigate in pyignite directories, I 
added a small README file to the 'examples' directory with a short 
description and a link to the docs:


https://github.com/nobitlost/ignite/blob/ignite-7782/modules/platforms/python/examples/readme.md

If you think this README is lacking something regardless of the main 
pyignite docs, please let me know.


On 9/17/18 8:59 PM, Igor Sapego wrote:

Dmitry,

Sorry, I was not clear enough. What I mean is that Ignite distributed
by both source and binary releases. Binary releases contain platforms
code, which is needed to write your own application in the language,
but does not contain developer stuff, such as tests, documentation
generating scripts, etc.

Of course, it is more common to use package managers when
developing your application, and of course, we are going to support
this approach. But still, we provide a way for a user to get any client
without any package manager.

I like the idea of supplying whl package in a binary release, though it
looks like it's going to take more effort to implement it. But I believe
except for the whl package, we will need to supply examples and
instructions, how user can run them.

What do you think?

Best Regards,
Igor


On Sat, Sep 15, 2018 at 7:03 AM Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:


Igor,

I am in doubt here because I am not fully comprehend the meaning of
"binary release". But if it is somehow related to the "distribution"
thing, I would dare to suggest the following options:

1. Copy nothing. Just do

```
$ python setup.py bdist_wheel
$ twine upload dist/*
```

during the build process (or separately) and let PyPI handle the
distribution.

This is the most natural and user-friendly way of distributing Python
packages. End user might only do

```
$ pip install pyignite
```

as it is suggested by my readme file.

2. Supply the wheel package. It is the file 'pyignite-*.whl' from 'dist'
directory, that should appear there as the "python setup.py bdist_wheel"
command finishes. Actually, it can be combined with the first option.

The wheel can then be installed by the end user:

```
$ pip install pyignite-*.whl
```

3. Supply the following files:

ignite/modules/platforms/python/pyignite/**
ignite/modules/platforms/python/requirements/**
ignite/modules/platforms/python/LICENSE
ignite/modules/platforms/python/README.md
ignite/modules/platforms/python/setup.py

(** stands for "all the files and sub-folders recursively, starting from
this folder".)

It is not uncommon or wrong to distribute Python programs as text
without prior packaging, but, in my personal experience, this is a way
more suitable for one-time scripts or proprietary programs. For example,
most of Python web apps relies on git for deployment, without any
packaging subsystem.

Here's a few words about wheels:

https://wheel.readthedocs.io/

Here's about uploading to PyPI with twine:


https://packaging.python.org/tutorials/packaging-projects/#uploading-the-distribution-archives

On 9/14/18 9:05 PM, Igor Sapego wrote:

Ok, good.

Now, what is about installation? Which directories/files
need to be copied to ignite's binary release?

Best Regards,
Igor

Re: ignite .net plugin for security

[wayne theron]

It will authenticate cluster nodes and clients as well as authorise cache 
permissions

On 18 Sep 2018, 09:23, at 09:23, Pavel Tupitsyn  wrote:
>Hi,
>
>Can you please elaborate, what does the plugin do?
>
>
>On Mon, Sep 17, 2018 at 10:49 PM wt  wrote:
>
>> I have almost completed a plugin in java and was wondering if this
>plugin
>> could be made available in .net. I have seen an example online (here
>-
>>
>>
>https://dzone.com/articles/implementing-ignitenet-plugin-distributed-semaphor
>> )
>> where plugin methods in java are made available in .net. Do you know
>if it
>> is possible to implement a security plugin in this fashion or is that
>level
>> of integration not possible between java and .net. I can see there
>are no
>> security classes available in .net so all the wiring would need to be
>> developed. would be great to get some of the gridgain developers with
>.net
>> knowledge to comment. Perhaps it is planned to be made available in
>the
>> future?
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>>

[jira] [Created] (IGNITE-9625) ML: Can't build ignite binaries because java doc exception

[Stepan Pilschikov (JIRA)]

Stepan Pilschikov created IGNITE-9625:
-

 Summary: ML: Can't build ignite binaries because java doc exception
 Key: IGNITE-9625
 URL: https://issues.apache.org/jira/browse/IGNITE-9625
 Project: Ignite
  Issue Type: Bug
  Components: ml
Affects Versions: 2.7
Reporter: Stepan Pilschikov


Exception:
{code}
Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run 
(javadoc-postprocessing-new) on project apache-ignite: An Ant BuildException 
has occured: Execution failed due to: Class doesn't have description in file: 
/target/javadoc/core/org/apache/ignite/ml/composition/boosting/GDBTrainer.GDBModel.html
Class doesn't have description in file: 
/target/javadoc/core/org/apache/ignite/ml/trainers/DatasetTrainer.EmptyDatasetException.html
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: ignite .net plugin for security

[Pavel Tupitsyn]

Hi,

Can you please elaborate, what does the plugin do?


On Mon, Sep 17, 2018 at 10:49 PM wt  wrote:

> I have almost completed a plugin in java and was wondering if this plugin
> could be made available in .net. I have seen an example online (here -
>
> https://dzone.com/articles/implementing-ignitenet-plugin-distributed-semaphor
> )
> where plugin methods in java are made available in .net. Do you know if it
> is possible to implement a security plugin in this fashion or is that level
> of integration not possible between java and .net. I can see there are no
> security classes available in .net so all the wiring would need to be
> developed. would be great to get some of the gridgain developers with .net
> knowledge to comment. Perhaps it is planned to be made available in the
> future?
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

[GitHub] ignite pull request #4772: Refine CacheAtomicityMode.TRANSACTIONAL_SNAPSHOT ...

[pavlukhin]

Github user pavlukhin closed the pull request at:

https://github.com/apache/ignite/pull/4772


---

[GitHub] ignite pull request #4777: Refine CacheAtomicityMode.TRANSACTIONAL_SNAPSHOT ...

[pavlukhin]

GitHub user pavlukhin opened a pull request:

https://github.com/apache/ignite/pull/4777

Refine CacheAtomicityMode.TRANSACTIONAL_SNAPSHOT javadoc



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9624

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4777.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4777


commit a6ba2870a581ace6b7f21511348136fb1fdd9d18
Author: ipavlukhin 
Date:   2018-09-17T13:52:41Z

correct CacheAtomicityMode.TRANSACTIONAL_SNAPSHOT javadoc




---

Re: Cache scan efficiency

[Alexey Goncharuk]

Dmitriy,

In my understanding, the proper fix for the scan query looks like a big
change and it is unlikely that we include it in Ignite 2.7. On the other
hand, the method suggested by Alexei is quite simple  and it definitely
fits Ignite 2.7, which will provide a better user experience. Even having a
proper scan query implemented this method can be useful in some specific
scenarios, so we will not have to deprecate it.

--AG

пн, 17 сент. 2018 г. в 19:15, Dmitriy Pavlov :

> As I understood it is not a hack, it is an advanced feature for warming up
> the partition. We can build warm-up of the overall cache by calling its
> partitions warm-up. Users often ask about this feature and are not
> confident with our lazy upload.
>
> Please correct me if I misunderstood the idea.
>
> пн, 17 сент. 2018 г. в 18:37, Dmitriy Setrakyan :
>
> > I would rather fix the scan than hack the scan. Is there any technical
> > reason for hacking it now instead of fixing it properly? Can some of the
> > experts in this thread provide an estimate of complexity and difference
> in
> > work that would be required for each approach?
> >
> > D.
> >
> > On Mon, Sep 17, 2018 at 4:42 PM Alexey Goncharuk <
> > alexey.goncha...@gmail.com>
> > wrote:
> >
> > > I think it would be beneficial for some Ignite users if we added such a
> > > partition warmup method to the public API. The method should be
> > > well-documented and state that it may invalidate existing page cache.
> It
> > > will be a very effective instrument until we add the proper scan
> ability
> > > that Vladimir was referring to.
> > >
> > > пн, 17 сент. 2018 г. в 13:05, Maxim Muzafarov :
> > >
> > > > Folks,
> > > >
> > > > Such warming up can be an effective technique for performing
> > calculations
> > > > which required large cache
> > > > data reads, but I think it's the single narrow use case of all over
> > > Ignite
> > > > store usages. Like all other
> > > > powerfull techniques, we should use it wisely. In the general case, I
> > > think
> > > > we should consider other
> > > > techniques mentioned by Vladimir and may create something like
> `global
> > > > statistics of cache data usage`
> > > > to choose the best technique in each case.
> > > >
> > > > For instance, it's not obvious what would take longer: multi-block
> > reads
> > > or
> > > > 50 single-block reads issues
> > > > sequentially. It strongly depends on used hardware under the hood and
> > > might
> > > > depend on workload system
> > > > resources (CPU-intensive calculations and I\O access) as well. But
> > > > `statistics` will help us to choose
> > > > the right way.
> > > >
> > > >
> > > > On Sun, 16 Sep 2018 at 23:59 Dmitriy Pavlov 
> > > wrote:
> > > >
> > > > > Hi Alexei,
> > > > >
> > > > > I did not find any PRs associated with the ticket for check code
> > > changes
> > > > > behind this idea. Are there any PRs?
> > > > >
> > > > > If we create some forwards scan of pages, it should be a very
> > > > intellectual
> > > > > algorithm including a lot of parameters (how much RAM is free, how
> > > > probably
> > > > > we will need next page, etc). We had the private talk about such
> idea
> > > > some
> > > > > time ago.
> > > > >
> > > > > By my experience, Linux systems already do such forward reading of
> > file
> > > > > data (for corresponding sequential flagged file descriptors), but
> > some
> > > > > prefetching of data at the level of application may be useful for
> > > > O_DIRECT
> > > > > file descriptors.
> > > > >
> > > > > And one more concern from me is about selecting a right place in
> the
> > > > system
> > > > > to do such prefetch.
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > > вс, 16 сент. 2018 г. в 19:54, Vladimir Ozerov <
> voze...@gridgain.com
> > >:
> > > > >
> > > > > > HI Alex,
> > > > > >
> > > > > > This is good that you observed speedup. But I do not think this
> > > > solution
> > > > > > works for the product in general case. Amount of RAM is limited,
> > and
> > > > > even a
> > > > > > single partition may need more space than RAM available. Moving a
> > lot
> > > > of
> > > > > > pages to page memory for scan means that you evict a lot of other
> > > > pages,
> > > > > > what will ultimately lead to bad performance of subsequent
> queries
> > > and
> > > > > > defeat LRU algorithms, which are of great improtance for good
> > > database
> > > > > > performance.
> > > > > >
> > > > > > Database vendors choose another approach - skip BTrees, iterate
> > > > direclty
> > > > > > over data pages, read them in multi-block fashion, use separate
> > scan
> > > > > buffer
> > > > > > to avoid excessive evictions of other hot pages. Corresponding
> > ticket
> > > > for
> > > > > > SQL exists [1], but idea is common for all parts of the system,
> > > > requiring
> > > > > > scans.
> > > > > >
> > > > > > As far as proposed solution, it might be good idea to add special
> > API
> > > > to
> > > > > > "warmup" partition with clear 

[jira] [Created] (IGNITE-9624) Refine CacheAtomicityMode.TRANSACTIONAL_SNAPSHOT javadoc

[Ivan Pavlukhin (JIRA)]

Ivan Pavlukhin created IGNITE-9624:
--

 Summary: Refine CacheAtomicityMode.TRANSACTIONAL_SNAPSHOT javadoc
 Key: IGNITE-9624
 URL: https://issues.apache.org/jira/browse/IGNITE-9624
 Project: Ignite
  Issue Type: Bug
  Components: mvcc
Reporter: Ivan Pavlukhin


Currently {{CacheAtomicityMode.TRANSACTIONAL_SNAPSHOT}} javadoc contains some 
typos. It might be good idea to proof read it and correct grammar.
See PR as starting point.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: PHP thin client

[Stepan Pilshchikov]

Sorry for interruption but im also review this feature and have couple
questions about documentation

1. Why exists second branches with all documentation? I mean they looks good
together, why need to separate?
Also, one place when i found this branch it is this thread, not in jira, not
in a github
2. Examples in readme.md have better look in php/examples. I mean
php/examples have one set of examples and readme have second one, they not
match and they both looks good. Why doesn't add readme examples in
php/examples? 



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[GitHub] asfgit closed pull request #12: Decode jira token

[GitBox]

asfgit closed pull request #12: Decode jira token
URL: https://github.com/apache/ignite-teamcity-bot/pull/12
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/HelperConfig.java 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/HelperConfig.java
index 01d6277..181afcb 100644
--- a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/HelperConfig.java
+++ b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/HelperConfig.java
@@ -149,12 +149,14 @@ public static File resolveWorkDir() {
  * @return Null or decoded auth token for Github.
  */
 @Nullable static String prepareJiraHttpAuthToken(Properties props) {
-String pwd = props.getProperty(JIRA_AUTH_TOKEN);
+String tok = props.getProperty(JIRA_AUTH_TOKEN);
 
-if (isNullOrEmpty(pwd))
+if (isNullOrEmpty(tok))
 return null;
 
-return pwd;
+tok = PasswordEncoder.decode(tok);
+
+return tok;
 }
 
 /**


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] SomeFire opened a new pull request #12: Decode jira token

[GitBox]

SomeFire opened a new pull request #12: Decode jira token
URL: https://github.com/apache/ignite-teamcity-bot/pull/12
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services