[GitHub] ignite pull request #3471: IGNITE-5779 Incorrect mapping of smallint cassand...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3471


---

[GitHub] ignite pull request #3471: IGNITE-5779 Incorrect mapping of smallint cassand...

[irudyak]

GitHub user irudyak opened a pull request:

https://github.com/apache/ignite/pull/3471

IGNITE-5779 Incorrect mapping of smallint cassandra type.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/irudyak/ignite ignite-5779

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3471.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3471


commit fdbfb67b47659b17755b2e3c1086446d6f351ec9
Author: Igor Rudyak 
Date:   2018-02-05T05:55:12Z

IGNITE-5779




---

[jira] [Created] (IGNITE-7620) Not able to run query in Ignite YARN

[Puviarasu (JIRA)]

Puviarasu created IGNITE-7620:
-

 Summary: Not able to run query in Ignite YARN
 Key: IGNITE-7620
 URL: https://issues.apache.org/jira/browse/IGNITE-7620
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.3
 Environment: Cloudera Enterprise 5.11.2

Ignite version 2.3.0

Deployed as Ignite YARN application.
Reporter: Puviarasu


Failed to run map query remotely. javax.cache.CacheException: Failed to run map 
query remotely.
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:748)
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$8.iterator(IgniteH2Indexing.java:1212)
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.processors.cache.QueryCursorImpl.iterator(QueryCursorImpl.java:95)
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.doDelete(DmlStatementsProcessor.java:553)
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.processDmlSelectResult(DmlStatementsProcessor.java:444)
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.executeUpdateStatement(DmlStatementsProcessor.java:420)
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.updateSqlFields(DmlStatementsProcessor.java:194)
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.processors.query.h2.DmlStatementsProcessor.updateSqlFieldsDistributed(DmlStatementsProcessor.java:229)
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryDistributedSqlFields(IgniteH2Indexing.java:1453)
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.processors.query.GridQueryProcessor$5.applyx(GridQueryProcessor.java:1909)
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.processors.query.GridQueryProcessor$5.applyx(GridQueryProcessor.java:1907)
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:36)
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:2445)
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.processors.query.GridQueryProcessor.querySqlFields(GridQueryProcessor.java:1914)
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:585)
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:560)
18/02/02 23:01:15 ERROR ProcessThread$$anonfun$run$1: 
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.query(GatewayProtectedCacheProxy.java:382)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Eviction policies with persistence

[Dmitry Pavlov]

Hi Alex,

Thank you for explanation. Do you mind if I copy this explanation to wiki
page?

Strongly +1 for new term for page replacement mechanism for PDS mode. I
like 'page swapping' because it has analogue in operating systems.

New term will also identify feature by class name PageEviction... and
PageSwap... will point to class purpose.

Sincerely,
Dmitriy Pavlov

вс, 4 февр. 2018 г. в 23:05, Alexey Goncharuk :

> Guys,
>
> To clarify the questions, I would like to reiterate over the mechanics of
> evictions and then suggest a renaming that should resolve such things in
> the future.
>
> First of all, Lucas is right - eviction policy only makes sense when native
> persistence is disabled because what it does is actually freeing up memory
> when a user hits the memory limit. The only way to do this is to destroy
> inserted data because there is no other way to free memory. Now as per the
> eviction mechanism, it is both per-page and per-entry: first, we choose a
> page which fits most for eviction, however, we cannot simply erase the page
> because quite a lot of data structures are referring to that page. To deal
> with it, we read keys that the chosen page contains and then clear
> individual cache entries, which in turn will clear up all necessary links.
> If there are no concurrent updates, the page becomes empty and will be
> reused for other user data. This is exactly what is explained on the wiki
> page (at least in my reading of the page).
>
> Second, at this point, I would rename page management mechanism with
> enabled persistence from 'eviction' to 'page replacement' or 'page
> swapping', because it does not destroy any data, but rather replaces a
> chosen buffer in memory from one page to another. The content of neither
> pages does not change during page replacement. This mechanism is unlikely
> to be selected by a user because the effectiveness of page replacements
> heavily depends on internal data structures and may change from version to
> version, and may even be adaptive depending on the load pattern.
>
> Hope this resolves the confusion.
>
> --AG
>
> 2018-02-03 1:03 GMT+03:00 Denis Magda :
>
> > Dmitriy,
> >
> > I’m surprised to hear that Random-LRU works at the entry level when
> Ignite
> > persistence is off. Frankly, this section confuses a lot:
> > https://cwiki.apache.org/confluence/display/IGNITE/
> > Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
> > underthehood-Entryeviction  > confluence/display/IGNITE/Ignite+Durable+Memory+-+under+
> > the+hood#IgniteDurableMemory-underthehood-Entryeviction>
> >
> > While it was assumed that the entry-based eviction works only for on-heap
> > caches:
> > https://apacheignite.readme.io/docs/evictions <
> > https://apacheignite.readme.io/docs/evictions>
> >
> > Alex G., please chime in and clarify the misunderstanding.
> >
> > —
> > Denis
> >
> > > On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov 
> wrote:
> > >
> > > Hi Val,
> > >
> > > I think it is quite accurate because eviction in case PDS enabled or
> not
> > > has quite different purposes.
> > >
> > > 1. Let us consider PDS enabled and page eviction occurs. First of all
> it
> > is
> > > page based eviction, but not entry-based. Actually data is not removed,
> > but
> > > only written to disk. We can address this page later by ID.
> > > PDS eviction is primarily the replacement of pages from memory to page
> on
> > > disk. This eviction policy should ensure the maximum performance of the
> > > solution in the future. There is no data removal from grid in this
> case.
> > > And Ignite does not allow users to configure the policy. If benchmarks
> > show
> > > that a change in policy results in increased performance, then we can
> > > switch policy.
> > >
> > > 2. Entry-based eviction (if there is no persistence is available) is
> > > slithly different. User will need to reload data into grid in case
> entry
> > is
> > > evicted but required in cache. In that case Ignite provides selection
> of
> > > policies.
> > >
> > > Sincerely,
> > > Dmirtiy Pavlov
> > >
> > >
> > >
> > > чт, 1 февр. 2018 г. в 22:24, Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com>:
> > >
> > >> Guys,
> > >>
> > >> Thanks for responses. But I still fail to understand how it affects a
> > user.
> > >>
> > >> Are you saying that with persistence enabled Random-LRU is always used
> > >> regardless of what is specified in pageEvictionMode, while in
> in-memory
> > >> only scenario we can also utilize Random-2-LRU for eviction? Is this
> > >> accurate? Are there any other differences?
> > >>
> > >> -Val
> > >>
> > >> On Thu, Feb 1, 2018 at 5:01 AM, Dmitry Pavlov 
> > >> wrote:
> > >>
> > >>> Hi Vyacheslav,
> > >>>
> > >>> Yes, this is right, but for now PDS page based eviction uses
> > Random-LRU,
> > >>> not Random-2-LRU. Something may be changed in future Ignite releases,
> > but
> > >>> now Random-LRU is used. I am going to research if there is some room
> > for
> > >>> impro

[jira] [Created] (IGNITE-7619) Documentation: using native persistence with CacheStore

[Alexey Goncharuk (JIRA)]

Alexey Goncharuk created IGNITE-7619:


 Summary: Documentation: using native persistence with CacheStore
 Key: IGNITE-7619
 URL: https://issues.apache.org/jira/browse/IGNITE-7619
 Project: Ignite
  Issue Type: Task
Affects Versions: 2.4
Reporter: Alexey Goncharuk
Assignee: Alexey Goncharuk
 Fix For: 2.4






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Eviction policies with persistence

[Alexey Goncharuk]

Guys,

To clarify the questions, I would like to reiterate over the mechanics of
evictions and then suggest a renaming that should resolve such things in
the future.

First of all, Lucas is right - eviction policy only makes sense when native
persistence is disabled because what it does is actually freeing up memory
when a user hits the memory limit. The only way to do this is to destroy
inserted data because there is no other way to free memory. Now as per the
eviction mechanism, it is both per-page and per-entry: first, we choose a
page which fits most for eviction, however, we cannot simply erase the page
because quite a lot of data structures are referring to that page. To deal
with it, we read keys that the chosen page contains and then clear
individual cache entries, which in turn will clear up all necessary links.
If there are no concurrent updates, the page becomes empty and will be
reused for other user data. This is exactly what is explained on the wiki
page (at least in my reading of the page).

Second, at this point, I would rename page management mechanism with
enabled persistence from 'eviction' to 'page replacement' or 'page
swapping', because it does not destroy any data, but rather replaces a
chosen buffer in memory from one page to another. The content of neither
pages does not change during page replacement. This mechanism is unlikely
to be selected by a user because the effectiveness of page replacements
heavily depends on internal data structures and may change from version to
version, and may even be adaptive depending on the load pattern.

Hope this resolves the confusion.

--AG

2018-02-03 1:03 GMT+03:00 Denis Magda :

> Dmitriy,
>
> I’m surprised to hear that Random-LRU works at the entry level when Ignite
> persistence is off. Frankly, this section confuses a lot:
> https://cwiki.apache.org/confluence/display/IGNITE/
> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
> underthehood-Entryeviction  confluence/display/IGNITE/Ignite+Durable+Memory+-+under+
> the+hood#IgniteDurableMemory-underthehood-Entryeviction>
>
> While it was assumed that the entry-based eviction works only for on-heap
> caches:
> https://apacheignite.readme.io/docs/evictions <
> https://apacheignite.readme.io/docs/evictions>
>
> Alex G., please chime in and clarify the misunderstanding.
>
> —
> Denis
>
> > On Feb 2, 2018, at 4:01 AM, Dmitry Pavlov  wrote:
> >
> > Hi Val,
> >
> > I think it is quite accurate because eviction in case PDS enabled or not
> > has quite different purposes.
> >
> > 1. Let us consider PDS enabled and page eviction occurs. First of all it
> is
> > page based eviction, but not entry-based. Actually data is not removed,
> but
> > only written to disk. We can address this page later by ID.
> > PDS eviction is primarily the replacement of pages from memory to page on
> > disk. This eviction policy should ensure the maximum performance of the
> > solution in the future. There is no data removal from grid in this case.
> > And Ignite does not allow users to configure the policy. If benchmarks
> show
> > that a change in policy results in increased performance, then we can
> > switch policy.
> >
> > 2. Entry-based eviction (if there is no persistence is available) is
> > slithly different. User will need to reload data into grid in case entry
> is
> > evicted but required in cache. In that case Ignite provides selection of
> > policies.
> >
> > Sincerely,
> > Dmirtiy Pavlov
> >
> >
> >
> > чт, 1 февр. 2018 г. в 22:24, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com>:
> >
> >> Guys,
> >>
> >> Thanks for responses. But I still fail to understand how it affects a
> user.
> >>
> >> Are you saying that with persistence enabled Random-LRU is always used
> >> regardless of what is specified in pageEvictionMode, while in in-memory
> >> only scenario we can also utilize Random-2-LRU for eviction? Is this
> >> accurate? Are there any other differences?
> >>
> >> -Val
> >>
> >> On Thu, Feb 1, 2018 at 5:01 AM, Dmitry Pavlov 
> >> wrote:
> >>
> >>> Hi Vyacheslav,
> >>>
> >>> Yes, this is right, but for now PDS page based eviction uses
> Random-LRU,
> >>> not Random-2-LRU. Something may be changed in future Ignite releases,
> but
> >>> now Random-LRU is used. I am going to research if there is some room
> for
> >>> improvements in this aspect (e.g. IGNITE-7507 recenly found).
> >>>
> >>> If clean page is evicted, nothing really happends. If dirty page is
> >> evicted
> >>> from region during checkpointing process run, page is stored in file
> >> before
> >>> eviction
> >>>
> >>> Some info about this can be found in
> >>> https://cwiki.apache.org/confluence/display/IGNITE/
> >>> Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-
> >>> underthehood-Pagebasedeviction
> >>>
> >>> Sincerely,
> >>> Dmitriy Pavlov
> >>>
> >>> чт, 1 февр. 2018 г. в 9:53, Vyacheslav Daradur :
> >>>
>  Hi Valentin,
> 
>  As far as I know, when persistence is enabled a

[GitHub] ignite pull request #3470: ignite-5804: ScanQuery transformer should be appl...

[sk0x50]

GitHub user sk0x50 opened a pull request:

https://github.com/apache/ignite/pull/3470

ignite-5804: ScanQuery transformer should be applied to all result pages



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sk0x50/ignite ignite-5804

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3470.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3470


commit 657255be916f0e05e55f73eb13c8becccfd144d3
Author: Slava Koptilin 
Date:   2018-02-04T19:48:31Z

ignite-5804: ScanQuery transformer applies to all result pages




---

[GitHub] ignite pull request #3410: IGNITE-7437 Partition based dataset implementatio...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3410


---