Re: IGNITE-3066 Set of Redis commands that can be easily implemented via existing REST commands

[Alexey Kuznetsov]

Roman,

I reviewed your code and now it looks good for me.
But I added two minor comments in JIRA.

Also I think Andrey Novikov should take a look, as he has some experience
in ignite-rest module.

Andrey, take a look:

Issue: https://issues.apache.org/jira/browse/IGNITE-3066
PR:  https://github.com/apache/ignite/pull/1212


On Tue, Nov 15, 2016 at 9:27 AM, Roman Shtykh 
wrote:

> Alexey,
> Thank you!I answered and pushed the changes.
> -Roman
>
>
> On Tuesday, November 15, 2016 12:14 AM, Alexey Kuznetsov <
> akuznet...@apache.org> wrote:
>
>
>  Roman,
>
> I made one more review,  see my comments in JIRA issue.
>
> On Mon, Nov 7, 2016 at 1:30 PM, Alexey Kuznetsov 
> wrote:
>
> > I will take a look on PR today.
> >
> > On Mon, Nov 7, 2016 at 11:35 AM, Roman Shtykh  >
> > wrote:
> >
> >>  Denis,
> >> It is https://github.com/apache/ignite/pull/1212
> >>
> >> Thank you,
> >> Roman
> >>
> >>
> >>On Saturday, November 5, 2016 4:56 AM, Denis Magda <
> >> dma...@gridgain.com> wrote:
> >>
> >>
> >>  Roman,
> >>
> >> Would you mind making a pull-request? It’s not clear and easy to review
> >> using the branch you provided
> >> https://github.com/apache/ignite/tree/ignite-2788 <
> >> https://github.com/apache/ignite/tree/ignite-2788>
> >>
> >> This link provides details how to achieve this
> >> https://cwiki.apache.org/confluence/display/IGNITE/How+to+
> >> Contribute#HowtoContribute-1.CreateGitHubpull-request <
> >> https://cwiki.apache.org/confluence/display/IGNITE/How+to+
> >> Contribute#HowtoContribute-1.CreateGitHubpull-request>
> >>
> >> Let us know if you have any issue preparing the pull-request.
> >>
> >> —
> >> Denis
> >>
> >> > On Nov 3, 2016, at 6:24 PM, Roman Shtykh 
> >> wrote:
> >> >
> >> > Igniters,
> >> > Please review the issue.https://issues.apache.or
> >> g/jira/browse/IGNITE-3066
> >> >
> >> > Thank you,Roman
> >>
> >>
> >>
> >>
> >
> >
> >
> > --
> > Alexey Kuznetsov
> >
>
>
>
> --
> Alexey Kuznetsov
>
>
>



-- 
Alexey Kuznetsov

Re: Request for review IGNITE-4198

[Roman Shtykh]

Hi Vladimir,
Thanks a lot for the review!I pushed the changes. Please let me know if it is 
good to merge now.
-Roman
 

On Friday, November 11, 2016 11:50 PM, Vladimir Ozerov 
 wrote:
 

 Hi Roman,

Reviewed. My comments are in the ticket.

Vladimir.

On Thu, Nov 10, 2016 at 8:00 AM, Roman Shtykh 
wrote:

> Igniters,
> Can anyone have a look and give comment on this issue?
> IGNITE-4198: Kafka Connect sink option to transform Kafka values.
> https://issues.apache.org/jira/browse/IGNITE-4198
>
> -Roman
>


   

Re: IGNITE-3066 Set of Redis commands that can be easily implemented via existing REST commands

[Roman Shtykh]

Alexey,
Thank you!I answered and pushed the changes.
-Roman
 

On Tuesday, November 15, 2016 12:14 AM, Alexey Kuznetsov 
 wrote:
 

 Roman,

I made one more review,  see my comments in JIRA issue.

On Mon, Nov 7, 2016 at 1:30 PM, Alexey Kuznetsov 
wrote:

> I will take a look on PR today.
>
> On Mon, Nov 7, 2016 at 11:35 AM, Roman Shtykh 
> wrote:
>
>>  Denis,
>> It is https://github.com/apache/ignite/pull/1212
>>
>> Thank you,
>> Roman
>>
>>
>>    On Saturday, November 5, 2016 4:56 AM, Denis Magda <
>> dma...@gridgain.com> wrote:
>>
>>
>>  Roman,
>>
>> Would you mind making a pull-request? It’s not clear and easy to review
>> using the branch you provided
>> https://github.com/apache/ignite/tree/ignite-2788 <
>> https://github.com/apache/ignite/tree/ignite-2788>
>>
>> This link provides details how to achieve this
>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+
>> Contribute#HowtoContribute-1.CreateGitHubpull-request <
>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+
>> Contribute#HowtoContribute-1.CreateGitHubpull-request>
>>
>> Let us know if you have any issue preparing the pull-request.
>>
>> —
>> Denis
>>
>> > On Nov 3, 2016, at 6:24 PM, Roman Shtykh 
>> wrote:
>> >
>> > Igniters,
>> > Please review the issue.https://issues.apache.or
>> g/jira/browse/IGNITE-3066
>> >
>> > Thank you,Roman
>>
>>
>>
>>
>
>
>
> --
> Alexey Kuznetsov
>



-- 
Alexey Kuznetsov

   

Re: Apache Ignite 1.8 Release

[Denis Magda]

Alexander P., Igor S.,

When will your merge all DML and ODBC (PDO) related changes into 1.8 branch? 
I’m looking forward to go through PDO [1] documentation and be sure that 
everything works as described on my side.

Pavel,

Do you think it will be possible to complete all the .NET usability tickets [2] 
under 1.8 and roll them out to the Apache Ignite users?

[1] https://issues.apache.org/jira/browse/IGNITE-3921 

[2] https://issues.apache.org/jira/browse/IGNITE-4114 


—
Denis

> On Nov 9, 2016, at 6:55 AM, Denis Magda  wrote:
> 
> Do we have a branch for ignite-1.8? Is there anyone who can take over the 
> release process of 1.8?
> 
> —
> Denis
> 
>> On Nov 8, 2016, at 9:01 PM, Alexander Paschenko 
>>  wrote:
>> 
>> Current status on DML:
>> 
>> - Basic data streamer support implemented (basicness is mostly about
>> configuration - say, currently there's no way to specify streamer's
>> batch size via JDBC driver, but this can be improved easily).
>> 
>> - Fixed all minor stuff agreed with Vladimir.
>> 
>> - There are some tests that started failing after binary hash codes
>> generation rework made by Vladimir in ignite-4011-1 branch, I will ask
>> him to look into it and fix those. Failing tests live in
>> GridCacheBinaryObjectsAbstractSelfTest, and are as follows:
>> - testPutWithFieldsHashing
>> - testCrossFormatObjectsIdentity
>> - testPutWithCustomHashing
>> I added them personally during working on first version of auto
>> hashing few weeks ago, and what they do is test these very hashing
>> features. Again, prior to Vlad's rework those tests passed. So could
>> you please take a look?
>> 
>> - Working on Sergey V.'s comments about current code.
>> 
>> - Alex
> 

Re: Code Review Tool Proposal: Upsource

[Sergi Vladykin]

I'd say that Github may be good for small changes, but when you trying to
review a big PR (like 100 files) it becomes almost unusable, because it
loads everything in one shot with no convenient navigation.

Also it happened to me that I lost some of my review comments on Github for
no obvious reason. It was really disappointing.

Thus I hope Upsource will do a better job here.

Anyways, I don't think we are going to enforce everyone to use Upsource, it
is just another tool in addition to existing ones.

Sergi

2016-11-14 22:16 GMT+03:00 Pavel Tupitsyn :

> Denis,
>
> Contributors will have to start a review on branch or pull request manually
> (a couple of clicks really), then attach an URL to the JIRA ticket.
> Example: https://issues.apache.org/jira/browse/IGNITE-4116
>
> > are there any examples of Apache projects that used some 3rd party tool
> for review process
> Some projects use Crucible: https://fisheye6.atlassian.com/
> Apache Hive used Phabricator in the past.
>
>
> Mike,
>
> > Why not / what is wrong with GitHub?
> Nothing is wrong with GitHub, I think it is the second best option.
> Still, Upsource is much nicer, so I'd like to explore this possibility.
>
> > commercial tool I have to pay for
> They provide open source license. We license TeamCity this way.
>
> On Mon, Nov 14, 2016 at 8:32 PM, Michael André Pearce <
> michael.andre.pea...@me.com> wrote:
>
> > Why not / what is wrong with GitHub?
> >
> > Code is there anyhow...
> >
> > I've found this seems to be the way a lot of projects have gone.
> >
> > It allows me to review the code without checkout
> >
> > I can comment inline with a pr or code commit
> >
> > I can fork a project to my own space and create a pr back to the main
> repo
> >
> > It updates when I make a commit
> >
> > Supports multiple reviewers.
> >
> > Eco system of bots
> >
> > It doesn't tie me into a commercial ide tool (I love IntelliJ like the
> > next person, but appreciate it is a commercial tool I have to pay for for
> > all the bells and whistles)
> >
> > Rgds
> > Mike
> >
> > > On 14 Nov 2016, at 17:03, Denis Magda  wrote:
> > >
> > > Pavel,
> > >
> > > How will the contribution process be affected if the community switches
> > to Upsource? Will Upsource introduce additional steps for those who want
> to
> > ask someone to review a branch or the tool simply intercepts all the
> > pull-requests automatically?
> > >
> > > Cos, Raul, Others,
> > >
> > > How this intention is aligned with Apache at all? In you experience,
> are
> > there any examples of Apache projects that used some 3rd party tool for
> > review process?
> > >
> > > —
> > > Denis
> > >
> > >> On Nov 14, 2016, at 4:08 AM, Pavel Tupitsyn 
> > wrote:
> > >>
> > >> Igniters,
> > >>
> > >> We have set up Upsource code review tool at
> > >> http://reviews.ignite.apache.org/
> > >>
> > >> I propose to evaluate it and see if it works for us.
> > >>
> > >>
> > >> * Why?
> > >> Current JIRA-based process is not very efficient. Anyone who have
> used a
> > >> review tool will probably agree:
> > >>
> > >> - No need to switch branches locally and interrupt your current work.
> > You
> > >> can see the code in one click.
> > >> - All current reviews are easily accessible
> > >> - Multiple reviewers
> > >> - Much better discussions: comments are right in the code; each point
> > can
> > >> be discussed and accepted separately
> > >> - Integrates with IDEA - open the diff in IDEA in one click, or see
> the
> > >> reviews there without opening the browser at all
> > >>
> > >>
> > >> * Why Upsource?
> > >> I've evaluated a bunch of tools (CodeCollaborator, ReviewBoard,
> > >> Phabricator, Crucible),
> > >> and Upsource looks like the best fit for us:
> > >> - PR-based code reviews. This is a major advantage: review for a PR
> can
> > be
> > >> created in one click, and it updates automatically when you push more
> > >> commits (fix review issues)
> > >> - Good Java support and IDEA integration
> > >> - Good performance (our code base is big, and tools like Crucible
> really
> > >> struggle with it)
> > >>
> > >>
> > >> Thoughts and suggestions are welcome.
> > >>
> > >> Thanks,
> > >>
> > >> Pavel
> > >
> >
>

[jira] [Created] (IGNITE-4222) Document the usage of ComputeJobContinuation API

[Denis Magda (JIRA)]

Denis Magda created IGNITE-4222:
---

 Summary: Document the usage of ComputeJobContinuation API
 Key: IGNITE-4222
 URL: https://issues.apache.org/jira/browse/IGNITE-4222
 Project: Ignite
  Issue Type: Task
Reporter: Denis Magda
 Fix For: 2.0


The usage and applicability of `ComputeJobContinuation` have to be documented 
on Apache Ignite Readme.io which will help out to avoid discussion like that 
[1]. The new page has to be created for the topic and placed here [2].

The content should consist not only of technical details but also provide use 
cases for this API:
- don't block a thread from the public pool if a job is waiting for some 
result. Put the thread back into the pool and wait for the result 
asynchronously.
- splitting a job into several pieces that can be executed in parallel. Refer 
to this example [3].
 
[1] 
http://apache-ignite-users.70518.x6.nabble.com/Remote-Server-Thread-Not-exit-when-Job-finished-Cause-out-of-memory-tp8934p8947.html
[2] https://apacheignite.readme.io/docs/compute-grid#section-features
[3] 
https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/computegrid/ComputeFibonacciContinuationExample.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4221) Document ComputeJobMasterLeaveAware interface usage

[Denis Magda (JIRA)]

Denis Magda created IGNITE-4221:
---

 Summary: Document ComputeJobMasterLeaveAware interface usage
 Key: IGNITE-4221
 URL: https://issues.apache.org/jira/browse/IGNITE-4221
 Project: Ignite
  Issue Type: Task
Reporter: Denis Magda
 Fix For: 2.0


The usage and applicability of `ComputeJobMasterLeaveAware` have to be 
documented on Apache Ignite Readme.io which will help out to avoid discussion 
like that [1]. The new page has to be created for the topic and placed here [2].

In advance, the following example has to be contributed to Apache Ignite
https://github.com/gridgain/gridgain-advanced-examples/blob/master/src/main/java/org/gridgain/examples/compute/masterleave/ComputeMasterLeaveAwareExample.java
 


[1] 
http://apache-ignite-users.70518.x6.nabble.com/Remote-Server-Thread-Not-exit-when-Job-finished-Cause-out-of-memory-tp8934p8947.html
[2] https://apacheignite.readme.io/docs/compute-grid#section-features



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: new committer: Igor Rudyak

[Igor Rudyak]

Thanks. Will try to do my best for the project.

Igor


On Mon, Nov 14, 2016 at 11:03 AM, Denis Magda  wrote:

> The Project Management Committee (PMC) for Apache Ignite
> has invited Igor Rudyak to become a committer and we are pleased
> to announce that he has accepted.
>
> Igor is known inside of Apache Ignite community for such a valuable and
> priceless contribution as Cassandra CacheStore implementation for Apache
> Ignite which was developed my him from scratch and being maintained for a
> while.
>
> Being a committer enables easier contribution to the
> project since there is no need to go via the patch
> submission process. This should enable better productivity.
> Being a PMC member enables assistance with the management
> and to guide the direction of the project.
>
> —
> Denis

Re: Code Review Tool Proposal: Upsource

[Pavel Tupitsyn]

Denis,

Contributors will have to start a review on branch or pull request manually
(a couple of clicks really), then attach an URL to the JIRA ticket.
Example: https://issues.apache.org/jira/browse/IGNITE-4116

> are there any examples of Apache projects that used some 3rd party tool
for review process
Some projects use Crucible: https://fisheye6.atlassian.com/
Apache Hive used Phabricator in the past.


Mike,

> Why not / what is wrong with GitHub?
Nothing is wrong with GitHub, I think it is the second best option.
Still, Upsource is much nicer, so I'd like to explore this possibility.

> commercial tool I have to pay for
They provide open source license. We license TeamCity this way.

On Mon, Nov 14, 2016 at 8:32 PM, Michael André Pearce <
michael.andre.pea...@me.com> wrote:

> Why not / what is wrong with GitHub?
>
> Code is there anyhow...
>
> I've found this seems to be the way a lot of projects have gone.
>
> It allows me to review the code without checkout
>
> I can comment inline with a pr or code commit
>
> I can fork a project to my own space and create a pr back to the main repo
>
> It updates when I make a commit
>
> Supports multiple reviewers.
>
> Eco system of bots
>
> It doesn't tie me into a commercial ide tool (I love IntelliJ like the
> next person, but appreciate it is a commercial tool I have to pay for for
> all the bells and whistles)
>
> Rgds
> Mike
>
> > On 14 Nov 2016, at 17:03, Denis Magda  wrote:
> >
> > Pavel,
> >
> > How will the contribution process be affected if the community switches
> to Upsource? Will Upsource introduce additional steps for those who want to
> ask someone to review a branch or the tool simply intercepts all the
> pull-requests automatically?
> >
> > Cos, Raul, Others,
> >
> > How this intention is aligned with Apache at all? In you experience, are
> there any examples of Apache projects that used some 3rd party tool for
> review process?
> >
> > —
> > Denis
> >
> >> On Nov 14, 2016, at 4:08 AM, Pavel Tupitsyn 
> wrote:
> >>
> >> Igniters,
> >>
> >> We have set up Upsource code review tool at
> >> http://reviews.ignite.apache.org/
> >>
> >> I propose to evaluate it and see if it works for us.
> >>
> >>
> >> * Why?
> >> Current JIRA-based process is not very efficient. Anyone who have used a
> >> review tool will probably agree:
> >>
> >> - No need to switch branches locally and interrupt your current work.
> You
> >> can see the code in one click.
> >> - All current reviews are easily accessible
> >> - Multiple reviewers
> >> - Much better discussions: comments are right in the code; each point
> can
> >> be discussed and accepted separately
> >> - Integrates with IDEA - open the diff in IDEA in one click, or see the
> >> reviews there without opening the browser at all
> >>
> >>
> >> * Why Upsource?
> >> I've evaluated a bunch of tools (CodeCollaborator, ReviewBoard,
> >> Phabricator, Crucible),
> >> and Upsource looks like the best fit for us:
> >> - PR-based code reviews. This is a major advantage: review for a PR can
> be
> >> created in one click, and it updates automatically when you push more
> >> commits (fix review issues)
> >> - Good Java support and IDEA integration
> >> - Good performance (our code base is big, and tools like Crucible really
> >> struggle with it)
> >>
> >>
> >> Thoughts and suggestions are welcome.
> >>
> >> Thanks,
> >>
> >> Pavel
> >
>

[jira] [Created] (IGNITE-4220) Allow to pass statements instead of strings to CacheStore.loadCache() implementations

[Valentin Kulichenko (JIRA)]

Valentin Kulichenko created IGNITE-4220:
---

 Summary: Allow to pass statements instead of strings to 
CacheStore.loadCache() implementations
 Key: IGNITE-4220
 URL: https://issues.apache.org/jira/browse/IGNITE-4220
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Affects Versions: 1.7
Reporter: Valentin Kulichenko


Currently both Cassandra store and POJO store allow user to provide a set of 
queries used for data loading. However, there are cases when user may want to 
provide a {{Statement}} instead (e.g., prepared statements with parameter 
binding are used). We should support {{Statement}} arguments in store 
implementations as well as strings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: IgniteCache.loadCache improvement proposal

[Valentin Kulichenko]

Hi Aleksandr,

Data streamer is already outlined as one of the possible approaches for
loading the data [1]. Basically, you start a designated client node or
chose a leader among server nodes [1] and then use IgniteDataStreamer API
to load the data. With this approach there is no need to have the
CacheStore implementation at all. Can you please elaborate what additional
value are you trying to add here?

[1] https://apacheignite.readme.io/docs/data-loading#ignitedatastreamer
[2] https://apacheignite.readme.io/docs/leader-election

-Val

On Mon, Nov 14, 2016 at 8:23 AM, Dmitriy Setrakyan 
wrote:

> Hi,
>
> I just want to clarify a couple of API details from the original email to
> make sure that we are making the right assumptions here.
>
> *"Because of none keys are passed to the CacheStore.loadCache methods, the
> > underlying implementation is forced to read all the data from the
> > persistence storage"*
>
>
> According to the javadoc, loadCache(...) method receives an optional
> argument from the user. You can pass anything you like, including a list of
> keys, or an SQL where clause, etc.
>
> *"The partition-aware data loading approach is not a choice. It requires
> > persistence of the volatile data depended on affinity function
> > implementation and settings."*
>
>
> This is only partially true. While Ignite allows to plugin custom affinity
> functions, the affinity function is not something that changes dynamically
> and should always return the same partition for the same key.So, the
> partition assignments are not volatile at all. If, in some very rare case,
> the partition assignment logic needs to change, then you could update the
> partition assignments that you may have persisted elsewhere as well, e.g.
> database.
>
> D.
>
> On Mon, Nov 14, 2016 at 10:23 AM, Vladimir Ozerov 
> wrote:
>
> > Alexandr, Alexey,
> >
> > While I agree with you that current cache loading logic is far from
> ideal,
> > it would be cool to see API drafts based on your suggestions to get
> better
> > understanding of your ideas. How exactly users are going to use your
> > suggestions?
> >
> > My main concern is that initial load is not very trivial task in general
> > case. Some users have centralized RDBMS systems, some have NoSQL, others
> > work with distributed persistent stores (e.g. HDFS). Sometimes we have
> > Ignite nodes "near" persistent data, sometimes we don't. Sharding,
> > affinity, co-location, etc.. If we try to support all (or many) cases out
> > of the box, we may end up in very messy and difficult API. So we should
> > carefully balance between simplicity, usability and feature-rich
> > characteristics here.
> >
> > Personally, I think that if user is not satisfied with "loadCache()" API,
> > he just writes simple closure with blackjack streamer and queries and
> send
> > it to whatever node he finds convenient. Not a big deal. Only very common
> > cases should be added to Ignite API.
> >
> > Vladimir.
> >
> >
> > On Mon, Nov 14, 2016 at 12:43 PM, Alexey Kuznetsov <
> > akuznet...@gridgain.com>
> > wrote:
> >
> > > Looks good for me.
> > >
> > > But I will suggest to consider one more use-case:
> > >
> > > If user knows its data he could manually split loading.
> > > For example: table Persons contains 10M rows.
> > > User could provide something like:
> > > cache.loadCache(null, "Person", "select * from Person where id <
> > > 1_000_000",
> > > "Person", "select * from Person where id >=  1_000_000 and id <
> > 2_000_000",
> > > 
> > > "Person", "select * from Person where id >= 9_000_000 and id <
> > 10_000_000",
> > > );
> > >
> > > or may be it could be some descriptor object like
> > >
> > >  {
> > >sql: select * from Person where id >=  ? and id < ?"
> > >range: 0...10_000_000
> > > }
> > >
> > > In this case provided queries will be send to mach nodes as number of
> > > queries.
> > > And data will be loaded in parallel and for keys that a not local -
> data
> > > streamer
> > > should be used (as described Alexandr description).
> > >
> > > I think it is a good issue for Ignite 2.0
> > >
> > > Vova, Val - what do you think?
> > >
> > >
> > > On Mon, Nov 14, 2016 at 4:01 PM, Alexandr Kuramshin <
> > ein.nsk...@gmail.com>
> > > wrote:
> > >
> > >> All right,
> > >>
> > >> Let's assume a simple scenario. When the IgniteCache.loadCache is
> > invoked,
> > >> we check whether the cache is not local, and if so, then we'll
> initiate
> > >> the
> > >> new loading logic.
> > >>
> > >> First, we take a "streamer" node, it could be done by
> > >> utilizing LoadBalancingSpi, or it may be configured statically, for
> the
> > >> reason that the streamer node is running on the same host as the
> > >> persistence storage provider.
> > >>
> > >> After that we start the loading task on the streamer node which
> > >> creates IgniteDataStreamer and loads the cache with
> > CacheStore.loadCache.
> > >> Every call to IgniteBiInClosure.apply simply
> > >> 

Re: Code Review Tool Proposal: Upsource

[Michael André Pearce]

Why not / what is wrong with GitHub?

Code is there anyhow...

I've found this seems to be the way a lot of projects have gone.

It allows me to review the code without checkout

I can comment inline with a pr or code commit

I can fork a project to my own space and create a pr back to the main repo 

It updates when I make a commit 

Supports multiple reviewers.

Eco system of bots

It doesn't tie me into a commercial ide tool (I love IntelliJ like the next 
person, but appreciate it is a commercial tool I have to pay for for all the 
bells and whistles)

Rgds
Mike

> On 14 Nov 2016, at 17:03, Denis Magda  wrote:
> 
> Pavel,
> 
> How will the contribution process be affected if the community switches to 
> Upsource? Will Upsource introduce additional steps for those who want to ask 
> someone to review a branch or the tool simply intercepts all the 
> pull-requests automatically?
> 
> Cos, Raul, Others,
> 
> How this intention is aligned with Apache at all? In you experience, are 
> there any examples of Apache projects that used some 3rd party tool for 
> review process?
> 
> —
> Denis
> 
>> On Nov 14, 2016, at 4:08 AM, Pavel Tupitsyn  wrote:
>> 
>> Igniters,
>> 
>> We have set up Upsource code review tool at
>> http://reviews.ignite.apache.org/
>> 
>> I propose to evaluate it and see if it works for us.
>> 
>> 
>> * Why?
>> Current JIRA-based process is not very efficient. Anyone who have used a
>> review tool will probably agree:
>> 
>> - No need to switch branches locally and interrupt your current work. You
>> can see the code in one click.
>> - All current reviews are easily accessible
>> - Multiple reviewers
>> - Much better discussions: comments are right in the code; each point can
>> be discussed and accepted separately
>> - Integrates with IDEA - open the diff in IDEA in one click, or see the
>> reviews there without opening the browser at all
>> 
>> 
>> * Why Upsource?
>> I've evaluated a bunch of tools (CodeCollaborator, ReviewBoard,
>> Phabricator, Crucible),
>> and Upsource looks like the best fit for us:
>> - PR-based code reviews. This is a major advantage: review for a PR can be
>> created in one click, and it updates automatically when you push more
>> commits (fix review issues)
>> - Good Java support and IDEA integration
>> - Good performance (our code base is big, and tools like Crucible really
>> struggle with it)
>> 
>> 
>> Thoughts and suggestions are welcome.
>> 
>> Thanks,
>> 
>> Pavel
> 

Re: IgniteCache.loadCache improvement proposal

[Dmitriy Setrakyan]

Hi,

I just want to clarify a couple of API details from the original email to
make sure that we are making the right assumptions here.

*"Because of none keys are passed to the CacheStore.loadCache methods, the
> underlying implementation is forced to read all the data from the
> persistence storage"*


According to the javadoc, loadCache(...) method receives an optional
argument from the user. You can pass anything you like, including a list of
keys, or an SQL where clause, etc.

*"The partition-aware data loading approach is not a choice. It requires
> persistence of the volatile data depended on affinity function
> implementation and settings."*


This is only partially true. While Ignite allows to plugin custom affinity
functions, the affinity function is not something that changes dynamically
and should always return the same partition for the same key.So, the
partition assignments are not volatile at all. If, in some very rare case,
the partition assignment logic needs to change, then you could update the
partition assignments that you may have persisted elsewhere as well, e.g.
database.

D.

On Mon, Nov 14, 2016 at 10:23 AM, Vladimir Ozerov 
wrote:

> Alexandr, Alexey,
>
> While I agree with you that current cache loading logic is far from ideal,
> it would be cool to see API drafts based on your suggestions to get better
> understanding of your ideas. How exactly users are going to use your
> suggestions?
>
> My main concern is that initial load is not very trivial task in general
> case. Some users have centralized RDBMS systems, some have NoSQL, others
> work with distributed persistent stores (e.g. HDFS). Sometimes we have
> Ignite nodes "near" persistent data, sometimes we don't. Sharding,
> affinity, co-location, etc.. If we try to support all (or many) cases out
> of the box, we may end up in very messy and difficult API. So we should
> carefully balance between simplicity, usability and feature-rich
> characteristics here.
>
> Personally, I think that if user is not satisfied with "loadCache()" API,
> he just writes simple closure with blackjack streamer and queries and send
> it to whatever node he finds convenient. Not a big deal. Only very common
> cases should be added to Ignite API.
>
> Vladimir.
>
>
> On Mon, Nov 14, 2016 at 12:43 PM, Alexey Kuznetsov <
> akuznet...@gridgain.com>
> wrote:
>
> > Looks good for me.
> >
> > But I will suggest to consider one more use-case:
> >
> > If user knows its data he could manually split loading.
> > For example: table Persons contains 10M rows.
> > User could provide something like:
> > cache.loadCache(null, "Person", "select * from Person where id <
> > 1_000_000",
> > "Person", "select * from Person where id >=  1_000_000 and id <
> 2_000_000",
> > 
> > "Person", "select * from Person where id >= 9_000_000 and id <
> 10_000_000",
> > );
> >
> > or may be it could be some descriptor object like
> >
> >  {
> >sql: select * from Person where id >=  ? and id < ?"
> >range: 0...10_000_000
> > }
> >
> > In this case provided queries will be send to mach nodes as number of
> > queries.
> > And data will be loaded in parallel and for keys that a not local - data
> > streamer
> > should be used (as described Alexandr description).
> >
> > I think it is a good issue for Ignite 2.0
> >
> > Vova, Val - what do you think?
> >
> >
> > On Mon, Nov 14, 2016 at 4:01 PM, Alexandr Kuramshin <
> ein.nsk...@gmail.com>
> > wrote:
> >
> >> All right,
> >>
> >> Let's assume a simple scenario. When the IgniteCache.loadCache is
> invoked,
> >> we check whether the cache is not local, and if so, then we'll initiate
> >> the
> >> new loading logic.
> >>
> >> First, we take a "streamer" node, it could be done by
> >> utilizing LoadBalancingSpi, or it may be configured statically, for the
> >> reason that the streamer node is running on the same host as the
> >> persistence storage provider.
> >>
> >> After that we start the loading task on the streamer node which
> >> creates IgniteDataStreamer and loads the cache with
> CacheStore.loadCache.
> >> Every call to IgniteBiInClosure.apply simply
> >> invokes IgniteDataStreamer.addData.
> >>
> >> This implementation will completely relieve overhead on the persistence
> >> storage provider. Network overhead is also decreased in the case of
> >> partitioned caches. For two nodes we get 1-1/2 amount of data
> transferred
> >> by the network (1 part well be transferred from the persistence storage
> to
> >> the streamer, and then 1/2 from the streamer node to the another node).
> >> For
> >> three nodes it will be 1-2/3 and so on, up to the two times amount of
> data
> >> on the big clusters.
> >>
> >> I'd like to propose some additional optimization at this place. If we
> have
> >> the streamer node on the same machine as the persistence storage
> provider,
> >> then we completely relieve the network overhead as well. It could be a
> >> some
> >> special daemon node for the cache loading assigned 

[jira] [Created] (IGNITE-4219) Hive job submsiion failed with exception ”java.io.UTFDataFormatException“

[Andrew Mashenkov (JIRA)]

Andrew Mashenkov created IGNITE-4219:


 Summary: Hive job submsiion failed with exception 
”java.io.UTFDataFormatException“
 Key: IGNITE-4219
 URL: https://issues.apache.org/jira/browse/IGNITE-4219
 Project: Ignite
  Issue Type: Bug
  Components: hadoop
Reporter: Andrew Mashenkov


Long property passing to Hadoop causes an exception:

{code}
Caused by: java.io.UTFDataFormatException 
   at 
java.io.ObjectOutputStream$BlockDataOutputStream.writeUTF(ObjectOutputStream.java:2144)
 
   at 
java.io.ObjectOutputStream$BlockDataOutputStream.writeUTF(ObjectOutputStream.java:1987)
 
   at java.io.ObjectOutputStream.writeUTF(ObjectOutputStream.java:865) 
   at 
org.apache.ignite.internal.util.IgniteUtils.writeUTFStringNullable(IgniteUtils.java:5029)
 
   at 
org.apache.ignite.internal.util.IgniteUtils.writeStringMap(IgniteUtils.java:4989)
 
   at 
org.apache.ignite.internal.processors.hadoop.HadoopDefaultJobInfo.writeExternal(HadoopDefaultJobInfo.java:137)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: IGNITE-3066 Set of Redis commands that can be easily implemented via existing REST commands

[Alexey Kuznetsov]

Roman,

I made one more review,  see my comments in JIRA issue.

On Mon, Nov 7, 2016 at 1:30 PM, Alexey Kuznetsov 
wrote:

> I will take a look on PR today.
>
> On Mon, Nov 7, 2016 at 11:35 AM, Roman Shtykh 
> wrote:
>
>>  Denis,
>> It is https://github.com/apache/ignite/pull/1212
>>
>> Thank you,
>> Roman
>>
>>
>> On Saturday, November 5, 2016 4:56 AM, Denis Magda <
>> dma...@gridgain.com> wrote:
>>
>>
>>  Roman,
>>
>> Would you mind making a pull-request? It’s not clear and easy to review
>> using the branch you provided
>> https://github.com/apache/ignite/tree/ignite-2788 <
>> https://github.com/apache/ignite/tree/ignite-2788>
>>
>> This link provides details how to achieve this
>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+
>> Contribute#HowtoContribute-1.CreateGitHubpull-request <
>> https://cwiki.apache.org/confluence/display/IGNITE/How+to+
>> Contribute#HowtoContribute-1.CreateGitHubpull-request>
>>
>> Let us know if you have any issue preparing the pull-request.
>>
>> —
>> Denis
>>
>> > On Nov 3, 2016, at 6:24 PM, Roman Shtykh 
>> wrote:
>> >
>> > Igniters,
>> > Please review the issue.https://issues.apache.or
>> g/jira/browse/IGNITE-3066
>> >
>> > Thank you,Roman
>>
>>
>>
>>
>
>
>
> --
> Alexey Kuznetsov
>



-- 
Alexey Kuznetsov

[GitHub] ignite pull request #1231: IGNITE-4138 .NET: document additional parameters ...

[ptupitsyn]

GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/1231

IGNITE-4138 .NET: document additional parameters usage in 
Apache.Ignite.exe.config



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-4138

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1231.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1231


commit 5e1437c8f316a9d4c0e50a6591a2996c7e9acfc6
Author: Pavel Tupitsyn 
Date:   2016-11-14T14:51:07Z

IGNITE-4138 .NET: document additional parameters usage in 
Apache.Ignite.exe.config

commit 665bc66816b6fcc0bb71ceafb90e1354d5d204aa
Author: Pavel Tupitsyn 
Date:   2016-11-14T14:57:49Z

Fix config




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Code Review Tool Proposal: Upsource

[Vladimir Ozerov]

Hi Pavel,

Very good idea! This will make review process transparent and more
community-friendly. So huge +1.

On Mon, Nov 14, 2016 at 4:17 PM, Andrey Gura  wrote:

> +1
>
> Great tool.
>
> On Mon, Nov 14, 2016 at 3:19 PM, Anton Vinogradov <
> avinogra...@gridgain.com>
> wrote:
>
> > +1
> >
> > On Mon, Nov 14, 2016 at 3:08 PM, Pavel Tupitsyn 
> > wrote:
> >
> > > Igniters,
> > >
> > > We have set up Upsource code review tool at
> > > http://reviews.ignite.apache.org/
> > >
> > > I propose to evaluate it and see if it works for us.
> > >
> > >
> > > * Why?
> > > Current JIRA-based process is not very efficient. Anyone who have used
> a
> > > review tool will probably agree:
> > >
> > > - No need to switch branches locally and interrupt your current work.
> You
> > > can see the code in one click.
> > > - All current reviews are easily accessible
> > > - Multiple reviewers
> > > - Much better discussions: comments are right in the code; each point
> can
> > > be discussed and accepted separately
> > > - Integrates with IDEA - open the diff in IDEA in one click, or see the
> > > reviews there without opening the browser at all
> > >
> > >
> > > * Why Upsource?
> > > I've evaluated a bunch of tools (CodeCollaborator, ReviewBoard,
> > > Phabricator, Crucible),
> > > and Upsource looks like the best fit for us:
> > > - PR-based code reviews. This is a major advantage: review for a PR can
> > be
> > > created in one click, and it updates automatically when you push more
> > > commits (fix review issues)
> > > - Good Java support and IDEA integration
> > > - Good performance (our code base is big, and tools like Crucible
> really
> > > struggle with it)
> > >
> > >
> > > Thoughts and suggestions are welcome.
> > >
> > > Thanks,
> > >
> > > Pavel
> > >
> >
>

Re: Code Review Tool Proposal: Upsource

[Andrey Gura]

+1

Great tool.

On Mon, Nov 14, 2016 at 3:19 PM, Anton Vinogradov 
wrote:

> +1
>
> On Mon, Nov 14, 2016 at 3:08 PM, Pavel Tupitsyn 
> wrote:
>
> > Igniters,
> >
> > We have set up Upsource code review tool at
> > http://reviews.ignite.apache.org/
> >
> > I propose to evaluate it and see if it works for us.
> >
> >
> > * Why?
> > Current JIRA-based process is not very efficient. Anyone who have used a
> > review tool will probably agree:
> >
> > - No need to switch branches locally and interrupt your current work. You
> > can see the code in one click.
> > - All current reviews are easily accessible
> > - Multiple reviewers
> > - Much better discussions: comments are right in the code; each point can
> > be discussed and accepted separately
> > - Integrates with IDEA - open the diff in IDEA in one click, or see the
> > reviews there without opening the browser at all
> >
> >
> > * Why Upsource?
> > I've evaluated a bunch of tools (CodeCollaborator, ReviewBoard,
> > Phabricator, Crucible),
> > and Upsource looks like the best fit for us:
> > - PR-based code reviews. This is a major advantage: review for a PR can
> be
> > created in one click, and it updates automatically when you push more
> > commits (fix review issues)
> > - Good Java support and IDEA integration
> > - Good performance (our code base is big, and tools like Crucible really
> > struggle with it)
> >
> >
> > Thoughts and suggestions are welcome.
> >
> > Thanks,
> >
> > Pavel
> >
>

[GitHub] ignite pull request #1230: IGNITE-4134 .NET: CacheConfiguration.ExpiryPolicy...

[ptupitsyn]

GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/1230

IGNITE-4134 .NET: CacheConfiguration.ExpiryPolicyFactory



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4134

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1230.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1230


commit 44003d5107198f5edb4e8e4804762a498ff64220
Author: Pavel Tupitsyn 
Date:   2016-11-11T08:48:21Z

IGNITE-4134 .NET: Expiration policy related improvements

commit 0604f1a24be14c5dd0a29f0d5d339ba3ba48adab
Author: Pavel Tupitsyn 
Date:   2016-11-11T08:55:39Z

Refactor expiry policy serialization

commit bd6c2ec0b049a5237bd80345d7635c7843bfa116
Author: Pavel Tupitsyn 
Date:   2016-11-11T08:59:59Z

wip

commit 17d28b8cf92ab2cbce4032f68ccb7a33abc5ee10
Author: Pavel Tupitsyn 
Date:   2016-11-11T09:00:16Z

wip

commit 82a1f3728b3c6461ea9c7a658222fbb7d14155ea
Author: Pavel Tupitsyn 
Date:   2016-11-11T09:37:30Z

Merge branch 'ignite-1.7.4' into ignite-4134

commit 7cfdb4ca90d31877080475444f93795027fae819
Author: Pavel Tupitsyn 
Date:   2016-11-11T09:44:48Z

wip

commit 2111f8c4089b058df67fac585dc02f950b2c8d7c
Author: Pavel Tupitsyn 
Date:   2016-11-11T10:14:07Z

Extract PlatformExpiryPolicy

commit 86630e20d585e2d4163107846ced0a3a866ec90c
Author: Pavel Tupitsyn 
Date:   2016-11-11T10:21:21Z

PlatformExpiryPolicyFactory

commit c58981757456f9a2e39649141b28df78c42e8cf7
Author: Pavel Tupitsyn 
Date:   2016-11-11T13:27:20Z

wip

commit 7c354129ace46c243e2afcfe634cf80814c821c7
Author: Pavel Tupitsyn 
Date:   2016-11-11T13:35:10Z

wip

commit d2b1cc03a5a3c78a92f1b344ca29753111ba3972
Author: Pavel Tupitsyn 
Date:   2016-11-11T13:37:18Z

wip

commit 00725a3f1ac7df3b6fdeed494fcc48531dcb39ab
Author: Pavel Tupitsyn 
Date:   2016-11-11T13:43:22Z

wip

commit b8f51c0702e254cedce472f819bf7314e28b3a09
Author: Pavel Tupitsyn 
Date:   2016-11-11T13:44:04Z

config read/write done

commit 7cabf1296e61321934e6531f8f3895bfd9f1f8c7
Author: Pavel Tupitsyn 
Date:   2016-11-11T13:58:13Z

read-write in Java done

commit b6886fa84b264388513d29e8ce2fde85b2d5ad0c
Author: Pavel Tupitsyn 
Date:   2016-11-11T14:11:35Z

wip tests

commit caba18ae19b1e9b4a4c99718428772e8b1cc03c7
Author: Pavel Tupitsyn 
Date:   2016-11-11T14:13:24Z

wip

commit 7a9c78a62b4a4553fe2cdeb6d69528150234a64b
Author: Pavel Tupitsyn 
Date:   2016-11-11T14:20:05Z

wip tests

commit 1ec9ae0f2046a21b3191ef6904696e2f27bc8155
Author: Pavel Tupitsyn 
Date:   2016-11-11T14:27:00Z

wip

commit 0dff7b0befdd2c81a604e3e587a9c319f293f60f
Author: Pavel Tupitsyn 
Date:   2016-11-11T15:23:21Z

wip test

commit db0317d86b64ee68a9edcf9d881a900d12c55d3a
Author: Pavel Tupitsyn 
Date:   2016-11-11T15:27:27Z

wip

commit cb50a807adcbd6ad9e40bacc96f0abe1a6c2b50f
Author: Pavel Tupitsyn 
Date:   2016-11-11T15:29:10Z

test works!

commit 85a4b966fdfb7018d1c91b73df1659082128f786
Author: Pavel Tupitsyn 
Date:   2016-11-14T10:38:33Z

IGNITE-4216 .NET: Fix PlatformAffinityFunction to inject resource into 
baseFunc

commit 4d754fe22501c11c3efda99d2c64979bd7cf2a21
Author: Pavel Tupitsyn 
Date:   2016-11-14T10:43:20Z

Merge branch 'ignite-1.7.4' into ignite-4134

commit dbf1119eb5834a2ee3ddf4069a1e042b2f43846e
Author: Pavel Tupitsyn 
Date:   2016-11-14T10:57:41Z

Cleanup

commit de7162d361442f550d7735928c10dbea14b91316
Author: Pavel Tupitsyn 
Date:   2016-11-14T11:49:34Z

Update schema

commit dd45f1da07b4ec12530193cc8c1a9633496fc06b
Author: Pavel Tupitsyn 
Date:   2016-11-14T12:30:55Z

wip

commit e52cb3053019095c32b5867bfcdb0d55087142c8
Author: Pavel Tupitsyn 
Date:   2016-11-14T12:34:12Z

tests

commit 798b740d55edd4b447e20f698eecf13b9889bb9b
Author: Pavel Tupitsyn 
Date:   2016-11-14T13:10:43Z

Tests done

commit 7f701452642191464c1ccd61126b5bf95cdceb10
Author: Pavel Tupitsyn 
Date:   2016-11-14T13:13:51Z

Merge branch 'master' into ignite-4134




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not 

Code Review Tool Proposal: Upsource

[Pavel Tupitsyn]

Igniters,

We have set up Upsource code review tool at
http://reviews.ignite.apache.org/

I propose to evaluate it and see if it works for us.


* Why?
Current JIRA-based process is not very efficient. Anyone who have used a
review tool will probably agree:

- No need to switch branches locally and interrupt your current work. You
can see the code in one click.
- All current reviews are easily accessible
- Multiple reviewers
- Much better discussions: comments are right in the code; each point can
be discussed and accepted separately
- Integrates with IDEA - open the diff in IDEA in one click, or see the
reviews there without opening the browser at all


* Why Upsource?
I've evaluated a bunch of tools (CodeCollaborator, ReviewBoard,
Phabricator, Crucible),
and Upsource looks like the best fit for us:
- PR-based code reviews. This is a major advantage: review for a PR can be
created in one click, and it updates automatically when you push more
commits (fix review issues)
- Good Java support and IDEA integration
- Good performance (our code base is big, and tools like Crucible really
struggle with it)


Thoughts and suggestions are welcome.

Thanks,

Pavel

[jira] [Created] (IGNITE-4218) Web console: wrong generation of collision SPIs with default values.

[Vasiliy Sisko (JIRA)]

Vasiliy Sisko created IGNITE-4218:
-

 Summary: Web console: wrong generation of collision SPIs with 
default values.
 Key: IGNITE-4218
 URL: https://issues.apache.org/jira/browse/IGNITE-4218
 Project: Ignite
  Issue Type: Bug
  Components: wizards
Affects Versions: 1.8
Reporter: Vasiliy Sisko
Assignee: Vasiliy Sisko
 Fix For: 1.8


Selected empty collision SPI bean is not generated in preview and on summary 
page.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: IgniteCache.loadCache improvement proposal

[Vladimir Ozerov]

Alexandr, Alexey,

While I agree with you that current cache loading logic is far from ideal,
it would be cool to see API drafts based on your suggestions to get better
understanding of your ideas. How exactly users are going to use your
suggestions?

My main concern is that initial load is not very trivial task in general
case. Some users have centralized RDBMS systems, some have NoSQL, others
work with distributed persistent stores (e.g. HDFS). Sometimes we have
Ignite nodes "near" persistent data, sometimes we don't. Sharding,
affinity, co-location, etc.. If we try to support all (or many) cases out
of the box, we may end up in very messy and difficult API. So we should
carefully balance between simplicity, usability and feature-rich
characteristics here.

Personally, I think that if user is not satisfied with "loadCache()" API,
he just writes simple closure with blackjack streamer and queries and send
it to whatever node he finds convenient. Not a big deal. Only very common
cases should be added to Ignite API.

Vladimir.


On Mon, Nov 14, 2016 at 12:43 PM, Alexey Kuznetsov 
wrote:

> Looks good for me.
>
> But I will suggest to consider one more use-case:
>
> If user knows its data he could manually split loading.
> For example: table Persons contains 10M rows.
> User could provide something like:
> cache.loadCache(null, "Person", "select * from Person where id <
> 1_000_000",
> "Person", "select * from Person where id >=  1_000_000 and id < 2_000_000",
> 
> "Person", "select * from Person where id >= 9_000_000 and id < 10_000_000",
> );
>
> or may be it could be some descriptor object like
>
>  {
>sql: select * from Person where id >=  ? and id < ?"
>range: 0...10_000_000
> }
>
> In this case provided queries will be send to mach nodes as number of
> queries.
> And data will be loaded in parallel and for keys that a not local - data
> streamer
> should be used (as described Alexandr description).
>
> I think it is a good issue for Ignite 2.0
>
> Vova, Val - what do you think?
>
>
> On Mon, Nov 14, 2016 at 4:01 PM, Alexandr Kuramshin 
> wrote:
>
>> All right,
>>
>> Let's assume a simple scenario. When the IgniteCache.loadCache is invoked,
>> we check whether the cache is not local, and if so, then we'll initiate
>> the
>> new loading logic.
>>
>> First, we take a "streamer" node, it could be done by
>> utilizing LoadBalancingSpi, or it may be configured statically, for the
>> reason that the streamer node is running on the same host as the
>> persistence storage provider.
>>
>> After that we start the loading task on the streamer node which
>> creates IgniteDataStreamer and loads the cache with CacheStore.loadCache.
>> Every call to IgniteBiInClosure.apply simply
>> invokes IgniteDataStreamer.addData.
>>
>> This implementation will completely relieve overhead on the persistence
>> storage provider. Network overhead is also decreased in the case of
>> partitioned caches. For two nodes we get 1-1/2 amount of data transferred
>> by the network (1 part well be transferred from the persistence storage to
>> the streamer, and then 1/2 from the streamer node to the another node).
>> For
>> three nodes it will be 1-2/3 and so on, up to the two times amount of data
>> on the big clusters.
>>
>> I'd like to propose some additional optimization at this place. If we have
>> the streamer node on the same machine as the persistence storage provider,
>> then we completely relieve the network overhead as well. It could be a
>> some
>> special daemon node for the cache loading assigned in the cache
>> configuration, or an ordinary sever node as well.
>>
>> Certainly this calculations have been done in assumption that we have even
>> partitioned cache with only primary nodes (without backups). In the case
>> of
>> one backup (the most frequent case I think), we get 2 amount of data
>> transferred by the network on two nodes, 2-1/3 on three, 2-1/2 on four,
>> and
>> so on up to the three times amount of data on the big clusters. Hence it's
>> still better than the current implementation. In the worst case with a
>> fully replicated cache we take N+1 amount of data transferred by the
>> network (where N is the number of nodes in the cluster). But it's not a
>> problem in small clusters, and a little overhead in big clusters. And we
>> still gain the persistence storage provider optimization.
>>
>> Now let's take more complex scenario. To achieve some level of
>> parallelism,
>> we could split our cluster on several groups. It could be a parameter of
>> the IgniteCache.loadCache method or a cache configuration option. The
>> number of groups could be a fixed value, or it could be calculated
>> dynamically by the maximum number of nodes in the group.
>>
>> After splitting the whole cluster on groups we will take the streamer node
>> in the each group and submit the task for loading the cache similar to the
>> single streamer scenario, except as the only keys will 

[jira] [Created] (IGNITE-4217) .NET: EventType.SwapspaceAll inconsistent naming

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-4217:
--

 Summary: .NET: EventType.SwapspaceAll inconsistent naming
 Key: IGNITE-4217
 URL: https://issues.apache.org/jira/browse/IGNITE-4217
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Reporter: Pavel Tupitsyn
 Fix For: 2.0


Should be SwapSpaceAll, similar to other events and SwapSpace classes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: IgniteCache.loadCache improvement proposal

[Alexey Kuznetsov]

Looks good for me.

But I will suggest to consider one more use-case:

If user knows its data he could manually split loading.
For example: table Persons contains 10M rows.
User could provide something like:
cache.loadCache(null, "Person", "select * from Person where id < 1_000_000",
"Person", "select * from Person where id >=  1_000_000 and id < 2_000_000",

"Person", "select * from Person where id >= 9_000_000 and id < 10_000_000",
);

or may be it could be some descriptor object like

 {
   sql: select * from Person where id >=  ? and id < ?"
   range: 0...10_000_000
}

In this case provided queries will be send to mach nodes as number of
queries.
And data will be loaded in parallel and for keys that a not local - data
streamer
should be used (as described Alexandr description).

I think it is a good issue for Ignite 2.0

Vova, Val - what do you think?


On Mon, Nov 14, 2016 at 4:01 PM, Alexandr Kuramshin 
wrote:

> All right,
>
> Let's assume a simple scenario. When the IgniteCache.loadCache is invoked,
> we check whether the cache is not local, and if so, then we'll initiate the
> new loading logic.
>
> First, we take a "streamer" node, it could be done by
> utilizing LoadBalancingSpi, or it may be configured statically, for the
> reason that the streamer node is running on the same host as the
> persistence storage provider.
>
> After that we start the loading task on the streamer node which
> creates IgniteDataStreamer and loads the cache with CacheStore.loadCache.
> Every call to IgniteBiInClosure.apply simply
> invokes IgniteDataStreamer.addData.
>
> This implementation will completely relieve overhead on the persistence
> storage provider. Network overhead is also decreased in the case of
> partitioned caches. For two nodes we get 1-1/2 amount of data transferred
> by the network (1 part well be transferred from the persistence storage to
> the streamer, and then 1/2 from the streamer node to the another node). For
> three nodes it will be 1-2/3 and so on, up to the two times amount of data
> on the big clusters.
>
> I'd like to propose some additional optimization at this place. If we have
> the streamer node on the same machine as the persistence storage provider,
> then we completely relieve the network overhead as well. It could be a some
> special daemon node for the cache loading assigned in the cache
> configuration, or an ordinary sever node as well.
>
> Certainly this calculations have been done in assumption that we have even
> partitioned cache with only primary nodes (without backups). In the case of
> one backup (the most frequent case I think), we get 2 amount of data
> transferred by the network on two nodes, 2-1/3 on three, 2-1/2 on four, and
> so on up to the three times amount of data on the big clusters. Hence it's
> still better than the current implementation. In the worst case with a
> fully replicated cache we take N+1 amount of data transferred by the
> network (where N is the number of nodes in the cluster). But it's not a
> problem in small clusters, and a little overhead in big clusters. And we
> still gain the persistence storage provider optimization.
>
> Now let's take more complex scenario. To achieve some level of parallelism,
> we could split our cluster on several groups. It could be a parameter of
> the IgniteCache.loadCache method or a cache configuration option. The
> number of groups could be a fixed value, or it could be calculated
> dynamically by the maximum number of nodes in the group.
>
> After splitting the whole cluster on groups we will take the streamer node
> in the each group and submit the task for loading the cache similar to the
> single streamer scenario, except as the only keys will be passed to
> the IgniteDataStreamer.addData method those correspond to the cluster group
> where is the streamer node running.
>
> In this case we get equal level of overhead as the parallelism, but not so
> surplus as how many nodes in whole the cluster.
>
> 2016-11-11 15:37 GMT+03:00 Alexey Kuznetsov :
>
> > Alexandr,
> >
> > Could you describe your proposal in more details?
> > Especially in case with several nodes.
> >
> > On Fri, Nov 11, 2016 at 6:34 PM, Alexandr Kuramshin <
> ein.nsk...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > You know CacheStore API that is commonly used for read/write-through
> > > relationship of the in-memory data with the persistence storage.
> > >
> > > There is also IgniteCache.loadCache method for hot-loading the cache on
> > > startup. Invocation of this method causes execution of
> > CacheStore.loadCache
> > > on the all nodes storing the cache partitions. Because of none keys are
> > > passed to the CacheStore.loadCache methods, the underlying
> implementation
> > > is forced to read all the data from the persistence storage, but only
> > part
> > > of the data will be stored on each node.
> > >
> > > So, the current implementation have two general drawbacks:
> > >
> > > 

[GitHub] ignite pull request #1195: IGNITE-4117 .NET: Fix ClientReconnectTask complet...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1195


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1193: Ignite 1.7.4

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1193


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1200: IGNITE-4118 .NET: Added OptimisticTransactionExam...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/1200


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: IgniteCache.loadCache improvement proposal

[Alexandr Kuramshin]

All right,

Let's assume a simple scenario. When the IgniteCache.loadCache is invoked,
we check whether the cache is not local, and if so, then we'll initiate the
new loading logic.

First, we take a "streamer" node, it could be done by
utilizing LoadBalancingSpi, or it may be configured statically, for the
reason that the streamer node is running on the same host as the
persistence storage provider.

After that we start the loading task on the streamer node which
creates IgniteDataStreamer and loads the cache with CacheStore.loadCache.
Every call to IgniteBiInClosure.apply simply
invokes IgniteDataStreamer.addData.

This implementation will completely relieve overhead on the persistence
storage provider. Network overhead is also decreased in the case of
partitioned caches. For two nodes we get 1-1/2 amount of data transferred
by the network (1 part well be transferred from the persistence storage to
the streamer, and then 1/2 from the streamer node to the another node). For
three nodes it will be 1-2/3 and so on, up to the two times amount of data
on the big clusters.

I'd like to propose some additional optimization at this place. If we have
the streamer node on the same machine as the persistence storage provider,
then we completely relieve the network overhead as well. It could be a some
special daemon node for the cache loading assigned in the cache
configuration, or an ordinary sever node as well.

Certainly this calculations have been done in assumption that we have even
partitioned cache with only primary nodes (without backups). In the case of
one backup (the most frequent case I think), we get 2 amount of data
transferred by the network on two nodes, 2-1/3 on three, 2-1/2 on four, and
so on up to the three times amount of data on the big clusters. Hence it's
still better than the current implementation. In the worst case with a
fully replicated cache we take N+1 amount of data transferred by the
network (where N is the number of nodes in the cluster). But it's not a
problem in small clusters, and a little overhead in big clusters. And we
still gain the persistence storage provider optimization.

Now let's take more complex scenario. To achieve some level of parallelism,
we could split our cluster on several groups. It could be a parameter of
the IgniteCache.loadCache method or a cache configuration option. The
number of groups could be a fixed value, or it could be calculated
dynamically by the maximum number of nodes in the group.

After splitting the whole cluster on groups we will take the streamer node
in the each group and submit the task for loading the cache similar to the
single streamer scenario, except as the only keys will be passed to
the IgniteDataStreamer.addData method those correspond to the cluster group
where is the streamer node running.

In this case we get equal level of overhead as the parallelism, but not so
surplus as how many nodes in whole the cluster.

2016-11-11 15:37 GMT+03:00 Alexey Kuznetsov :

> Alexandr,
>
> Could you describe your proposal in more details?
> Especially in case with several nodes.
>
> On Fri, Nov 11, 2016 at 6:34 PM, Alexandr Kuramshin 
> wrote:
>
> > Hi,
> >
> > You know CacheStore API that is commonly used for read/write-through
> > relationship of the in-memory data with the persistence storage.
> >
> > There is also IgniteCache.loadCache method for hot-loading the cache on
> > startup. Invocation of this method causes execution of
> CacheStore.loadCache
> > on the all nodes storing the cache partitions. Because of none keys are
> > passed to the CacheStore.loadCache methods, the underlying implementation
> > is forced to read all the data from the persistence storage, but only
> part
> > of the data will be stored on each node.
> >
> > So, the current implementation have two general drawbacks:
> >
> > 1. Persistence storage is forced to perform as many identical queries as
> > many nodes on the cluster. Each query may involve much additional
> > computation on the persistence storage server.
> >
> > 2. Network is forced to transfer much more data, so obviously the big
> > disadvantage on large systems.
> >
> > The partition-aware data loading approach, described in
> > https://apacheignite.readme.io/docs/data-loading#section-
> > partition-aware-data-loading
> > , is not a choice. It requires persistence of the volatile data depended
> on
> > affinity function implementation and settings.
> >
> > I propose using something like IgniteDataStreamer inside
> > IgniteCache.loadCache implementation.
> >
> >
> > --
> > Thanks,
> > Alexandr Kuramshin
> >
>
>
>
> --
> Alexey Kuznetsov
>



-- 
Thanks,
Alexandr Kuramshin

[jira] [Created] (IGNITE-4216) .NET: PlatformAffinityFunction does not inject resource into baseFunc

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-4216:
--

 Summary: .NET: PlatformAffinityFunction does not inject resource 
into baseFunc
 Key: IGNITE-4216
 URL: https://issues.apache.org/jira/browse/IGNITE-4216
 Project: Ignite
  Issue Type: Bug
  Components: platforms
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
 Fix For: 1.8


PlatformAffinityFunction does not inject resource into baseFunc. This causes 
NPE within RendezvousAffinityFunction:

* start multiple nodes
* start a new cache with RendezvousAffinityFunction



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)