[GitHub] ignite pull request #1395: ignite-4496

[akuramshingg]

GitHub user akuramshingg opened a pull request:

https://github.com/apache/ignite/pull/1395

ignite-4496

Review all logging for sensitive data leak

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4496

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1395.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1395


commit 1c82cd04fadc61ab5da02e1d395aedfb28a039da
Author: Pavel Tupitsyn 
Date:   2016-08-23T10:28:40Z

IGNITE-3279 .NET: NLog logger

commit 48b293db4d443b4d8739f709ff12f19aad008b84
Author: Pavel Tupitsyn 
Date:   2016-10-03T11:08:07Z

IGNITE-3279 .NET: NLog logger

commit bba019fd5076412ca43c10a32fd300b6031ccd0b
Author: vozerov-gridgain 
Date:   2016-10-03T14:25:20Z

IGNITE-3980: Processed failing tests in query suites. This closes #1137.

commit 02f48d72364fb0c52e95aef8ed383a14ee531bf6
Author: ptupitsyn 
Date:   2016-10-04T08:17:54Z

IGNITE-3820: .NET: Added log4net integration. This closes #1138.

commit bfdb5c3b374fd3512481cf16779d227d7f96e569
Author: Saikat Maitra 
Date:   2016-10-04T09:40:35Z

IGNITE-3841 Web console added check for eviction policy max mem and max 
size consistency. Fixes #1136.

commit a92f20b5cc75e6b80b2731da0192723526b0c1dc
Author: vozerov-gridgain 
Date:   2016-10-04T11:10:26Z

IGNITE-3597: Removed static work directory.

commit d45383b69cc68c0ec967ebd673b197e437720214
Author: Pavel Tupitsyn 
Date:   2016-10-04T15:48:25Z

.NET: Fix code analysis warnings

commit 23461b8d33922772ef8e7217e9e87b3f3b0b37b1
Author: vozerov-gridgain 
Date:   2016-10-06T07:14:59Z

IGNITE-4001: Timeouts for threads in Ignite pools. This closes #1130.

commit b94b0aeae4c42b1d35128c6b1de97e3fa318d497
Author: tledkov-gridgain 
Date:   2016-10-06T07:22:50Z

IGNITE-3163 IGFS: Added working directory support to 
IgniteHadoopIgfsSecondaryFileSystem. This closes #1030. This closes #1058. This 
closes #1132.

commit e6317e01fa8a0de03e15dcdd84a575c6b06ce701
Author: vozerov-gridgain 
Date:   2016-10-06T09:03:48Z

IGNITE-3593 .NET: IgniteConfiguration.WorkDirectory has no effect. This 
closes #903. This closes #1145.

commit 952be8b995050b34379006dd6e739da3fe3b49e3
Author: Dmitriy Govorukhin 
Date:   2016-10-07T12:00:09Z

Squashed commit of the following:

commit 566881b695b8bc00e618fe9a9b4c86a8fd563cc1
Author: sboikov 
Date:   Fri Oct 7 13:08:38 2016 +0300

minor

commit 7fe88a1cb21f794ee55a176ab36d895cbf916528
Author: Dmitriy Govorukhin 
Date:   Thu Oct 6 11:11:24 2016 +0300

ignite-update-notifier fix after review

(cherry picked from commit a10d2ff)

commit f2de749f958a3b18dc479f8a5517d7bf9362b933
Author: Dmitriy Govorukhin 
Date:   Tue Oct 4 12:12:08 2016 +0300

ignite-2079-2 optimize import and change url path

(cherry picked from commit 830a3cf)

commit 0d1be85ad55b0aa91224690d6c112ae92e8bc0a9
Author: Dmitriy Govorukhin 
Date:   Thu Sep 29 19:54:54 2016 +0300

update-notifier remove parse xml, now parse like properties

(cherry picked from commit 9ecaa29)

commit e43bca6fb4528a7fc0dcb804a74fca1c59d7468b
Author: Dmitriy Govorukhin 
Date:   Tue Sep 27 17:07:21 2016 +0300

remove dom parser

(cherry picked from commit d1653b2)

commit b9c776a8423471706ecb1dc6176b38f23e799077
Author: Anton Vinogradov 
Date:   2016-10-10T08:52:57Z

IGNITE-3235 Failed to initialize primitive boolean cache property of 
superclass

commit f9a0676fad7fd6c23e3c91c10d7e0412ccb27c06
Author: vozerov-gridgain 
Date:   2016-10-11T07:23:01Z

IGNITE-4041: Created separate processor for thread pools and refactored IO 
manager. This closes #1150.

commit 9a6cfce659df40b0a4624f19fd91c217b74bafea
Author: nikolay_tikhonov 
Date:   2016-10-11T10:59:57Z

IGNITE-4014 Fixed "Transaction hangs if entry processor failed during 
serialization". This closes #1148.

commit 1938a17b01fac1e08c30011180bbcc3ed7374d83
Author: Andrey V. Mashenkov 
Date:   2016-10-11T11:50:18Z

IGNITE-3948: Fixed a bug causing TTL manager to continue tracking of 
evicted entries. This closes #1101.

commit 60d27547dfc6bd27c6d39cbcc523d0d1e872a821
Author: vozerov-gridgain 

Re: Google Summer Of Code 2017

[Denis Magda]

Igor R., is there anything that can be improved & refined in the existed 
integration of Ignite and Cassandra and outsourced to Google Summer of Code? 

—
Denis

> On Dec 22, 2016, at 10:29 AM, Denis Magda  wrote:
> 
> This is great!
> 
> So, we already have three mentors in total (Val, Dmitriy, Denis). Anyone else?
> 
> Any other tasks you think are suitable for this summer program?
> 
> —
> Denis
> 
>> On Dec 21, 2016, at 4:26 PM, Valentin Kulichenko 
>>  wrote:
>> 
>> I can take over DataFrames integration as I already did some investigation
>> in this area. Spark also has GraphX and GraphFrames for graph processing,
>> we can think about integration with them as well.
>> 
>> Machine learning also sounds very interesting.
>> 
>> -Val
>> 
>> On Wed, Dec 21, 2016 at 4:22 PM, Dmitriy Setrakyan 
>> wrote:
>> 
>>> I can also do mentoring for any of the listed features.
>>> 
>>> On Wed, Dec 21, 2016 at 2:50 PM, Denis Magda  wrote:
>>> 
 Alex, thanks for bringing this out!
 
 This is a great opportunity for our community to find talented students
 who are capable of adding new features to Ignite. I can take care of all
 the paperwork. The features list won’t be a big deal as well. But we need
 mentors who will voluntarily dedicate their time helping the
>>> participants.
 
 Well, here is a list of features that pop up immediately out of my head
 and that I would like to offer to GSoC (Google Summer Of Code).
 Native client for Python (data grid, compute grid, continuous queries).
 Native client for Node.JS (https://issues.apache.org/
 jira/browse/IGNITE-961 ).
 Spark Data Frames API implementation.
 Machine Learning Grid implementation or integration with Spark MLib.
 
 Any other features?
 
 As for the mentoring I can take over one or two of the following -
 Kubernetis, Machine Learning or Node.JS.
 
 Is there anyone else who can try a role of a mentor?
 
 —
 Denis
 
> On Dec 19, 2016, at 8:15 PM, Alexey Kuznetsov 
 wrote:
> 
> Igniters!
> 
> What do you think about participating in Google Summer Of Code 2017?
> See: https://developers.google.com/open-source/gsoc
> 
> The Google Summer of Code, often abbreviated to GSoC, is an
>>> international
> annual program, first held from May to August 2005, in which Google
 awards
> stipends (of US$5,500, as of 2016 ) to all students who successfully
> complete a requested free and open-source software coding project
>>> during
> the summer.
> 
> Students will work on some issues in Ignite.
> Ignite community will provide mentors for those students and answer
> students questions on dev list.
> Google will pay students. :)
> 
> We need to prepare list of tasks that will be useful for Ignite and
>>> could
> be finished in three months by students.
> 
> What do you think?
> --
> Alexey Kuznetsov
 
 
>>> 
> 

Re: [ANNOUNCE] Apache Ignite 1.8.0 Released

[Denis Magda]

Sally, looks like you got confused with this

> On Dec 29, 2016, at 2:38 PM, Sally Khudairi  wrote:
> 
> And thanks in advance, Roman Shtykh, for acting as project spokesperson in 
> your role as release manager. Participation from the community at-large is 
> greatly appreciated!

Roman have never been a release manager yet. However, this is just a matter of 
time and his interest/intention ;)

—
Denis

Re: [ANNOUNCE] Apache Ignite 1.8.0 Released

[Sally Khudairi]

Thanks, Denis. Hello, everyone!
Indeed, I think releasing a "highlight" announcement makes sense here, and I'll 
be happy to work with you over the next week or so to get that drafted.
And thanks in advance, Roman Shtykh, for acting as project spokesperson in your 
role as release manager. Participation from the community at-large is greatly 
appreciated!
Here's hoping everyone has a happy New Year!
Warmly,
Sally = = = = =  vox +1 617 921 8656gvox +1 646 598 4616skype sallykhudairi

  From: Denis Magda 
 To: "s...@apache.org"  
Cc: ASF Marketing & Publicity ; dev@ignite.apache.org; Roman 
Shtykh 
 Sent: Thursday, December 29, 2016 4:24 PM
 Subject: Re: [ANNOUNCE] Apache Ignite 1.8.0 Released
   
CC-ing Apache Ignite dev@ in the loop.
Igniters, PMC members had a talk with Sally Khudairi who represents ASF 
Marketing & Publicity group. Sally, thanks a lot for finding a time to jump on 
the call. 
One of the main topics of the call was our latest 1.8.0 release. It was agreed 
that Sally would help the community to prepare an additional announcement so 
that we can expand awareness of the following features:
   
   - In-Memory SQL Grid. Sally, hope that I could convey the main benefit of 
this feature to you. Just in case, here is my recent blog post [1] where it’s 
explained how PHP users can leverage from this new component.

   
   - Redis protocol support. Roman S., could you find a time and write a blog 
post about it or recount the main benefits that Sally can include in the 
announcement referring to you?

   
   - .NET features. Already well described in this post [2]. Just in case, 
Pavel T. will be able to provide more details if needed.


Also we discussed marketing related activities, issues and approaches related 
to Apache Ignite in general. At the moment community members maintain the 
following sources that increase awareness of Ignite:   
   - Blogs: https://ignite.apache.org/blogs.html
   - News: https://ignite.apache.org/news.html
   - Twitter: https://twitter.com/ApacheIgnite

I will deep dive into this topic the next year relying on you, Igniters, and 
Sally trying to find a way to get all we can from existed channels as well as 
implementing other approaches.

[1] 
https://dzone.com/articles/apache-ignite-enables-full-fledged-sql-support-for[2]
 https://dzone.com/articles/whats-new-in-apache-ignitenet-18
Regards,Denis

On Dec 28, 2016, at 3:25 PM, Sally Khudairi  wrote:
Excellent --thank you, Denis!


[From the mobile; please excuse top-posting, spelling/spacing errors, and 
brevity]


 
 
 On Wed, Dec 28, 2016 at 17:24, Denis Magda wrote:  
Absolutely,
Anyone from Apache Ignite PMC is welcomed to join the call. Here are the 
details. Please book the call in your calendar.
1.  Please join my meeting, Dec 29, 2016 at 11:30 AM PST.
https://global.gotomeeting.com/join/537803613

2.  Use your microphone and speakers (VoIP) - a headset is recommended. Or, 
call in using your telephone.

Dial +1 (872) 240-3311
Access Code: 537-803-613
Audio PIN: Shown after joining the meeting

Meeting ID: 537-803-613

GoToMeeting®
Online Meetings Made Easy®

Not at your computer? Click the link to join this meeting from your iPhone®, 
iPad®, Android® or Windows Phone® device via the GoToMeeting app.
—Denis

On Dec 28, 2016, at 2:00 PM, Sally Khudairi  wrote:
Thanks, Denis.
Yes --that works for me. I appreciate your following up on this.
Should any non-GridGainers/additional PMCers wish to join the call, I presume 
that would be possible, yes?
Until then,Sally = = = = =  vox +1 617 921 8656gvox +1 646 598 4616skype 
sallykhudairi

  From: Denis Magda 
 To: Nikita Ivanov  
Cc: priv...@ignite.apache.org; Sally Khudairi ; ASF Marketing 
& Publicity 
 Sent: Wednesday, December 28, 2016 4:57 PM
 Subject: Re: [ANNOUNCE] Apache Ignite 1.8.0 Released
  
Sally,
I will schedule 30 minutes screen sharing call via GoToMeeting tool for 11.30 
AM PST so that both Dmitriy and Nikita can attend.
Please confirm that the time works for you.
—Denis

On Dec 28, 2016, at 1:53 PM, Nikita Ivanov  wrote:
Tomorrow's morning works for me too. 
Thanks!
--Nikita Ivanov

On Wed, Dec 28, 2016 at 12:56 PM, Sally Khudairi  wrote:

Thanks, Denis.
I'll be happy to chat with you tomorrow after 10AM PT, if that works for you.
Ideally there will be other members of the PMC on the call as well, so we can 
discuss possible ways to help the project with outreach strategy and tactics.
Just so you know, I've worked with the GridGain PR/Comms team (Brigit, 
Samantha, Elyce) on Ignite through the incubation process, as well as with 
GridGain announcements (to ensure that they are in alignment with ASF 
guidelines), so it'll be a good idea to loop them in at some point if they'll 
be providing a supporting role 

Re: About RoundRobinLoadBalancingSpi

[Valentin Kulichenko]

Fixed.

On Thu, Dec 29, 2016 at 6:08 AM, 李玉珏@163 <18624049...@163.com> wrote:

> Val，
>
> Javadoc also need to be modified.
>
>
>
> 在 2016/12/29 00:08, Valentin Kulichenko 写道:
>
>> Hi,
>>
>> Thanks for pointing this out. I fixed the documentation.
>>
>> -Val
>>
>> On Tue, Dec 27, 2016 at 7:50 PM, 李玉珏  wrote:
>>
>> Hi，
>>>
>>>
>>> In RoundRobinLoadBalancingSpi，according to the description of the
>>> document,per-task mode is the default configuration and should fit most
>>> of
>>> the use cases as it provides a fairly well-distributed split and also
>>> ensures that jobs within a single task are spread out across nodes to the
>>> maximum.
>>>
>>>
>>> But from the source code, global mode is the default value.
>>> I want to ask,does this belong to the document description issue, or is a
>>> bug?
>>>
>>
>
>

Re: Error in JDBC store

[Valentin Kulichenko]

Hi Dmitry,

My opinion is that this is not a valid case and we should throw an
exception on cache startup if two Java fields are mapped to the same DB
field. Even if user needs such duplication on objects level (which I also
doubt, BTW), the mapping in the store must be correct.

-Val

On Thu, Dec 29, 2016 at 12:23 AM, Dmitriy Karachentsev <
dkarachent...@gridgain.com> wrote:

> Hi all!
>
> According to this thread [1] in JDBC store configuration:
> 1. Map key and value fields to the same columns in DB.
> 2. Try to update data.
> 3. Got invalid SQL.
>
> Let's see an pseudocode example of described use case.
> KeyClass { field1; field2; field3 }, ValClass { field1; field2; field3 }
>
> Map fields to DB table TABLE_NAME columns: field1 -> col1, field2 -> col2,
> field3 -> col3.
>
> User expects the following update request built by Ignite:
> UPDATE TABLE_NAME SET col1=?, col2=?, col3=? WHERE (col1=?, col2=?,
> col3=?);
>
> But Ignite checks that value object fields have the same mappings, throws
> them away and builds wrong query:
> UPDATE TABLE_NAME SET WHERE (col1=?, col2=?, col3=?);
> That is obviously wrong.
>
> Is there any reason to do so?
> Probably, it is better to build query according to user mapping and
> delegate verification to DB.
>
> [1]
> http://apache-ignite-users.70518.x6.nabble.com/Error-
> while-writethrough-operation-in-Ignite-td9696.html
>
> Thanks!
> Dmitry.
>

Re: PageMemory approach for Ignite 2.0

[Dmitriy Setrakyan]

Denis, this is not "dreaming". We can't have a release with less features than 
1.8. We can only add features, not subtract.

Dmitriy



> On Dec 29, 2016, at 9:49 AM, Denis Magda  wrote:
> 
> Alex,
> 
>> On Dec 29, 2016, at 1:37 AM, Alexey Goncharuk  
>> wrote:
>> 
>> The subject of this discussion is to determine whether the PageMemory
>> approach is a way to go, because the this implementation is almost 2x
>> slower than current 2.0 branch. 
> 
> What is the main reason for that? Some architecture flaw you didn’t consider 
> before or performance issues that have to be cleaned up?
> 
> —
> Denis

Re: PageMemory approach for Ignite 2.0

[Denis Magda]

Alex,

> On Dec 29, 2016, at 1:37 AM, Alexey Goncharuk  
> wrote:
> 
> The subject of this discussion is to determine whether the PageMemory
> approach is a way to go, because the this implementation is almost 2x
> slower than current 2.0 branch. 

What is the main reason for that? Some architecture flaw you didn’t consider 
before or performance issues that have to be cleaned up?

—
Denis

Re: IGNITE-4487 - NPE on query execution

[Denis Magda]

Alexander, thanks.

Please move the ticket from “open” into “patch available” state in JIRA and run 
the tests on TeamCity. Refer to the details covered there
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-1.CreateGitHubpull-request
 


—
Denis

> On Dec 29, 2016, at 3:45 AM, Александр Меньшиков  wrote:
> 
> Alexey, I'm already make pull request where throw exception in that place.
> 
> https://github.com/apache/ignite/pull/1388/commits
> 
> 2016-12-29 11:16 GMT+03:00 Alexey Goncharuk :
> 
>> I think that If fallbacks(...) returns an empty nodes collection, then we
>> should fail with an exception.
>> 
>> 2016-12-28 22:06 GMT+03:00 Denis Magda :
>> 
>>> Alexander, added you to the contributors list. Please check that you can
>>> assign the ticket on yourself.
>>> 
>>> —
>>> Denis
>>> 
 On Dec 28, 2016, at 2:15 AM, Александр Меньшиков >> 
>>> wrote:
 
 
 Username: sharpler
 
 Full Name: Alexander Menshikov
 
 
 
 2016-12-27 22:57 GMT+03:00 Denis Magda > dma...@apache.org>>:
 Alexander,
 
 I need to know your JIRA ID in order to add you to the contributors
>> list.
 
 As for your questions, this situation might be caused by the race when
>> a
>>> cache is being stopped and there are still scan queries running in
>>> parallel. So, in general it’s not about data loss.
 
 Sam, Alex G., could you share your thoughts in regards to the proper
>> fix?
 
 —
 Denis
 
> On Dec 26, 2016, at 2:43 AM, Александр Меньшиков <
>> sharple...@gmail.com
>>> > wrote:
> 
> Hello everyone.
> 
> I want to pick up *https://issues.apache.org/jira/browse/IGNITE-4487
>> <
>>> https://issues.apache.org/jira/browse/IGNITE-4487>
> >> https://issues.apache.org/jira/browse/IGNITE-4487>>* as my
> first issue.
> 
> Please add me as contributor.
> 
> I already found that: in inner class
> 'GridCacheQueryAdapter.ScanQueryFallbackClosableIterator' in
>>> constructor is
> called with method 'init()', but method 'init()' cannot be called
>> with
>>> an
> empty field 'nodes'. In source code it looks like:
> 
> private ScanQueryFallbackClosableIterator(int part,
>>> GridCacheQueryAdapter
> qry,
>   GridCacheQueryManager qryMgr, GridCacheContext cctx) {
>   this.qry = qry;
>   this.qryMgr = qryMgr;
>   this.cctx = cctx;
>   this.part = part;
> 
>   nodes = fallbacks(cctx.discovery().topologyVersionEx());
>   // !!! Here nodes.isEmpty()==true, and init() will fail in
>>> the
> future. !!!
>   init();
>   }
> 
> I can fix it by adding some check in code, but i must know what
>>> behavior
> are best in this case? As I understand it, the list of nodes is empty
>>> if
> there are no nodes with the current partition, which means data loss,
>>> and
> either need to return a meaningful exception, or ignore this
>>> situation. But
> maybe I missed something.
 
 
>>> 
>>> 
>> 

Re: About RoundRobinLoadBalancingSpi

[李玉珏]

Val，

Javadoc also need to be modified.


在 2016/12/29 00:08, Valentin Kulichenko 写道:

Hi,

Thanks for pointing this out. I fixed the documentation.

-Val

On Tue, Dec 27, 2016 at 7:50 PM, 李玉珏  wrote:


Hi，


In RoundRobinLoadBalancingSpi，according to the description of the
document,per-task mode is the default configuration and should fit most of
the use cases as it provides a fairly well-distributed split and also
ensures that jobs within a single task are spread out across nodes to the
maximum.


But from the source code, global mode is the default value.
I want to ask,does this belong to the document description issue, or is a
bug?

Re: Sort nodes in the ring in order to minimize the number of reconnections

[Александр Меньшиков]

I think that in the weeks after the 'new year' holidays or sooner.

2016-12-29 13:28 GMT+03:00 Yakov Zhdanov :

> Guys, I have updated the ticket.
>
> Alexander Menshikov, when do you expect the implementation to be finished?
>

[jira] [Created] (IGNITE-4512) Doesnt work sql-delete benchmark

[Ilya Suntsov (JIRA)]

Ilya Suntsov created IGNITE-4512:


 Summary: Doesnt work sql-delete benchmark
 Key: IGNITE-4512
 URL: https://issues.apache.org/jira/browse/IGNITE-4512
 Project: Ignite
  Issue Type: Bug
  Components: general
Affects Versions: 1.8
 Environment: Benchmark class: IgniteSqlDeleteBenchmark
Yardsitck configuration: 1 client, 4 servers, 64 threads, 1 backup, primary sync
Reporter: Ilya Suntsov
 Fix For: 2.0


I got the following exception when tried to run sql-delete benchmark:
{noformat}
<13:18:49> DELETE setUp: have successfully put 0 items
<13:18:50> DELETE setUp: have successfully put 10 
items
<13:18:51> DELETE setUp: have successfully put 20 
items
<13:18:52> Probe writer is not configured (using default CSV 
writer)
<13:18:52> ThroughputLatencyProbe is started.
<13:18:52> DStatProbe is started. Command: 'dstat -m --all 
--noheaders --noupdate 1'
<13:18:52> PercentileProbe is started.
[13:18:53,373][INFO ][grid-timeout-worker-#23%null%][IgniteKernal].
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=84f60fbc, name=null, uptime=00:00:05:000]
^-- H/N/C [hosts=5, nodes=5, CPUs=80]
^-- CPU [cur=0%, avg=0%, GC=0%]
^-- Heap [used=56MB, free=97.26%, comm=2047MB]
^-- Non heap [used=56MB, free=-1%, comm=57MB]
^-- Public thread pool [active=0, idle=6, qSize=0]
^-- System thread pool [active=0, idle=16, qSize=0]
^-- Outbound messages queue [size=7]
Finishing main test [ts=1483006734702, date=Thu Dec 29 13:18:54 MSK 2016]
ERROR: Shutting down benchmark driver to unexpected exception.
Type '--help' for usage.
java.util.NoSuchElementException
<-->at java.util.AbstractQueue.remove(AbstractQueue.java:117)
<-->at 
org.apache.ignite.yardstick.cache.dml.IgniteSqlDeleteBenchmark.test(IgniteSqlDeleteBenchmark.java:73)
<-->at 
org.yardstickframework.impl.BenchmarkRunner$2.run(BenchmarkRunner.java:176)
<-->at java.lang.Thread.run(Thread.java:745)
[13:18:54,740][INFO ][Thread-72][GridCacheProcesso
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: IGNITE-4487 - NPE on query execution

[Александр Меньшиков]

Alexey, I'm already make pull request where throw exception in that place.

https://github.com/apache/ignite/pull/1388/commits

2016-12-29 11:16 GMT+03:00 Alexey Goncharuk :

> I think that If fallbacks(...) returns an empty nodes collection, then we
> should fail with an exception.
>
> 2016-12-28 22:06 GMT+03:00 Denis Magda :
>
> > Alexander, added you to the contributors list. Please check that you can
> > assign the ticket on yourself.
> >
> > —
> > Denis
> >
> > > On Dec 28, 2016, at 2:15 AM, Александр Меньшиков  >
> > wrote:
> > >
> > >
> > > Username: sharpler
> > >
> > > Full Name: Alexander Menshikov
> > >
> > >
> > >
> > > 2016-12-27 22:57 GMT+03:00 Denis Magda  dma...@apache.org>>:
> > > Alexander,
> > >
> > > I need to know your JIRA ID in order to add you to the contributors
> list.
> > >
> > > As for your questions, this situation might be caused by the race when
> a
> > cache is being stopped and there are still scan queries running in
> > parallel. So, in general it’s not about data loss.
> > >
> > > Sam, Alex G., could you share your thoughts in regards to the proper
> fix?
> > >
> > > —
> > > Denis
> > >
> > > > On Dec 26, 2016, at 2:43 AM, Александр Меньшиков <
> sharple...@gmail.com
> > > wrote:
> > > >
> > > > Hello everyone.
> > > >
> > > > I want to pick up *https://issues.apache.org/jira/browse/IGNITE-4487
> <
> > https://issues.apache.org/jira/browse/IGNITE-4487>
> > > >  > https://issues.apache.org/jira/browse/IGNITE-4487>>* as my
> > > > first issue.
> > > >
> > > > Please add me as contributor.
> > > >
> > > > I already found that: in inner class
> > > > 'GridCacheQueryAdapter.ScanQueryFallbackClosableIterator' in
> > constructor is
> > > > called with method 'init()', but method 'init()' cannot be called
> with
> > an
> > > > empty field 'nodes'. In source code it looks like:
> > > >
> > > > private ScanQueryFallbackClosableIterator(int part,
> > GridCacheQueryAdapter
> > > > qry,
> > > >GridCacheQueryManager qryMgr, GridCacheContext cctx) {
> > > >this.qry = qry;
> > > >this.qryMgr = qryMgr;
> > > >this.cctx = cctx;
> > > >this.part = part;
> > > >
> > > >nodes = fallbacks(cctx.discovery().topologyVersionEx());
> > > >// !!! Here nodes.isEmpty()==true, and init() will fail in
> > the
> > > > future. !!!
> > > >init();
> > > >}
> > > >
> > > > I can fix it by adding some check in code, but i must know what
> > behavior
> > > > are best in this case? As I understand it, the list of nodes is empty
> > if
> > > > there are no nodes with the current partition, which means data loss,
> > and
> > > > either need to return a meaningful exception, or ignore this
> > situation. But
> > > > maybe I missed something.
> > >
> > >
> >
> >
>

[jira] [Created] (IGNITE-4511) Set QueryIndexType.SORTED by default for an index

[Steve Hostettler (JIRA)]

Steve Hostettler created IGNITE-4511:


 Summary: Set QueryIndexType.SORTED by default for an index 
 Key: IGNITE-4511
 URL: https://issues.apache.org/jira/browse/IGNITE-4511
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 1.7
 Environment: Windows Server + Java 8
Reporter: Steve Hostettler
Priority: Minor


The code snippet below creates a index of type FULL TEXT which much slower and 
that I do not need;
QueryIndex idx1 = new QueryIndex();
LinkedHashMap idxFlds1 = new LinkedHashMap<>();
idxFlds1.put("field1", true);
idxFlds1.put("field2", true);
idx1.setFields(idxFlds1);
idxs.add(idx1);

To avoid this I must explicitly set
idx1.setIndexType(QueryIndexType.SORTED);

This is because with the above code snippet, the Index Type is null and that 
null is interpreted as FULL TEXT in  GridQueryProcessor.java 
 if (idx.getIndexType() == QueryIndexType.SORTED || idx.getIndexType() == 
QueryIndexType.GEOSPATIAL) { 
 
} else { 
assert idx.getIndexType() == QueryIndexType.FULLTEXT; 

for (String field : idx.getFields().keySet()) { 
String alias = aliases.get(field); 

if (alias != null) 
field = alias; 

d.addFieldToTextIndex(field); 
} 
} 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Sort nodes in the ring in order to minimize the number of reconnections

[Yakov Zhdanov]

Guys, I have updated the ticket.

Alexander Menshikov, when do you expect the implementation to be finished?

Re: Sort nodes in the ring in order to minimize the number of reconnections

[Yakov Zhdanov]

I am OK with CLUSTER_REGION_ID. However I would like this name -
DISCOVERY_REGION_ID - more.

>>
Do you think we will ever care about the order of nodes within the same
region, e.g. does the order of nodes within the same rack matter?
>>

I think this is too much, but if you care you still can. Imagine you have
1000..1010 values for machines in rack 1 and 2000..2010 values for machines
in rack 2. So you can set exact position for each machine. This approach
provides full flexibility.

I will update https://issues.apache.org/jira/browse/IGNITE-4501 shortly.

--Yakov

[jira] [Created] (IGNITE-4510) SQL: co-located query may calculate target partition in advance in some cases

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-4510:
---

 Summary: SQL: co-located query may calculate target partition in 
advance in some cases
 Key: IGNITE-4510
 URL: https://issues.apache.org/jira/browse/IGNITE-4510
 Project: Ignite
  Issue Type: Task
  Components: SQL
Affects Versions: 1.8
Reporter: Vladimir Ozerov
 Fix For: 2.0


Consider the following SQL query with {{distributedJoins=false}}:
{code}
SELECT ... 
FROM Employee e
INNER JOIN Department d on d.id = e.dept_id
WHERE e.dept_id = ?
{code}

If {{dept_id}} is affinity key, and this should be certainly so in case of 
non-colocated joins (otherwise query will return incorrect result), then we can 
do the following:
1) Determine partition for this affinity key.
2) Send query execution request to this partition only.

Same technique can be applied to {{WHERE e.dept_id IN (?, ...)}} case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: PageMemory approach for Ignite 2.0

[Sergi Vladykin]

I agree, we should fix all the outstanding issues and resolve the
performance problems before merging it into 2.0

Sergi

2016-12-29 12:37 GMT+03:00 Alexey Goncharuk :

> Folks,
>
> I pushed an initial implementation of IGNITE-3477 to ignite-3477 branch for
> community review and further discussion.
>
> Note that the implementation lacks the following features:
>  - On-heap deserialized values cache
>  - Full LOCAL cache support
>  - Eviction policies
>  - Multiple memory pools
>  - Distributed joins support
>  - Off-heap circular remove buffer
>  - Maybe something else I missed
>
> The subject of this discussion is to determine whether the PageMemory
> approach is a way to go, because the this implementation is almost 2x
> slower than current 2.0 branch. There is some room for improvement, but I
> am not completely sure we can gain the same performance numbers as in 2.0.
>
> I encourage the community to review the code and architecture and share
> their thoughts here.
>
> Thanks,
> AG
>

[jira] [Created] (IGNITE-4509) SQL: query with condition on affinity columns and without joins and subqueries can be replaced with GET

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-4509:
---

 Summary: SQL: query with condition on affinity columns and without 
joins and subqueries can be replaced with GET
 Key: IGNITE-4509
 URL: https://issues.apache.org/jira/browse/IGNITE-4509
 Project: Ignite
  Issue Type: Task
  Components: SQL
Affects Versions: 1.8
Reporter: Vladimir Ozerov
 Fix For: 2.0


Simple SQL queries usually demonstrate negative scalability when more nodes are 
added. 

Consider the following query:
{code}
SELECT * FROM Person p
WHERE p.id = ?
{code}

If {{Person.id}} is affinity field, then query can be replaced with plain 
{{IgniteCache.get}} or some specialized version of GET, which will return only 
desired fields. This optimization will guarantee smooth scale with any number 
of nodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

PageMemory approach for Ignite 2.0

[Alexey Goncharuk]

Folks,

I pushed an initial implementation of IGNITE-3477 to ignite-3477 branch for
community review and further discussion.

Note that the implementation lacks the following features:
 - On-heap deserialized values cache
 - Full LOCAL cache support
 - Eviction policies
 - Multiple memory pools
 - Distributed joins support
 - Off-heap circular remove buffer
 - Maybe something else I missed

The subject of this discussion is to determine whether the PageMemory
approach is a way to go, because the this implementation is almost 2x
slower than current 2.0 branch. There is some room for improvement, but I
am not completely sure we can gain the same performance numbers as in 2.0.

I encourage the community to review the code and architecture and share
their thoughts here.

Thanks,
AG

Error in JDBC store

[Dmitriy Karachentsev]

Hi all!

According to this thread [1] in JDBC store configuration:
1. Map key and value fields to the same columns in DB.
2. Try to update data.
3. Got invalid SQL.

Let's see an pseudocode example of described use case.
KeyClass { field1; field2; field3 }, ValClass { field1; field2; field3 }

Map fields to DB table TABLE_NAME columns: field1 -> col1, field2 -> col2,
field3 -> col3.

User expects the following update request built by Ignite:
UPDATE TABLE_NAME SET col1=?, col2=?, col3=? WHERE (col1=?, col2=?, col3=?);

But Ignite checks that value object fields have the same mappings, throws
them away and builds wrong query:
UPDATE TABLE_NAME SET WHERE (col1=?, col2=?, col3=?);
That is obviously wrong.

Is there any reason to do so?
Probably, it is better to build query according to user mapping and
delegate verification to DB.

[1]
http://apache-ignite-users.70518.x6.nabble.com/Error-while-writethrough-operation-in-Ignite-td9696.html

Thanks!
Dmitry.

Re: IGNITE-4487 - NPE on query execution

[Alexey Goncharuk]

I think that If fallbacks(...) returns an empty nodes collection, then we
should fail with an exception.

2016-12-28 22:06 GMT+03:00 Denis Magda :

> Alexander, added you to the contributors list. Please check that you can
> assign the ticket on yourself.
>
> —
> Denis
>
> > On Dec 28, 2016, at 2:15 AM, Александр Меньшиков 
> wrote:
> >
> >
> > Username: sharpler
> >
> > Full Name: Alexander Menshikov
> >
> >
> >
> > 2016-12-27 22:57 GMT+03:00 Denis Magda >:
> > Alexander,
> >
> > I need to know your JIRA ID in order to add you to the contributors list.
> >
> > As for your questions, this situation might be caused by the race when a
> cache is being stopped and there are still scan queries running in
> parallel. So, in general it’s not about data loss.
> >
> > Sam, Alex G., could you share your thoughts in regards to the proper fix?
> >
> > —
> > Denis
> >
> > > On Dec 26, 2016, at 2:43 AM, Александр Меньшиков  > wrote:
> > >
> > > Hello everyone.
> > >
> > > I want to pick up *https://issues.apache.org/jira/browse/IGNITE-4487 <
> https://issues.apache.org/jira/browse/IGNITE-4487>
> > >  https://issues.apache.org/jira/browse/IGNITE-4487>>* as my
> > > first issue.
> > >
> > > Please add me as contributor.
> > >
> > > I already found that: in inner class
> > > 'GridCacheQueryAdapter.ScanQueryFallbackClosableIterator' in
> constructor is
> > > called with method 'init()', but method 'init()' cannot be called with
> an
> > > empty field 'nodes'. In source code it looks like:
> > >
> > > private ScanQueryFallbackClosableIterator(int part,
> GridCacheQueryAdapter
> > > qry,
> > >GridCacheQueryManager qryMgr, GridCacheContext cctx) {
> > >this.qry = qry;
> > >this.qryMgr = qryMgr;
> > >this.cctx = cctx;
> > >this.part = part;
> > >
> > >nodes = fallbacks(cctx.discovery().topologyVersionEx());
> > >// !!! Here nodes.isEmpty()==true, and init() will fail in
> the
> > > future. !!!
> > >init();
> > >}
> > >
> > > I can fix it by adding some check in code, but i must know what
> behavior
> > > are best in this case? As I understand it, the list of nodes is empty
> if
> > > there are no nodes with the current partition, which means data loss,
> and
> > > either need to return a meaningful exception, or ignore this
> situation. But
> > > maybe I missed something.
> >
> >
>
>

[jira] [Created] (IGNITE-4507) Hadoop: allow direct shuffle mode when combiner is set

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-4507:
---

 Summary: Hadoop: allow direct shuffle mode when combiner is set
 Key: IGNITE-4507
 URL: https://issues.apache.org/jira/browse/IGNITE-4507
 Project: Ignite
  Issue Type: Task
  Components: hadoop
Affects Versions: 1.8
Reporter: Vladimir Ozerov
 Fix For: 2.0


Currently "direct" shuffle mode is not used when combiner is set. It is 
necessary to investigate root cause of this limitation and remove it if 
possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)