PR IGNITE-1178 fix for NPE in GridCacheProcessor.onKernalStop()

[ALEKSEY KUZNETSOV]

Plz, review my PR : https://github.com/apache/ignite/pull/1517

https://issues.apache.org/jira/browse/IGNITE-1178
-- 

*Best Regards,*

*Kuznetsov Aleksey*

ignite analysis with jepsen

[Evgeniy Stanilovskiy]

Hello Igniters,

There is fresh doc about mongodb analysis with jepsen tool:
https://jepsen.io/analyses/mongodb-3-4-0-rc3
Looks like mongo soon would be production ready database )
And what do you think about the same analysis for ignite?

I have no deal with jepsen but has a lot of positive feed backs.
Spend a little time for introduction into
https://github.com/jepsen-io/jepsen
found all testing config under clojure language witch may cause some  
initial troubles i think ..

Re: DML data streaming

[Dmitriy Setrakyan]

On Wed, Feb 15, 2017 at 2:41 PM, Alexander Paschenko <
alexander.a.pasche...@gmail.com> wrote:

> Folks,
>
> Regarding INSERT semantics in JDBC DML streaming mode - I've left only
> INSERTs supports as we'd agreed before.
>
> However, current architecture of streaming related internals does not
> give any clear way to intercept key duplicates and inform the user -
> say, I can't just throw an exception from stream receiver (which is to
> my knowledge the only place where we could filter erroneous keys) as
> long as it will lead to whole batch remap and it's clearly not what we
> want here.
>
> Printing warning to log from the receiver is of little to no use as it
> will happen on data nodes so the end user won't see anything.
>

However, you still must do it. You should try throttling the identical log
messages, so we don't flood the log.


>
> What I've introduced for now is optional config param that turns on
> allowOverwrite on the streamer used in DML operation.
>

Agree, sounds like a good use of the flag. Are you setting it via JDBC/ODBC
connection flag?


> Does anyone have any thoughts about what could/should be done
> regarding informing user about key duplicates in streaming mode? Or
> probably we should just let it be as it is now?
>

In my view, we should introduce some generic error trap callback, e.g.
onSqlError(...), for all unhandled SQL errors. User should provide it in
the configuration, before startup. What do you think?


>
> Regards,
> Alex
>
> 2017-02-15 23:42 GMT+03:00 Dmitriy Setrakyan :
> > On Wed, Feb 15, 2017 at 4:28 AM, Vladimir Ozerov 
> > wrote:
> >
> >> Ok, let's put aside current fields configuration, I'll create separate
> >> thread for it. As far as _KEY and _VAL, proposed change is exactly about
> >> mappings:
> >>
> >> class QueryEntity {
> >> ...
> >> String keyFieldName;
> >> String valFieldName;
> >> ...
> >> }
> >>
> >> The key thing is that we will not require users to be aware of our
> system
> >> columns. Normally user should not bother about existence of hidden _KEY
> and
> >> _VAL columns. Instead, we just allow them to optionally reference the
> whole
> >> key and/or val through predefined name.
> >>
> >>
> > Vladimir, how will it work from the DDL perspective. Let's say whenever
> > user wants to create a table in Ignite?
>

Re: Inaccurate documentation about transactions

[Evgeniy Stanilovskiy]

postgres has the different viewpoint, i hope.

https://www.postgresql.org/docs/9.1/static/transaction-iso.html

Read Committed Isolation Level

Read Committed is the default isolation level in PostgreSQL. When a  
transaction uses this isolation level, a SELECT query (without a FOR  
UPDATE/SHARE clause) sees only data committed before the query began; it  
never sees either uncommitted data or changes committed during query  
execution by concurrent transactions. In effect, a SELECT query sees a  
snapshot of the database as of the instant the query begins to run.  
However, SELECT does see the effects of previous updates executed within  
its own transaction, even though they are not yet committed.



Alexandr,

If you PUT some value within transaction, this is absolutely normal that
you will see it on successive GET in the _same_ transaction. DIRTY_READ  
is

a different thing - it allows reads of not-yet-committed changes from
_another_ transaction.

On Wed, Feb 15, 2017 at 9:41 AM, Alexandr Kuramshin  


wrote:

After doing some tests with transactions I've found transactions work  
not

as expected after reading the documentation [1].

First of all, nowhere's written which methods of the cache are
transactional and which are not. Quite the contrary, after reading
documentation we get know that each TRANSACTIONAL cache is fully
ACID-compliant without exceptions.

Only after deep multi-thread testing, and consulting with other  
developers,

I get know that only get and put methods are running within transaction,
but iterator and query methods are running outside (in autonomous)
transaction with READ_COMMITTED isolation level.

Later I've understood that only methods throwing
TransactionTimeoutException/TransactionRollbackException/
TransactionHeuristicException
are fully transactional. I think all methods on page [2] should be  
directly

described - are they transactional or not. Btw, why these exceptions are
not derived from the common base class, e.g. TransactionException?

Secondary, using the transactional get() method inside the  
READ_COMMITTED
transaction we expect to get the committed value, as the documentation  
[1]

claims:

* READ_COMMITTED - Data is read without a lock and is never cached in  
the

transaction itself.

Ok, but what about put()? After doing the put() a new value, we get
successive reads of the new value, that is actually DIRTY READ. Hence  
the

value is cached within transaction. It's not documented behavior.

[1] https://apacheignite.readme.io/docs/transactions

[2]
https://ignite.apache.org/releases/1.8.0/javadoc/org/
apache/ignite/IgniteCache.html

--
Thanks,
Alexandr Kuramshin

Re: DML data streaming

[Alexander Paschenko]

Folks,

Regarding INSERT semantics in JDBC DML streaming mode - I've left only
INSERTs supports as we'd agreed before.

However, current architecture of streaming related internals does not
give any clear way to intercept key duplicates and inform the user -
say, I can't just throw an exception from stream receiver (which is to
my knowledge the only place where we could filter erroneous keys) as
long as it will lead to whole batch remap and it's clearly not what we
want here.

Printing warning to log from the receiver is of little to no use as it
will happen on data nodes so the end user won't see anything.

What I've introduced for now is optional config param that turns on
allowOverwrite on the streamer used in DML operation.

Does anyone have any thoughts about what could/should be done
regarding informing user about key duplicates in streaming mode? Or
probably we should just let it be as it is now?

Regards,
Alex

2017-02-15 23:42 GMT+03:00 Dmitriy Setrakyan :
> On Wed, Feb 15, 2017 at 4:28 AM, Vladimir Ozerov 
> wrote:
>
>> Ok, let's put aside current fields configuration, I'll create separate
>> thread for it. As far as _KEY and _VAL, proposed change is exactly about
>> mappings:
>>
>> class QueryEntity {
>> ...
>> String keyFieldName;
>> String valFieldName;
>> ...
>> }
>>
>> The key thing is that we will not require users to be aware of our system
>> columns. Normally user should not bother about existence of hidden _KEY and
>> _VAL columns. Instead, we just allow them to optionally reference the whole
>> key and/or val through predefined name.
>>
>>
> Vladimir, how will it work from the DDL perspective. Let's say whenever
> user wants to create a table in Ignite?

error - Failed to wait for partition release future

[Cameron Braid]

Hi,

I am getting the following error in Ignite version 1.8.1 :

"Failed to wait for partition release future"

The related logs lines are at the bottom.

I was curious if it could be related to
https://issues.apache.org/jira/browse/IGNITE-3212 which has fix for version
1.9.  Could this be the case ?

Otherwise, is this likely to be a bug in ignite, or something specific to
my environment?

Cheers

Cameron


   - Feb 16 08:23:28 webapp-92-50ik3 webapp 15287661 WARN
   o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture
   [exchange-worker-#26%webapp%] Failed to wait for partition release future
   [topVer=AffinityTopologyVersion [topVer=21, minorTopVer=2],
   node=79d74dfa-9903-42e3-af94-524ab08c1f97]. Dumping pending objects that
   might be the cause:
   - Feb 16 08:23:28 webapp-92-50ik3 webapp 15287661 WARN
   o.a.i.i.p.c.GridCachePartitionExchangeManager
   [exchange-worker-#26%webapp%] Ready affinity version:
   AffinityTopologyVersion [topVer=21, minorTopVer=1]
   - Feb 16 08:23:28 webapp-92-50ik3 webapp 15287667 WARN
   o.a.i.i.p.c.GridCachePartitionExchangeManager
   [exchange-worker-#26%webapp%] Last exchange future:
   GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
   reassign=false, discoEvt=DiscoveryCustomEvent
   [customMsg=DynamicCacheChangeBatch [reqs=[DynamicCacheChangeRequest
   [deploymentId=3f757f24a51-ee74fcc5-5cea-4548-9cd6-8a201c29fdf9,
   startCfg=CacheConfiguration [name=userTable.cmsDataVehicles.query,
   storeConcurrentLoadAllThreshold=5, rebalancePoolSize=2,
   rebalanceTimeout=1, evictPlc=null, evictSync=false,
   evictKeyBufSize=1024, evictSyncConcurrencyLvl=4, evictSyncTimeout=1,
   evictFilter=null, evictMaxOverflowRatio=10.0, eagerTtl=true,
   dfltLockTimeout=0, startSize=150, nearCfg=null, writeSync=PRIMARY_SYNC,
   storeFactory=null, storeKeepBinary=false, loadPrevVal=false,
   aff=o.a.i.cache.affinity.rendezvous.RendezvousAffinityFunction@7e1c4f4,
   cacheMode=REPLICATED, atomicityMode=ATOMIC, atomicWriteOrderMode=PRIMARY,
   backups=2147483647, invalidate=false, tmLookupClsName=null,
   rebalanceMode=ASYNC, rebalanceOrder=0, rebalanceBatchSize=524288,
   rebalanceBatchesPrefetchCount=2, offHeapMaxMem=-1, swapEnabled=false,
   maxConcurrentAsyncOps=500, writeBehindEnabled=false,
   writeBehindFlushSize=10240, writeBehindFlushFreq=5000,
   writeBehindFlushThreadCnt=1, writeBehindBatchSize=512, maxQryIterCnt=1024,
   memMode=ONHEAP_TIERED,
   
affMapper=o.a.i.i.processors.cache.CacheDefaultBinaryAffinityKeyMapper@6ca896dd,
   rebalanceDelay=0, rebalanceThrottle=0, interceptor=null,
   longQryWarnTimeout=3000, qryDetailMetricsSz=0, readFromBackup=true,
   
nodeFilter=o.a.i.configuration.CacheConfiguration$IgniteAllNodesPredicate@db2015b,
   sqlSchema=null, sqlEscapeAll=false, sqlOnheapRowCacheSize=10240,
   snapshotableIdx=false, cpOnRead=true, topValidator=null], cacheType=USER,
   initiatingNodeId=79d74dfa-9903-42e3-af94-524ab08c1f97, nearCacheCfg=null,
   clientStartOnly=false, stop=false, close=false, failIfExists=true,
   template=false, rcvdFrom=null, exchangeNeeded=true, cacheFutTopVer=null,
   cacheName=userTable.cmsDataVehicles.query]], clientNodes=null,
   id=4f757f24a51-ee74fcc5-5cea-4548-9cd6-8a201c29fdf9,
   clientReconnect=false], affTopVer=AffinityTopologyVersion [topVer=21,
   minorTopVer=2], super=DiscoveryEvent [evtNode=TcpDiscoveryNode
   [id=79d74dfa-9903-42e3-af94-524ab08c1f97, addrs=[0:0:0:0:0:0:0:1%lo,
   10.130.1.43, 127.0.0.1], sockAddrs=[webapp-92-50ik3/10.130.1.43:47500,
   /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500], discPort=47500, order=21,
   intOrder=12, lastExchangeTime=1487193806549, loc=true,
   ver=1.8.1#20161208-sha1:a8fcb7b2, isClient=false], topVer=21,
   nodeId8=79d74dfa, msg=null, type=DISCOVERY_CUSTOM_EVT,
   tstamp=1487193798024]], crd=TcpDiscoveryNode
   [id=659ea029-ca80-4413-af47-67b446ffd3e7, addrs=[0:0:0:0:0:0:0:1%lo,
   10.128.1.11, 127.0.0.1], sockAddrs=[/0:0:0:0:0:0:0:1%lo:47500, /
   127.0.0.1:47500, /10.128.1.11:47500], discPort=47500, order=18,
   intOrder=10, lastExchangeTime=1487178534506, loc=false,
   ver=1.8.1#20161208-sha1:a8fcb7b2, isClient=false],
   exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
   [topVer=21, minorTopVer=2], nodeId=79d74dfa, evt=DISCOVERY_CUSTOM_EVT],
   added=true, initFut=GridFutureAdapter [resFlag=0, res=null,
   startTime=1487193798024, endTime=0, ignoreInterrupts=false, state=INIT],
   init=false, topSnapshot=null, lastVer=null,
   partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=3,
   done=false, cancelled=false, err=null, futs=[true, true, false, true]],
   affChangeMsg=null, skipPreload=false, clientOnlyExchange=false,
   initTs=1487193798024, centralizedAff=false, evtLatch=0,
   remaining=[0940d968-a901-4c78-8354-8171628d41aa,
   659ea029-ca80-4413-af47-67b446ffd3e7], srvNodes=[TcpDiscoveryNode
   [id=659ea029-ca80-4413-af47-67b446ffd3e7, addrs=[0:0:0:0:0:0:0:1%lo,
   10.128.1.11, 127.0.0.1], 

Re: DML data streaming

[Dmitriy Setrakyan]

On Wed, Feb 15, 2017 at 4:28 AM, Vladimir Ozerov 
wrote:

> Ok, let's put aside current fields configuration, I'll create separate
> thread for it. As far as _KEY and _VAL, proposed change is exactly about
> mappings:
>
> class QueryEntity {
> ...
> String keyFieldName;
> String valFieldName;
> ...
> }
>
> The key thing is that we will not require users to be aware of our system
> columns. Normally user should not bother about existence of hidden _KEY and
> _VAL columns. Instead, we just allow them to optionally reference the whole
> key and/or val through predefined name.
>
>
Vladimir, how will it work from the DDL perspective. Let's say whenever
user wants to create a table in Ignite?

Re: Inaccurate documentation about transactions

[Denis Magda]

This is even better. Alexandr, could you do this since you already know what’s 
transaction and what isn’t?

—
Denis

> On Feb 15, 2017, at 10:50 AM, Dmitriy Setrakyan  wrote:
> 
> Why not just have @IgniteTransactional annotation and attach it to all the
> transactional methods?
> 
> On Wed, Feb 15, 2017 at 9:48 AM, Denis Magda  > wrote:
> 
>> Alexander,
>> 
>> Thanks for extensive feedback. I do support your idea that we need to
>> specify explicitly which methods are transactional and which aren’t.
>> 
>>> Only after deep multi-thread testing, and consulting with other
>> developers, I get know that only get and put methods are running within
>> transaction, but iterator and query methods are running outside (in
>> autonomous) transaction with READ_COMMITTED isolation level.
>> 
>> As far as I know, a family of “invoke” methods is also fully
>> transactional. As for SQL queries we state loud and clear that they’re non
>> transactional at the moment:
>> https://apacheignite.readme.io/docs/sql-queries#section-transactional-sql
>> > 
>>> 
>> 
>>> I think all methods on page [2] should be directly described - are they
>> transactional or not. Btw, why these exceptions are not derived from the
>> common base class, e.g. TransactionException?
>> 
>> Agree. Could you label all the methods at Java Doc layer and send it for
>> review to the community? Once the review has been passed I’ll update the
>> readmeio documentation.
>> 
>> —
>> Denis
>> 
>>> On Feb 14, 2017, at 10:41 PM, Alexandr Kuramshin 
>> wrote:
>>> 
>>> After doing some tests with transactions I've found transactions work
>> not as expected after reading the documentation [1].
>>> 
>>> First of all, nowhere's written which methods of the cache are
>> transactional and which are not. Quite the contrary, after reading
>> documentation we get know that each TRANSACTIONAL cache is fully
>> ACID-compliant without exceptions.
>>> 
>>> Only after deep multi-thread testing, and consulting with other
>> developers, I get know that only get and put methods are running within
>> transaction, but iterator and query methods are running outside (in
>> autonomous) transaction with READ_COMMITTED isolation level.
>>> 
>>> Later I've understood that only methods throwing
>> TransactionTimeoutException/TransactionRollbackException/TransactionHeuristicException
>> are fully transactional. I think all methods on page [2] should be directly
>> described - are they transactional or not. Btw, why these exceptions are
>> not derived from the common base class, e.g. TransactionException?
>>> 
>>> Secondary, using the transactional get() method inside the
>> READ_COMMITTED transaction we expect to get the committed value, as the
>> documentation [1] claims:
>>> 
>>> * READ_COMMITTED - Data is read without a lock and is never cached in
>> the transaction itself.
>>> 
>>> Ok, but what about put()? After doing the put() a new value, we get
>> successive reads of the new value, that is actually DIRTY READ. Hence the
>> value is cached within transaction. It's not documented behavior.
>>> 
>>> [1] https://apacheignite.readme.io/docs/transactions <
>> https://apacheignite.readme.io/docs/transactions 
>> >
>>> 
>>> [2] https://ignite.apache.org/releases/1.8.0/javadoc/org/ 
>>> 
>> apache/ignite/IgniteCache.html > 
>> releases/1.8.0/javadoc/org/apache/ignite/IgniteCache.html>
>>> 
>>> --
>>> Thanks,
>>> Alexandr Kuramshin

Re: IGNITE-4534 - Ignite 2.0 eviction design

[Dmitriy Setrakyan]

On Wed, Feb 15, 2017 at 9:06 AM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> Dmitriy,
>
> For the current off-heap approach, we only have a per-entry LRU eviction
> policy which does not fit well page memory architecture. I like the
> adaptation of clock pro algorithm because it requires a significantly
> smaller amount of memory to track page updates and is scan-resistant.
>

Agree.


>
> I want to back Ivan here and discuss whether we need synchronized evictions
> at all. Given that current implementation has flaws and correct
> implementation should work with the same protocol guarantees as a cache
> update (meaning two-phase commit for transactional caches on every
> operation, including reads). I would rather drop synchronous evictions at
> all.
>
> Thoughts?
>

Agree again. Synchronous evictions only make sense in pure in-memory
approach, where there is no database at all. AFAIK, most of the Ignite
deployments are backed by a database. If not, then users should control the
evictions themselves.

However, I am always concerned when removing a certain feature. If we start
getting complaints, we may have to add it in 2.0.


>
> 2017-02-15 0:59 GMT+03:00 Dmitriy Setrakyan :
>
> > Ivan, thanks for the detailed explanation. My preference would be to
> > support all of the suggested eviction policies and control them from the
> > configuration. However, it does sound the K-random-LRU or K-randome-LRU-2
> > will be much easier to support than LIRs.
> >
> > Don't we already have something like K-random-LRU in place?
> >
> > D.
> >
> > On Tue, Feb 14, 2017 at 9:13 AM, Ivan Rakov 
> wrote:
> >
> > > Hello Igniters,
> > >
> > > I want to share some thoughts about supporting eviction on new page
> > memory
> > > storage.
> > >
> > > I think, we should reconsider eviction policies in their current state.
> > > Current pluggable eviction policy mechanism allows to determine which
> key
> > > will be evicted from the cache and when eviction should be performed.
> > There
> > > are two issues about it:
> > >
> > > 1) Such mechanism tends to make off-heap storage highly fragmented:
> batch
> > > of small evicted entries can accumulate needed total amount of free
> space
> > > in FreeList, but it still would be impossible to store one big entry.
> > >
> > > 2) Growth of cache size implies creating O(n) heap objects and links to
> > > maintain eviction policy, which causes linear growth of GC load. That
> > > doesn't suit us: getting rid of GC overhead was one of the main reasons
> > for
> > > keeping cache data in off-heap memory.
> > >
> > > I propose to use "whole page eviction, off-heap eviction metadata" as
> > > default mode for eviction. It's almost non-deterministic, because
> actual
> > > eviction result depends on how data is packed into pages, but is more
> > > efficient, and shouldn't cause long-term performance degradation due to
> > > fragmentation.
> > >
> > > I can suggest two ways to implement it:
> > >
> > > 1) *K-random-LRU
> > > *The idea is to allocate big off-heap array, to track timestamp of last
> > > usage for each data page.
> > > When data page on address X is accessed, we store current timestamp in
> X
> > /
> > > PAGE_SIZE array position.
> > > When there's a need for eviction, we randomly choose K indices of array
> > > (if some of indices point to non-data pages, re-choose them) and evict
> > data
> > > page with minimum timestamp. In case K=5, we'll evict page from 17%
> least
> > > recently used set with 50% probability  > > /questions/786392/expectation-of-minimum-of-n-i-i-d-uniform-
> > > random-variables>.
> > > Non-data pages can be simply recognized by having zero in corresponding
> > > position of timestamp array.
> > > If entry is too big and occupies more than one page, only first page
> will
> > > be tracked, and other will be considered as non-data pages.
> > > *K-random-LRU-2* is perspective variant of this approach. We'll evict
> > page
> > > with minimum time of penultimate access. Implementation is mostly the
> > same,
> > > the difference is that we should store two timestamps for each data
> page.
> > > LRU-2 outperforms  > > stos/courses/721-resources/p297-o_neil.pdf> LRU by resolving "one-hit
> > > wonder" problem: if page is accessed very rarely, but accidentally
> > accessed
> > > once, it's "protected" from eviction for long time.
> > > Memory overhead: timestamp can be packed into 4 bytes. 100GB off-heap
> > > cache with standard 2KB pages requires 200MB of additional memory
> (400MB
> > in
> > > case of LRU-2 strategy).*
> > > *
> > >
> > > 2)*Adaptation of CLOCK-Pro
> > > *CLOCK-Pro (article  > > /event/usenix05/tech/general/full_papers/jiang/jiang_html/html.html>,
> > > presentation ) is modern
> > > page replacement algorithm, used in actual OS kernels. It is an
> 

Is there a faster way to launch an ignite client ?

[Danny Shemesh]

Hey,

Launching an ignite client on the same machine as the ignite server, with
the simplest configuration (VmIpFinder on localhost)
through the java interface can take anywhere between 5 and 10 seconds.

I'm refactoring an ignite based CLI utility and am focusing on improving
its statup time.
Any recommendations on where I could look on (preferably not via REST as
it's an external module) ?

- Danny

Re: IGNITE-2741 - spring session design

[Valentin Kulichenko]

Hi Rishi,

Thanks, I'll take a look.

-Val

On Wed, Feb 15, 2017 at 9:07 AM, Rishi Yagnik  wrote:

> Hi Val,
>
> As promised, please find attached code for spring boot integration with
> spring security along with Ignite.
>
> Some more information on project -
>
>- It is a maven project ( Ignite 1.7.0, SB 1.4.3 )
>- spring security integrated with boot project along with ignite
>- HttpSessionCookieCsrfTokenRepository does not work, gives
>intermediate errors on single instance so used CookieCsrfTokenRepository
>for CSRF token, again I think we need a fix here from Ignite.
>
> I cant reproduce this errors while I am running on single instance, you
> need to run this app on 2 spring boot instance having proxy in front ( F5,
> OR any proxy ) with round robin fashion ( no sticky session on F5 OR
> proxies ).
>
> We were thinking with round robin the user session will active since we
> used session replication on backend.
>
> Do let me know if you need more information here.
>
> Thanks,
>
> Rishi
>
>
>
>
> On Tue, Feb 14, 2017 at 9:57 PM, Rishi Yagnik 
> wrote:
>
>> Val,
>>
>> My SB sample project is ready however I have asked for an approval to
>> submit sample project to you, it would take day or two.
>>
>> I will keep you posted.
>>
>> Thanks for all your help,
>>
>> On Tue, Feb 14, 2017 at 3:51 PM, Rishi Yagnik 
>> wrote:
>>
>>> Let me build an example app for you and send it across to you.
>>>
>>> Thanks,
>>>
>>> On Tue, Feb 14, 2017 at 3:28 PM, Valentin Kulichenko <
>>> valentin.kuliche...@gmail.com> wrote:
>>>
 Rishi,

 No I don't, and I think that's what we should start with. I want to
 understand a use case that is currently not supported (if any) and then
 find the best solution. And I would like to reuse existing code as much
 as
 possible.

 Do you have any code that reproduces the problem you had and how you
 tried
 to utilize current web session clustering? Can you share it with us?

 -Val

 On Tue, Feb 14, 2017 at 11:28 AM, Rishi Yagnik 
 wrote:

 > Hi Val,
 >
 > I am working on SB platform with spring security and we found out
 that the
 > web session filter ignite provides does not work for session
 management on
 > 2 node spring boot cluster.
 >
 > Somehow, spring security filter kicks in result in some weird errors
 with
 > web session filter.
 >
 > So making compatible with spring security somehow, we need to write
 > implementation on spring session.
 >
 > Do you have any test cases that says web session filter would work
 with
 > spring security on boot platform ?
 >
 > Thanks,
 >
 >
 > On Tue, Feb 14, 2017 at 1:03 PM, Valentin Kulichenko <
 > valentin.kuliche...@gmail.com> wrote:
 >
 > > Hi Rishi,
 > >
 > > Can you please take a look at web session clustering feature [1]
 provided
 > > by Ignite? I'm looking at Spring Session docs and it seems to me it
 does
 > > exactly the same - replaces HttpSession with custom implementation
 that
 > has
 > > a backend storage. If it doesn't provide any additional API or
 > > functionality, I'm not sure I understand the benefit of this
 feature.
 > >
 > > Let me know if I'm missing something.
 > >
 > > [1] https://apacheignite-mix.readme.io/docs/web-session-clustering
 > >
 > > -Val
 > >
 > > On Mon, Feb 13, 2017 at 2:41 PM, Rishi Yagnik <
 rishiyag...@gmail.com>
 > > wrote:
 > >
 > > > I would like to discuss session replication / fail over design on
 > spring
 > > > boot platform and wanted to find what is the best out to get
 started
 > > here ?
 > > >
 > > > Possible approaches are as follows -
 > > >
 > > >- Make use of Spring Session for session replication and fail
 over
 > > >- Extend the web session filter and make it work on spring boot
 > > >application
 > > >
 > > >
 > > > I am thinking that best approach would be to get started here with
 > spring
 > > > session design however I am open for feedback here.
 > > >
 > > > --
 > > > Rishi Yagnik
 > > >
 > >
 >
 >
 >
 > --
 > Rishi Yagnik
 >

>>>
>>>
>>>
>>> --
>>> Rishi Yagnik
>>>
>>
>>
>>
>> --
>> Rishi Yagnik
>>
>
>
>
> --
> Rishi Yagnik
>

Re: Inaccurate documentation about transactions

[Dmitriy Setrakyan]

Why not just have @IgniteTransactional annotation and attach it to all the
transactional methods?

On Wed, Feb 15, 2017 at 9:48 AM, Denis Magda  wrote:

> Alexander,
>
> Thanks for extensive feedback. I do support your idea that we need to
> specify explicitly which methods are transactional and which aren’t.
>
> > Only after deep multi-thread testing, and consulting with other
> developers, I get know that only get and put methods are running within
> transaction, but iterator and query methods are running outside (in
> autonomous) transaction with READ_COMMITTED isolation level.
>
> As far as I know, a family of “invoke” methods is also fully
> transactional. As for SQL queries we state loud and clear that they’re non
> transactional at the moment:
> https://apacheignite.readme.io/docs/sql-queries#section-transactional-sql
>  >
>
> >  I think all methods on page [2] should be directly described - are they
> transactional or not. Btw, why these exceptions are not derived from the
> common base class, e.g. TransactionException?
>
> Agree. Could you label all the methods at Java Doc layer and send it for
> review to the community? Once the review has been passed I’ll update the
> readmeio documentation.
>
> —
> Denis
>
> > On Feb 14, 2017, at 10:41 PM, Alexandr Kuramshin 
> wrote:
> >
> > After doing some tests with transactions I've found transactions work
> not as expected after reading the documentation [1].
> >
> > First of all, nowhere's written which methods of the cache are
> transactional and which are not. Quite the contrary, after reading
> documentation we get know that each TRANSACTIONAL cache is fully
> ACID-compliant without exceptions.
> >
> > Only after deep multi-thread testing, and consulting with other
> developers, I get know that only get and put methods are running within
> transaction, but iterator and query methods are running outside (in
> autonomous) transaction with READ_COMMITTED isolation level.
> >
> > Later I've understood that only methods throwing
> TransactionTimeoutException/TransactionRollbackException/TransactionHeuristicException
> are fully transactional. I think all methods on page [2] should be directly
> described - are they transactional or not. Btw, why these exceptions are
> not derived from the common base class, e.g. TransactionException?
> >
> > Secondary, using the transactional get() method inside the
> READ_COMMITTED transaction we expect to get the committed value, as the
> documentation [1] claims:
> >
> > * READ_COMMITTED - Data is read without a lock and is never cached in
> the transaction itself.
> >
> > Ok, but what about put()? After doing the put() a new value, we get
> successive reads of the new value, that is actually DIRTY READ. Hence the
> value is cached within transaction. It's not documented behavior.
> >
> > [1] https://apacheignite.readme.io/docs/transactions <
> https://apacheignite.readme.io/docs/transactions>
> >
> > [2] https://ignite.apache.org/releases/1.8.0/javadoc/org/
> apache/ignite/IgniteCache.html  releases/1.8.0/javadoc/org/apache/ignite/IgniteCache.html>
> >
> > --
> > Thanks,
> > Alexandr Kuramshin
>
>

Re: Python Ignite client (Requires ignite-rest-http)

[Denis Magda]

Hi Danny,

Excellent news, thanks for sharing! Cross-posting to the dev list.

By the way, don’t you want take part or take over the full-fledged Python lib 
development?
https://issues.apache.org/jira/browse/IGNITE-4600 


—
Denis

> On Feb 14, 2017, at 10:29 PM, Danny Shemesh  wrote:
> 
> Hey Ignite users,
> 
> I've written a REST based ignite client for python - thought some of you 
> could find it helpful.
> https://github.com/dany74q/python-apache-ignite 
> 
> pip install ignite
> 
> Please feel free to contribute, open issues, pull requests and message me 
> directly if needed.
> 
> Have a great, productive day!
> - Danny

Re: Is there a faster way to launch an ignite client ?

[Denis Magda]

Hi Danny,

Try to reduce the port range in your VmIpFinder. If the range is long and 
you’re on a Windows machine, then it can take time to iterate over every 
IP:port combination in condition that some of IP:ports are not presently used.

—
Denis

> On Feb 15, 2017, at 9:12 AM, Danny Shemesh  wrote:
> 
> Hey,
> 
> Launching an ignite client on the same machine as the ignite server, with the 
> simplest configuration (VmIpFinder on localhost)
> through the java interface can take anywhere between 5 and 10 seconds.
> 
> I'm refactoring an ignite based CLI utility and am focusing on improving its 
> statup time.
> Any recommendations on where I could look on (preferably not via REST as it's 
> an external module) ?
> 
> - Danny 

Re: Inaccurate documentation about transactions

[Denis Magda]

Alexander,

Thanks for extensive feedback. I do support your idea that we need to specify 
explicitly which methods are transactional and which aren’t. 

> Only after deep multi-thread testing, and consulting with other developers, I 
> get know that only get and put methods are running within transaction, but 
> iterator and query methods are running outside (in autonomous) transaction 
> with READ_COMMITTED isolation level.

As far as I know, a family of “invoke” methods is also fully transactional. As 
for SQL queries we state loud and clear that they’re non transactional at the 
moment:
https://apacheignite.readme.io/docs/sql-queries#section-transactional-sql 


>  I think all methods on page [2] should be directly described - are they 
> transactional or not. Btw, why these exceptions are not derived from the 
> common base class, e.g. TransactionException?

Agree. Could you label all the methods at Java Doc layer and send it for review 
to the community? Once the review has been passed I’ll update the readmeio 
documentation.

—
Denis

> On Feb 14, 2017, at 10:41 PM, Alexandr Kuramshin  wrote:
> 
> After doing some tests with transactions I've found transactions work not as 
> expected after reading the documentation [1].
> 
> First of all, nowhere's written which methods of the cache are transactional 
> and which are not. Quite the contrary, after reading documentation we get 
> know that each TRANSACTIONAL cache is fully ACID-compliant without exceptions.
> 
> Only after deep multi-thread testing, and consulting with other developers, I 
> get know that only get and put methods are running within transaction, but 
> iterator and query methods are running outside (in autonomous) transaction 
> with READ_COMMITTED isolation level.
> 
> Later I've understood that only methods throwing 
> TransactionTimeoutException/TransactionRollbackException/TransactionHeuristicException
>  are fully transactional. I think all methods on page [2] should be directly 
> described - are they transactional or not. Btw, why these exceptions are not 
> derived from the common base class, e.g. TransactionException?
> 
> Secondary, using the transactional get() method inside the READ_COMMITTED 
> transaction we expect to get the committed value, as the documentation [1] 
> claims:
> 
> * READ_COMMITTED - Data is read without a lock and is never cached in the 
> transaction itself.
> 
> Ok, but what about put()? After doing the put() a new value, we get 
> successive reads of the new value, that is actually DIRTY READ. Hence the 
> value is cached within transaction. It's not documented behavior.
> 
> [1] https://apacheignite.readme.io/docs/transactions 
> 
> 
> [2] 
> https://ignite.apache.org/releases/1.8.0/javadoc/org/apache/ignite/IgniteCache.html
>  
> 
> 
> -- 
> Thanks,
> Alexandr Kuramshin

Re: Apache Ignite 1.9

[Denis Magda]

Anton,

The implementation is correct and was reviewed by Val long time ago. The only 
issue that is left is the compilation in *TeamCity* only.
You fixed it for one scala version and now should simply do the same for other 
scala version by adding a new suite to TeamCity and adjusting the pom.

—
Denis

> On Feb 15, 2017, at 1:20 AM, Anton Vinogradov  
> wrote:
> 
> Denis,
> 
> I don't know what is the plan for this issue.
> You asked me to fix compilation and thats what I done :) , but I'm not sure
> that issue implemented in proper way. I don't like changes I've done, they
> looks redundant although it works.
> Do we have somebody more familiar with spark inside the community?
> We have to review what we already done and figure out what's left.
> 
> On Wed, Feb 15, 2017 at 5:01 AM, Denis Magda  wrote:
> 
>> Continuing Spark related topic...
>> 
>> This ticket definitely must be included in 1.9:
>> https://issues.apache.org/jira/browse/IGNITE-3710 <
>> https://issues.apache.org/jira/browse/IGNITE-3710>
>> 
>> We can’t longer afford torture Ignite Shared RDD users who have to use an
>> outdated Spark version.
>> 
>> Anton, could you finalize it this week? As far as I remember, the only
>> thing that is left is to add a missing test suite to Team City.
>> 
>> —
>> Denis
>> 
>>> On Feb 14, 2017, at 5:00 PM, Denis Magda  wrote:
>>> 
>>> Reviewed and merged the first (!) Ignite Shared RDD examples contributed
>> by Manish:
>>> 
>>> https://issues.apache.org/jira/browse/IGNITE-4526
>>> 
>>> Java:
>>> https://github.com/apache/ignite/blob/b461cb47882861356ede58775bd9e2
>> 53dcf26202/examples/src/main/java8/org/apache/ignite/examples/java8/spark/
>> SharedRDDExample.java
>>> 
>>> Scala:
>>> https://github.com/apache/ignite/blob/b461cb47882861356ede58775bd9e2
>> 53dcf26202/examples/src/main/scala/org/apache/ignite/
>> scalar/examples/spark/ScalarSharedRDDExample.scala
>>> 
>>> Manish, thanks for making the life of Apache Ignite users easier. No
>> they can refer to your examples while learning Ignite Shared RDD API.
>>> 
>>> —
>>> Denis
>>> 
 On Feb 14, 2017, at 1:21 AM, Anton Vinogradov 
>> wrote:
 
> 
> Can we push to 1.9 and then merge/rebase to master?
 
 
 Yes, that's *correct *too, but only *megre *should be used.
 
 On Tue, Feb 14, 2017 at 12:19 PM, Vladimir Ozerov >> 
 wrote:
 
> Can we push to 1.9 and then merge/rebase to master? It should be more
> reliable than cherry-picking.
> 
> On Tue, Feb 14, 2017 at 12:15 PM, Anton Vinogradov <
> avinogra...@gridgain.com
>> wrote:
> 
>> Denis,
>> 
>> Merge to the master first and then cherry-pick to 1.9 from there is a
>> *correct
>> *case.
>> 
>> On Tue, Feb 14, 2017 at 5:24 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
>> wrote:
>> 
>>> Thanks, Denis!
>>> 
>>> On Mon, Feb 13, 2017 at 5:49 PM, Denis Magda 
> wrote:
>>> 
 Well, merged Kubernetes integration to the master and cherry-picked
> to
 1.9. The docs and getting started for readme.io to be done soon.
 
 —
 Denis
 
> On Feb 13, 2017, at 9:10 AM, Denis Magda 
> wrote:
> 
> Do I merge to the master first and then cherry-pick to 1.9 from
>> there?
> 
> —
> Denis
> 
>> On Feb 13, 2017, at 8:21 AM, Anton Vinogradov <
>>> avinogra...@gridgain.com>
 wrote:
>> 
>> Igniters,
>> 
>> We're starting release stabilization process.
>> Branch 1.9 created and equals to master branch right now.
>> 
>> Please make sure that your changes will be merged to both
> branches:
>> *ignite-1.9* and *master*.
>> 
>> On Fri, Feb 10, 2017 at 4:53 PM, Sergey Kalashnikov <
 zkilling...@gmail.com>
>> wrote:
>> 
>>> Anton,
>>> 
>>> Here is the status for IGNITE-4523.
>>> The work is completed, tests are OK.
>>> Most of the changes were made by Alexey and were reviewed.
>>> I only authored few last commits for which I would appreciate a
>>> review.
>>> The PR is https://github.com/apache/ignite/pull/1508
>>> 
>>> Thank you
>>> --
>>> Sergey
>>> 
>>> 
>>> 
>>> 2017-02-09 23:58 GMT+03:00 Denis Magda :
>>> 
 Anton,
 
 You can expect Kubernetes support from me in time.
 
 —
 Denis
 
> On Feb 9, 2017, at 1:52 AM, Anton Vinogradov 
>>> wrote:
> 
> Igniters,
> 
> We're on a half way to Ignite 1.9.
> 

Re: IGNITE-4534 - Ignite 2.0 eviction design

[Alexey Goncharuk]

Dmitriy,

For the current off-heap approach, we only have a per-entry LRU eviction
policy which does not fit well page memory architecture. I like the
adaptation of clock pro algorithm because it requires a significantly
smaller amount of memory to track page updates and is scan-resistant.

I want to back Ivan here and discuss whether we need synchronized evictions
at all. Given that current implementation has flaws and correct
implementation should work with the same protocol guarantees as a cache
update (meaning two-phase commit for transactional caches on every
operation, including reads). I would rather drop synchronous evictions at
all.

Thoughts?

2017-02-15 0:59 GMT+03:00 Dmitriy Setrakyan :

> Ivan, thanks for the detailed explanation. My preference would be to
> support all of the suggested eviction policies and control them from the
> configuration. However, it does sound the K-random-LRU or K-randome-LRU-2
> will be much easier to support than LIRs.
>
> Don't we already have something like K-random-LRU in place?
>
> D.
>
> On Tue, Feb 14, 2017 at 9:13 AM, Ivan Rakov  wrote:
>
> > Hello Igniters,
> >
> > I want to share some thoughts about supporting eviction on new page
> memory
> > storage.
> >
> > I think, we should reconsider eviction policies in their current state.
> > Current pluggable eviction policy mechanism allows to determine which key
> > will be evicted from the cache and when eviction should be performed.
> There
> > are two issues about it:
> >
> > 1) Such mechanism tends to make off-heap storage highly fragmented: batch
> > of small evicted entries can accumulate needed total amount of free space
> > in FreeList, but it still would be impossible to store one big entry.
> >
> > 2) Growth of cache size implies creating O(n) heap objects and links to
> > maintain eviction policy, which causes linear growth of GC load. That
> > doesn't suit us: getting rid of GC overhead was one of the main reasons
> for
> > keeping cache data in off-heap memory.
> >
> > I propose to use "whole page eviction, off-heap eviction metadata" as
> > default mode for eviction. It's almost non-deterministic, because actual
> > eviction result depends on how data is packed into pages, but is more
> > efficient, and shouldn't cause long-term performance degradation due to
> > fragmentation.
> >
> > I can suggest two ways to implement it:
> >
> > 1) *K-random-LRU
> > *The idea is to allocate big off-heap array, to track timestamp of last
> > usage for each data page.
> > When data page on address X is accessed, we store current timestamp in X
> /
> > PAGE_SIZE array position.
> > When there's a need for eviction, we randomly choose K indices of array
> > (if some of indices point to non-data pages, re-choose them) and evict
> data
> > page with minimum timestamp. In case K=5, we'll evict page from 17% least
> > recently used set with 50% probability  > /questions/786392/expectation-of-minimum-of-n-i-i-d-uniform-
> > random-variables>.
> > Non-data pages can be simply recognized by having zero in corresponding
> > position of timestamp array.
> > If entry is too big and occupies more than one page, only first page will
> > be tracked, and other will be considered as non-data pages.
> > *K-random-LRU-2* is perspective variant of this approach. We'll evict
> page
> > with minimum time of penultimate access. Implementation is mostly the
> same,
> > the difference is that we should store two timestamps for each data page.
> > LRU-2 outperforms  > stos/courses/721-resources/p297-o_neil.pdf> LRU by resolving "one-hit
> > wonder" problem: if page is accessed very rarely, but accidentally
> accessed
> > once, it's "protected" from eviction for long time.
> > Memory overhead: timestamp can be packed into 4 bytes. 100GB off-heap
> > cache with standard 2KB pages requires 200MB of additional memory (400MB
> in
> > case of LRU-2 strategy).*
> > *
> >
> > 2)*Adaptation of CLOCK-Pro
> > *CLOCK-Pro (article  > /event/usenix05/tech/general/full_papers/jiang/jiang_html/html.html>,
> > presentation ) is modern
> > page replacement algorithm, used in actual OS kernels. It is an
> > approximation of LIRS  > /hpcs/WWW/HTML/publications/papers/TR-02-6.pdf> with significant
> > advantage for our case: instead of doubly linked list, which is hard to
> > maintain in off-heap memory, CLOCK-Pro uses only circular list and three
> > pointers. However, we can't use CLOCK-Pro as is: there's difference
> between
> > OS paged memory and Ignite off-heap storage. In OS, if memory page is
> > swapped out, it still contains the same data, with same relevance and
> > patterns of access. That's why keeping metadata for swapped-out page (see
> > Non-resident Cold Pages set) works. In Ignite, in case we evict page, all
> > data is 

[GitHub] ignite pull request #1539: IGNITE-1624 [not ready, just for testing]

[SharplEr]

GitHub user SharplEr opened a pull request:

https://github.com/apache/ignite/pull/1539

IGNITE-1624 [not ready, just for testing]

Not ready. just for testing in CI.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SharplEr/ignite ignite-1624

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1539.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1539


commit 4e38acf2757d76ed848adaa476aab0faba1a8fc0
Author: Alexander Menshikov 
Date:   2017-02-15T15:41:28Z

Add repeating to test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-4707) Add documentation for HA IGFS client

[Vladimir Ozerov (JIRA)]

Vladimir Ozerov created IGNITE-4707:
---

 Summary: Add documentation for HA IGFS client
 Key: IGNITE-4707
 URL: https://issues.apache.org/jira/browse/IGNITE-4707
 Project: Ignite
  Issue Type: Task
  Components: documentation, IGFS
Reporter: Vladimir Ozerov
Assignee: Vladimir Ozerov
Priority: Minor
 Fix For: 2.0


We need to document HA feature added in IGNITE-2356.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] ignite pull request #1538: Ignite-gg-11901 package renaming

[DmitriyGovorukhin]

GitHub user DmitriyGovorukhin opened a pull request:

https://github.com/apache/ignite/pull/1538

Ignite-gg-11901 package renaming



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-gg-11901

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1538.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1538


commit eec3456933ce9f04dfdec17cb64455b2aaa8526d
Author: AKuznetsov 
Date:   2016-12-13T11:51:30Z

Removed deprecated code. Updated VisorCache metrics.

commit ba4f72283658831394ddd5acafd33ba1d7f23806
Author: AKuznetsov 
Date:   2016-12-13T13:33:49Z

Merge branches 'ignite-gg-8.0.1.ea7' and 'ignite-gg-8.0.2.ea1' of 
https://github.com/gridgain/apache-ignite into ignite-gg-8.0.2.ea1

commit 4731bee70704e6a092a48adf1497f11d3544210b
Author: Dmitriy Govorukhin 
Date:   2016-12-14T09:06:00Z

ignite-gg-11650 add lientChangeGlobalStateComputeRequest in properties 
classpath +  datastreamer test

commit 87a701813b6abec5dc174694f3f0963b1641fa8e
Author: Dmitriy Govorukhin 
Date:   2016-12-14T09:48:52Z

ignite-gg-11650 comment for code

commit dc677c9fc86c68cc82d980c33e420da701a4954b
Author: sboikov 
Date:   2016-12-14T10:57:18Z

ignite-db-x minor rename

commit cc675d9b75696cbdcc3af994ad3fed30c032c10c
Author: Dmitriy Govorukhin 
Date:   2016-12-14T12:41:00Z

ignite-gg-11650 move  IgniteChangeGlobalStateSupport  to internal package 
+javadoc

commit 024bb4e1142128102b8c3faf09b08d544baab53a
Author: Dmitriy Govorukhin 
Date:   2016-12-14T12:56:46Z

ignite-gg-11650 minor fix

commit 5036d31fef1957226b9395493af808788b202e3a
Author: Dmitriy Govorukhin 
Date:   2016-12-14T14:30:31Z

ignite-gg-11650 sh bat activate/deactivate, move .gitignore to platform

commit b961fdb41e1b4a1ccb2e769ff64e3b8f09609958
Author: Ilya Lantukh 
Date:   2016-12-14T16:10:10Z

Assertions for pageId == effectivePageId.

commit fdd272e0bf956df3f70a3310923fd41967697261
Author: Alexey Goncharuk 
Date:   2016-12-14T16:14:09Z

EA7 - Fixed security tests polluting logs.

commit 893ddf084a687bb0044b8fa7d3dbf5904a774bb7
Author: Alexey Goncharuk 
Date:   2016-12-14T16:14:23Z

Merge remote-tracking branch 'community/ignite-gg-8.0.1.ea7' into 
ignite-gg-8.0.1.ea7

commit 291d2a5b23616fc26f1af4c890a87b9b7d34b5c6
Author: Dmitriy Govorukhin 
Date:   2016-12-15T07:46:13Z

ignite-gg-11650 control sh update

commit d9855b66d28aa5888b5d3a12f263d09ae96597d7
Author: Dmitriy Govorukhin 
Date:   2016-12-15T08:38:04Z

ignite-gg-11650 bat script update

commit 0aff7d68d906dfad1afcb8f407546a8fe92a3514
Author: EdShangGG 
Date:   2016-12-15T09:26:44Z

Merge branch 'ignite-gg-8.0.1.ea7' into ignite-gg-8.0.2.ea1

commit c6b01009bdf0507183aa4b05db062c182b5ff22e
Author: Dmitriy Govorukhin 
Date:   2016-12-15T12:31:41Z

Merge branch 'ignite-gg-8.0.2.ea1' into ignite-gg-11650

# Conflicts:
#   modules/core/src/main/java/org/apache/ignite/IgniteSystemProperties.java
#   
modules/core/src/main/java/org/apache/ignite/internal/managers/communication/GridIoMessageFactory.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCachePartitionExchangeManager.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/database/IgniteCacheDatabaseSharedManager.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPartitionsExchangeFuture.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPartitionsFullMessage.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPartitionsSingleMessage.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/preloader/GridDhtPreloader.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/odbc/OdbcProcessor.java

commit 689cc4ef98cf31fdc76f6a3eabb94716d3f0bbe0
Author: Dmitriy Govorukhin 
Date:   2016-12-15T13:22:48Z

ignite-gg-11650 resolve merge changes

commit 12c731e28ce1b33a9dbebb6c9c8251cd5e574136
Author: Dmitriy Govorukhin 
Date:   2016-12-15T14:23:11Z

Merge 

Re: general question

[ALEKSEY KUZNETSOV]

hmm, why and how EvictionManager tracks an entry ?

вт, 14 февр. 2017 г. в 23:07, Andrey Mashenkov :

> Hi Aleksey,
>
> EvictableEntry.meta is used by EvictionManager to track entry.
> Actually it may contains Node of EvictionPolicy queue to manupulate queue
> items in more efficient way.
>
> On Tue, Feb 14, 2017 at 4:26 PM, ALEKSEY KUZNETSOV <
> alkuznetsov...@gmail.com
> > wrote:
>
> > need help. What is the use of EvictableEntry.meta  ? Why do we need to
> ship
> > some meta to entry ?
> > --
> >
> > *Best Regards,*
> >
> > *Kuznetsov Aleksey*
> >
>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
-- 

*Best Regards,*

*Kuznetsov Aleksey*

[GitHub] ignite pull request #1537: IGNITE-1948

[SharplEr]

GitHub user SharplEr opened a pull request:

https://github.com/apache/ignite/pull/1537

IGNITE-1948

I divided ClusterTopologyCheckedException into two: 
ClusterTopologyCheckedException (with readyFut) and 
ClusterTopologyLocalException (without readyFut). 
ClusterTopologyCheckedException extends ClusterTopologyLocalException.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SharplEr/ignite ignite-1948

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1537.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1537


commit 814c92620fd92cd9a245d71e8e310c4a5df0abe5
Author: Alexander Menshikov 
Date:   2017-02-14T11:11:53Z

Add ClusterGroupEmptyLocalException and ClusterTopologyLocalException

commit edf38a248011a56691605dfd7acaf8d2d51536e7
Author: Alexander Menshikov 
Date:   2017-02-14T12:00:23Z

Add license

commit 19fc62af997d54dcffcf8df84e76736018bbb7f8
Author: Alexander Menshikov 
Date:   2017-02-14T12:05:46Z

Remove some commented lines

commit 7a280a8c5253c94f7238f8078114494835f66957
Author: Alexander Menshikov 
Date:   2017-02-14T12:25:34Z

Remove some commented code and add asserts

commit 9015c9b3d54d266c12450a8c211dbb4531799b63
Author: Alexander Menshikov 
Date:   2017-02-14T12:29:41Z

Add assert for testing

commit 1cbd9bd9502ba534dd7ad6d8adad9bed447be669
Author: Alexander Menshikov 
Date:   2017-02-14T13:50:22Z

Cleanup code

commit b02be8fc99408aa0a12eb0ef355ea93b0173d7e0
Author: Alexander Menshikov 
Date:   2017-02-15T10:37:48Z

Expanded use of ClusterTopologyLocalException




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

IGNITE-3422 - ready for review

[Vyacheslav Daradur]

Hello everyone.

Please, review implemented solution.

https://issues.apache.org/jira/browse/IGNITE-3422 - No way to control
object initialization during deserialization/unmarshalling

ci.tests 

[GitHub] ignite pull request #1530: IGNITE-4693 Add possibility to wrap Java plugin e...

[ptupitsyn]

Github user ptupitsyn closed the pull request at:

https://github.com/apache/ignite/pull/1530


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: DML data streaming

[Sergi Vladykin]

Vladimir,

Looks good to me.


Pavel,

No worries, it will work exactly like you described: hidden _key and _val
fields will be always accessible.

Sergi

2017-02-15 15:56 GMT+03:00 Pavel Tupitsyn :

> I have no particular opinion on how we should handle _key/_val,
> but we certainly need a way to select entire key and value objects via
> SqlFieldsQuery,
> and this should work without any additional configuration.
>
> We can rename these, turn them into system functions, whatever.
>
> Ignite.NET LINQ provider heavily relies on this possibility - users often
> want to select the entire entry value.
>
> On Wed, Feb 15, 2017 at 3:28 PM, Vladimir Ozerov 
> wrote:
>
> > Ok, let's put aside current fields configuration, I'll create separate
> > thread for it. As far as _KEY and _VAL, proposed change is exactly about
> > mappings:
> >
> > class QueryEntity {
> > ...
> > String keyFieldName;
> > String valFieldName;
> > ...
> > }
> >
> > The key thing is that we will not require users to be aware of our system
> > columns. Normally user should not bother about existence of hidden _KEY
> and
> > _VAL columns. Instead, we just allow them to optionally reference the
> whole
> > key and/or val through predefined name.
> >
> > On Wed, Feb 15, 2017 at 3:07 PM, Sergi Vladykin <
> sergi.vlady...@gmail.com>
> > wrote:
> >
> > > I don't see any improvement here. Usability will only suffer with this
> > > change.
> > >
> > > I'd suggest to just add mapping for system columns like _key, _val ,
> > _ver.
> > >
> > > Sergi
> > >
> > > 2017-02-15 13:18 GMT+03:00 Vladimir Ozerov :
> > >
> > > > I think the whole QueryEntity class require rework to allow for this
> > > > change. I would start with creating QueryField class which will
> > > encapsulate
> > > > all field properties which are currently set through different
> setters:
> > > >
> > > > class QueryField {
> > > > String name;
> > > > String type;
> > > > String alias;
> > > > boolean keyField;
> > > > }
> > > >
> > > > class QueryEntity {
> > > > String tableName;
> > > > String keyType;
> > > > String valType;
> > > > Collection fields;
> > > > Collection indexes;
> > > > }
> > > >
> > > > Then we can add optional key and value field names to top-level
> config.
> > > If
> > > > set, key and/or value will have names and will be included into
> SELECT
> > *
> > > > query in the same way as we do this for _KEY and _VAL at the moment:
> > > >
> > > > class QueryEntity {
> > > > String tableName;
> > > > String keyType;
> > > > String valType;
> > > > *String keyFieldName;*
> > > > *String valFieldName;*
> > > > Collection fields;
> > > > Collection indexes;
> > > > }
> > > >
> > > > Any other ideas?
> > > >
> > > > On Tue, Feb 14, 2017 at 9:19 PM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > > wrote:
> > > >
> > > > > Vova,
> > > > >
> > > > > Agree about the primitive types. However, it is not clear to me how
> > the
> > > > > mapping from a primitive type to a column name will be supported.
> Do
> > > you
> > > > > have a design in mind?
> > > > >
> > > > > D.
> > > > >
> > > > > On Tue, Feb 14, 2017 at 6:16 AM, Vladimir Ozerov <
> > voze...@gridgain.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Dima,
> > > > > >
> > > > > > This will not work for primitive keys and values as currently the
> > > only
> > > > > way
> > > > > > to address them is to use "_KEY" and "_VAL" aliases respectively.
> > For
> > > > > this
> > > > > > reason I would rather postpone UPDATE/DELETE implementation until
> > > > "_KEY"
> > > > > > and "_VAL" are hidden from public API and some kind of mapping is
> > > > > > introduced. AFAIK this should be handled as a part of IGNITE-3487
> > > ]1].
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-3487
> > > > > >
> > > > > > On Sat, Feb 11, 2017 at 3:36 AM, Dmitriy Setrakyan <
> > > > > dsetrak...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > On Fri, Feb 10, 2017 at 3:36 AM, Vladimir Ozerov <
> > > > voze...@gridgain.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I propose to ship streaming with INSERT support only for now.
> > > This
> > > > is
> > > > > > > > enough for multitude cases and will add value to Ignite 1.9
> > > > > > immediately.
> > > > > > > We
> > > > > > > > can think about correct streaming UPDATE/DELETE architecture
> > > > > separately
> > > > > > > .It
> > > > > > > > is much more difficult thing, we cannot support it in a clean
> > way
> > > > > right
> > > > > > > now
> > > > > > > > due to multiple "_key" and "_val" usages over the code base.
> > > > > > > >
> > > > > > >
> > > > > > > Vova, I disagree. If all parts of the key are present, then we
> > can
> > > > > always
> > > > > > > construct a key in all cases. For these operations we can
> always
> > > > > support
> > > > > > > streaming. For all other 

Re: IGNITE-13

[Вадим Опольский]

Hello everybody!

Valentin, yes, I can  create a micro benchmark and check if it gives any
improvement.

Vadim

2017-02-15 0:52 GMT+03:00 Valentin Kulichenko :

> Vladimir,
>
> I think we misunderstood each other. My understanding of this optimization
> is the following.
>
> Currently string serialization is done in two steps (see
> BinaryWriterExImpl#doWriteString):
>
> strArr = BinaryUtils.strToUtf8Bytes(val); // Encode string into byte
> array.
> out.writeByteArray(strArr);  // Write byte array into
> stream.
>
> What this ticket suggests is to write directly into stream while string is
> encoded, without intermediate array. This both reduces memory consumption
> and eliminates array copy step.
>
> I updated the ticket and added this explanation there.
>
> Vadim, can you create a micro benchmark and check if it gives any
> improvement?
>
> -Val
>
> On Sun, Feb 12, 2017 at 10:38 PM, Vladimir Ozerov 
> wrote:
>
>> Hi,
>>
>> It is hard to say whether it makes sense or not. No doubt, it could speed
>> up marshalling process at the cost of 2x memory required for strings. From
>> my previous experience with marshalling micro-optimizations, we will hardly
>> ever notice speedup in distributed environment.
>>
>> But, there is another sied - it could speedup our queries, because we
>> will not have to unmarshal string on every field access. So I would try to
>> make this optimization optional and then measure query performance with
>> classes having lots of strings. It could give us interesting results.
>>
>> On Mon, Feb 13, 2017 at 5:37 AM, Valentin Kulichenko <
>> valentin.kuliche...@gmail.com> wrote:
>>
>>> Vladimir,
>>>
>>> Can you please take a look and provide your thoughts? Can this be
>>> applied to binary marshaller? From what I recall, it serializes string a
>>> bit differently from optimized marshaller, so I'm not sure.
>>>
>>> -Val
>>>
>>> On Fri, Feb 10, 2017 at 5:16 PM, Dmitriy Setrakyan <
>>> dsetrak...@apache.org> wrote:
>>>
 On Thu, Feb 9, 2017 at 11:26 PM, Valentin Kulichenko <
 valentin.kuliche...@gmail.com> wrote:

 > Hi Vadim,
 >
 > I don't think it makes much sense to invest into OptimizedMarshaller.
 > However, I would check if this optimization is applicable to
 > BinaryMarshaller, and if yes, implement it.
 >

 Val, in this case can you please update the ticket?


 >
 > -Val
 >
 > On Thu, Feb 9, 2017 at 11:05 PM, Вадим Опольский <
 vaopols...@gmail.com>
 > wrote:
 >
 > > Dear sirs!
 > >
 > > I want to resolve issue IGNITE-13 -
 > > https://issues.apache.org/jira/browse/IGNITE-13
 > >
 > > Is it actual?
 > >
 > > Vadim Opolski
 > >
 >

>>>
>>>
>>
>

Re: DML data streaming

[Pavel Tupitsyn]

I have no particular opinion on how we should handle _key/_val,
but we certainly need a way to select entire key and value objects via
SqlFieldsQuery,
and this should work without any additional configuration.

We can rename these, turn them into system functions, whatever.

Ignite.NET LINQ provider heavily relies on this possibility - users often
want to select the entire entry value.

On Wed, Feb 15, 2017 at 3:28 PM, Vladimir Ozerov 
wrote:

> Ok, let's put aside current fields configuration, I'll create separate
> thread for it. As far as _KEY and _VAL, proposed change is exactly about
> mappings:
>
> class QueryEntity {
> ...
> String keyFieldName;
> String valFieldName;
> ...
> }
>
> The key thing is that we will not require users to be aware of our system
> columns. Normally user should not bother about existence of hidden _KEY and
> _VAL columns. Instead, we just allow them to optionally reference the whole
> key and/or val through predefined name.
>
> On Wed, Feb 15, 2017 at 3:07 PM, Sergi Vladykin 
> wrote:
>
> > I don't see any improvement here. Usability will only suffer with this
> > change.
> >
> > I'd suggest to just add mapping for system columns like _key, _val ,
> _ver.
> >
> > Sergi
> >
> > 2017-02-15 13:18 GMT+03:00 Vladimir Ozerov :
> >
> > > I think the whole QueryEntity class require rework to allow for this
> > > change. I would start with creating QueryField class which will
> > encapsulate
> > > all field properties which are currently set through different setters:
> > >
> > > class QueryField {
> > > String name;
> > > String type;
> > > String alias;
> > > boolean keyField;
> > > }
> > >
> > > class QueryEntity {
> > > String tableName;
> > > String keyType;
> > > String valType;
> > > Collection fields;
> > > Collection indexes;
> > > }
> > >
> > > Then we can add optional key and value field names to top-level config.
> > If
> > > set, key and/or value will have names and will be included into SELECT
> *
> > > query in the same way as we do this for _KEY and _VAL at the moment:
> > >
> > > class QueryEntity {
> > > String tableName;
> > > String keyType;
> > > String valType;
> > > *String keyFieldName;*
> > > *String valFieldName;*
> > > Collection fields;
> > > Collection indexes;
> > > }
> > >
> > > Any other ideas?
> > >
> > > On Tue, Feb 14, 2017 at 9:19 PM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > > wrote:
> > >
> > > > Vova,
> > > >
> > > > Agree about the primitive types. However, it is not clear to me how
> the
> > > > mapping from a primitive type to a column name will be supported. Do
> > you
> > > > have a design in mind?
> > > >
> > > > D.
> > > >
> > > > On Tue, Feb 14, 2017 at 6:16 AM, Vladimir Ozerov <
> voze...@gridgain.com
> > >
> > > > wrote:
> > > >
> > > > > Dima,
> > > > >
> > > > > This will not work for primitive keys and values as currently the
> > only
> > > > way
> > > > > to address them is to use "_KEY" and "_VAL" aliases respectively.
> For
> > > > this
> > > > > reason I would rather postpone UPDATE/DELETE implementation until
> > > "_KEY"
> > > > > and "_VAL" are hidden from public API and some kind of mapping is
> > > > > introduced. AFAIK this should be handled as a part of IGNITE-3487
> > ]1].
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-3487
> > > > >
> > > > > On Sat, Feb 11, 2017 at 3:36 AM, Dmitriy Setrakyan <
> > > > dsetrak...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > On Fri, Feb 10, 2017 at 3:36 AM, Vladimir Ozerov <
> > > voze...@gridgain.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I propose to ship streaming with INSERT support only for now.
> > This
> > > is
> > > > > > > enough for multitude cases and will add value to Ignite 1.9
> > > > > immediately.
> > > > > > We
> > > > > > > can think about correct streaming UPDATE/DELETE architecture
> > > > separately
> > > > > > .It
> > > > > > > is much more difficult thing, we cannot support it in a clean
> way
> > > > right
> > > > > > now
> > > > > > > due to multiple "_key" and "_val" usages over the code base.
> > > > > > >
> > > > > >
> > > > > > Vova, I disagree. If all parts of the key are present, then we
> can
> > > > always
> > > > > > construct a key in all cases. For these operations we can always
> > > > support
> > > > > > streaming. For all other operations, we can delegate to standard
> > MR,
> > > > but
> > > > > > still perform most operations on the same node, as I suggested in
> > > > another
> > > > > > email.
> > > > > >
> > > > >
> > > >
> > >
> >
>

[GitHub] ignite pull request #1524: IGNITE-4685 Propagate platform plugin configurati...

[ptupitsyn]

Github user ptupitsyn closed the pull request at:

https://github.com/apache/ignite/pull/1524


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: DML data streaming

[Vladimir Ozerov]

Ok, let's put aside current fields configuration, I'll create separate
thread for it. As far as _KEY and _VAL, proposed change is exactly about
mappings:

class QueryEntity {
...
String keyFieldName;
String valFieldName;
...
}

The key thing is that we will not require users to be aware of our system
columns. Normally user should not bother about existence of hidden _KEY and
_VAL columns. Instead, we just allow them to optionally reference the whole
key and/or val through predefined name.

On Wed, Feb 15, 2017 at 3:07 PM, Sergi Vladykin 
wrote:

> I don't see any improvement here. Usability will only suffer with this
> change.
>
> I'd suggest to just add mapping for system columns like _key, _val , _ver.
>
> Sergi
>
> 2017-02-15 13:18 GMT+03:00 Vladimir Ozerov :
>
> > I think the whole QueryEntity class require rework to allow for this
> > change. I would start with creating QueryField class which will
> encapsulate
> > all field properties which are currently set through different setters:
> >
> > class QueryField {
> > String name;
> > String type;
> > String alias;
> > boolean keyField;
> > }
> >
> > class QueryEntity {
> > String tableName;
> > String keyType;
> > String valType;
> > Collection fields;
> > Collection indexes;
> > }
> >
> > Then we can add optional key and value field names to top-level config.
> If
> > set, key and/or value will have names and will be included into SELECT *
> > query in the same way as we do this for _KEY and _VAL at the moment:
> >
> > class QueryEntity {
> > String tableName;
> > String keyType;
> > String valType;
> > *String keyFieldName;*
> > *String valFieldName;*
> > Collection fields;
> > Collection indexes;
> > }
> >
> > Any other ideas?
> >
> > On Tue, Feb 14, 2017 at 9:19 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > > Vova,
> > >
> > > Agree about the primitive types. However, it is not clear to me how the
> > > mapping from a primitive type to a column name will be supported. Do
> you
> > > have a design in mind?
> > >
> > > D.
> > >
> > > On Tue, Feb 14, 2017 at 6:16 AM, Vladimir Ozerov  >
> > > wrote:
> > >
> > > > Dima,
> > > >
> > > > This will not work for primitive keys and values as currently the
> only
> > > way
> > > > to address them is to use "_KEY" and "_VAL" aliases respectively. For
> > > this
> > > > reason I would rather postpone UPDATE/DELETE implementation until
> > "_KEY"
> > > > and "_VAL" are hidden from public API and some kind of mapping is
> > > > introduced. AFAIK this should be handled as a part of IGNITE-3487
> ]1].
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-3487
> > > >
> > > > On Sat, Feb 11, 2017 at 3:36 AM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > > wrote:
> > > >
> > > > > On Fri, Feb 10, 2017 at 3:36 AM, Vladimir Ozerov <
> > voze...@gridgain.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > I propose to ship streaming with INSERT support only for now.
> This
> > is
> > > > > > enough for multitude cases and will add value to Ignite 1.9
> > > > immediately.
> > > > > We
> > > > > > can think about correct streaming UPDATE/DELETE architecture
> > > separately
> > > > > .It
> > > > > > is much more difficult thing, we cannot support it in a clean way
> > > right
> > > > > now
> > > > > > due to multiple "_key" and "_val" usages over the code base.
> > > > > >
> > > > >
> > > > > Vova, I disagree. If all parts of the key are present, then we can
> > > always
> > > > > construct a key in all cases. For these operations we can always
> > > support
> > > > > streaming. For all other operations, we can delegate to standard
> MR,
> > > but
> > > > > still perform most operations on the same node, as I suggested in
> > > another
> > > > > email.
> > > > >
> > > >
> > >
> >
>

Re: DML data streaming

[Sergi Vladykin]

I don't see any improvement here. Usability will only suffer with this
change.

I'd suggest to just add mapping for system columns like _key, _val , _ver.

Sergi

2017-02-15 13:18 GMT+03:00 Vladimir Ozerov :

> I think the whole QueryEntity class require rework to allow for this
> change. I would start with creating QueryField class which will encapsulate
> all field properties which are currently set through different setters:
>
> class QueryField {
> String name;
> String type;
> String alias;
> boolean keyField;
> }
>
> class QueryEntity {
> String tableName;
> String keyType;
> String valType;
> Collection fields;
> Collection indexes;
> }
>
> Then we can add optional key and value field names to top-level config. If
> set, key and/or value will have names and will be included into SELECT *
> query in the same way as we do this for _KEY and _VAL at the moment:
>
> class QueryEntity {
> String tableName;
> String keyType;
> String valType;
> *String keyFieldName;*
> *String valFieldName;*
> Collection fields;
> Collection indexes;
> }
>
> Any other ideas?
>
> On Tue, Feb 14, 2017 at 9:19 PM, Dmitriy Setrakyan 
> wrote:
>
> > Vova,
> >
> > Agree about the primitive types. However, it is not clear to me how the
> > mapping from a primitive type to a column name will be supported. Do you
> > have a design in mind?
> >
> > D.
> >
> > On Tue, Feb 14, 2017 at 6:16 AM, Vladimir Ozerov 
> > wrote:
> >
> > > Dima,
> > >
> > > This will not work for primitive keys and values as currently the only
> > way
> > > to address them is to use "_KEY" and "_VAL" aliases respectively. For
> > this
> > > reason I would rather postpone UPDATE/DELETE implementation until
> "_KEY"
> > > and "_VAL" are hidden from public API and some kind of mapping is
> > > introduced. AFAIK this should be handled as a part of IGNITE-3487 ]1].
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-3487
> > >
> > > On Sat, Feb 11, 2017 at 3:36 AM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > > wrote:
> > >
> > > > On Fri, Feb 10, 2017 at 3:36 AM, Vladimir Ozerov <
> voze...@gridgain.com
> > >
> > > > wrote:
> > > >
> > > > > I propose to ship streaming with INSERT support only for now. This
> is
> > > > > enough for multitude cases and will add value to Ignite 1.9
> > > immediately.
> > > > We
> > > > > can think about correct streaming UPDATE/DELETE architecture
> > separately
> > > > .It
> > > > > is much more difficult thing, we cannot support it in a clean way
> > right
> > > > now
> > > > > due to multiple "_key" and "_val" usages over the code base.
> > > > >
> > > >
> > > > Vova, I disagree. If all parts of the key are present, then we can
> > always
> > > > construct a key in all cases. For these operations we can always
> > support
> > > > streaming. For all other operations, we can delegate to standard MR,
> > but
> > > > still perform most operations on the same node, as I suggested in
> > another
> > > > email.
> > > >
> > >
> >
>

IGNITE-602

[Дмитрий Рябов]

Hello, can you answer me a simple question?

We have 2 looping lists:

ArrayList list1 = new ArrayList<>();
ArrayList list2 = new ArrayList<>();

list2.add(list1);
list1.add(list2);

Class GridToStringBuilder represent them into string.

Simple public static  String toString(Class cls, T obj) method
returns string "ArrayList [size=1]" for each list.

*Question is* how should look the result of toString(Class cls, T obj,
String name, @Nullable Object val) method and other methods with additional
values?
"ArrayList [size=1, name=ArrayList[size=1]]" or something else?

The trouble is in GridStringBuilder.a(Object obj) method - it tries to
append collection's toString() method which invokes toString() on every
element.

[jira] [Created] (IGNITE-4706) Rewrite test GridTaskExecutionSelfTest::testJobIdCollision to use multiple JVMs

[Sergey Chugunov (JIRA)]

Sergey Chugunov created IGNITE-4706:
---

 Summary: Rewrite test 
GridTaskExecutionSelfTest::testJobIdCollision to use multiple JVMs
 Key: IGNITE-4706
 URL: https://issues.apache.org/jira/browse/IGNITE-4706
 Project: Ignite
  Issue Type: Test
Reporter: Sergey Chugunov


h2. Notes
Test emulates ID collision by tricks with *IgniteUuid* local counter which 
leads to failures of important components like discovery messaging.
It makes test unstable; sometimes it hangs or fails in a various ways.

h2. Acceptance Criteria
Test exploits multiple JVMs to execute its logic, all tricks with IgniteUuid 
are removed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (IGNITE-4705) Atomic cache protocol change: notify client node from backups

[Semen Boikov (JIRA)]

Semen Boikov created IGNITE-4705:


 Summary: Atomic cache protocol change: notify client node from 
backups
 Key: IGNITE-4705
 URL: https://issues.apache.org/jira/browse/IGNITE-4705
 Project: Ignite
  Issue Type: Task
  Components: cache
Reporter: Semen Boikov
Assignee: Semen Boikov
 Fix For: 2.0


Currently in atomic cache backup nodes first reply to primary and then primary 
replies to client, need check if it is possible to notify client directly from 
backups.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] ignite pull request #1535: IGNITE-4694 Add tests to check there are no memor...

[gvvinblade]

GitHub user gvvinblade opened a pull request:

https://github.com/apache/ignite/pull/1535

IGNITE-4694 Add tests to check there are no memory leaks in PageMemory



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4694

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1535.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1535


commit 692cd6c8ec619001716e81a80001e9570cb154c5
Author: Igor Seliverstov 
Date:   2017-02-15T10:41:08Z

IGNITE-4694 Add tests to check there are no memory leaks in PageMemory




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Questions of the Ignite Configurations

[Vyacheslav Daradur]

Vladimir, thanks for your answers.

2017-02-15 13:32 GMT+03:00 Vladimir Ozerov :

> Hi Vyacheslav,
>
> Answering your questions:
>
> 1. Yes
> 2. Yes
> 3. Only binary infrastructure
> 4. Yes
>
> On Wed, Feb 15, 2017 at 12:32 PM, Vyacheslav Daradur 
> wrote:
>
> > Hello everybody.
> >
> > I have some questions about classes of configuration in the Ignite.
> >
> > 1. IgniteConfiguration - is it only one instance per Ignite-node?
> >
> > 2. BinaryConfiguration  - is it only one instance per Ignite-node?
> >
> > 3. Does BinaryConfiguration  relates only to binary-infrastructure or I
> can
> > use it with non-binary classes such as OptimizedMarshaller?
> >
> > 4. BinaryContext  - is it only one instance per BinaryMarshaller?
> >
>

Re: Questions of the Ignite Configurations

[Vladimir Ozerov]

Hi Vyacheslav,

Answering your questions:

1. Yes
2. Yes
3. Only binary infrastructure
4. Yes

On Wed, Feb 15, 2017 at 12:32 PM, Vyacheslav Daradur 
wrote:

> Hello everybody.
>
> I have some questions about classes of configuration in the Ignite.
>
> 1. IgniteConfiguration - is it only one instance per Ignite-node?
>
> 2. BinaryConfiguration  - is it only one instance per Ignite-node?
>
> 3. Does BinaryConfiguration  relates only to binary-infrastructure or I can
> use it with non-binary classes such as OptimizedMarshaller?
>
> 4. BinaryContext  - is it only one instance per BinaryMarshaller?
>

Re: DML data streaming

[Vladimir Ozerov]

I think the whole QueryEntity class require rework to allow for this
change. I would start with creating QueryField class which will encapsulate
all field properties which are currently set through different setters:

class QueryField {
String name;
String type;
String alias;
boolean keyField;
}

class QueryEntity {
String tableName;
String keyType;
String valType;
Collection fields;
Collection indexes;
}

Then we can add optional key and value field names to top-level config. If
set, key and/or value will have names and will be included into SELECT *
query in the same way as we do this for _KEY and _VAL at the moment:

class QueryEntity {
String tableName;
String keyType;
String valType;
*String keyFieldName;*
*String valFieldName;*
Collection fields;
Collection indexes;
}

Any other ideas?

On Tue, Feb 14, 2017 at 9:19 PM, Dmitriy Setrakyan 
wrote:

> Vova,
>
> Agree about the primitive types. However, it is not clear to me how the
> mapping from a primitive type to a column name will be supported. Do you
> have a design in mind?
>
> D.
>
> On Tue, Feb 14, 2017 at 6:16 AM, Vladimir Ozerov 
> wrote:
>
> > Dima,
> >
> > This will not work for primitive keys and values as currently the only
> way
> > to address them is to use "_KEY" and "_VAL" aliases respectively. For
> this
> > reason I would rather postpone UPDATE/DELETE implementation until "_KEY"
> > and "_VAL" are hidden from public API and some kind of mapping is
> > introduced. AFAIK this should be handled as a part of IGNITE-3487 ]1].
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-3487
> >
> > On Sat, Feb 11, 2017 at 3:36 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> > > On Fri, Feb 10, 2017 at 3:36 AM, Vladimir Ozerov  >
> > > wrote:
> > >
> > > > I propose to ship streaming with INSERT support only for now. This is
> > > > enough for multitude cases and will add value to Ignite 1.9
> > immediately.
> > > We
> > > > can think about correct streaming UPDATE/DELETE architecture
> separately
> > > .It
> > > > is much more difficult thing, we cannot support it in a clean way
> right
> > > now
> > > > due to multiple "_key" and "_val" usages over the code base.
> > > >
> > >
> > > Vova, I disagree. If all parts of the key are present, then we can
> always
> > > construct a key in all cases. For these operations we can always
> support
> > > streaming. For all other operations, we can delegate to standard MR,
> but
> > > still perform most operations on the same node, as I suggested in
> another
> > > email.
> > >
> >
>

Re: Sort nodes in the ring in order to minimize the number of reconnections

[Александр Меньшиков]

Need to do code review until February 17, if we want to get this feature in
version 1.9.

2017-02-08 22:14 GMT+03:00 Александр Меньшиков :

> Done. Please look.
>
> JIRA: https://issues.apache.org/jira/browse/IGNITE-4501
> PR: https://github.com/apache/ignite/pull/1436/files
> Tests: http://ci.ignite.apache.org/project.html?projectId=IgniteTes
> ts=projectOverview_IgniteTests=pull/1436/head
>

Questions of the Ignite Configurations

[Vyacheslav Daradur]

Hello everybody.

I have some questions about classes of configuration in the Ignite.

1. IgniteConfiguration - is it only one instance per Ignite-node?

2. BinaryConfiguration  - is it only one instance per Ignite-node?

3. Does BinaryConfiguration  relates only to binary-infrastructure or I can
use it with non-binary classes such as OptimizedMarshaller?

4. BinaryContext  - is it only one instance per BinaryMarshaller?

Re: Apache Ignite 1.9

[Anton Vinogradov]

Denis,

I don't know what is the plan for this issue.
You asked me to fix compilation and thats what I done :) , but I'm not sure
that issue implemented in proper way. I don't like changes I've done, they
looks redundant although it works.
Do we have somebody more familiar with spark inside the community?
We have to review what we already done and figure out what's left.

On Wed, Feb 15, 2017 at 5:01 AM, Denis Magda  wrote:

> Continuing Spark related topic...
>
> This ticket definitely must be included in 1.9:
> https://issues.apache.org/jira/browse/IGNITE-3710 <
> https://issues.apache.org/jira/browse/IGNITE-3710>
>
> We can’t longer afford torture Ignite Shared RDD users who have to use an
> outdated Spark version.
>
> Anton, could you finalize it this week? As far as I remember, the only
> thing that is left is to add a missing test suite to Team City.
>
> —
> Denis
>
> > On Feb 14, 2017, at 5:00 PM, Denis Magda  wrote:
> >
> > Reviewed and merged the first (!) Ignite Shared RDD examples contributed
> by Manish:
> >
> > https://issues.apache.org/jira/browse/IGNITE-4526
> >
> > Java:
> > https://github.com/apache/ignite/blob/b461cb47882861356ede58775bd9e2
> 53dcf26202/examples/src/main/java8/org/apache/ignite/examples/java8/spark/
> SharedRDDExample.java
> >
> > Scala:
> > https://github.com/apache/ignite/blob/b461cb47882861356ede58775bd9e2
> 53dcf26202/examples/src/main/scala/org/apache/ignite/
> scalar/examples/spark/ScalarSharedRDDExample.scala
> >
> > Manish, thanks for making the life of Apache Ignite users easier. No
> they can refer to your examples while learning Ignite Shared RDD API.
> >
> > —
> > Denis
> >
> >> On Feb 14, 2017, at 1:21 AM, Anton Vinogradov 
> wrote:
> >>
> >>>
> >>> Can we push to 1.9 and then merge/rebase to master?
> >>
> >>
> >> Yes, that's *correct *too, but only *megre *should be used.
> >>
> >> On Tue, Feb 14, 2017 at 12:19 PM, Vladimir Ozerov  >
> >> wrote:
> >>
> >>> Can we push to 1.9 and then merge/rebase to master? It should be more
> >>> reliable than cherry-picking.
> >>>
> >>> On Tue, Feb 14, 2017 at 12:15 PM, Anton Vinogradov <
> >>> avinogra...@gridgain.com
>  wrote:
> >>>
>  Denis,
> 
>  Merge to the master first and then cherry-pick to 1.9 from there is a
>  *correct
>  *case.
> 
>  On Tue, Feb 14, 2017 at 5:24 AM, Dmitriy Setrakyan <
> >>> dsetrak...@apache.org>
>  wrote:
> 
> > Thanks, Denis!
> >
> > On Mon, Feb 13, 2017 at 5:49 PM, Denis Magda 
> >>> wrote:
> >
> >> Well, merged Kubernetes integration to the master and cherry-picked
> >>> to
> >> 1.9. The docs and getting started for readme.io to be done soon.
> >>
> >> —
> >> Denis
> >>
> >>> On Feb 13, 2017, at 9:10 AM, Denis Magda 
> >>> wrote:
> >>>
> >>> Do I merge to the master first and then cherry-pick to 1.9 from
>  there?
> >>>
> >>> —
> >>> Denis
> >>>
>  On Feb 13, 2017, at 8:21 AM, Anton Vinogradov <
> > avinogra...@gridgain.com>
> >> wrote:
> 
>  Igniters,
> 
>  We're starting release stabilization process.
>  Branch 1.9 created and equals to master branch right now.
> 
>  Please make sure that your changes will be merged to both
> >>> branches:
>  *ignite-1.9* and *master*.
> 
>  On Fri, Feb 10, 2017 at 4:53 PM, Sergey Kalashnikov <
> >> zkilling...@gmail.com>
>  wrote:
> 
> > Anton,
> >
> > Here is the status for IGNITE-4523.
> > The work is completed, tests are OK.
> > Most of the changes were made by Alexey and were reviewed.
> > I only authored few last commits for which I would appreciate a
> > review.
> > The PR is https://github.com/apache/ignite/pull/1508
> >
> > Thank you
> > --
> > Sergey
> >
> >
> >
> > 2017-02-09 23:58 GMT+03:00 Denis Magda :
> >
> >> Anton,
> >>
> >> You can expect Kubernetes support from me in time.
> >>
> >> —
> >> Denis
> >>
> >>> On Feb 9, 2017, at 1:52 AM, Anton Vinogradov 
> > wrote:
> >>>
> >>> Igniters,
> >>>
> >>> We're on a half way to Ignite 1.9.
> >>> Development phase will be finished 17th of February.
> >>>
> >>> Here  >> Apache+Ignite+1.9> is
> >>> a status page for upcoming release.
> >>>
> >>> According to this page we have some Issues we *especially
> >>> *expect
> > in
> > 1.9:
> >>>
> >>> IGNITE-4106 SQL: parallelize sql queries over cache local
> > partitions
> >> Andrew
> 

Re: Inaccurate documentation about transactions

[Vladimir Ozerov]

Alexandr,

If you PUT some value within transaction, this is absolutely normal that
you will see it on successive GET in the _same_ transaction. DIRTY_READ is
a different thing - it allows reads of not-yet-committed changes from
_another_ transaction.

On Wed, Feb 15, 2017 at 9:41 AM, Alexandr Kuramshin 
wrote:

> After doing some tests with transactions I've found transactions work not
> as expected after reading the documentation [1].
>
> First of all, nowhere's written which methods of the cache are
> transactional and which are not. Quite the contrary, after reading
> documentation we get know that each TRANSACTIONAL cache is fully
> ACID-compliant without exceptions.
>
> Only after deep multi-thread testing, and consulting with other developers,
> I get know that only get and put methods are running within transaction,
> but iterator and query methods are running outside (in autonomous)
> transaction with READ_COMMITTED isolation level.
>
> Later I've understood that only methods throwing
> TransactionTimeoutException/TransactionRollbackException/
> TransactionHeuristicException
> are fully transactional. I think all methods on page [2] should be directly
> described - are they transactional or not. Btw, why these exceptions are
> not derived from the common base class, e.g. TransactionException?
>
> Secondary, using the transactional get() method inside the READ_COMMITTED
> transaction we expect to get the committed value, as the documentation [1]
> claims:
>
> * READ_COMMITTED - Data is read without a lock and is never cached in the
> transaction itself.
>
> Ok, but what about put()? After doing the put() a new value, we get
> successive reads of the new value, that is actually DIRTY READ. Hence the
> value is cached within transaction. It's not documented behavior.
>
> [1] https://apacheignite.readme.io/docs/transactions
>
> [2]
> https://ignite.apache.org/releases/1.8.0/javadoc/org/
> apache/ignite/IgniteCache.html
>
> --
> Thanks,
> Alexandr Kuramshin
>